re:sonance is a self initiated, digital exhibition project that takes viewers on an interactive, audiovisual 3D journey with sounds based on naturalistic soundscape and harmonising, captivating visuals.
When the second wave of the pandemic hit Sweden I thought to myself stress levels are at an all time high with financial insecurity, uncertainty about the future and social isolation. Spending all our time indoors has led to more people wanting to explore the outdoors and nature, allowing people to meet others at a safe distance and breaking out of our own four walls.
How could we take a part of nature and combine it with a feeling of connectedness? What if we could make it easily accessible to take a break, explore and relax?
My role / Company
Lead UX Designer / Self initiated project
– Visuals (WebGL)
– Website (HTML, CSS and JS)
Duration / Year
3 months / 2021
Pen and Paper
Adob AI, PS
1 Sound Designer
Check it out yourself before reading further – Use desktop and Chrome for best experience:
Close-up on snow texture
Evolving a curiosity into a real project
It all started with me attending a digital exhibition and it made me think if I could create one myself. A few days earlier I stumbled across Cables a visual development tool for interactive WebGL, which opened a whole new world of possibilities to me. Wanting to learn this new tool, I set up to brainstorm ideas on what would be of value and also would give me the space to create something a bit more artsy – within constraints. One day being outside with my dog listening to the soft crushing of ocean waves on rocks it hit me – a multi-sensory experience with visuals and sounds. I wanted it to be interactive, with beautiful visuals and relaxing sounds. For the sound part I definitely needed some help, so I reached out to my good friend Emily Kornaat and explained her my idea with a sketch (the one you see on the top with just dots). My idea was to create a universe where organic shapes exist and you can wander around, choosing which one you want to listen to, and that the visuals would reflect the sounds. Luckily she was as excited as me!
Kick-starting the design process through desk research
Down the rabbit hole of researching nature sounds and the effects it has on our brains! After reading lots of articles about our mind, auditive stimulation, relaxation & stressor sounds and psycho-evolutionary theory I nailed down our concept and vision:
“An immersive, interactive audiovisual 3D exhibition. Each interactive shape has a unique base layer of a naturalistic soundscape, recorded by family and friends all across the globe. Added onto the base layer is a unique interpretation of the feeling the sound evokes. When tapped or clicked selected bass sounds ring out. The shapes look and animations are a visual interpretation of the naturalistic base and also remind the viewer of it.”
Summary and notes
Research of scholarly articles
Creating value for viewers and ourselves
([…] restorative influences of nature involve a shift towards a more positively-toned emotional state, positive changes in physiological activity levels, and that these changes are accompanied by sustained attention/intake.)
Creating a feeling of connectedness in a time where physical closeness is not possible with features naturalistic sound patterns from all around the globe
(Your brain interprets them as non-threatening noises […] stress reducing influences of nature are derived from a psycho-evolutionary theory.)
Designing and producing a visually interesting exhibition, that makes people curious
Capturing the feeling of naturalistic sounds and adding on to them
Visually coding and interpreting sounds as WebGLs
Oh, the highs and lows of learning a new tool. It was quite difficult at first to get a hang of Cables. Cables is visual coding, you have OPs that you place and then connect to each other – similar to nodes. After lots of tutorials, creating ugly things and looking at other peoples patches I was finally able to set up a shape base. I could reuse this same base but tweak or exchange effects somewhat easily. Every shape is visually unique and my own artistic interpretation of the naturalistic base sound or finished audio from Emily. I sometimes created the shape first and it inspired Emily, and sometimes her audio inspired my visuals.
After finishing the shape, I added the code for the interaction, which is visually a bit different depending on the shape, but it is always a deform effect of some kind – and it plays the bass of the track. I wanted first that the viewer is able to drag, and click and interact in many different ways, but that turned out to be very difficult and time consuming, and too much for this scope.
All element shapes (LTR: wind, fire, ice, snow, water)
Organic Patterns and Interactions
Cables.gl visual coding of ice element
Evoking feelings through uniquely crafted sound
All tracks have naturalistic soundscapes that were kindly provided by friends and family all over the world – or we recorded them ourselves. The sounds are Emilys personal interpretations of what feelings these sounds provoke. She produced the finished tracks using different methods like Musical Typing in GarageBand or even singing to achieve different ranges of sounds.
Emily recording waterfall, my dog being of great assistance
GarageBand and Musical Typing
Shape with sound base and interaction overlay
Giving special meaning through a name and logo
We named this exhibition RE:SONANCE combining the words resonate (“evoke images, memories, and emotions”) and sonance (“Sound”). I designed the logo for it, keeping it very clean, in black & white to let the colorful shapes stand out. I first played around with different fonts, then decided on the strong sans-serif Bebas Neue. I really wanted to give this logo a 3D space, and move away from flat design. I achieved that with the blend effect and playing with opacities and the 3D effect in Illustrator. Also applying a bit of monochromatic noise in photoshop, gave this logo more character, instead of a perfect finish.
Timelapse of me designing the logo
Design in process
Embedding a digital exhibition and hoping for the best
Surprise, I also coded the website! Simply by using HTML and CSS to create a very simple base to be able to click through the different shapes. Then came the realisation that browsers have specific rules about playing audio. I first tried to somehow work around it, which was very difficult because by no means am I a developer – I found a script code snippet online that would play the audio after the DOM Content has loaded. It works, not perfectly – but it works.
For implementing cables on HTML, it was quite easy – you can download a .zip with all your files for your patch. After finishing all the pages, embedding everything I created a subdomain and uploaded all through a FTP. Well, as you can imagine, a 3D interactive patch with sound is quite heavy on a server, so it was so slow and not smooth at all. Change of plans: I found a way to host it on Cables server itself and embed just an iframe into my HTML. The result now being a smooth running digital exhibition!
Unfortunately there are major problems on mobile (from Cables side). Textures do not get loaded, so the shapes are just dark – I added a little note for users when on mobile that this experience is not designed for mobile and to please switch to a desktop. I wanted to minimise frustrations here and give the user insights of why they would encounter problems, they can still look at it on mobile, but understand it is not intended this way.
Coding in Visual Studio with Live Server
Video of full exhibition
Use desktop and Chrome for best experience:
“This made me realise how much I miss going on art exhibitions! Really liked the look and feel of it and together with the sounds it was a very nice experience.”
“Wow! I took my time playing around and I will probably go back again soon, they are like digital stress balls!“
“I found the results of this so inspiring that I had to take a look at the case study for this: and it’s such a well documented and interesting story of exploration. For anyone interested its a really worthwhile read!! And great example of a first testable product.”
“How beautifully done! You guys have so much talent.”
– Quotes from visitors
Honestly, at some point in this project I thought to myself that I bit off more than I can chew. And I am so happy I pulled through. I learned so much through this incredible project and was able to really let my creativity go wild – visually and through with problem solving. Having to code visually and the website was a nice reminder of how parts work together, and how often it can be difficult to navigate through technical constraints that you did not consider before. I’m really happy and got an amazing positive response from so many lovely people too. Big thanks to my friend Emily too, it was so much fun to work on this!
Dream big start small
It’s easy to get overwhelmed, when you have big expectations and then you don’t reach them. Through this project I realised that I love being creative and dreaming, but equally important it is to set yourself up for success by starting with a smaller scope of a project.
Finding creativity in a new medium
I really needed some fresh, creative wind in my sails. I noticed how easily we get into a routine and how as designers we need to try out new tools and have projects where we just can be creative. I really enjoyed working with a 3D tool and also interactivity! More of that!