Augmented Reality

Visualization of Sounds: Improving the Listening Experience Through Digital Visualization

Abstract
The effectiveness of supporting visuals such as charts, tables, and infographics to communicate complex data has long been proven and accepted. In this project we will seek to observe the persuasiveness of visualization when pertaining to audio data. Thus, we seek to examine how visualizing sound data can allow for a better understanding of the information communicated. Consequently, we use digital data visualization technics to convert originally composed music, with no lyrics or context, into visuals that would allow the interpretation of said composition. However, as infographics offer no new information - simply a representation of the original data – it was important to this exercise that the audio data visualization be done with a similar intention. Thus, steps were taken using data bending techniques, as well as 3D modeling and animation to convert audio files into visuals with as little creative bias as possible.
About the Author
Maeliss Usher

I am a multi-platform artist, digital marketer and brand consultant currently living in Toronto, Canada. In 2015, I graduated with a B.comm in Marketing from The John Molson School of Business at Concordia University in Montreal. Moving to Toronto four years ago, I was determined to grow in the digital space. And this year, through the MDM program, I was able to harness my passion and translate it into a portfolio.

About the Project

Development Process

During the initial stage of this project, multiple visualization techniques were studied to determine a suitable development process. Based on the research, the following process was undertaken to convert the audio data into a digital visualization experience. 

Data Bending 

Data bending is the process of manipulating a media file of a certain format, using software meant to edit files of another format (Word Finder, n.d.). Thus, through the Audacity 2.3.2 software, the MP3 files were converted into the RAW formats that would generate the deepest available description (signed 32-bit) and the widest dynamic range (A-law encoded).


Glitch Effect

Glitch images from an audio file generated from Data Bending


The Glitch Effect is created when there is an error or a flaw in an image file (Leigh, 2019). In this research however, the Glitch Effect is caused intentionally through the data bending process. Then an image is created from the RAW file when photoshop depicts the binary code as pixels and reads the colors into the image – 0 being Black and 225 representing white (File Formats, n.d).

Animation & Immersion

Once the images were generated from the RAW File, they were brought over to Blender to create a data visualization experience:


Representation of a textured room as it would have been displayed via projection mapping.


3D Textured Room

The Glitch images are transformed into textures to create a 3D environment. Note that, prior to the pandemic, the textures would have been showcased using projection mapping and artificial reality to allow for immersion.


Sound Animation

Displacement animation generated by glitch image video from an audio file
Wave animation generated by glitch image video from an audio file

Displacement

Animations were created by compiling the glitch images to create a video. This video was then used to inform the displacement of objects in Blender. In this exercise, the Glitch image texture’s RGB values were used to displace the mesh of the object in the XYZ direction.

Wave animation

In this animation, the glitch image video was used to inform the direction of the wave. while the soundwaves were used to inform the strength of the waves.


MY NEONORMAL STORY
''
Prior to the Quarantine, I had planned on putting together an immersive audio-visual experience. It was going to be an event in which all participants could be immersed in a musical composition – an art exhibit that included 3D printed sculptures and projection mapping. Given that it was no longer possible for the event to take place, I had to pivot and re-strategize in a major way to make sure I produced an experience that still had a similar effect. Yet, I have been able to learn and grow my digital skills in a way that I believe will be beneficial to my career in this current environment.