Method: Audiovisual

Basics

Let's start on a very fundamental level and look at the scientific and perceptual differences of sound and visuals.

Another important difference is their "presence" according to our sensory equipment. While vision is very much dependending on the direction we look at and where we place our focus, hearing works less directional - we pick up sounds that are behind our backs easily. In addition, our ears don't have a dedicated shutting mechanism. So in a way we are much more exposed to sound than to visuals, where we can instantly turn away or close our eyes.

This becomes naturally relevant if your work is based on an audiovisual immersion. Cinema's strategy is the black box, where every "external" visual stimulus is eliminated and your viewing direction is fixed. But for more dynamic spaces (like a club or an installation) this needs consideration.

Correspondences

Color Tone Correspondences

Througout history attempts have been made to find universal links between sound and images, which are based on an inner concordance. A the core of this endeavor is the mapping of colors to (musical) tones. The immediate comparison of color and tone dates back to the early antiquity:

• Pythagoras "Geometry is music solidified" (570BC-510BC)
• Archytas v. Tarent - Chromatic Scale
• Aristoteles - Color-Tone Harmonics In the Renaissance these first attempts were further developed by e.g. Leonardo da Vinci, Franchino Gaffurio and Giuseppe Arcimboldo (Greyscale Color Notation)

In 1704 Isaac Newton envisioned that colors are harmonically proportioned like the notes in a musical scale (in his treatise "Opticks" ). He relates muscical notes to his color wheel.

Chromesthesia or sound-to-color synesthesia is a type of synesthesia in which sound involuntarily (and consistently) evokes an experience of color

Studies to date have reported that synesthetes and non-synesthetes alike associate high pitched sounds with lighter or brighter colors and low pitched sounds with darker colors, indicating that a common mechanism may underly those associations in normal adult brains. (Wikipedia, Ward J)

Synaesthesia and Temporal Analogies

Viking Eggeling: Symphonie Diagonale, 1921-25

AV Oddities

McGurk Effect

Strategies

Hierarchy
A first important distinction is the type of hierarchy - is one medium in control, one medium the foundation for the other (eg music video) or are they equal in theire creation process, etc.

• Scientific Image and Sound Analysis
  Spectrums, Waveforms, sequence diagrams, oscilloscopes, and methods that are grounded in data visualisation practices with an "neutral", objective goal.

• Collage & Montage
  A topic that has developed a rich repertoir in the cinematic arts. Synchresis, counterpoint/emphasis, etc.
  Sound and image occur together and develop their effect through this parallel experience. They can support each other or even tell a different story. Think about film music, sound effects, etc.

• Parallel Sound and Image Synthesis
  One meta-system controls/creates both sound and images. Therefore there a direct relationship is inherently established while no direct interaction between the two media is necessary

• Parameter Mapping / Syaesthetic Analogies
  Instead of one meta system sound and image both have their own constructive system, where some parameters are picked and mapped. This is a key concept for live AV performances. An attempt is made to construct a consistent mapping between sound and visual properties, like the construction of a common alphabet.

• Crosscoding
  The strategy of treating one media like the other often with brute force. eg. interpreting a sound file as an image in photoshop or plugging a video signal into a audio input, etc

• Structural Analogies (conceptual interchange)
  Using the concept of one domain in the other. eg the Visual Music genre or usign the method of the (musical) fugue as a way to construct images (Paul Klee,František Kupka,...), etc

• Gesamtkunstwerk
  Made famous by Richard Wagner - the peak attempt to incorporate every possible element (sound, light, images, dance, acting, text, sculpture, scenography, costumes) into one artistic vision and often aim for to dissolve the border between art and reality.

• Graphical Scores, Eye Music
  using graphical properties as a way to "compose" music and thereby become a unique aesthetic artefact in itself (see Cornelius Cardew, Roman Haubenstock-Ramati, Iannis Xenakis, John Cage, Brian Eno,...)

Basic AV Data Links

A combination of both methods can certainly bring together the best of both worlds, but this is not always a feasible effort.

Sound Analysis

The three most fundamental data that you can get from your digital audio are

• waveform
• amplitude/RMS (global volume at a given time over the whole spectrum)
• energy levels for specific frequencies, via FFT

Because they are relatively easy to obtain you will find methods for them in most frameworks - so it will be useful to know how to apply them.

Additional Resources

Basic Audiovisualisation Slides, Jason Igal

Visual Music and early AV pieces

Baranoff-Rossiné - Piano Optiphonique (youtube) - 1904
Thomas Wilfred (~1920) - Master of Light - (2018)
Thomas Wilfred's Lumia compostions Opus 140. (1948)
Hans Richter - Rhythmus 21 (1921)
Hans Richter - Rhythmus 23 (1921)
Walther Ruttmann - Lichtspiel Opus I (1921)
Walther Ruttmann - Lichtspiel Opus II (1921)
Viking Eggeling: Symphonie Diagonal
Oskar Fischinger - An Optical Poem (1938)
Norman McLaren - Dots (1940)
Norman McLaren - A Phantasy in Colors (1949)
Norman McLaren: Pen Point Percussion (1951)
John Whitney - Experiments in motion graphics (1968)
John Whitney - Permutations (1968)
Jordan Belson - Allures (1961) "space-iest film that had been done until then"

Scanimate: The Origins of Computer Motion Graphics

Articles

Thomas Wilfred, Lumia - Hyperallergic
Data Visualization in Music- Nightingale