Handout for Visual Music

Visual Music

Abstract visualizations, or “visual music”—generative patterns and audio responsive graphics that have a direct symbiotic relationship between image and sound—can evoke a sense of space, environment, rhythm and illuminate concepts in performance without representational imagery. In a live context of improvised music and improvised visuals, abstract visualization allows music and image to be partners in the creation of an aesthetically meaningful shared experience.

Visual music is an art form inspired by and in some cases driven by synaesthesia—the production of a sense impression relating to one sense or part of the body by stimulation of another sense or part of the body. In common cases of synaesthesia, a person will associate a sound with a color.

There is a theory that we are all synesthetes on some level. Take for instance Booba/Kiki effect, a non-arbitrary mapping between speech sounds and the visual shape of objects. This suggests that the human brain somehow attaches abstract meanings to the shapes and sounds in a consistent way.

Lesson 1: Abstract Visualization / Color Organ

The synaesthetic relationship between sound and vision continues to be explored by scientists artists working today. “Color scales,” dating back to Isaac Newton, attempt to scientifically correlate musical scales with colors. ”Color Organs,” instruments that generate colors based on notes, date back to the 1850s and continue to be developed. Wassily Kandinksy, one of the founders of non-objective painting, went so far as to create a color code for sounds. He intended for his pieces to be both seen and heard, titling them as “compositions.”

This week, we will create a shape-based color organ, creating and performing forms and colors that represent specific sounds using our inner synesthete.

Videos and Slides

Lesson Overview

  • Triggering content with sound/MIDI impulse
    • Prev / next / rand
    • Triggering a specific clip
    • Quantized triggering
  • Manipulating color with sound/MIDI
    • RGB vs HSV (HSB)
    • ‘Audio Level to Color’
    • Color organ generator and sound synth

Lecture Notes

  • What does is the color of a sound / tone?
    • Color scales
    • Translations between sounds and imagery
  • Color representations – RGB vs HSV

Resources

Homework

  • Create and record a ~60 second ‘color organ’ using selected music

Lesson 2: Audio Visualizers and the Shape of Sounds

As discussed in Stills To Motion module, we are hardwired to find patterns in shapes that are similar and arranged closely together. In this module, we can explore pattern making as a synaesthetic device, influencing the frequency, amplitude and continuity of repetitive shapes and images to rhythmic effect.

Videos and Slides

Lesson Overview

  • What is the shape of a sound?
  • Audio Visualizers:
    • VU Meters
    • Waveform and FFT Visualizers
    • Audio Spectrograms
    • ProjectMilkSyphon
    • Booba/Kiki ISFs
    • Rutt Etra
  • Create two layers/media bins of shapes (i.e., Booba/Kiki):
    • Organic
    • Polygonal
  • Augment with audio visualizers (waveforms, FFT, spectrogram)

Lecture Notes

  • What is the shape of a sound?
    • Waveform visualization
    • FFT visualization
    • Audio Spectrograms

Resources

Homework

  • Create and record three Booba/Kiki shape-based sequences ~10 seconds each using different forms of music and sounds effects.

Lesson 3: Generative Patterns

As discussed in Stills To Motion module, we are hardwired to find patterns in shapes that are similar and arranged closely together. In this module, we can explore pattern making as a synaesthetic device, influencing the frequency, amplitude and continuity of repetitive shapes and images to rhythmic effect.

Videos and Slides

Lesson Overview

  • Tile effects
  • Patterns with code (eg Goto10, Sol LeWitt)
  • Introduction to the ISF Editor / GLSL (optional, advanced)
    • Making a solid color generator
    • Making a basic pattern generator (strips, checkerboards)
    • Remixing compositions
  • Use tiling effects and data-sources to create animated generative patterns.
  • Syncing tile generators / effects to music

Lecture Notes

  • Patterns
    • Properties of patterns
      • Symmetry
      • Rotation
      • Chaos, flow, meanders
      • Fractals
    • Patterns in nature
      • Snowflakes
      • Spirals
      • Waves, dunes
      • Bubbles, foam
      • Cracks
      • Spots and stripes
    • Geometric patterns
      • Tiling
      • Visual motifs
  • Introduction to GLSL (optional)
    • What are shaders?
    • Why are they fast?
    • Tools for writing shaders
    • Examples of patterns made with GLSL

Resources

Homework

  • Create and record black and white or color patterns:
    • four still image capture
    • two video sequences, about 10 seconds each
Tags: