Module 2: Montage

Introduction to Montage Overview

In his essay “The Photographic Message,” Roland Barthes sees the newspaper as “a complex of concurrent messages with the photograph as centre and surrounds constituted by the text, the title, the caption, the layout and … by the very name of the paper.” According to Barthes, a newspaper is a sea of text with pictures in the middle of it. What live performance, but a sea of text or music with pictures in the middle of it?

Choosing images is the central undertaking of video design. The audience will decode these images, so how will they make use of all the meanings? We can use major cinematic devices to clarify the emotion and meaning in audio/visual performance.

A central technique of VJ’ing is the remix, a collection of cuts. In cinema, we refer to a collection of cuts, or edits, as a montage or assembly. Montage was introduced to cinema primarily by Sergei Eisenstein in the 1920s, and early Soviet directors used it as a synonym for creative editing.

In our first module, Montage, you will learn a range of basic editing and layering principles, developing their timing skills as they develop sensitivity to successiveness, temporal order, and simultaneity. These will be used to contrast clock-time with subjective time, duration, temporal continuity, the feelings of anticipation and expectation, and the shifting of temporal perspectives as it relates to content.

Lesson 1: The Cut

Handout: The Cut

One of the oldest examples of montage editing is called the “Kuleshov Effect.” This experiment demonstrated cinema’s unique capacity as an art form to conjure emotional reactions from the relationship between indexical images.

You will use this cornerstone of editing as your first VJ’ing exercise, eliciting different responses and correlations by strategically ordering content. Finally, you will experiment with different ‘soundtracks’ to greater influence emotion and meaning.

Lesson Overview

  • What is the Kuleshov Effect?
  • Using the Kuleshov example project
    • Loading or relinking media files into Kuleshov project
  • Capturing video clips
  • Recording output in Kuleshov project
  • Adding soundtracks
    • Using Soundflower
    • Using other audio sources via audio capture
  • Upload recordings to Vimeo / Youtube
  • Discussion: Silence vs Music
  • The Shot and the Language of Film

Special Equipment

Required:

Recommended:

Lecture Notes

Discussions

  • Silence vs Music

Demonstrations

Using the Kuleshov example project

  1. Open the Kuleshov example project
    • Relink media files using automatic or manual relinking
    • Or, start from an empty version of the project and load the media files
  2. Demonstation the Kuleshov Effect using sample clips
  3. Record a sample movie in the style of a Kuleshov experiment
  4. Prepare audio capture
    • Recording from iTunes / Spotify / web browser / etc using Soundflower:
      • Set System Audio Device to Soundflower
      • Set Movie Recorder to capture from Soundflower
      • Use Audio Analysis for optional sound play-thru
    • Recording from audio interface:
      • Set Movie Recorder to capture from specified device
      • Play audio
      • Use Audio Analysis for optional sound play-thru
  5. Demonstrate and record the Kuleshov experiments again with:
    • Silence
    • Music (Dramatic)
    • Music (Silly)

Exercises

Using the Kuleshov example project

  1. Open the Kuleshov example project
    • Relink media files using automatic or manual relinking
    • Or, start from an empty version of the project and load the media files
  2. Demonstation the Kuleshov Effect using sample clips
  3. Record a sample movie in the style of a Kuleshov experiment
  4. Prepare audio capture
    • Recording from iTunes / Spotify / web browser / etc using Soundflower:
      • Set System Audio Device to Soundflower
      • Set Movie Recorder to capture from Soundflower
      • Use Audio Analysis for optional sound play-thru
    • Recording from audio interface:
      • Set Movie Recorder to capture from specified device
      • Play audio
      • Use Audio Analysis for optional sound play-thru
  5. Demonstrate and record the Kuleshov experiments again with:
    • Silence
    • Music (Dramatic)
    • Music (Silly)
  6. Optional: Add a second Movie Recorder plugin to capture from the web-cam and record new short reaction shots, without sound

Note: In a classroom setting, you may find it useful to skip having the part of the exercise where the students use Soundflower and instead choose the music selection to play over speakers.

Lesson 2: Rhythmic Sequence

Handout: The Rhythmic Sequence

Eisenstein’s montage theories are based on the idea that montage originates in the “collision” between different shots in an illustration of the idea of thesis and antithesis. His collisions of shots were based on conflicts of scale, volume, rhythm, motion (speed, as well as direction of movement within the frame), as well as more conceptual values such as class.

Rhythmic editing is when the relations between shots function to control visual pace and meaning. A shot’s physical length corresponds to a measurable duration. Rhythmic function occurs when several shot lengths form a discernable pattern. For instance, equal length between cuts will create a steady metrical beat.

You’re creating rhythm almost immediately after you begin performing – it’s inevitable once multiple elements appear on the screen. In this case, we’d like that rhythm to be a little more planned, instead of placing elements randomly. There are three primary types of rhythm you can plan for:

  • Regular rhythm: Intervals between images are the same in duration (i.e., one second per image)
  • Progressive rhythm: The duration of images are changed over a progression, getting faster towards the end (2 sec, 1 sec, ½ sec, and so on)
  • Flowing (organic) rhythm: Occurs when the images or intervals are organic, used to create a feeling of visual polyphony. Think VJ’ing to wind chimes.

Lesson Overview

  • Sequencing content using Media Bins, Layer Controls, and Data Sources
    • Linear and non-linear playback
    • Cycles and Loops
    • Linking Data Sources
      • Keyboard
      • Clock
      • Audio
  • Create the three primary rhythmic sequences using provided video clips
  • Record edits with provided audio samples, experimenting with different tempos and structures (beats per minute, seconds, etc)

Special Equipment

Required:

Recommended:

Lecture Notes

Discussions

Demonstrations

Basic Regular rhythm

  1. Use the Step Sequencer to trigger media files
    • Load the “Simple Player” template
    • Add provided sample media files to media bin
    • Assign the next or random button in the media bin to Clock 1 > Every Beat
  2. Using the Clock
    • From the Workspace Inspector > Plugins, open the clock into its own window
    • Adjust Clock BPM to adjust rate of sequence playback
    • Note that clock can be synchronized to MIDI clock / sound input
  3. Basic quantization
    • Disable existing automations
    • In the Media Bin use the T: menu and set to On Measure
    • Trigger clips; observe the triggers are quantized to the measure
    • In the Layer Source Controls, click the clock button to enable quantization
    • Trigger clips; observe the triggers and playback are quantized to the measure
  4. Clock time to seconds:
    • At 120 BPM, 1 beat is 0.5 seconds (60 seconds / 120 beats)
      • At 4 beats per measure, at 120 BPM, 1 measure is 2 seconds
    • At 60 BPM, 1 beat is 1 seconds (60 seconds / 60 beats)
      • At 4 beats per measure, at 60 BPM, 1 measure is 4 seconds

File management in VDMX

Continuing from above…

  1. Selecting clips
    • Hold the shift key and click on a clip to select it
    • Hold the shift key and drag to select multiple clips
  2. Moving and copying clips
    • Click and drag a selection to move it
    • Hold option while clicking and drag a selection to copy it
  3. Pages
    • The sidebar in the media bin can be used to organize clips in pages
    • Use the +/- buttons to add and remove pages
    • Pages can be renamed
    • Drag / copy-drag media from visible page to sidebar
    • Inspect media bin
      • In the Menus tab, switch to the list of pages
      • Use controls to adjust which pages are visible in the Pages menu
    • Sidebar can be expanded or hidden
  4. The Workspace Inspector > Files
    • cmd+click on a media file to inspect it
    • Default settings for source, FX and composition controls can be set
    • File / page organization can also be accomplished from the workspace inspector

Create the three primary rhythmic sequences using provided video clips

Follow the below steps, or use the provided sample project and skip to step 3

  1. Use the Step Sequencer to trigger media files
    • Load the “Simple Player” template
    • Add provided sample media files to media bin
    • From the Workspace Inspector > Plugins, add a Step Sequencer plugin
      • Set the number of rows for the default track to match number of page clips
      • Use the track sub-inspector > options to disable normalized values
    • In the Media Bin inspector > Control, set the “Trigger by Index” data-receiver to the step sequencer track
  2. Adjust step sequencer track to adjust patterns
    • Change the number of measures (M:)
    • Change the total number of columns
    • cmd+click on a column to have it skipped
    • Use the Section Preset bar to create presets
  3. Create sequences based on,
    • Regular rhythm: Intervals between images are the same in duration (i.e., one second per image)
    • Progressive rhythm: The duration of images are changed over a progression, getting faster towards the end (2 sec, 1 sec, ½ sec, and so on)
    • Flowing (organic) rhythm: Occurs when the images or intervals are organic, used to create a feeling of visual polyphony. Think VJ’ing to wind chimes.

Exercises

Create the three primary rhythmic sequences using provided video clips

Follow the below steps, or use the provided sample project and skip to step 3

  1. Use the Step Sequencer to trigger media files
    • Load the “Simple Player” template
    • Add provided sample media files to media bin
    • From the Workspace Inspector > Plugins, add a Step Sequencer plugin
      • Set the number of rows for the default track to match number of page clips
      • Use the track sub-inspector > options to disable normalized values
    • In the Media Bin inspector > Control, set the “Trigger by Index” data-receiver to the step sequencer track
  2. Adjust step sequencer track to adjust patterns
    • Change the number of measures (M:)
    • Change the total number of columns
    • cmd+click on a column to have it skipped
    • Use the Section Preset bar to create presets
  3. Create sequences based on,
    • Regular rhythm: Intervals between images are the same in duration (i.e., one second per image)
    • Progressive rhythm: The duration of images are changed over a progression, getting faster towards the end (2 sec, 1 sec, ½ sec, and so on)
    • Flowing (organic) rhythm: Occurs when the images or intervals are organic, used to create a feeling of visual polyphony. Think VJ’ing to wind chimes.
  4. Use the Movie Recorder plugin to capture edits with provided audio samples, experimenting with different tempos and structures (beats per minute, seconds, etc)

Lesson 3: Cinéma Pur

Handout: Cinéma Pur

From the 1930s to the 1950s, montage sequences often combined numerous short shots with special optical effects (fades, dissolves, split screens, double and triple exposures) dance and music.

The Cinéma Pur movement was to create a cinema that focused on the pure elements of film like motion, light, visual composition, and rhythm. It was begun by European filmmakers René Clair, Fernand Léger, Hans Richter, Viking Eggeling, Walter Ruttmann, Man Ray and Marcel Duchamp. Often these abstract art films showed patterns in motion, creating visually pleasing or intriguing compositions that reflected cinema’s essential power. Films like Man With A Movie Camera, Ballet Mécanique, Symphony Diagonale, and Berlin: Symphony of a Great City are quintessential examples of Cinéma Pur.

For this exercise, you will experiment with the layering of indexical material to create new visual associations and meanings using a select range of optical effects. Experiment superimposing contrasting imagery — abstract and realistic, light and dark, graphic and photographic, fast and slow, human and machine. Be prepared to discuss the emergence of this third image with a statement of intent.

Lesson Overview

  • Two-Channel Mixer
    • Layer Modes (Add, et al)
    • Composition Modes
    • Alpha and Transparency
      • Layer Masks
      • Chroma Key
    • Hi-contrast graphics
  • Create three montage sequences using the provided video clips
    • Layer Opacity
    • Additive Mode
    • Layer Mask

  • Record edits with provided audio samples

  • Using longer clips as a backing track and mixing in elements over it.
  • Using clips with high motion vs slow motion, motion in different directions.

Special Equipment

Required:

Lecture Notes

  • Cinéma Pur Lecture Video

  • Cinéma Pur Lecture Slides

  • Cinéma Pur
    • Story/Narrative Reduction
    • Dziga Vertov, Man with a Movie opening shot
    • Film / cinema as an art form separate from literature / theater
    • Signature techniques include:
      • Stop motion
      • Slow motion
      • Sped up film
      • Timelapse
  • Introduction to Composition
    • Mixing
    • Fading
    • Additive vs Overlay
    • Transparency / Alpha channels
    • Masking
      • Luma Key (brightness)
      • Chroma Key (color)

Discussions

  • What are some of your favorite examples of usage of stop motion, slow motion, sped up, timelapse, etc?

Demonstrations

Two-Channel Mixer

Steps using templates,

  1. Load the ‘Simple Mixer’ template from the Templates menu
    • Add provided media files set 1 to project
      • Note that pages are shared between bins
    • Create a new page in bin 2 and add media files set 2
  2. Demonstrate Two Channel Mixer
    • Show that adjusting the mixer position changes the fade levels for each layer
    • Show fade / cut and auto-fade features
    • Inspect the mixer
      • Adjust auto-fade time
        • Optional: Assign MIDI or OSC control to Fade Duration slider
      • Adjust fade curves
        • Note that mixing defaults to additive mode
        • Reset fade curves to be full opacity for each layer in the middle (default for this template)
  3. Use the section preset for the mixer to demonstrate auto mixer by data-sources
  4. Technique tip: Using longer clips as a backing track and mixing in elements over it

OR

Steps starting from new project,

  1. Prepare layer arrangement
    • Add a layer to the project
    • Rename Layer 1 to Left and Layer 2 to Right
    • Open Layer Src controls and Layer FX windows for each layers
    • Use the ‘Use Source’ menu to select sample media for each layer
  2. Prepare the plugins
    • Switch to the Plugins section of the Workspace Inspector
    • Using the + button menu:
      • Add a second media bin
        • Set the second media bin to target Layer 2
        • Add provided media files set 1 to project
        • Create a new page in bin 2 and add media files set 2
      • Add 1 or 3 preview windows
        • Set the first preview to the main output
        • If using 3 previews, set preview’s 2 & 3 to layers 1 and 2
      • Add the Two Channel Mixer plugin
        • Set the left side to Layer 1 and the right side to Layer 2
  3. Arrange plugin and layer windows
    • Optional: Demonstrate more than one window arrangement
  4. Demonstrate Two Channel Mixer
    • Show that adjusting the mixer position changes the fade levels for each layer
    • Show fade / cut and auto-fade features
    • Inspect the mixer
      • Adjust auto-fade time
        • Optional: Assign MIDI or OSC control to Fade Duration slider
      • Adjust fade curves to be full opacity for each layer in the middle
        • Note that mixing defaults to additive mode
      • Create a section preset to store the default state
  5. Add an LFO plugin and use the sine or cosine waveforms to control the mixer position
    • Create a section preset to store the automated state
    • Demonstrate adjusting the duration / rate of the LFO
  6. Technique tip: Using longer clips as a backing track and mixing in elements over it

Layer Blending / Composition Modes, Alpha and Transparency

  1. Continuing from above, set the mixer to the mid-point so that the layers are evenly mixed at full opacity
  2. Demonstrate blend / composition modes
    • Switch to the Layers section of the Workspace Inspector
    • Inspect the layer composition controls for Left layer
      • Note that layers render from bottom to top; only the composition mode of the Left layer is needed here
  3. Over Mode and Alpha / Transparency
    • Switch to OpenGL Over mode
      • Note that Left layer now completely covers the Right layer
      • Switch to using content that has transparency
        • Movie with alpha channel
        • Interactive generator with transparency background
        • Interactive Checkerboard.fs generator, adjusting the alpha of the each color
    • Note that OpenGL Add / Over modes are most efficient, but all layers within a group must be using them
  4. Composition Modes
    • Switch composition mode to GLSL > ‘Difference.fs’
    • Switch composition mode to GLSL > ‘Multiply Blend.fs’

Masking

  1. What is masking?
    • Masking is the process of adding alpha channels to an existing stream; useful for compositing layers together in ‘over’ mode
    • Imagine poking holes in a piece of paper so it shows through to what is behind it
    • Masking category of FX includes options such as Chroma Mask (replace a color with alpha), Quad Mask and Shape Mask (cut out shapes and patterns from an image), and the Layer Mask (use a secondary video stream as a luma key)
  2. Luma Key examples
    • Load ‘Luma Key Example’ template
      • Note that the ‘Layer Mask’ FX is applied to ‘Front’ and only the sections where the ‘Mask’ layer is white show through
      • Demonstrate basic Layer Mask controls
        • Use ‘bright’ control to transition between ‘Front’ and ‘Background’ being totally visible, reset to middle when finished
        • Switch ‘Contrast between 1.0 and -1.0 to show inverting
        • Note that other controls can be used when stacking multiple masking FX
      • From the Workspace Inspector > Layers, inspect the Mask layer and note that the layer is hidden, but can still be previewed and used as a mask
    • Load the ‘Luma Key Mixer’ example template
      • Add the hi-con clips for the luma key bin
        • Hi-con, or ‘high contrast’ clips often work well for luma keys
      • Note that Layer Masks and Two Channel Mixers can be used together to create advanced setups
  3. Chroma Key example
    • Load the ‘Chroma Key Example’ template
      • Add provided blue screen clips to the Front Trigger bin
      • Trigger blue screen clip
      • Use Chroma Mask controls
        • Adjust blue color or use ‘sample’ button to set to a pixel from the video stream
        • Use ‘Show Alpha’ to preview the alpha stream
        • Try using ‘Hard Cutoff’ when dealing with noisy footage
        • Dilate and blur options can be used to tweak edges

Exercises

Create montage sequences using the provided video clips

  1. Basic additive mix
    • Load the Simple Mixer Template
    • Optional: Add ‘Movie Recorder’ plugin for capture
    • Create montages by adjusting mixer position:
      • Using longer clips as a backing track and mixing in elements over it
      • Using clips with high motion vs slow motion, motion in different directions
  2. Layer mask
    • Load the ‘Luma Key Mixer’ example template
    • Optional: Add ‘Movie Recorder’ plugin for capture
    • Create montages by adjusting mixer position and masking clips:
      • Using longer clips as a backing track and mixing in elements over it
      • Using clips with high motion vs slow motion, motion in different directions
Tags: