Introduction to Digital Signal Processing: Interactive Sound and Video

Instructor: Frank Spigner

Email: fas268@nyu.edu

Course Description:

This course will demonstrate, and put into practice, an interdisciplinary approach to computer-based signal processing for the purpose of creating real-time interactive media for use in visual, sonic, performative, and time-based arts practices. The course will provide an introduction to the history and theory of digital signal processing (DSP), aesthetic issues, contemporary practices, and will focus on Cycling ’74’s Max/MSP/Jitter, a visual programming environment first developed by Miller Puckett at IRCAM in the mid-80s and continuing today as being an industry standard and ‘lingua franca’ of interactive and multimedia artists and researchers.

Course Objectives:

• To introduce students to the history, materials, and techniques of computer-based digital signal processing.
• To introduce students to Cycling ’74’s Max/MSP/Jitter.
• To provide students with opportunities to develop proficiency in interactive and real-time media production using DSP.
• To orient students to a range of creative coding and hardware components to augment their use of DSP in Max/MSP/Jitter.
• To provide students with a framework to build upon after the completion of the course, as to allow them to further adapt Max/MSP/Jitter to their own specific needs and to be prepared to integrate new technologies into their creative practice as they emerge.
• To give students a resource for building custom tools in their own creative practice, such as creating Max for Live virtual instruments and effects, or as a platform for multichannel and interactive instruments, installations, or live processing.

Organization:

This is a hybrid/lab course with in-class lectures and discussion with assignments to be completed outside of class. There are several demonstration Max patches for each week to demonstrate new concepts, which are available to students for download.

Grading is as follows:

10% class participation

20% in-class presentations and assignments

30% midterm

40% final

Required materials:

Students are expected to attend class with a personal computer running Windows or Mac with Max/MSP/Jitter installed (Educational discount subscription available on official Cycling ’74 website)

Course Schedule:

Week 1

Course overview

Introduction to digital signal processing and examples of DSP used in art and music.

Introduction to Cycling ’74’s Max/MSP/Jitter.

Examples of music and art that use digital signal processing, physical computing, etc.

Assignment: Make sure Max/MSP/Jitter is installed and working properly

 Week 2

– Max Interface – the console, the inspector window, locking and unlocking patch/edit mode, placing objects

– Max/MSP basic syntax and signal flow

– Message ordering

– Types of data used in Max/MSP/Jitter – Numerical data, MIDI, Signals, Jitter matrices

– Types of messages – Bangs, Numbers (int, float), Words (or symbols), Lists

Assignment:

Make a patch using the button, toggle, number, float, comment, and print objects. The patch should demonstrate understanding of signal flow, all the objects covered in class, and the ability to print messages and numerical data to the console.

See also:

– Max Tutorial 1 “Hello” and tutorial patch “01mHello” (Help>Reference>Max Tutorials>”Hello”>”Open Tutorial”)

– Max Tutorial 5 “Message Ordering” and tutorial patch “05mMessageOrdering” (Help>Reference>Max Tutorials>”Message Ordering”>”Open Tutorial”)

-Basic objects

Objects introduced:

• Button
• Toggle
• Metro
• Messages
• Random
• Makenote
• Noteout
• Mathematical operators
• Number
• Flonum
• Jit.qt.movie
• Jit.pwindow
• Jit.fpsgui
• Jit.brcosa
• Mousestate
• Scale
• Adc~

Week 3

-Playing videos

-Basic video operation

Objects introduced:

• route
• key
• itoa
• print
• select
• pack
• counter
• coll

Week 4

-Camera live feed

-Jitter matrices

Objects introduced:

• jit.qt.grab
• loadbang
• jit.matrix
• jit.window

Week 5

-Recording video

-Compositing video

-Chromakey

Objects introduced:

• jit.qt.record
• jit.plur
• jit.op
• jit.lumakey
• jit.chromakey
• jit.xfade
• jit.charmap
• jit.map
• jit.repos
• jit.pack

Week 6

-Basic Midi

-Lists

-3D objects

Objects introduced:

• midiin
• noteout
• stripnote
• notein
• mtof
• attrui
• playbar
• qmetro
• trigger
• jit.gl.render
• pak
• jit.gl.gridshape
• patcher
• screensize
• unpack
• drunk
• umenu
• prepend
• jit.gl.texture
• jit.rgb2luma
• jit.dimmap
• jit.gl.handle
• inlet
• outlet

Week 7

-Midterm Presentations

-Manipulating video with sound

-Audio filters

Objects introduced:

• adc~
• peakamp~
• sfplay~
• change
• reson~
• f
• bucket
• clip

Week 8

-Sending and receiving values

-More about subpatches

Objects introduced:

• value
• send
• receive
• jit.submatrix
• fffb~

Week 9

-Physical computing with camera feed

-Presence detection

-Motion detection

Objects introduced:

• jit.3m
• pgmout
• jit.scissors
• jit.findbounds
• vexpr
• prepend

Week 10

-Multiple camera feeds

-Intro to computer vision

Objects introduced:

• jit.change
• cv.jit.centroids
• cv.jit.dilate
• pictslider
• cv.jit.open
• cv.jit.label
• cv.jit.blobs
• cv.jit.LKflow

Week 11

-“Sketching” objects with open GL

-Intro to JavaScript for Jitter

Objects introduced:

• jit.gl.sketch
• js

Week 12

-Interact with file system outside Max

Objects introduced:

• udpreceive
• udpsend
• folder
• thispatcher
• tosymbol
• sprintf
• filepath
• itoa
• delay
• bucket

Week 13

-More about 3D rendering

-Procedural geometry

Objects introduced:

• jit.noise
• jit.gl.nurbs
• jit.expr
• qmetro
• jit.slide
• jit.gl.videoplane
• jit.gl.slab
• jit.gl.mesh
• phasor~
• jit.poke~

Week 14

-Image to sound

Objects introduced:

• jit.peek~
• sig~
• snapshot~
• rate~
• oscbank~
• mtof~
• jit.linden
• jit.str.tosymbol
• kslider

Week 15

-Final Presentations

-Miscellaneous

Objects introduced:

• split
• gate
• coll

ATTENDANCE / PARTICIPATION

Attendance is mandatory. Absences for religious holidays & family emergencies must be cleared with the instructor in advance.  

Absence due to illness will only be excused with a doctor’s note.  

Arrival to class more than 5 minutes late or leaving early will be counted as ½ of an absence, subsequently affecting your participation grade. 

If you miss an exercise presentation because of an unexcused absence, you will receive a grade of 0 

Three absences will result in failure of the course, in keeping with Tisch policy.

TISCH POLICY ON ACADEMIC INTEGRITY       

The core of the educational experience at the Tisch School of the Arts is the creation of original academic and artistic work by students for the critical review of faculty members. Any attempt to commit plagiarism or cheat is in violation of Tisch School of the Arts community standards. All confirmed violations of the Academic Integrity Policy will be reported to the department chair and the associate dean for student affairs and subject to disciplinary action. Penalties for violations of the Academic Integrity Policy may range from being required to redo an assignment to dismissal from the school. The full Academic Integrity policy can be found at: 

Click to access Tisch_Academic_Integrity_Policy.pdf

Your health and safety are a priority at NYU.  If you experience any health or mental health issues during this course, we encourage you to utilize the support services of the 24/7 NYU Wellness Exchange 212-443-9999.  Also, all students who may require an academic accommodation due to a qualified disability, physical or mental, please register with the Moses Center 212-998-4980. Please let your instructor know if you need help connecting to these resources