Technische Universität München Lecture

I have a month to work on a 45 minute lecture on my work at the Technische Universität München. I will also be doing a lecture at TU Darmstadt in November. I would like to use this opportunity to try to bring my video experiments into focus and incorporate it into my previous time-lapse projects.

This could take the form of a video series of me playing the prepared video buffer, possibly with explanations about what is causing what effect and demonstrative images / videos ? It could be contrasted with a “typical” workflow using the constraints of a software platform like photoshop? I could prepare this a similarly focused investigation of how the prepared piano altered sound, how Sonia Sheridan altered the photocopier, Paul Schafer made custom concrete music machines, etc.

EDIT* : I had a try !

• It’s kind of long and I also need to get a better top view.
• Adding text to describe what’s happening ?
• I also see that when I’m turning knobs the device is kind of hidden. I haven’t really designed the interface to be playable and to look good and be legible while being played.
• Looked into some other examples of video synth top down videos :

This is making me think more about the physical presence of my devices, and how I could incorporate more mechanical parts – instead of purely making electronics. I may have to play the game a bit more, make things that look like what they do and help people understand the associations I’m trying to make.

I could have micrometer heads for the potentiometers :

I could have fiber optic cables too :

I think if I every try to make a product though I would need to team up with someone who has skills I don’t.

***

Why make hardware ?

• it can be analog and digital
• it’s faster at doing things with video than software
• it can be custom
• it has a live-playable interface

***

What effects am I creating ? (check out Premiere to see what kinds of effects exist).

• Recording and playing things back at different speeds (video transposition), assigning to different color channels
• desynchronizing recordings with the screen sync so that they jump around
• recording videos side by side in memory so that they bleed over one another.
• Video collage and montage composer (video sampling, video looping), video plastifier, étude II, micro-montage, noise research, play, do and see, polyvideo layering, defamiliarizing of the image + reflecting the mediascape we inhabit, video as “palpable”, “nontheoretical”, and “experiential” (quotes from musique concrete wiki).
• The structure of the videos is based on the indexicality of the medium (the SRAM and how it stores). So the recording technique is also the synthesis technique.

Struggling to represent the painterliness of the raster videos without showing them as a video. It is completely different from my experiments with lines and triangles, and is less intuitive for me. An attempt at a kind of chronogram of a video :

It gives some idea of the range of images I guess?

Here are some notes I took about a year ago after visiting a painting expo that relate to my painterly video experiments :

• Layering
• Contour
• Surface
• Saturation
• Flow vs. Choppiness
• Roughness, relief 
• Edges versus centers
• Movement vs stasis 
• Enriching of detail
• Abstracting, cosmic making
• Fine lines versus coarse
• Exposure 
• Color fields
• Texture
• Deep friedness
• Films of cows, ocean. Stills of still life? Flowers, birds, drapes folding, mold, fruit, bones
• Color mixing vs melting vs meeting
• Paint viscosity (runny vs clotted)
• Direction of brush strokes 
• Contour
• Focus direction in canvas
• Translucent flesh, shine vs glow from interior, waxiness 
• Graininess 
• Composition, color pairings and form
• Foreground displaced to background vice versa 
• Meeting pattern planes 
• Lighting and shadow
• Illusion of depth
• Cloudiness 
• Turbulence

Check out this “glitch” video artist Jacques Perconte that Marc shared with me : https://www.youtube.com/watch?v=8_Xhu9Vx5XM

He works, since the 1990s, with compression and achieves a painterly, impressionistic effects and exposes them on immense projection surfaces.

Check out also Tauba Auerbach’s work :

Jack Whitten :

*************

Some quotes from Art21 series :

James Turrel – what you are perceiving is perception itself. We don’t receive the world around us we create it.

Jeff Wall – Art gives you an experience that alters something, it doesn’t tell you or convince you.

Jeff Wall – When I started photography there were potential energies in the medium that weren’t being realized.

Roland Barthes – The punctum points to those features of a photograph that seem to produce or convey a meaning without invoking any recognizable symbolic system.

Olafur Eliasson – Art provokes a negotiation : why I am seeing this the way that I am seeing it, what does looking mean? Instead of questioning the object you are questioning yourself. Art offers the opportunity for self-evaluation.

Tauba Auerbach – I want to learn things all the time, I want to understand the patterns behind things. I experience science through craft. With craft I try to cultivate sensitivity in a practiced, purposeful way. If you work with marbling you know just as much about viscosity and flow than a scientist, through your fingertips in a different way.

Tauba Auerbach – Ideally I want to make an image that has a tiny effect on all the future images that you see.

Tauba Auerbach – The pursuit of the sweet spot, cultivate the place of boundaries, limits of fraying, not hard edges.

Cindy Sherman – I don’t know what I’m looking for until I see it.

Richard Serra – Artists invent strategies, tools, techniques, that allow themselves to see in a way they haven’t seen before to extend their vision beyond the standardized classic reflex actions. These help us to see inside what we’re doing so we don’t get into a lock step notion of how to do what you do.

Louise Despont – If you’re always making work for someone, and not for yourself, maybe you don’t let yourself make the mistakes that are necessary.

Louise Despont – Each drawing is a series of tiny discoveries, it reveals/unfolds the drawing from total control. I own at most 1/4 of the drawing. The rest is something else, this is what’s exciting.

Louise Despont – It’s best explained in the drawings, words are clumsy to describe something like the concrete feeling of spirituality.

Liz Magor – My Studio is a space to reconcile dissonance, no one knows I’m here. When I’m not in my studio I want to be in my studio working. My tools are rudimentary. I’m here for pleasure but it’s not fun. Everyone should have a studio for mental health. I can filter out the noise and see the ever present under the radar stuff.

Liz Magor – I’m not an animist but objects have stuff in them that comes out. I try to resurrect these objects from the netherworld.

Liz Magor – The slowness of the material process matches the slowness of my thinking process.

Liz Magor – I’m creating an experience for looking.

Liz Magor – I give myself my own program, I give myself my own assignments. And art is the choices I make.

Liz Magor – Maintaining the conditions for this production, which is not very logical, uncalled for. No one is asking me to do this, I’m barely asking me to do this. To do this I have to overlook my making with the journey of the things into the world.

Liz Magor – After the casting I unwrap it like a little gift, there are surprises.

Elliott Hundley – This photoshoot is so elaborate but not because I want a certain effect, because I want to not control the effects, and have a layer of unexpected results. In relinquishing control, the piece gives me something back I didn’t expect.

Jack Whitten – I’m not a narrative painter.

Jack Whitten – I built a device to move large amounts of acrylic paint in one gesture.

Katherina Gross – Painting is not linear. The synchronicity in painting is compelling for your thought process. I’m trying to grasp some of those fast thoughts that are moving through my brain.

*************

Reading about the history of video synthesis.

From https://museumzero.blogspot.com/2013/12/its-all-baseball-nam-june-paik-starts.html)

Nam June Paik sees Taoist energy in working with the flow of TV video :

“To receive simultaneously the parallel flows of many independent movements is, as Paik pointed out at the time, a classical Taoist way of meditating: by becoming aware of everything going on in the present, you discover eternity right now.” 

“I love Chinese history, surprise, nonlinearity.  You know what judo means?  A way to be soft.  You let other guy do all the work.  I think in some Oriental way.  In TV I try not to be boss, to stay as small as possible.  So that seems to work. “ 

“So then when I started TV, the best decision I made in life is not really to go into TV, but to do work inside TV.”

“Imagine everything that exists is flowing, like the tao, in a perpetual motion, with endless waves, like the ocean.  We come from it; we go back to it.  Whatever surges forward falls back, so Lao-tse, who wrote the Tao Te Ching (On the Nature of the Way), urges us not to try too hard.  Relax; be like a baby; follow whatever path presents itself.  That is the way. ”

Not striving to fit yourself to some preconceived ideal, like a Buddhist or a Confucian; simply being, following breath wherever it blows.  In this sense, the television signal is like the tao; endlessly variable, continually reversing itself, dying, being reborn, it is never idle, yet it is so vast and perpetual that it often seems calm as the sea.

Beginning with whatever live programs came along, distorting them, then recombining his distortions like waves on the ocean surface, Paik created an indeterminate and endless show for the Galerie Parnass, in Wuppertal, Germany, in 1963.  

From Scanimate: The Origins of Computer Motion Graphics :

• This is about the history of television, these were the first ways to put text and make graphics on screen (Media Archaeology!). Nowadays visuals are gratuitous. What did we lose in the transition to digital ? The imperfections, the proximity to the functioning.
• Had a role in music videos, and news segments, so was part of culture.
• “Blooming” is the term for the glowing edges in CRT Scanimate.
• It’s real life artefacts. It’s like exploding scale models instead of GCI.
• They made cookbooks with the recipes of different effects that created while working with clients.

From https://wearethemutants.com/2018/01/09/a-sloppy-machine-like-me-the-history-of-video-synthesizers/

Like its close temporal and conceptual counterpart, the audio synthesizer, the video synthesizer was created iteratively by academics, artists, and tinkerers, then eagerly snatched up by the world’s biggest media producers once prototypes had proved their power and versatility.

• Started with experimental musicians like John Cage and la musique concrète.
• Early pirate TV station in NYC.
• Amiga supplants the Scanimate and analog synths.
• The 1967 Portapak release and it’s importance for activism and experimental art.
• Video comes from the root video of Latin video (“I see”).
• There is a relationship between images and waves  :

CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=641492

This is so hard for me to understand but this somehow relates to chirping also :

By Drummyfish – Own work, CC0, https://commons.wikimedia.org/w/index.php?curid=77608151

More cool matrix math image modifications here : https://en.wikipedia.org/wiki/Digital_image_processing

Also this website which describes it all visually ! https://setosa.io/ev/image-kernels/

From Video Art : An Anthology :

• • The synthesizer let’s you see things that exist only in the mind’s eye.
  • visual ingredients
  • video synthesizers churn out…images based on their own electronic structure.
  • We make the image with our eyes and brain. It’s psycho-visual
  • video as concretized imagination.
  • video is surreal, the images are ephemeral products of micro electronic pulses.
  • “electronic imagery”
  • video is a medium which works with time

The Vasulkas

*************

Here is an attempt to describe what could be a model of a potential “practice” :

*EDIT* This should include me blogging ! Maybe add a RESEARCH and/or DOCUMENTATION phase ?

*******

Check out this pyramid of types of work :

I feel like my project is about analysing (and curating), I’m not sure if I take a real position.

*******

Check out this amazing NIME article called The Concentric Sampler: A musical instrument from a repurposed floppy disk drive by Timothy Tate about “the redundancy and physicality of magnetic recording media” which utilizes “time-based granular synthesis” and lo-fi:

https://nime.pubpub.org/pub/uh76shf0/release/1

Some quotes :

• Physical media offer an opportunity to “foreground the physical medium as a tool for musical expression.”
• “The deterioration of old magnetic tape as it is looped, defining a … musical structure.” The “failure of the medium is in focus, rather than the mechanism.”
• “as soon as something’s on tape, it becomes a substance which is malleable and mutable and cuttable and reversible in ways that discs aren’t.”
• “This study emphasises working within a medium’s limitation, rather than prescribing a compositional framework or sonic agenda upon it”
• Motivation to “foreground musical and technical possibilities…and generative sonic potentials…and novel, unique expressive and
• compositional possibilities”…to be “discovered…explored”
• The device is a “performable instrument and a generative sonic tool”
• “Rather than focusing on the authenticity of the sound reproduction, the Concentric Sampler draws focus to the byproducts and the resultant indexicality and generative sonic possibilities of the floppy disk as a medium.”
• “the term ‘musical expression’ here places value on the ability to shape and define a musical grammar from the indexicality of the …medium and mechanism.”
• “subverting traditional modes of … expression“
• “Each section is accompanied by a demonstrative video” !!!
• The live playable instrument can “achieve the dissociation of pitch and time“
• “imperfections in … reproduction”

• “possibilities within a framework of … making”

Now following up on some of the references from this article:

• You can think of an instrument as a framework that has a predisposed range (affordances). Our tools provide a certain range of suggested usages. Some things are easier and some things harder to do with an instrument, this is the instrument’s spectrum. If you stay within those ranges, you accept the silent grain of the instrument. If you make an unusual tool then, do you risk finding yourself in an unusual affordance ? (J. Mooney) Could it be possible to make videos subtractively, instead of additively ?
• From New Media Reader : The different between representation and simulation. P.18″Digital media process the physical properties of the input data, light and sound waves, are not converted into another object but into numbers; that is, into abstract symlbols rather than analogous objects and physical surfaces. Hence, media processes are brought into the symbolic realm of math rather than physics or chemistry. Once coded numerically, the input data in a digital media production can immediately be subjected to the math processes of addition, subtraction, multiplication and division…” AND compressed AND manipulated more easily than analogue forms because they are more mutable than objects. Also easy access and quick movement of the data of an enormous mass of data. This causes the materiality of the world around us to become unsteady…The televisual
• Because everything else is also digitized, there is a “convergence of previously discreet media forms, now a new fluid area of media decentralized production and consumption. Remediation, Multilinearity.
• Hand-made videos. Exploring the boundary between hand tools and large-scale machines which is a distinction Marx makes P.91.

Check out these Bauhaus era artist’s work with the audio band in video :

Paul Schaeffer (coiner of the “club d’essai“) designed specific tools like the morphophone and phonogène for modifying audio for musique concrete :

*******

Watching ENSCI diploma presentations I’ve thought about how nice the sequence is : first you write something that explores a topic and cites thinkers on the subject, and then you work on a design project that relates to this somehow. Also looking back at some videos, there is already a lot of content I have produced that I could try to learn from ! Like that Amsterdam-based artist told me, I could also focus on using my tools.

I am also doing a synth workshop in late September at Villette Makerz but I’m not sure if I can connect to that or not here. *EDIT* an FPGA board that generates patterns based on audio input is a GO !*

I also want to make a next FPGA board which will be a software reconfigurable device so that I can move to verilog for the summer instead of making a new board for every experiment. Not sure if this could also take the form of a simple video synth for Villette Makerz or at least a simplified version perhaps.

*******

Some quick looking in to GPUs :

GPUs are all about direct memory access (DMA) not mediated through the CPU. A thought about the FPGA board – perhaps I should first have Arduino mess with the clock while playing back / recording something on the 16MB board (but I’m limited to the number of pins for address and I/O – it would essentially be a counter that could change in frequency on the fly)? Having memory connected to the screen (a screen buffer) and modifying it is the fundamental computer display setup. From 1951 a memory pattern on CRT :

It would be cool to experiment with sprites and copying parts of memory from one bank to another too like a blitter https://en.wikipedia.org/wiki/Blitter .

It looks like what makes shaders cool is that they are programmable and have their own code languages now. Before shaders there were hardware rendering pipelines called fixed-function. Different units would be responsible for specific functions. It seems like these units would take in vertex data then output pixel colors.

I looked in to how to do matrix multiplication in hardware. It actually seems to be more natural as an analog circuit with resistor arrays. It can also be done by multiplying two 2-bit binary numbers using this kind of logic setup :

**********

I am becoming more and more curious about the difference from having a screen buffer that is written to during the blanking period and having a massive bank of memory that can be recorded to like in the 128Kbit. Check out these articles about how writing and reading to video memory is dealt with in different systems :

https://en.wikipedia.org/wiki/Tiled_rendering

https://en.wikipedia.org/wiki/Multiple_buffering

https://en.wikipedia.org/wiki/Vertical_blank_interrupt ,quoted from : 

“During the vertical blanking interval, the driver orders the video card to either rapidly copy the off-screen graphics area into the active display area (double buffering), or treat both memory areas as displayable, and simply switch back and forth between them (page flipping). [The two buffers are called the front and the back buffer] Some graphics systems let the software perform its memory accesses so that they stay at the same time point relative to the display hardware’s refresh cycle, known as raster interrupt or racing the beam.” [i.e. you can also draw things during the horizontal blanking interval]

**********

The Real-Time Corrupter is fascinating https://redscientist.com/rtc . The kinds of memory modifications it allows for are fascinating and far more dynamic than static, text-based find-and-replace experiments I’ve done in workshops. Looking at this video I learned some cool stuff (https://www.youtube.com/watch?v=n9HS6zftuSk&list=PLItZ3jvJKD7rnoxmqJJ0B5E1B9_WW9L5E&index=2). For instance, copying values from one address (in video RAM, scrolling backdrops memory space a.k.a. “nametables”, NES RAM, ROM etc.) and then reapplying them to another address every frame (called piping) or just one time, the ability to isolate specific effects with the sanitize function. It can generate noise between certain values, or shift values up or down on (called tilting), listen to a value and then reapply it after a fixed number of frames, freezing the current value somewhere for future frames, replacing values intelligently with values in certain sets (for 3D glitching). It’s like performing brain surgery while the patient is fully conscious and telling a story. All of these memory transformations could be applied with the FPGA board !

**********

In a similar spirit, this game called Code War involves creating a program that competes with others to write to all of the memory space. There are competing strategies (quoted from https://corewar-docs.readthedocs.io/en/latest/corewar/strategies/) for warriors :

Rock - a warrior which rapidly bombs the core with dat instructions
Paper - a warrior which replicates, creating multiple, parallel copies
Scissors - a warrior which scans the core looking for other warriors

The battle is visualized in a memory grid (https://corewar-docs.readthedocs.io/en/latest/corewar/visualisation/)

You can simulate and battle warrior code here : https://crypto.stanford.edu/~blynn/play/redcode.html

**********

Visualizing memory access in time. From https://bling.kapsi.fi/blog/x86-memory-access-visualization.html:

*********

Some thoughts on HCI from people at work :

it’s research if it follows scientific method and has possible applications.

Find your peers, try to publish in their journals.

The idea of publishing is to get feedback from your peers.

If you make a pedagogical tool, then you could test how effective it is at helping people learn.

Based on this conversation, I thought about how I don’t necessarily want to promote engineering education and that I always want to be on the design/art side of things. I also have learned that I find doing something I already know how to do less exciting, and so making simple kits or small series of artworks that are guaranteed to work for days on end, is not always super rewarding for me intellectually. This leads me to the current conclusion : I want to teach and then do a kind of artistic research on the side. I should then keep building my teaching career and try to get expos of my work. I am still trying to figure out how exactly my artistic research should be communicated (through articles in design magazines, through conferences with expos connected to them?).

**********

I am preparing a solar workshop at Villette Makerz based on the simple solar engine with optimized SMD components. The idea is an art bot that will draw, paint, in the sun. I am working on testing the various combinations of solar panels, capacitors and voltage supervisors.

We are also making a version 2 of the video synth modules which are all Eurorack compatible in terms of dimensions, use 3.5mm audio jacks, the eurorack power supply. The idea is that the modules be both manual and automatable for the purpose of demos and to be didactic for students learning about analog electronics.

Marc is doing a great job reimagining this project – he is good at compromising and imposing the vision of the final thing on each step of the construction. I am, in contrast, driven astray I think by my fascination by what “the machine itself wants to express”…

********

Finally getting around to testing the Arduino-controllable analog switch to activate record or READ mode on the SRAM 16MB board. The idea is to test taking a recorded video and then doing some bit manipulations with it and saving it back to the memory. Predictably though, I’m having issues just writing anything to the thing…

I/O pins on Arduino 0-7
CLK on Arduino 12
REC on Arduino 13 connected to analog switch connecting the WR pin on SRAM to either VCC or !CLK

unsigned long memory; //total number of 8bit words on the 16Mb SRAM = 2048000

const byte IO_0 = 7; 
const byte IO_1 = 6; 
const byte IO_2 = 5; 
const byte IO_3 = 4; 
const byte IO_4 = 3; 
const byte IO_5 = 2; 
const byte IO_6 = 1; 
const byte IO_7 = 0; 

void setup() {
//Serial.begin(115200);
pinMode(13, OUTPUT);
pinMode(12, OUTPUT);

pinMode(IO_0, OUTPUT);
pinMode(IO_1, OUTPUT);
pinMode(IO_2, OUTPUT);
pinMode(IO_3, OUTPUT);
pinMode(IO_4, OUTPUT);
pinMode(IO_5, OUTPUT);
pinMode(IO_6, OUTPUT);
pinMode(IO_7, OUTPUT);

//WRITE
while(memory<= 2048000){
memory++;

// if (memory%10000==0){
// Serial.println(memory);
// }

if (memory%5==0){
PORTD = B11110000; // the IO pins. simple pattern test.

}
else{
PORTD = B00001111;// the IO pins. simple pattern test.
}
//WRITE MODE
PORTB = B11101111; // PIN 12 ARDUINO CLK goes LOW/HIGH
PORTB = B11111111; // PIN 13 ARDUINO WR STAYS HIGH

}
// Serial.println("finished writing");
}

void loop() {
pinMode(IO_0, INPUT);
pinMode(IO_1, INPUT);
pinMode(IO_2, INPUT);
pinMode(IO_3, INPUT);
pinMode(IO_4, INPUT);
pinMode(IO_5, INPUT);
pinMode(IO_6, INPUT);
pinMode(IO_7, INPUT);

digitalWrite(IO_0, LOW);
digitalWrite(IO_1, LOW);
digitalWrite(IO_2, LOW);
digitalWrite(IO_3, LOW);
digitalWrite(IO_4, LOW);
digitalWrite(IO_5, LOW);
digitalWrite(IO_6, LOW);
digitalWrite(IO_7, LOW);

//PIN 13 REC IN LOW READ MODE
PORTB = B11011111; // PIN 12 ARDUINO CLK goes LOW/HIGH
PORTB = B11001111; // PIN 12 ARDUINO CLK goes LOW/HIGH
}

unsigned long memory; //total number of 8bit words = 2048000
byte cell; // store the word at this address

void setup() {

//PD0 - PD7 are IO
pinMode(13, OUTPUT); // REC - PB5
pinMode(12, OUTPUT);// CLK - PB4

 while(memory<= 2048000){
    memory++;
//READ
        DDRD = B00000000; // all IOs in INPUT
        PORTD = B00000000; // no pull-ups

        PORTB = B11011111; // WR stays LOW + CLK HIGH
        delayMicroseconds(10);
        cell = PIND; // read the cell at this memory address and store it in the variable cell
        delayMicroseconds(10);
        PORTB = B11001111; // WR stays LOW + CLK LOW
        delayMicroseconds(10);

//WRITE
       
        DDRD = B11111111; // all IOs in OUTPUT

        PORTD = ~cell; // write  a transformed cell back to memory
        PORTB = B11111111; // WR stays HIGH + CLK HIGH
        delayMicroseconds(10);
        PORTB = B11101111; // WR stays HIGH + CLK goes LOW
        delayMicroseconds(10);
  }
}


void loop() {
//READ MODE TO SEE WHAT WE HAVE IN MEMORY
        PORTB = B11011111; // WR stays LOW + CLK HIGH
        delay(500);
        PORTB = B11001111; //  WR stays LOW +  CLK LOW
        delay(500);
}
Not working either...