Should video be interactive?
I wrote this informal essay for my advanced video class as part of my MFA studies a few years back, responding to Ann-Sergeant Wooster’s essay in the volume Illuminating Video — “Reach Out and Touch someone: The Romance of Interactivity”.
Two Decades Later: Is the romance of interactive art still a complicated relationship?
“Reach Out and Touch Someone”
Imagine this now defunct advertising campaign being the slogan for Facebook with all of its unintended implications. Ann-Sargent Wooster, by the way, as of February 17, at 10:26PM, updated her Facebook status thusly:
I just signed up for a running class. Another of the things I never thought I’d do in the 8th grade.
I neither know Ann-Sargent but the settings on her “wall” are sufficiently public enough that I can get a striking micro-snapshot of her, not just personal and intimate, but also timely. When Ann-Sargent Wooster wrote her essay “Reach Out and Touch someone: The Romance of Interactivity” the world was on the precipice of a communication revolution, and reflecting on a technology driven essay from two decades past feels simultaneously like tabulating buggy whip production data in 1915, and looking at DaVinici’s helicopter sketches. In the year 2011, we are in the know of changes set to occur, and we praise the authors of the past who are prescient of the changes to come.
We also point to artists who seem to know what is to come next and congratulate them for such judiciousness. It comes as no surprise that Nam Jun Paik coined “electronic superhighways“1, as he knew the business he was in was communication, and Marshal McLuhan emphatically concurred, “The artist picks up the message of cultural and technological challenge decades before its transforming impact occurs.“2 Yet the romance of interactivity is still a mysterious and unknowable tryst for many artists and media makers. In some ways, it’s a failed and doomed relationship.
Wooster’s humorous description of the interactive fiction games from the late 1970’s is familiar to anyone who has frustratingly screamed at a screen. She describes the process of engaging with the computer’s “700 word vocabulary”. Methodologies have advanced to give the computer more input for parsing what the user is attempting to tell it, comparing lexicons with expansive tuples. A lexicon is simply a list of vocabulary words. A tuple is an ordered list of any data type. A scanner method can be built that convert a series of data input from a user into chunks, and checks the lexicon against the tuple, and then parses the data to simpler results for the program to understand.3 Stored lexicons have become so large and useful and the methods which compare data have become so refined that we are now at the point where a computer program can parse words enabling it to defeat even the staunchest of “Jeopardy” contestants.4
Now that our machines can certainly understand us a bit better, has this interactive control helped us get a grip on two way communication in context with technology and video? Wooster could see the way we would come to consume interactive art and media.
It was not only until the development of computers’ capacity for random access memory and the most complex interactive disks that they could fully realize their vision of a complex simultaneous web of images and ideas.
Her description of the mechanisms of Interactive Video Disks felt real and relevant in 1991. Oddly, the basic methodologies and technologies of interactive video playback in a physical media form hasn’t changed much in the two decades since. Artists and video makers are still wrapping their heads around how to produce real and relevant statements using the promise of interactivity. Interactive discs (with the rise of the DVD in the 2000’s) have changed only modestly in technology and approach, and have not brought rise to a deluge of new means of expression, storytelling or modes.
Conversely, the internet and the exponential increase of personal computing power has brought on huge advances in many fields, art and media emphatically so. So why did interactive film and disks stall? The basic fact remains that this kind of information parsing still yields pre-recorded video, or pre-written text for the user or audience to engage with. Wooster states:
Interactive fiction offers neither the pleasure of being swept away given by watching a film or reading a book or the joy of creating something new.
While working on Hasbro’s VideoNow, I was tasked with a team of other artists to create a small series of interactive video discs for popular children’s shows. We had several restrictions of our hardware, namely, the RAM of our small system was limited to one byte (one number we could increment or decrement from 0–255) we had a small number of inputs the system could recognize, and we couldn’t put in any orders to have custom content created from the show’s production teams. Any new content had to be created by our small team of artists. Interactivity seemed nearly impossible with such little resources. While flowcharting out our first experience for The Batman, we had long branches of possible streams of video on the whiteboard. The game essentially progressed as video playback that would have timing windows. Clues on screen when Batman was fighting Joker during a fight scene in the show would indicate with an on screen display when to press a certain button (a joypad and action buttons) and if the button wasn’t depressed and stored by the end of a video segment, you essentially had to start the game over. This is exactly the game mechanic of Interactive Video Disk’s most popular game title, Dragon’s Lair.5
Our early versions of The Batman wasn’t particularly well received. It was far, far too difficult. Dragon’s Lair was maddening in its difficulty, as well, one reason it got so many quarters from obsessed nine year olds. It made up for its difficulty in the engaging animation produced by the legendary Don Bluth for its many streams of video. So our first lesson learned of interactivity when concerned with video was an incredibly important one. If your audience can’t reach your content, you might as well not produce it. Wooster quotes Leo Steinberg claiming this is less of an issue for artists. “Unlike the clear, direct messages of a street sign, one of the qualities of great art is its ambiguity”. Ambiguity is useful for creating subtext and layers of meaning, but it is useless when this content is hidden away on a secret compartment of a disc your audience can’t access. To remedy the difficulty of Batman, we had to introduce check gates, sometimes called “save points” in game speak. This meant the user could at least start over from midway through the disk, as long as the unit was not turned off, and slowly make their way from checkpoint to checkpoint. This entire structure made for an incredibly tedious experience. We began investigating new forms of interactivity. We created a menu system, or home screen.
Creating a home screen, or in the world of the game, the central computer used by Batman in the batcave, gave a place for the user to rest and branch off on mini games. One of the mini games was an “action” sequence which used the Dragon’s Lair mechanic. A second mini-game provided a chance for the user to engage in analysis of on screen data. Batman (acting as the user’s digital representation) would investigate a detectives office for clues. There were several options to choose- a computer, a file cabinet, and a ledger. The ledger contained a rewritten report, readable by the user, on the color of Joker gas (green), and the file cabinet contained mysterious symbols. The computer had two screens that had multiple choices for the user to push a button. The first screen was a series of symbols in sequence that was the password into the next screen. The sequence of symbols was the same from the cabinet. The next screen was answering a question — “what color was the joker gas?” so the retention of the information from the ledger was needed. If the player chose the computer before choosing the ledger or the file cabinet, they couldn’t answer the questions and would return to the home screen. If they investigated, but couldn’t make sense of the data, they would probably also get sent back. This is the classic “Puzzle” mechanic, used exclusively by the classic interactive experience Myst. We learned another major rule of interactivity: The best computer in the room is the user’s brain. While code can go through a series of incredibly complex conditionals at the breakneck speed of microprocessing, it can’t make leaps of logic, or compare the similarities between two systems. This requires two humans, the designer, and a user to engage in the experience. This makes for a system of reciprocity, essential to the engagement of art.
On our third title, Teen Titans, we gained another vital lesson. While The Batman was playable, there was very little incentive to return to the experience once you got to the end of the story. With Teen Titans we introduced a scoreboard metaphor. This ranked the performance of the interactive video as the user played through the mini games. Your ranking could improve or decrease as you progressed through the experience. This encouraged the user to go back to the beginning of the experience and apply what they knew. This is a building block of an important interactive concept- engaging in a system, altering elements in the field of play, but not radically re-arranging the whole system.6 We use scoring to create a system of play, and gave the user reasons to engage with this system.
When the game or experience is purely systemic, an interactive experience is essentially simulation. This creates problem with authorship, voice, and communication of idea. The specific voice and concept of the initial artist or designer has the tendency to disappear. Artists, game designers, and technologists have struggled with this dilemma since interactivity began. How much of the experience is predetermined for sake of consistency of idea and style of the original work, and how much is created whole cloth by the user engaging with a system? While a curator would not put software like Adobe Photoshop in the Guggenhiem, it is an elegant system of user feedback that intends to let artists focus purely on creation while attempting to be as transparent as possible. Does the act of using photoshop itself preclude new types of form? When is a system restrictive as opposed to enabling?
As computer artists have approached this language, the choice of how much of the art is simulation as opposed to mapping datasets often becomes the art itself. While Marius Watz asserts that “Code is the language of the digital world, where algorithms articulate form as a function of process where each images is just one of an infinite number of results”.7 The same concept aligns neatly with the manifestos of fluxus artists of the 20th century who utilized performance and happenings. This belief, that the art they produce is essentially interactive art with infinite possibilities and variation, is more engaged with the audience and therefore more tangible or constitutes the true mimesis as opposed to the previous established visual language based upon one-way communication. Audience participation has a long history, but without a human brain on view to guide the proceedings, how much of the artist is left in the art? Video playback has to take the role of the performance of an artist.
Let’s apply the rules of interactivity that I learned as a designer for VideoNow to interactive modes of creation in video art.
1. If your audience can’t reach your content, you might as well not produce it.
One of the issues with branching playback is that there is a strong possibility that the user won’t be exposed to whole swaths of the work. This dissatisfies many artists and designers. While reading “choose your own adventure” books as a child, I often times became frustrated with the mechanic of choice, and read the books from front to back, to explore the alternate endings and branches of the stories that seemed unobtainable with the structure of the books. Even in the decidedly low tech form of print, without the addition of pixels or time based media, this maxim made experiencing the interactivity frustrating and limiting.
2. The best computer in the room is the user’s brain.
The best cinema challenges us. The worst cinema insults us. “I pity the French Cinema because it has no money. I pity the American Cinema because it has no ideas.“8 Pausing the playback of time-based media actually makes the dialogue less engaging for the audience because they are now the smartest computer in the room, and the artwork is the dumbest. Interrupting the director or artist’s intention of experience. The viewer should always be challenged, and the interaction with the artwork should elucidate the concept, rather then hinder.
3. Engage in a system, alter the elements, but don’t break the system.
The system of controlling time-based media streams has been a primitive system from the beginning. Users can pause and stop, choose playback, and alter occlusion (volume, picture, and distortion of the media). This system allows for editing, which is important for creators to create montage and build larger ideas, but the building blocks are more basic then the twenty six glyphs that comprise the letters of the alphabet. This proves problematic when handing the keys of the editing Ferarri to the teenagers of our audience. Subtext and subtlety are lost.
Wooster is excited for the possibilities of Interactive Disk Media, and she views the allure of interactivity as rife with romance for the creator. Many artists have been seduced by the romance of interactivity and have presented challenging and engaging experiences. One of the ways of working through these issues is to further explore and examine video’s nature of reproduction on computer systems, essentially, that of an array of pixel values in a dataset. The values of these pixels can be manipulated in interesting and coherent ways that add up to more then simple distortion.
Aaron Koblin and Chris Milk’s The Johnny Cash Project presents an edited video of the late icon Johnny Cash, and then isolates every frame with an integrated paint system that “encourages participants to draw their own portrait of Johnny Cash to be integrated into a collective whole. As people all over the world contribute, the project will continue to evolve and grow, one frame at a time.“9, thus the director retains ownership of the overall stream of the work, but the physical surface is changing and interactive as new artists discover the site and add their contributions, which are tagged and added to the database. The creators show their content, the computer serves the user’s brain, and everyone engages with the system when they alter the elements, but don’t break the haunting flow of the video.
Wooster would be impressed. Perhaps I’ll post the link on her wall.
Notes:
1. Paik, Nam June. Media Planning for the Post-industrial Society.( Reprinted in Nam June Paik, Kenworth Moffett, et. al, The Electronic Superhighway. New York and Seoul: Holly Solomon Gallery and Hyundai Gallery) 1995, p. 39–47.
2. Marshall McLuhan, Understanding Media (London, 1964, pp.64–66)
3. Zed A. Shaw, Learning Python the Hard Way (Nov 01, 2010)
4. http://www.nytimes.com/2011/02/17/science/17jeopardy-watson.html?_r=1&pagewanted=2
5. http://en.wikipedia.org/wiki/Dragon’s_Lair
6. Andy Cameron, Dissimulations www.mfj-online.org/journalpages/MFJ28/dissimulations.html
7. Marius Watz, Beautiful Systems: Software as aesthetic medium 2009.1030
8. Richard Brody, Everything Is Cinema: The Working Life of Jean-Luc Godard, (Metropolitan Books; 1St Edition edition (May 13, 2008)
9. http://www.thejohnnycashproject.com/