What does it feel like to move and be moved? Where does play become play-full, and make new meaning? Why is the body so often forgotten in front of our screens? Nathaniel Stern shares the story of his interactive art, and why it led him to the blockchain.
More than two decades ago — after excitedly monologuing to Professor Dan O’Sullivan’s class about an idea I had for an interactive installation — Dan O, as we affectionately called him, asked if I could create the installation such that the art’s participants “moved like I move when I talk.” This led me to so many questions: what does it feel like to be moved? Where does play become play-full? Why are our bodies so often forgotten in front of our screens?
In a post-pandemic and web3 world, I find myself again posing questions about bodies and screens, matter and meaning — but with new and different twists.
How does remote presence change our bodies, online and in real life? Can blockchain-based ownership of digital assets be physically felt in a moment? Can meanings be transformed by how we experience them, physically? By sitting down, via our laptops in bed, at a desk in a not-so-private cubicle at work? How do virtual spaces like Zoom, Twitter Spaces, or Decentraland create unique experiences? Where can generative AI amplify the work of interactivity?
Why are our bodies so often forgotten in front of our screens?
This article briefly looks back on some of my interactive art and writing over the last twenty-five-plus years. And it calls for new critical art and writing, now that the blockchain has such expanded potential in the digital realm.
Interactive art combines three pieces of technology: some form of sensor, like a camera, but beyond the everyday mouse and keyboard; a computer to process that input; and any form of sensory output — audiovisual, tactile, olfactory, mechanical, or otherwise. These three pillars are placed together in a system that responds to the participation of its viewers, and that participation is required for the system to realise its qualities as an artwork. The transition from viewer to participant occurs concurrently with the transition from system to artwork.
But this framing establishes a problematic hierarchy: an emphasis on the sensor, computer processing, and output, an emphasis on the tools we use, rather than the situations they create. If we explain what interactive art is primarily through technology, then we experience it merely as a technological object. Instead, we should emphasise what interactive art does — and what we do with it.
Affect is, most simply, unqualified emotion. My palms are sweaty; my heart is racing; I have butterflies in my stomach. These are embodied sensations that do not have a name or category until we “decide” on what we are feeling. Affect is not-yet emotion, which is abstract as much as it is embodied by us. It is intensity without direction.
Our pre-conscious affection is a moving-thinking-feeling that accompanies our conscious reflection. We sense affection when viewing beautiful works of art. We feel them immediately, only then to reflect on where and what and why, on their context and meaning. But affection and reflection are continuously at work, always influencing each other in our daily lives and relationships.
enter, the piece I eventually produced for Dan O’s class, attempted to explore the entanglements of affection and reflection. As I initially explained to my classmates in the year 2000, the work’s participants (I performed a half-spin) use their full bodies to grab animated words that constantly retreat from them in a large projection. If they touch any one, it stops, turns red, and recites related spoken word in the space. Words run away from us, as we turn on a phrase, or reach for the end of a sentence.
enter is not about reading words projected on a screen, and nor is it simply a choreographed performance in front of an image. It is a situation that highlights embodiment and meaning as always together. Embodiment: how bodies transform over time. Signification: the process of making meaning. Embodiment and signification emerge together, and enter makes that co-emergence felt.
I firmly believe art amplifies who we are, and more importantly asks who we could be. Across hundreds of exhibitions and articles, I always seem to wander and wonder around ecology, around affect — both human and non-human — and its effects.
Wait… nonhuman affect? It is actually an easy leap to make when we consider that there is nothing human at the centre of my human body. My body is always acting and reacting both to its environment (affection… then reflection), and as an environment (with the millions of particles, bacteria and cells, both living and otherwise, constantly working inside me). Nonhuman affect refers to the sensations and responses of physical, nonhuman matter. Like a human body, matter has various bodies that also sense and react in the world. We are always more than the boundaries of what we know, or feel, or make.
An ecological approach, then, takes account of various agents, processes, and thoughts. We concern ourselves with how, for example, humans and nonhumans; matter and not-yet-things; past, present, and future, are all actively shaped by their interrelations.
What happens when we add blockchain to that ecological mix? Which new forms, affects, and effects might the distributed ledger technology that underpins web3 make possible? In other words, what does critical interaction feel like, onchain? How can the ecological forces at play through consensus mechanisms, like proof of stake — which keep every participant aligned with the state of the distributed ledger — change what we embody and mean? How might that change what we see, what we do, what we are?
According to Kate Mondloch, the space between bodies and screens is always worthy of study. Screens, she says, are both windows into other worlds, and physical things themselves. Even before Mondloch, Nicolas Bourriaud asserted that the ‘aura’ of contemporary art lies in the relationships it develops in the gallery — the social space in front of the work.
We are more than the data we leave behind.
So what better way to explore the new performances that occur at home and work, on Discord and Twitter, through Instagram and TikTok, Excel, and Outlook… than in how we physically act in front of our computers? Why do we only think of these machines as windows, of ourselves as brains uploading and downloading to and from ‘faraway’ virtual spaces? I’m tapping my foot and whispering to myself as I type… yet I so often forget the flesh I cannot see when wrapped up in my laptop, until I feel the pain of sitting in one position for too long. “Here I am,” I say aloud to the room as I type this, feeling like an idiot but proving my point.
My laptop-chained movements are vastly different from how I may engage with art in the gallery. But, in some ways, that makes them even more worthy of study. We are more than the eye and the finger that most computers ‘see’ pressing keys, reading screens, clicking mice. We are more than the data we leave behind; we are also how our bodies twitch and creak, how our eyes dash and how our necks and shoulders follow suit; we lie down and spill our coffee, get up to use the loo and return half-naked without a care, scream “I’m coming!” to our kids or partners while commenting on just one more Twitter post. And each affect on us and effect on our environment, each action and reaction, changes who and how we are.
Networked interactive art, where the participants connect to each other through a blockchain that is inherently connected and immutable, is uniquely positioned to have us move, think, feel, and reflect anew, with our behaviours around our computers and their always-tethered screens.
A second gallery-bound interactive installation, stuttering, was initially produced in 2003 in Johannesburg. Developed and shown in dialogue alongside people with stutters, participants use a real-time outline of their entire bodies to touch and trigger 34 invisible buttons laid out in a grid inspired by Piet Mondrian’s art. When activated, each rectangle in the work’s projected image is not filled with primary colours, but animated text and spoken word. The saturation of these virtual buttons creates an inverse relationship: move quickly, and the piece will itself stutter in a barrage of audiovisual verbiage; move slowly, even cautiously, and stutter with your body, to listen with all of your self, and hear its words.
What if we performed our bodies, concepts, and materials with this level of care all the time?
We feel a potency within that interactive space. We must navigate our limbs laboriously back and forth, on and off each individual button. We listen with our entire bodies. stuttering asks, ‘what if we performed our bodies, concepts, and materials with this level of care all the time? In parliaments or business meetings, with our families or in nature? What could that be and do?’ stuttering invites us to rehearse speaking and listening more carefully.
In another interactive artwork, elicit, every movement of the participant, small or sweeping, births fluidly animated text on screen. And these “characters,” in turn, elicit fluid performances from us. The software responds to small movements, writing letters on the screen slowly for us to read, and to fast ones, like rapid passers-by, whose bodies birth hundreds of flying characters, impossible to decode. While enter performatively entwines text and activity, and stuttering asks us to listen with our bodies, elicit has us experiment with the continuity between text, technology, and touch. It situates us as part of a language of constant movement, where meaning and motion are always in flux, and in synthesis.
elicit would have very different affects and effects in a home or office space, but after some consideration and experimentation, I realised that it would be no less worth exploring. And so STILL MOVING was born. Like elicit, with STILL MOVING, motion-tracking software births animated text in front of you, but it is a browser-based interactive artwork, so works ubiquitously, including on smaller, more intimate screens — phones, tablets, and laptops —, and each version has unique attributes created by code. It ‘unreads’ a poem written by my collaborator Sasha Stiles (and her AI alter-ego, Technelegy, this time additionally trained on my writing about interactive art).
The animated texts in STILL MOVING — which might go in any number of directions, depending on your mint — completely stop whenever you do. The text, your attributes, the code, and its relation to how you move and are moved, live entirely on chain, making a paradoxical pact between movement and immutability, connecting your piece, and how you engage with it, with each and every other edition and their participants. Here, because of sensors and sensitivity, even in stillness, we are moved and moving; we are together, apart, and a part of something much larger than ourselves.
We listen with our entire bodies.
The poem, also called STILL MOVING, continues mine and Sasha’s explorations of language and bodies, how the networked intermingling of thought and movement are an important part of identity and culture. Each iteration might select as much as three lines to interact with, or only one word, and might display these as full words or single characters. The preview images are a different kind of text-to-image than in the AI world, with the poem’s text laid out in a low resolution grid, appearing as pixels that make up your randomly-selected extract. The interactions I have seen thus far still raise so many questions for me. I, too, am still moving after all this time.
enter, elicit, and stuttering, along with scripted — where we literally perform the physical shape and sound of language with our bodies — make up a suite of interactive installations called Body Language. I try to always show them all together, facilitating a more complex experience of embodiment and affect through the subtle differences in how we perform with each.
In addition to having us explore movement on smaller screens in different environments, STILL MOVING accomplishes the same potential of Body Language, but with a single code base and variable editions. Iterations might ask, ‘what does it feel like to physically emphasise, or to explore the artwork’s font, Times New Roman, with varying probabilities affecting the typeface in each piece? Does (font) size matter? Do different words, characters, or lines prompt different affects and effects? Do various colour palettes play out mood ‘swings’? Might speed and direction influence our own speed and direction?’
Rarer attributes like curved animations, trails, and hand-drawn graphics, add different affective forces. Our interactions might be awkwardly mirrored or located upside-down on screen. It feels like a mistake when we play with it — is this a bug? Who changed my Zoom settings? — and makes us all the more aware of our physical bodies and movements in space. STILL MOVING becomes its own large-scale interactive art suite, right in front of our screens.
We are together, apart, and a part of something much larger than ourselves.
And, STILL MOVING only begins to explore interactive art’s potential via the blockchain. Even with the data-storage limitations of operating onchain, edge detection, light tracking, motion tracking, and many other simple computer vision techniques that work with a built-in webcam can be used. And alternative decentralised platforms allow for even greater possibilities, such as body-and-amplitude-tracking, face detection, or the triggering of image or sound-based assets. Perhaps even display screens (like Infinite Objects) can one day incorporate cameras themselves, or Kinect-like tech, creating another kind of exploratory and interactive NFT space.
Blockchain opens up new opportunities for interactive and generative art. Artists must seize opportunities to make work that actively entangles affects and effects between participants and their environment. Have us move and and think and feel what it means to be a human body.
Make us act what it means to be a human body, online and in real life, with the actions and transactions empowered by the blockchain. The possibilities themselves are the work of the art.
Affect is not-yet emotion, which is abstract as much as it is embodied by us.
Nathaniel Stern is a Professor of Art, Engineering, and Entrepreneurship at the University of Wisconsin-Milwaukee, and an Associate Researcher at the University of Johannesburg. He and his wife, Kate, have 5 kids and 3 cats — so there’s never a dull moment.
