CIID · 2024
Mirrored Canvas
Turning facial expressions and voice into a live, glitching painting.
The short version
About 1 minute
- Situation
- A four-day programming course at CIID, working in pairs.
- Problem
- Emotions show up across the whole face, body and voice, but most tools treat expression as a handful of fixed faces. We wanted something that let people play with how they express themselves.
- What made it hard
- Facial tracking glitched for some people because their features fell outside the tracker's expected range.
- The piece had to make people comfortable being silly in front of others.
- My role
Built with Wayne Kim. I worked on the creative coding, interaction mapping, testing with people and the concept video.
With Wayne Kim
Tools: Processing, FaceOSC, Premiere Pro, Lightroom Classic
- Key decisions
- Mapped each expression to one visible effect, so people learn the system by playing with it.
- Added voice pitch as a paintbrush, because voice is another way people release emotion.
- Dropped eye tracking after testing showed it failed for some people, and kept eyebrows instead.
- Outcome
- People laughed, made odd faces and voices, and kept playing. The tracking glitches also showed us why datasets need diverse representation.
Details
In a world where emotions are expressed through complex, dynamic patterns across the face and body, Mirrored Canvas bridges facial expression and artistic self-expression. It turns fleeting expressions, like a smile or raised eyebrows, into a live visual artwork. Inspired by how mirrors show us and canvases capture art, it helps people see their emotions in new ways: participants watch their expressions become an evolving artwork projected onto a translucent canvas, making the invisible visible.
Why the face matters
Humans have 43 facial muscles, with which they can produce up to 10,000 different expressions, making the human face one of the most powerful communicative tools we have (Rinn, 1984).
Most studies focus on a few fixed facial expressions, but real emotions are much more complex. People use their whole bodies, not just faces, to show how they feel, and if we don’t take that into account, we can’t capture the nuanced, dynamic and multimodal nature of genuine human emotion.
So why do we sometimes fail to understand or express emotions?
Mirrored Canvas at work
A webcam tracks facial expressions in real time with FaceOSC. A program built in Processing turns them into colourful visuals, and a projector casts the moving art onto a see-through acrylic canvas, so you watch your emotions come alive.
Each expression triggers a unique visual:
- Raising your eyebrows makes the canvas glitch, more the higher you raise them.
- Tilting your head writes “I am a body bender”, to show your flexibility.
- Dropping your jaw shows an image from a database.
- Your voice pitch drives a flowing paintbrush: low pitch paints orange, high pitch blue.
- The centre of your head anchors the brush.
- Pressing S saves the artwork.
Making people laugh was a joy, and the laughter was contagious. It was great to see people embrace a playful, childlike curiosity with their movements while feeling free to be goofy: not shy about making weird voices, and exploring different pitches. Creating something absurd brings out the absurdity in those trying it.
Iterating expressions and technical details
We wanted the tool to feel Dada: weird, absurd, expressive and intentional, so people could be free and do embarrassing things without fear of judgement. We started by making as many strange faces as we could in a mirror, to discover what those 43 facial muscles can do.
Next, we used FaceOSC for facial tracking, capturing eye, eyebrow, mouth and head movements and mapping them to the outputs we wanted. We also used voice, since it’s another tool we often use to release or convey emotion.
Mapping the eyes caused glitches when testing with different users: some people’s eye measurements fell outside FaceOSC’s standard range. That showed us why datasets need diverse representation, so we removed eye tracking and focused on eyebrow movements.
Voices from the canvas
“W*f is that? OMG it’s so funny.”
“Oh niceeee… I love it.”
Lessons from the mirror
Mirrored Canvas shows that emotions are never still or simple. Our faces and bodies tell stories that shift and blend in real time. The project makes those stories visible and fun, opening new ways to explore how we feel and express ourselves.