Category Archives: perception hacking

On the rules for VR

SIGGRAPH attendees are a sophisticated audience, so demoing Pearl in the VR Village last week led to some really interesting conversations.  One thing I heard more than once was this idea that to do storytelling in VR, we have to throw out all the rules of traditional cinema. While I appreciate the swashbuckling spirit of that sentiment, I don’t think it’s actually true.

drawing by Elizabeth Floyd
drawing by Elizabeth Floyd

I had a life drawing instructor in college who used to teach us rules like “highlights are circular, and shadows are triangular.” As a math major, this really bothered me at the time, because taken literally it was provably false– just give me a flashlight and a grapefruit and I’ll show you! But that was missing his point. The human body is made of smooth, convex masses, and the highlights on them do indeed tend to be round. And when one limb casts a shadow on another, the contour of the shadow’s edge wraps around and hits the silhouette at an angle, forming a sharp point. In other words, “triangular”. So my teacher’s rule, within the context of human figure drawing, was totally valid and actually pretty insightful. But it wasn’t a law of nature, it was something he invented. And to construct it, he had to synthesize knowledge from human anatomy, physics, geometry, and visual perception.

The rules of filmmaking seem atomic and universal to us, but they’re not. Like the “triangular/circular” rule, they’re chimaeras, hybrid creatures assembled from bits of wisdom from different disciplines. They’re not real the way math and biology are real, we’re just so used to them that we mistake them for reality.

illustration of the 180-degree ruleFor example, take film’s 180º rule. That’s the rule that says if you’re shooting a conversation between two characters, there’s an imaginary line connecting them, and when you cut from shot to shot, you always have to keep the camera on the same side of it. Cross that line, and you risk confusing your audience. This rule has elements of geometry (projecting 3D space to a plane), perception (how humans construct mental models of 3D space) and psychology (how we organize those models based on relationships between people). That’s a lot of moving parts! Now imagine trying to apply this rule to a VR experience where you can walk around the scene. Some of those elements change (the flat screen becomes a volume) but the perception and psychology parts are still there. So the question is not whether to keep the 180º rule or throw it away.  The question to ask is which parts do we keep, and what else do we add into the mix, to construct a new rule that works for VR?

For VR storytelling, we shouldn’t have to throw out the rules of the mediums we know and love. But we can unpack them, dismantle them into their component parts, and analyze them at a deeper level than we’re used to doing. And that’s going to be a fun way to spend the next few years, for all of us.

Inceptionism and learning envy

Inceptionist squirrel is watching you!
Inceptionist squirrel is watching you!

A few days ago the image above started going around the social networks, attributed to “a friend working on AI”. Apparently a deliberate leak, now we know where it came from: a research group at Google working on neural networks for image recognition. Their research is worth reading about, and the images are fascinating: Inceptionism.

I have to admit, machine learning makes me jealous. Why does the machine get to do the learning? Why can’t I do the learning? But projects like this make me feel a little better. When the black box opens and we see the writhing mass inside, we get to learn how the machine does the learning. Everyone wins.

And the machines still have some catching up to do. As soon as I saw the amazing gallery of inceptionist images, I recognized the “inceptionist style” from the mysterious grey squirrel. Could a neural network do that?

Fun with Pseudocolor, Part Two

A more perceptually-uniform, if less pretty, pseudocolor scheme.
A more perceptually-uniform, though arguably less pretty, pseudocolor scheme.

Inspired by this brilliant interactive demo of the perceptually uniform CIE L*a*b* color space, I decided to try a L*a*b* version of my pseudocolor scheme. I don’t find this version as pretty to look at, but it has the advantage that higher values are always mapped to colors that are perceptually brighter than lower values. In other words, if you squint at the image above, the bright and dark regions correspond pretty much exactly to what you’d see if it were greyscale. (For the L*a*b* to RGB conversion, I grabbed pseudocode from this handy page.)

L*a*b* space is much bigger than RGB space, so the spiral gets clipped against the edge of the cube in some places.
L*a*b* space is much bigger than RGB space, so the spiral gets clipped by the sides of the cube in a few places.
If you crank up the saturation, you do get more vivid colors, at the cost of a lot more clipping.
If you crank up the saturation, you do get more vivid colors, at the cost of a lot more clipping.

Meta-Perceptual Helmets


Cleary Connolly, an artist in Ireland, has built a lovely series of perception-altering helmets, including the “Hammerhead” (above), which is effectively a head-mounted, lens-based version of the Telestereoscope. The helmets look beautifully crafted and durable. The site has some great photos of the work in progress, and don’t miss the drawings of what they’ve got planned for the next helmets in the series. (I especially love the Siamese Helmet concept– great fun!)

Thanks to Sasha Magee for the link.

Eyeteleporter and Pinhole Selfies

From a planet not far from the Telestereoscope, two projects have just entered our universe…

The Eyeteleporter, a wearable cardboard periscope that displaces your vision about two feet in any direction:


And Pinhole Selfies, a delightful mashup of retro tech with millennial idiom:



Hat tip to Brock Hanson for the links.

Update: I just realized that the pinhole selfie photographer, Ignas Kutavicius, is the same fellow who invented the amazing solargraphy technique of capturing the sun’s movement with a long exposure pinhole camera. Brilliant!


A homegrown telestereoscope

Once in a while I get a random email from someone interested in checking out the Telestereoscope. If they’re local, I usually direct them to the CuriOdyssey museum, where we have a small one installed. But lately I’ve been encouraging anyone who’s interested to try building one for themselves. Our first prototype cost just a few dollars in materials, and can be put together in minutes. (Calibrating it takes a bit longer, but the process is educational, and ultimately quite rewarding.)

Here’s a working telestereoscope built by Will Rogers. He used metal C-clamps and some very interestingly shaped mirrors (maybe reclaimed from an old car?) giving his version a really distinctive style. I love it!

spozbo's telestereoscope

A camera lucida for all my friends!

My old friend Golan Levin and his collaborator Pablo Garcia have updated the camera lucida for the 21st century. Being a big fan of arcane optical devices, I had to have one. But this would be good for anyone who draws from life or is interested in learning to do so. Judging by the breathtaking rate at which this project’s getting backed, you probably have only a few hours to jump in. Back it here!

Joshua Foer’s Topophonic Telestereoscope

Science writer, TED talker and all-around interesting fellow Joshua Foer has built himself a device that does for sight and hearing what our Telestereoscope did for sight alone: a Topophonic Telestereoscope. You have to scroll down a bit on his single-page website to find it (apparently Foer doesn’t believe in permalinks) but I assure you there’s nothing but inspiring stuff between here and there, so it’s worth a scroll.

Meanwhile, I’ll be off on one of Foer’s tangents, trying to teach myself Mandarin Chinese.

Animation Hypnosis

Sometimes, when you’ve worked on one shot for too long, you can go a bit blind. It’s a very specific kind of blindness, one that prevents you from seeing mistakes you’ve made and opportunities you may have missed. It seems to happen to every animator at some point, and it is deadly to the creative process.

"Day Nine", photo by Jerry Cooke

There are tricks you can use to get around it. You can hold a mirror up to the screen to see the shot from a fresh point of view. You can step away for a few minutes, or a few days, or longer. (One time I was able to step away from a shot for a full year: boy did I see it with fresh eyes at that point!) And of course you can show other people. But once the effect has set in, your own perceptive powers are severely diminished.

I have a hunch that the main cause of this is the simple act of watching your shot over and over again. In psychology there’s a concept called habituation: any stimulus repeated long enough reduces your sensitivity to that stimulus. It’s the reason why you notice the sound of the dishwasher when you first walk into the kitchen, but eventually it fades into the background.

In the early days of animation, watching a shot in progress play back at full speed was a luxury animators didn’t have. To do that required shooting key drawings onto film, developing the film, threading it into a projector, and so on: an expensive and time consuming process. And yet, great animation still got made: animators planned very carefully and learned to do most of the work in their heads. In the early days of computer animation, it was much the same, though for different reasons: we’d have to wait for an overnight render to see our work at speed.

"Hypnotizing", photo by Patrick Breen.

Nowadays, animators have digital tools that allow for instant, real-time feedback, which for the most part is a tremendous aid. But it also makes it very easy to hypnotize yourself with all those looping stimuli.

If you want to stay sharp, it’s critical that you delay the onset of animation hypnosis as long as you possibly can. So what I try to do is avoid watching my shot while I’m working on it. I’ll watch it once, twice, maybe three times, and then jot down my thoughts about what needs doing. Then I go to work, keeping narrowly focused on each detail as I go. If I have to play some part of the shot at speed to judge some nuance of weight or gesture, I’ll hide everything but the body part I’m working on, so as not to get distracted. Once I’ve addressed all of my notes, only then will I watch the shot as a whole again. It takes a kind of discipline that I can’t always muster. But when I succeed, it feels great. And as a side benefit, I find I get more real work done in less time: after all, time spent looking at your shot is not time spent working on it.

"Sissi ipnotizzata", photo by screanzatopo.

I’d love to have some way of counting how many times I’ve looped my shot, to see if there’s a certain magic number where hypnosis sets in. What would that number be? 500? 5,000? You could make a game of it, a la “Name That Tune”: challenge yourself to finish a shot with the absolute minimum of viewings. How low could you go? Ten loops? Three loops? Zero?

What would 120fps mean for animation?

Following up on yesterday’s post about higher frame rates in movies, there’s another question looming. If high frame rates catch on industry-wide, what will it mean for animators?

We won’t really know for sure until the movies start coming out. But we can guess. There are televisions on the market that will play movies at 120fps regardless of how the movie was shot. They do this by creating the inbetween frames automatically, in real time. (How exactly it’s done, I’m not sure, but it’s probably some sort of optical flow technique, like Twixtor.) When you see this done to a live action movie shot at 24fps, the effect is impressive: movement really does feel incredibly smooth, and the strobing/juddering problem is minimal. But if you watch an animated movie on one of these TVs, the results are not good. Timing that felt snappy at 24fps feels mushy at 120. Eyes look bizarre during blinks. And don’t get me started on smear frames.

Of course, this is just a machine trying its best to interpolate frames according to some fixed set of rules. Animators will be able to make more intelligent choices, which of course it’s our job to do. But that’s where it gets interesting. How many frames should a blink take at 120fps? What’s the upper limit on how snappy a move can be, if you can potentially get from one pose to another in a mere 8 milliseconds? It could open up new creative possibilities too. Take staggers for example: at 24fps, if you want something to vibrate really quickly, your only option is to do it “on ones” (that is, alternating single frames). But at 120fps, you could potentially have staggers vibrating on anything from ones to fives. How will those different speeds feel to the audience?

One thing seems pretty certain: animating at 120fps would be a lot more work. For animators who agonize over every frame, it will mean five times the agony. It will certainly mean more reliance on computer assistance: more spline interpolation, fewer hand-crafted inbetweens, and forget about hand-drawing every frame! I look forward to hearing animators’ stories from the trenches on Hobbit. Will they find 48fps twice as hard, or more, or less? What tricks will they have to invent to make their job manageable?