Machine Learning and Creativity

It’s an interesting time to be an artist. As machine learning becomes part of the toolkit, in different ways for different people, new ideas are shaking loose, and I feel compelled to write about them as a way of wrapping my head around the whole thing.

The most recent headquake hit me by way of the ML-assisted album Chain Tripping by post-punk-pop band YACHT. Here’s a great Google I/O talk by bandleader Claire Evans that describes just how they made it. (Tl;dr: no, the machines are not coming for your jobs, songwriters! Using ML actually made the process slower: it took them three years to finish the album.) This case is interesting for what it tells us about not just the limitations of current AI techniques, but also the creative process, and what makes people enjoy music.

In music there’s this idea that enjoyment comes from a combination of the familiar and the unexpected. For example, a familiar arrangement of instruments, a familiar playing style, with a surprising melody or bass line. Maybe it works like visual indeterminacy: it keeps you interested by keeping you guessing.

As genres go, pop music is particularly information-sparse. What I take from YACHT’s example is that low level noise— nearly random arrangements of words and notes— can produce occasional bursts of semi-intelligible stuff. By manually curating the best of that stuff and arranging it, they pushed the surprise factor well above the threshold of enjoyability for a pop song. And then they provided the familiarity piece by playing their instruments and singing in their own recognizable style. The result: it’s pretty damn catchy.

So if you like the album, what is it exactly that you like? It sounds to me like what you’re enjoying is not so much the ML algorithm’s copious output of melodies and lyrics, but YACHT’s taste in selecting the gems from within it. So far, so good. But there’s another piece of this puzzle that makes me question whether this analysis is going deep enough.

Screenshot of the video for SCATTERHEAD, showing the lyrics.
Lyrics from SCATTERHEAD:
Time flies and I feel / but I can’t hear / palm of your eye / is it empty, memory?

The first time I watched the video for SCATTERHEAD, one lyric fragment jumped out at me: “palm of your eye”. I’m not alone: NPR Music’s review calls it out specifically as a “lovely phrase … which pins the lilting chorus into place”. But it jumped out at me for a rather different reason: I’d heard those exact words before. I immediately recognized them from Joanna Newsom’s 2004 song Peach, Plum, Pear.

I have read the right books / to interpret your look / you were knocking me down / with the palm of your eye

At the time, not knowing anything about YACHT’s process, I assumed they were making an overt, knowing reference to Newsom’s song. But then I learned how they generated their lyrics: they trained the ML model on the lyrics of their own back catalog plus the entire discography of all of the artists that influenced them. This opens up another plausible explanation: it could be that Newsom was among those influencers, the model lifted her lyric whole cloth, and YACHT simply failed to recognize it. If that’s the case, it would mean the ML model performed a sort of money-laundering operation on authorship. YACHT gets plausible deniability. Everyone wins.

This sounds like a scathing indictment of YACHT or of ML, but I honestly don’t mean it that way. It really isn’t that different from what happens in the creative process normally. Humans are notoriously bad at remembering where their own ideas come from. It’s all too common for two people to walk away from a shared conversation, each thinking he came up with a key idea. For example: witness the recent kerfuffle about the Ganbreeder images, created by one artist using software developed by another artist, unknowingly appropriated by a third artist who thought he had “discovered” it in latent space, and then exhibited and sold in a gallery. So, great, now we have yet one more way that ML can cloud questions of authorship in art.

But maybe authorship isn’t actually as important as we think it is. Growing up in our modern capitalist society, we’ve been trained to value the idea of intellectual property. It’s baked into how working artists earn their living, and it permeates all kinds of conversations around art and technology. We assume that coming up with an original idea means you own that idea (dot dot dot, profit!) But capitalism is a pretty recent invention, and for most of human history this is not how culture worked. Good ideas take hold in a culture by being shared, repeated, modded and remixed. Maybe there’s a way forward from here, to a world where culture can be culture, and artists can survive and even thrive, without the need to cordon off and commodify every little thing they do. It’s a nice dream, at any rate.

At some level this is just me, sticking a toe in the water, as I get ready to add ML to my own toolkit. (It’s taken me this long to get over my initial discomfort at the very thought of it…) When I do jump in, we’ll see how long I can keep my eyes open.

Animation Day at Infinity Festival

"Infinity Festival Hollywood" logo

I’m heading to LA this coming weekend to do a retrospective talk about Spotlight Stories. It’s part of an ASIFA-organized “Animation Day” event at Infinity Festival Hollywood. starting Saturday, November 9th at 10am. (Our friends from Baobab will also be doing a talk about some of their latest work, so it should be a really interesting morning!) Angelenos, swing by and say hello!

FMX and Expressive 2019

https://fmx.de/fileadmin/fmx18/img/logos/logo_2019.png
expressive-logo

Next week I’m heading to Europe for a couple of conferences: FMX (Stuttgart) where Jan Pinkava will be giving two talks about our work at Spotlight Stories, and Expressive (Genoa) where I’m presenting a paper about our look development work for Age of Sail: Non-Photorealistic Animation for Immersive Storytelling. I’m beyond excited to meet up with old colleagues and new ones, learn about the latest graphics techniques, mangle two foreign languages, and explore some cities I’ve never been to before. If you’ll be at either of these events, let me know!

Dragon Wrap 360

An animated loop of the directors, producers and PDI-based crew of "How to Train Your Dragon", made from photos taken at the wrap party in March 2010.
Click the image to see the full animated GIF!

Here’s a thing that happened. Remember that crazy accidental stereo photo that we shot at the LA wrap party for “How to Train Your Dragon”, back in 2010? Well, not long after that, we had a second wrap party in Palo Alto, for the PDI part of the crew. At one point I was talking with Chris Sanders, and I showed him that stereo photo from the other party. His eyes got really big, and his inner ten-year-old, always very close to the surface but particularly so in that moment, looked at me very seriously and said “we have to do this again… right now… with EVERYBODY HERE!” Without any kind of plan, we just snapped into action, moving tables, herding animators, passing on instructions in a game of telephone as everyone gathered in a big circle with Chris, Dean, Bonnie and Bruce in the middle. Cameras and phones out and ready, on a count of three, we all snapped a shot– as simultaneously as a crowd of reveling filmmakers can manage (which turns out to be not simultaneous at all, but hey, we’ll fix it in post!) I got everyone to email me their photos the next day, and spent way too many hours truing them up over the following weeks. I even did some very bad morphing at one point. I never quite got it to a state that felt good enough to share, so this sat on my hard drive for the better part of a decade without anybody seeing it.

But the third chapter in the trilogy comes out today! So in honor of that, and all the amazing artists who were there in that room nine years ago, and the many others who have worked on these movies before and since, here it finally is, in the form of an animated GIF: Dragon Wrap 360!

Photos by: Jennifer Yip, Craig Rittenbaum, Kathy Altieri, Craig Ring, Gil Zimmerman, Andy Wheeler, Susan Hayden, Ronman Ng, Melanie Cordan, Jennifer Dahlman, Rebecca Huntley, Ben Andersen, Janet Breuer, John Batter, Andrew Pearce, Katrina Conwright, Toshi Otsuka, Lou Dellarosa, Nara Youn, Michel Kinfoussia, Kevin Andrus, Dave Torres, Michael Baula, Tanner Owen, Karen Dryden, April Henley, Kate Spencer, Cassidy Curtis, Ron Pucherelli, Scott LaFleur, Simon Otto, and Dane Stogner.

Polarized Rainbow!

Maybe this should have been obvious, but it took me totally by surprise: rainbows are made entirely of polarized light! (I’m guessing this is because of how the light bounces off the insides of the raindrops on its way back to you.) So if you put on polarized lenses (like some sunglasses) and tilt your head sideways, you can make them disappear— or make them look twice as bright against the non-polarized sky!

“Age of Sail” at the Annies and the VES Awards

It’s not every week you have to fly down to LA twice, but what a great reason to do it. “Age of Sail” was nominated in a bunch of categories, and won Outstanding Production Design at the Annie Awards, and Outstanding Visual Effects in a Real-Time Project at the VES Awards.  I’m so grateful to have worked with this amazing team of artists, and and so proud of what we’ve accomplished together!

Jasmin Lai, Céline Desrumaux, Sikand Srinivas and Neth Nom at the Annie Awards.
John Kahrs, Kevin Dart, yours truly and Theresa Latzko at the VES Awards.
(Apparently I still don’t know how to keep a bow tie straight.)

Non-Photorealistic Animation (SIGGRAPH 1999 course notes)

Way back in 1999, I had the pleasure of contributing a segment to a SIGGRAPH course on non-photorealistic rendering. By that point I had made a whopping two-and-a-half animated short films with different visual styles (Brick-a-Brac, Fishing, and a never-finished The New Chair) which in those innocent times made me an authority on the subject. So I threw together a loose framework based on what I’d learned from those experiences, and built my piece of the course around that.

I went back and re-read it the other day, and was surprised to find a lot of it still holds true. In particular, one lesson that we carried through in both Pearl and Age of Sail is that if you plan ahead and you’re smart about it, committing to a stylized look can also save you a lot of time and money.

So if you’re interested in making a film with a new visual style, but you just don’t know where to start, have a look!

SIGGRAPH 99 Course 17: Non-Photorealistic Animation

Cassidy Curtis's splendid display of colorful things.