SIGGRAPH 1999 Course 17
Section: "Non-Photorealistic Animation"
Instructor: Cassidy Curtis
A practical explanation of how to approach an animation production that calls for non-photorealistic imaging (NPI). To make things concrete, we will walk through the different key stages of NPI-based character animation, using some of the author's own projects as examples. The focus will be on the ways in which NPI requires alterations to the now-standard "traditional pipeline" for CG animation.
Audience: animators, technical directors, art directors, producers, production managers, and software developers.
Take-away knowledge: some guidelines for applying non-photorealistic techniques to an animation production, with lots of examples from real projects.
Non-photorealistic animation may be quite new to the graphics community, but author Vladimir Nabokov seems to have anticipated the concept by decades. In his 1939 novel Laughter in the Dark, the main character, an art collector, is inspired by a "beautiful idea":
It had to do with colored animated drawings -- which had just begun to appear at the time. How fascinating it would be, he thought, if one could use this method for having some well-known picture, preferably of the Dutch School, perfectly reproduced on the screen in vivid colors and then brought to life -- movement and gesture graphically developed in complete harmony with their static state in the picture . . . and the colors . . . they would be sure to be far more sophisticated than those of animated cartoons. What a tale might be told, the tale of an artist's vision, the happy journey of eye and brush, and a world in that artist's manner suffused with the tints he himself had found! [NAB89]
It certainly is a beautiful idea, and one that still holds currency today. In the time since Nabokov's book, traditional animators have experimented with various techniques for bringing paintings to life. Alexander Petrov is known for the rich, hand-painted look of his oil-on-glass animations such as "The Cow" and "The Mermaid". Another example is Joan Gratz, whose lighthearted film, "Mona Lisa Descending the Staircase" (1992) deftly morphs through dozens of works of modern art including Picasso's nudes and Van Gogh's self-portraits, always maintaining a painterly quality thanks to her innovative "claypainting" technique.
Computer animation is only beginning to catch up to the variety of styles found in traditional and experimental animation. Non-photorealistic rendering (NPR) is a step in the right direction, but like any tool, it can be used skillfully or clumsily. Nabokov's character articulates this perfectly:
...the designer would not only have to possess a thorough knowledge of the given painter and his period, but be blessed with talent enough to avoid any clash between the movements produced and those fixed by the old master: he would have to work them out from the picture -- oh, it could be done.
As recent work shows, it can indeed be done. But how? The existing literature on NPR techniques provides plenty of implementation details, but says little about how they might be used in production. The natural assumption is that an NPR algorithm can simply be tacked on to the end of the traditional CG pipeline as a post process, and that everything will work just fine. In my experience, this is almost never the case.
Animation requires a lot of design work and planning. Many of the most important decisions are made long before a finger touches a keyboard. If a project calls for a non-photorealistic look, it is essential to consider that look in every stage of preproduction and production, rather than trying to "fix it in post".
In this document, I hope to provide a few guidelines for using existing NPR techniques for animation, and for developing new techniques for specific projects. I will walk through some of the key stages of character animation production, and show examples of how an NPR technique can induce substantial changes in their planning and execution. The examples are drawn primarily from three short animations I have worked on: The New Chair, Fishing, and Brick-a-Brac.
Since each stage is typically the responsibility of a different person on a large production, each of the sections below is geared toward a different audience: art directors, technical directors, software developers, and producers. However, since there is often some overlap between these roles, it is worth reading beyond your own area to get a sense of the big picture.
There are two terms used throughout this document: non-photorealistic imaging (NPI), which refers to the goal of creating images by any means that resemble some medium other than photography, and non-photorealistic rendering (NPR), an umbrella term comprising a set of digital techniques for achieving that goal. The distinction between the two is relevant in the context of animation production, since many animations these days employ a combination of digital and hand-drawn techniques.
A feature common to most current NPR methods is the use of some type of screen-space marks to construct an image. These marks may be small relative to the image being rendered, as in the case of pen-and-ink lines [SAL97] or oil paint brushstrokes [MEI96], or they may occupy nearly the entire image, as in the case of a watercolor wash [CUR97].
Screen-space marks, as their name implies, must obey rules relating to the two-dimensional space of the image. They may optionally also represent some aspect of a three-dimensional model, but they are fundamentally two-dimensional objects. An important distinction between screen-space marks and the two-dimensional polygons used in techniques like the Z-buffer is that the former are not necessarily projected from a three-dimensional original.
Section 1. Defining a visual goal
If computer animation is still a young medium, NPR is in its infancy. At this early stage, it's hard to know how any technique will look in motion because most have never been tried before. In this sense, every non-photorealistic animation project is a new experiment.
The greatest threat to the success of such an experiment is vagueness of purpose. I have seen many NPR animation projects revert to a traditional CG look, or fail entirely, simply because they were started without a clear visual goal in mind. Directors and animators can become locked in a pointless cycle of revisions, tweaking small parameters to little effect: "Can you make the lines thinner? Hmm, that's no good. Try making them thicker... No, that's still wrong. I guess this look just won't work after all." Such a situation is frustrating for everyone involved.
The key to avoiding such pitfalls is strong art direction. Strong art direction boils down to two factors: vision and communication.
The art director has to have a vision. By this I don't mean some kind of supernatural revelation, but simply a clear mental image of what the finished product should look like, down to the tiniest detail.
For a non-photorealistic project, this means thinking of a certain class of details that aren't necessarily present in photorealistic CG. These are the specific qualities that make a painting look like a painting, a drawing like a drawing, and so on. They are also the details that give a particular image a "style". They include:
In addition to these static qualities, the fact that the image will be animated brings on the need to consider its dynamic qualities. This is where things get most experimental: With current technology it's not yet possible to "sketch out" a flock of moving brushstrokes to see how they'll behave, so intuition and imagination are crucial. The dynamic qualities include:
These properties will all be plainly visible to the viewer, whether you've chosen them or not. For this reason, it's essential to consider them all, even the ones that don't seem relevant at first glance. Imagine what it would look like one way or the other, and decide which is more appropriate. When in doubt about a particular feature, it's always good to check it against the story content, and to ask: Does this feature fit the content at this point? What might fit better?
You need to consider these properties separately for each character, prop and environment in every scene. What works for one subject may not work for them all. If there are important distinctions between characters and background, or between scenes that have different moods, try to identify which properties make those distinctions clear.
What you're doing is essentially aesthetic problem-solving. Ultimately, your job will entail a balancing act between the rules of composition [GLA98] for a static image, and the rules of animation [THO81] and cinematography [CAL96]. But given the directive to achieve a certain look in every frame while allowing motion in the scene, there may be multiple solutions, or there may be none at all! In the latter case, it's necessary to go back and analyze the initial directive, and try to ferret out what's really behind it.
For example: in Brick-a-Brac, the paper texture was originally meant to make the animation look as if it had really been drawn on paper. To mimic the entire animation process faithfully, this would have meant changing the texture from frame to frame, since each image would have had to be drawn separately. However, doing this caused a distracting amount of noise when the results were played back. The richer I made the texture, the 'louder' the noise became -- but without the texture, it no longer looked 'real' enough. This was an unacceptable aesthetic compromise.
Looking back at my own motivation, I realized that my real goal was not to fool people into believing that the animation was done by hand, but rather to acquaint them with the characters in the story. So I chose to keep the texture prominent but static from frame to frame. I felt like I had sacrificed the "handmade" illusion to protect the story. But much to my surprise, when I started showing the piece, many viewers still thought I had done it by hand.
This particular problem seems easy enough to resolve. But as the look of a piece becomes more complex, the coherence-related dilemmas tend to grow in number. The crucial step is to look at each such question as it comes up, and make an aesthetic choice that's consistent with and motivated by the story.
A clear vision in and of itself is not enough to get the job done. You have to communicate that vision to the rest of the team-- again, supplying all of the necessary details. Here are some suggestions:
Figures 1 and 2 are reference images from The New Chair.
Figure 1 shows some drawings I used for line reference. My goal was
to emulate the looseness and hastiness of these lines, if not their
literal shape. Figure 2 (color) shows a collage I made to provide
reference for texture and color palette.
Figures 3-5 (color) show some reference images from David Gainey's
film Fishing, for which I provided a watercolor look. Gainey
is an accomplished watercolorist, and was able to paint me a set of
pictures that showed exactly how he wanted the finished product
to look, in every respect. He pointed out the functions of the
different marks he had used, showing how some conveyed shading while
others acted to fill negative space. He also described how the color
palette should change over the course of the film, to convey the
changing time of day. This is the best art direction one can hope
for, and it is also the greatest challenge!
On that note, it's important to prioritize (unless you have an infinite budget.) Choose which properties define the style, and which ones don't matter as much. In the case of Fishing, the high priorities included random variation in line thickness, and the use of color to indicate time of day. In The New Chair, it was most important to provide a way to vary the line style according to the character's mood.
A final word about the timing of this process: Don't think that it's necessary to have your vision worked out to the tiniest detail before the project begins. Obviously, the more decisions you can make in advance, the clearer the task will be for the rest of the team. But every production has its little surprises. You may find that the work your team produces pulls you in a direction that you couldn't have imagined ahead of time. Your vision should continue to become clearer and more refined throughout production until, hopefully, it is matched by the finished piece.
Section 2. Defining the problem space
Once a visual goal is clearly defined, someone, usually a lead technical director, has to translate that goal into something that a team of animators can accomplish. This may require developing some new software, or it may mean simply making clever use of tools you already have.
At this point, you should ask yourself the following questions:
The end product of this stage should be (a) a specification for any software tools that need to be developed, and (b) a rough idea of the pipeline through which each shot will be pushed into production.
For Fishing, the director's specification for the look could be broken down to the following criteria:
Was CG the right tool? Yes. The director's painting skills and draftsmanship were definitely up to the task of creating this animation by hand. However, he felt the end product would be much better if he could take advantage of the perspective and keyframe animation benefits of working in CG. Also, the project could then take place in an environment already streamlined for animation production, and make use of the skills of other CG animators.
Was additional software necessary to achieve the watercolor look? No. A heavy simulation approach to watercolor rendering [CUR97] would have been complete overkill in this case. The main benefit of such a simulation is the complex interaction between different flows of water and pigment within a reasonably-sized region. But in this case, the brushstrokes were all too narrow to show such effects. The edge-darkening effect could easily be faked using edge-detection and other filters available in a generic image processing toolkit, and the textures could be achieved with clever application of procedural noise.
The New Chair
For The New Chair, my initial criteria were somewhat less clearly defined, because the project grew organically from what was originally a simple motion test. By the time it became a full-fledged animation project, however, I had come to some clear decisions about how I wanted it to look:
Was CG the right tool? Yes. Even if I had started from scratch, it would not have been possible for me to render this animation entirely by hand, because my drawing skills were not consistent enough to achieve the kind of subtle motions I wanted from the character. Working in a 3D keyframe-based system enabled me to refine the performance to the level I needed.
Was additional software necessary to achieve the look? Yes. It would not have been possible to achieve such a broad range of line styles using a simple image processing package. I needed some kind of tool that could create drawings in a wide variety of styles based on the 3D models generated by the animation package I was using.
Would this change the animation and rendering pipeline? Definitely. In Section 4, you will find a detailed explanation of what became easier and what became more difficult as a result.
Section 3. Writing usable tools
If you are going to need new software, it's important to keep it simple. It may be impossible to write a tool that automatically replicates the style of a Van Gogh. But if you can provide a simpler tool that lets a skilled user do the job pretty quickly, then you're on the right track.
The goal should be to reduce the tedium the users experience. This is not the same thing as reducing the amount of work they have to do! This is a very important distinction. If you eliminate work at the cost of making the interface obscure or confusing, you will cripple your users. But if you make the work engaging, it will go quickly and the end product will be vastly better.
The computer should make only the decisions that would be too tedious for a person to do by hand. Placing thousands of brushstrokes for every frame of a film is a task that clearly falls in this category. But the computer does not have to make all of the decisions. Of the properties listed in the previous chapter, certain ones lend themselves to automation, while others do not. The most abstract properties, such as the emotional expressiveness of a mark, are naturally the most difficult to automate. Luckily, they can also be the most fun to set by hand.
In cases where there are many dimensions of variation to the style, it may be difficult or even impossible to write an algorithm that is guaranteed to work under all possible inputs. If that's the case, don't bang your head against the problem for too long. If it's possible to screen for bad combinations of parameters, do so. But if that proves too complex a task, it's probably better to risk the occasional bad frame than to impose too narrow a restriction on the range of possible styles. A good TD can always find a way around a glitch.
In the case of "The New Chair", the right implementation turned out
to be a simple image filter that turned a depth map into a line
drawing. (See Appendix A for a brief description of the technique.)
The interface was text-based, a compromise between ease of development
and ease of use. (Since the user in question was a programmer and not
much of a painter, this was an appropriate way to go.) Figures 6 and
7 show examples of the simple formats used.
Section 4. Optimizing the pipeline
Using NPR in production affects every stage of the process, including art direction, design, modeling, shading, lighting, and even motion. Depending on the technique used, some of these stages may be made more difficult, while others will become simpler or even disappear entirely. Consider this carefully before you start to budget your team's time.
Figure 8 shows a synposis of how the "Loose and Sketchy" process changed the production pipeline for The New Chair.
The task of modeling the main character and all of the props was reduced from potential weeks to a few days, because it was no longer necessary to worry about intersecting surfaces, parametric texture coordinates, or smoothness. For example, the main character consisted of a handful of surfaces of revolution jammed together in space in such a way that they would have a nice silhouette. (Figure 9.)
Surfacing became almost nonexistent: I spent a few hours refining the paper texture, and that was all. For a stylized 3D look it would have been necessary to do at the very least some work on every single surface, if only to give the viewer's eye something to attach to. And for a photorealistic look, it would have taken substantially longer.
Lighting was likewise simplified. Only one shadow-casting light was needed, to create ground shadows for the character and props. Two fill lights were added to get the balance right. The entire process took a matter of minutes. Had the animation had a traditional CG look, much more time would have been needed: for example, I would have had to add "kicker" highlights to separate foreground from background.
The task of animating the character was made easier by the new look in two different ways. First, since the objects were rendered in different layers (see Appendix B for why this was necessary), it was alright to let them intersect each other. (Notice that in Figure 9, the character's leg goes right through his desk!) Secondly, the look actually made the character more appealing to look at, and easier to identify with. Without the distracting qualities imparted by poor photorealism, it much easier to evaluate and improve the performance.
Really, the only task that got more complex was compositing.
Appendix B describes the compositing process in detail. Integrating
several different line styles into a single image required managing
many layers, and sometimes switching the order of the layers mid-shot.
This made the shell scripts I used for rendering substantially longer
and more complicated than they might have been otherwise.
After working on some very unusual projects, I've come to the conclusion that computer animation is a truly limitless medium: any wild idea that you can imagine can be rendered digitally... but it's probably beyond your budget. For those of you in the production world, I hope these notes provide a bit of help in making your wilder ideas more feasible.
Appendix A: Loose and Sketchy method.
The "loose and sketchy" filter automatically draws the visible silhouette edges of a 3-D model using image processing and a stochastic, physically-based particle system. For input, it requires only a depth map of the model (Figure A1) and a few simple parameters set by the user. First, the depth map is converted into two images:
Next, particles are generated, one at a time, for a fixed number of
particles. Each particle's initial position is chosen at random from
within the template image, with a bias toward areas that need more
ink. No particles are ever born in areas that need no ink.
Acceleration at each timestep is based on the force field, with
additional coefficients for randomness and drag. The particle is
rendered onto the canvas as an antialiased line segment. If a
particle wanders into an area that needs no more ink, it dies and a
new one is born in another random place. The particle also erases the
dark pixels from the template image as it travels, so that those edges
will not be drawn again. (This aspect of the technique is similar to
previous approaches to pen-and-ink rendering [SAL97, TUR96].)
Figures A7-A11 show just a few of the other styles available by
varying these and other parameters. Each of these images took only
10-60 seconds to compute.
[CAL96] Sharon Calahan. "Pixel Cinematography", SIGGRAPH 1996 Course #30
[CUR97] Cassidy Curtis, Sean Anderson, Josh Seims, Kurt Fleischer, and David Salesin. "Computer-Generated Watercolor", Proceedings of SIGGRAPH 1997.
[KIL97] David Kilmer. "The Animated Film Collector's Guide" (John Libbey & Company, Sydney, Australia, 1997)
[GLA98] Andrew Glassner, Barbara Kerwin, Jeff Callender, James Mahoney, and Mat Gleason. "Art for Computer Graphicists", SIGGRAPH 1998 Course #30
[MEI96] Barbara Meier. "Painterly Rendering for Animation", Proceedings of SIGGRAPH 1996.
[NAB89] Vladimir Nabokov. "Laughter in the Dark" (Vintage International Press, 1989), pp. 8-10
[SAL97] Michael P. Salisbury, Michael T. Wong, John F. Hughes and David H. Salesin, "Orientable Textures for Image-Based Pen-and-Ink Illustration", Proceedings of SIGGRAPH 1997.
[THO81] Frank Thomas and Ollie Johnston. "Disney Animation: The Illusion of Life", Abbeville Press, New York, 1981.
[TUR96] Greg Turk and David Banks, "Image-Guided Streamline Placement", Proceedings of SIGGRAPH 1996.