Weta’s four-time Oscar-winning senior VFX supervisor talks about the complexities of creating believable digital ape performances in the final film in Fox’s ‘Apes’ trilogy reboot, director Matt Reeves’ futuristic action adventure ‘War for the Planet of the Apes.’
‘War for the Planet of the Apes’ directed by Matt Reeves. TM and © 2017 Twentieth Century Fox Film Corporation. All images courtesy of Weta Digital.
With Oscar Sunday less than a week away, industry hand-wringing continues as to which artists and films will walk away with the coveted gold. This year, the five best visual effects nominees are particularly strong, each amply demonstrating both the artistry and magnitude of Big Hollywood VFX-driven visual spectacle.
For Weta Digital studio director and senior VFX supervisor Joe Letteri, himself a four-time Oscar winner, War for the Planet of the Apes culminates almost a decade of work on the Fox trilogy, a reboot of the classic Planet of the Apes movie franchise done entirely with digital characters based on human performances captured with sophisticated cameras and computers.
While Weta received Oscar nominations for each film, they came home empty handed for both Rise of the Planet of the Apes and Dawn of the Planet of the Apes. So far this awards season, Letteri and his team seem the group to beat, taking top honors for their work on War at the recent VES Awards, among others. It’s worth noting the VES also recognized Letteri’s three decades of stellar work and contributions to filmmaking by presenting him with the prestigious Georges Méliès Award.
Weta’s work on director Matt Reeves’ film showed emotionally nuanced and gripping performances by Caesar (Andy Serkis), Maurice (Karin Konoval) and other main characters that surpassed anything seen before in such digitally created characters. Does this mean the third time’s the charm for the studio and their work on the trilogy?
AWN had a chance to talk with Letteri at the most recent VIEW Conference, in Turin, Italy, as well as the VES Awards, about his work on the Apes trilogy. He spoke at length about the unique challenges of creating believable digital apes and how, in a sense, his team’s effort on War began back with his first assessments of what was needed for Rise, and how they strove to stay focused on animating the best performances possible across all three films.
AWN: Weta Digital handled all visual effects production duties on the three new Planet of the Apes films, starting with Rise and culminating with the latest, War for the Planet of the Apes, all of which received best visual effects Oscar nominations. You were in charge of the studio production and VFX on all three. It’s almost as if you had one long, ongoing production from which three films were created….
Joe Letteri: Well, this film grew into the third film going back to the first one. My role on the film started with the original script for Rise, talking to the executives at Fox. John Kilkenny [head of 20th Century Fox’s visual effects department] sent me the script and said, “Is this even possible? Can we do it? How would we do it?” I loved the script so much, being a fan of the original [films], that just thought, “Well, we have to do it. We have to figure out a way to do this.”
The story required that we start with apes that looked like real apes…[to then show] how they get intelligence from this virus. We made the decision early on that we were not going to be using any prosthetics or real chimps, that everything we did would be digital, because we needed to establish that baseline to then show the evolution of the characters.
That was the early focus. How do you make them look real? How do you make them look believable? And then on top of that, how do you give Caesar enough characteristics so he can start to add this kind of human performance, especially the ability to speak? What happens when you’re shooting with performance capture? How do you stage all of that? I was really lucky to have Dan Lemmon as visual effects supervisor, starting on Rise and working through all three films. Dan really focused a lot on the day-to-day. Once we worked out the ideas for our technology, he would bring that to the production and shepherd it through. He worked everything out as it needed to happen on stage and then afterwards. So, I gradually shifted from, “How are we going to do it,” to focusing on the character performances. And that’s really true in this third film.
Because we had three films with these characters, we really got to know them. There was so much inward emotion in this third film, so much tension and drama. We really had to focus on just capturing the performances and bringing them through to the characters. And when I say capturing the performance, I’m using the words in a slightly different sense. Everyone talks about performance capture. There are two aspects to that. One is recording the performance on stage with the motion-capture cameras and facial head rigs. In a way, that’s a mechanical process. It’s highly technical. It has to be done very accurately, especially in very challenging conditions. On this third film, Matt had us out in the snow, the wet, the rain and the mud. These were pretty difficult circumstances to make it all work, but that’s our baseline.
Then, we looked at that data and started to apply it to our characters. There’s an artistic process then that needs to happen. Does this translation give you the same feel emotionally as when you’re watching the performers in the raw footage, like Andy [Serkis as Caesar], or Karin Konoval doing Maurice, or any of them? That’s the point where the artistry comes in.
We spent a lot of time with the animation and modeling teams getting the nuance and changes of the shape correct, because animation is not pose to pose. Animation is the flow of the emotion across the shot, and that’s really important to get right. So, we focused on that aspect probably more than anything else.
AWN: Weta is known for excellence in digital characters. Andy Serkis obviously has been a seminal character in the acting behind some of the most iconic performances you’ve produced. Each film you’ve done has raised the bar with regards to pushing the technology, allowing you to capture performances of all types in extremely rough conditions. But the focus still remains on the performance and how it’s animated. How do you know you’ve reached a point that your animation of the performance is good enough for the final film?
JL: It’s a bit of a cliché, but it’s in the eyes. On Rise, the very first film we did, we were trying to work out how Caesar would look and how his performance would come through. We had a shot that Fox wanted for the trailer, where Caesar was sitting in his cell in the ape facility after deciding he wasn’t going back home. He was trying to figure out what he needed to do in this new environment. He was basically cut adrift and starting to scope out the other apes and humans in his environment. You see him sitting in the cell, his eyes watching the humans go by, just thinking about what he’s going to do next. When we saw that for the first time, when we knew we had the thought behind the eyes, that was the moment where everything clicked for us.
That’s when we knew this was going to work. Anything else you do, the big, broad strokes, apes fighting, things like that, for sure you focus on the realism. But, when you’re focusing on those inward shots, that’s where the audience has to come along with you. And if they don’t, if they’re not with you in that moment, they’re not with you for the rest of the film.
We knew we had it in Andy’s performance. Obviously, he and Rupert Wyatt, who directed Rise, had that worked [the performance] out. They knew they had it. Rupert had his moment in the cut. For us, it was when we watched Caesar and just kept refining and refining [the animation] until we felt the same thing. It really comes down to the feeling. There’s a technical aspect to it where you’re matching the micromovements piece by piece. But, it’s all bit of a jigsaw puzzle. If they move in slightly the wrong way, if it doesn’t lock in to you, it doesn’t feel right. It can easily feel rubbery, even at a very small level. So, these refinements are really where we focus, using the actor’s performance as a blueprint. You know exactly what you’re going for. But you kind of have to just intuit when it works.
AWN: When you first started breaking down the script for War, what did you think would be the most challenging parts of the film?
JL: Well, the most challenging thing we thought going in was the dialogue. As much as we think of chimps being very human-like, they’ve got really elongated muzzles, and if you have a chimp start to speak using a natural human voice or human cadences, it looks like a man in a suit. That breaks down the illusion right off the bat. So, a lot of the performance came down to how Andy worked out the evolution of the dialogue. The fact that it was very guttural and very hard to do, that Caesar had to think about it before he could say a word, we built that cadence over the three films. But, that’s really what we were always watching for.
And in a similar way, you had the same thing happen in the body. Apes are quadrupedal. They have long arms and short legs, and the actors trained with arm extensions to do their locomotion. But physically, it’s very difficult. So, humans naturally want to fall back into more of a bipedal pose. You could tell when you crossed a line and it looked like an actor in a suit. The realism gets completely destroyed, even though it looks like a realistic ape -- the pose tells you that it’s not. So, you’re back to basic animation principles in what you’re looking for. You’re having the actors rehearse and perform, watching that they stay on the side of the line that they need to stay on, maintaining that level throughout the films.
AWN: Were you able to reuse any of the digital assets from the previous films, or did you rebuild everything from scratch?
JL: That’s a good question, because this is probably the first film we’ve done in a series like this where we did not have to rebuild the characters from scratch. Going into Rise, we knew what we needed [for the series], so we built a pretty high level of fidelity into all the characters. We were able to use them each as the starting point. The differences were subtle in that from film to film, there was some time that elapsed, so we aged them subtly.
The more important difference though is throughout the course of the films, we knew there was going to be more dialogue, not only from Caesar, but from the other apes. So, on the first film, Caesar might look like a real chimp, but we actually shortened his muzzle a little bit so that he could do the dialogue a little more naturally. When we knew that needed to happen…we just hid it in the aging process. We shortened the muzzle slightly for the chimps that were going to speak. It made it a little bit easier for us to launch the dialogue in a more comprehensible fashion.
AWN: Were there any new tech innovations you implemented on War that either made things more efficient or gave you more capabilities?
JL: Well one of the things that really helped the animators is we built a new puppet system called Koru, which allowed us to do a large number of puppets almost in real-time. That was really helpful for War because we had so many apes in each scene. Previously, a puppet could get pretty heavy -- when you’re doing one, that’s not too bad. You’re doing two or three, it tends to slow you down. You’re doing five or six, now you’re having to do one or two at a time and reload them into the scene. Animators need speed. They need feedback. So, with Koru, we were able to get practically real-time feedback on half a dozen puppets. The animators could work with the scenes all together. That’s one of those things that happens behind the scenes. But, for the animators, it’s what you need to iterate and check your performance. Turnaround time, speed and flow is everything. You don’t want to break that when you’re animating.
AWN: For people who don’t know how these films are made, there is an assumption that one, because you use computers, well, the computer does the work, and two, because you’ve got someone acting in a suit, you just record them and there’s the performance. Even people in our industry don’t understand how much artistry goes into the animation of these performances. How does the degree of difficulty animating a character like Caesar compare to other characters you create?
JL: It’s pretty much the same across any character. When you’re working with performance capture and you have a lot of actor reference, that’s a really nice blessing. The actors have been able to work everything out with the director and with each other. You get moments that are hard to anticipate when you’re thinking outside the realm of where that drama happens. But, those also require a lot of fidelity to make work, especially reinterpreting body language and facial performance into a character that as I said, looks very much human but is not quite human.
Whereas, take a completely different kind of character like Smaug, where we recorded Benedict Cumberbatch’s facial performance. But, it was really just to give us an idea of the character itself. We had to develop a lot of the characteristics of the body language and performance ourselves. So, you’re not matching exactly what an actor does. Obviously, you can’t really use performance capture for a character like that. The body language, the body type is different, so for us these things exist on a spectrum.
I don’t think we weigh one more or less heavily than the other. They all still require a lot of attention by the animation team to make them work. It’s just a question of where you have to focus, and that all comes down to the character design. But, the performance has to work either way, whatever your blueprint is.
AWN: Each character you’ve created, starting with Gollum on up to the most recent performance of Caesar, you’ve continually pushed the merging of performance and technology. On War, the animation is truly spectacular. What’s next for this type of work? How would you make Caesar even better the next time?
JL: Look, it’s all story-driven -- what the requirements are of the story and the film. I remember, going back to Gollum, Andy was brought in really just to do the voice. But, because he was a trained actor, he and Peter thought, “Well, why don’t you get in the scene and work with the other actors? Then we’ll have that interaction, which will be beneficial to everyone.” Which it truly was. When it came time to animate and we had Andy in the shots, we could see what he was doing -- we had multiple reference cameras. When the animators were trying to match that performance, we realized it was really difficult. We were trying to match a very human performance. So, we thought we would try motion-capture and have Andy recreate those scenes to see if that was actually viable.
It was really a bit of a guess at the time if that was even going to work for us, to recreate the performance and lock it back in. It gave us a lot of Andy’s performance, but even so, because you’re matching to things that were shot on plates, animators had to make a lot of changes to fit in contact points and everything. There’s still a lot of handwork that goes into it, and Gollum’s whole facial performance was all keyframed. We were lucky to be able to capture his body in those days. There was no way we could figure out how to capture his face, so that was all done by keyframe animation.
But, the principles that we’d learned, applying the FACS [Facial Action Coding System] system, things like that, we started building this library of knowledge right then and there. That evolved for us on our next film, which was King Kong. We thought, “Okay, now that we got the body capture down, can we try to capture the face.” And so, we glued reflective markers all over Andy’s face and captured it and figured out a way to integrate that onto the FACS system.
Again, that was a great baseline for us, to have Andy’s performance, But, it still required a lot of animation to be able to do that work. When we did Avatar, Jim [Cameron, the film’s director] wanted to break free of that. He wanted to mount head rigs in front of the actor and give them more mobility. Now you don’t have 3D information anymore. You just have a visual image that you’re matching to. So, all these improvements are driven by the needs of the story as well as how the director wants to shoot the film.
But, the breakthrough for us on the first Apes film was to be able to take all the technology onto a stage and just say, “Okay, we don’t want to ask Andy to do his performance again. We’ve done that for several films. How do we make this a viable part of a motion picture stage?” And that’s what we worked out and that’s why we were able to take that technology from a stage out onto more and more difficult locations. All these innovations were driven by leapfrogging from what we could do to what the next challenge was from the story, what the nuance was, what the director was after, always working those two together. That’s a long way of saying I don’t really know where it’s going, because you don’t know until you see the script.
Dan Sarto is Publisher and Editor-in-Chief of Animation World Network.
