In undertaking a third Avengers film, Endgame, Marvel visual effects supervisor Dan DeLeeuw relied on his previous experiences at Marvel Studios. “The Marvel system is that they bring you in and get you immersed,” DeLeeuw stated. “I started on second unit on Iron Man 3 and learned the system. Then, [Avengers directors] Joe and Anthony [Russo] brought us on for [Captain America] Winter Soldier, so you learn what was going on, and then you come in and do [Avengers] Infinity War and it got bigger, then Endgame, which is as big as you can go. So each film trains you for the next film.”
Simultaneously, Marvel’s visual effects producer Jen Underdah began assembling the vendors and production pipeline required for Avengers: Endgame. “One of the things we did was look at what the key development items were going to be, and what teams and technology we were going to need,” she said, “so, immediately we started on Smart Hulk and Professor Hulk – getting those characters because we knew we had to sell those; they had to be believable, sympathetic characters in their own way.”
To execute many of Endgame’s principal setpieces, the Russo brothers wanted to formulate a virtual production system, including acquiring motion-capture footage for Hulk and Thanos, among other characters, including face cams necessary to translate actors’ performances into digital characters.
After Captain America: Civil War, which featured many characters in the Avengers films, the Russo brothers developed the scripts for both Avengers: Infinity War and Avengers: Endgame. “We teamed up and started breaking down what we think are the things that will take the longest to develop,” DeLeeuw recalled. “What are the things that matter most, and how are we going to achieve it technically. It was just trying to mastermind a plan—also knowing it would change over time, all the time. We would have to be prepping while we were shooting and posting to get these two things done at the same time.”
Thus, many scenes in Infinity War and Endgame were executed at the same time. “They were trying to interweave the scenes that were similar,” DeLeeuw revealed of Avengers’ directing brothers. “So if you were at Avengers Compound in Infinity War, they would shoot the Endgame scenes at the compound. But that changed as soon as the Infinity War script became more Thanos-centric. The first version was a straight ahead Avengers movie, and then as Thanos became more developed as a character, he took over and drove the narrative. Once you start doing that, then the ability to shoot them concurrently didn’t work anymore. So, we shot as much scheduled out Infinity War as we could, got done, started post-production; all the while they were pre-prepping Endgame, so they finished Infinity War, and then went back and shot the rest of Endgame, and then finished it.”
At Marvel Studios, the core effects group spent three years on Endgame. “We are privileged as a department because we do so much heavy-lifting throughout the entire picture,” said Underdah. “We touch almost every shot in some form. We are, by necessity, in early production meetings and script meetings, just as the movie is getting fleshed out. Dan has to be there to technically advise, and the brothers are also super open to creatively what everyone is bringing to the table. We’re very early on in small groups of people prepping for what we need to achieve.”
At Weta Digital in New Zealand, Matt Aitken, visual effects supervisor for Weta, pointed to the enormity of scope in Endgame as a foremost challenge. “If you stood back and looked at the task you’ve been handed, it would be too daunting,” Aitken conveyed, “so you break it down and look at the individual components, and start to take it piece-by-piece. We have a great team at Weta Digital, so there’s a lot of strength there as far as taking on the asset builds and the animation, and exploring the characters.”
Unquestionably, one of Weta Digital’s critical tasks was the realization of Thanos. “We were doing motion capture at the same time the live-action photography is going on,” Aitken detailed. “So Josh Brolin is on set performing [as Thanos], but he’s able to interact with Robert Downey, Jr. as Iron Man and Chris [Hemsworth] as Thor, so we’re getting this great interplay between the virtual character and the live-action one, which is really important, not just to the Thanos performance, but to the live-action characters as well, so they can build those performances together.”
Once Brolin’s facial and bodily performances were recorded, a digital likeness of Thanos replaced Brolin, albeit a much taller, bulkier character, in the film’s finally rendered scenes. “We’ll be basing his motion off Josh’s on-set motion, obviously,” said Aitken. “If it’s a big stunt, there’s a stunt team standing in for Josh, but, even then, we’ll need to dial Thanos into that captured performance because the way a person who’s eight-feet-tall will move will never be accurately captured by a live-action performer. So we need to dial Thanos’ physicality—he’s huge, but also agile.”
With work spanning 500 Weta Digital artists, Thanos was brought to life, not only with valuable information from Brolin’s acting on set, but with the additional expert abilities of digital animators. “That’s a keyframe in the animation that we were dialing on top of the performance capture, and finding Thanos’ character from that process,” Aitken related of the Weta animators’ toolbox. “It’s more of a by-hand process; it can work with performance capture—you can use keyframe animation to modify a captured performance. It doesn’t have to start from scratch, but it’s the artistry of animator being brought to bear.”
In addition to Weta Digital and 12 additional vendors, Industral Light and Magic (ILM), a Lucasfilm company, was brought in to handle major visual effects tasks on Avengers: Engame. “I was hoping that we would have the opportunity to work on it, having worked on some of the prior [Marvel] films, starting with Winter Soldier,” said ILM visual effects supervisor Russell Earl. “If I hadn’t done that, I probably would have been overwhelmed. Winter Soldier led into Endgame, and one of the things I’ll never forget: Winter Soldier started as a much smaller picture. By the time we got to Endgame, the first meeting, I’m like, ‘Okay, don’t get too attached to anything; it’s all going to change—we’re going to get through it.’ I felt like we were really prepared to tackle any task at hand.”
With Thanos being realized by Weta Digital, ILM was given the job of the look and performance of digital Hulk. “Creating that character and making sure he can live on-screen with our live-action actors,” said Earl of his principal concern at ILM. “We did that, and revisited some of our time travel or quantum-realm stuff, then we played a part in the end battle too. We were sharing, cutting back and forth between Weta shots and ILM shots.”
As with Brolin’s performance informing the digital performance of Thanos, Mark Ruffalo’s inhabiting of his character led to the final actions and reactions of digital Hulk. “We built a new toolset to capture his performance, and then, also, at the same time, a new toolset to re-target balance,” said Earl of his team, variably 400 to 500 artists. “You’d find that Mark could do a smile, and it would look normal like Mark. Then, as soon as we transferred that to Hulk, it looks too goofy and weird. So our animators had full control — get Mark’s performance, capture that, pass it on to animation. Then, the animators had to enhance the performance, or dial certain areas back, so we could bring his performance in or out, and modify it to capture the essence Mark was doing.”
Lastly, DeLeeuw unveiled an unforgettable element in Avengers: Endgame which was notably not a visual effect: the appearance of the entire principal Marvel Cinematic Universe cast at Iron Man’s funeral. “It was an incredible day,” he said. “It was pretty spectacular. No digital effects. Four Oscar winners were on the set that day [Brie Larson, Gwyneth Paltrow, Marisa Tomei and Michael Douglas].”
