Last week, Below the Line shared an interview with Production Designer Ramsey Avery, who snagged the coveted role of designing Amazon’s Lord of the Rings series, which just wrapped up its first season.
The second installment of our interview series for The Rings of Power introduces someone who worked very closely with Avery — VFX Supervisor Jason Smith, who was instrumental in helping to create Middle Earth, as well as its creatures and various inhabitants. He worked with a number of VFX houses across the globe to create the scale of that world and distinguish the many different kinds of people who populate it.
Below the Line jumped on Zoom with Smith a few weeks back for the following interview, in which he talks about his duties as well as a number of key sequences driven by the show’s dazzling visual effects.
Below the Line: Production designer Ramsey Avery mentioned having to work with artists in many different time zones. Are you used to that as a VFX Supe?
Jason Smith: I’ve done that before [on] shows where I’ve got crews in Singapore and Vancouver and Montreal. But I would say that this project has been the most massive in terms of just the areas of the world where we’ve been working. We’ve had crews in San Francisco, Los Angeles, [and] everywhere from Montreal and parts of Canada, all the way through to Singapore [and] Australia. You name it, we basically had a crew there. I could really review shots around the clock. If that was the way I wanted to work, I could do it around the clock. It was pretty crazy.
Of course, the COVID thing is a part of this, too — the fact that we’ve all been kind of forced into realizing we can work so well remotely. Visual effects is really one of those disciplines [where] we can do a lot of the work that way. There’s always something that you gain from being together and in the same place, and of course, we’re sensitive to all the negatives around the world that the pandemic brought. At the same time, it’s one of those things that did push us toward this cloud workflow. It pushed us towards this idea of [how] everything that we do shouldn’t be in a single place that’s in a certain geographic location. It [should be] accessible all over the world to artists all over the world. So, it’s been a pretty incredible system.
BTL: I presumed wrongly that you’re from the Weta camp, but you’re actually a former Industrial Light & Magic (ILM) guy. So how did you end up involved with The Rings of Power?
Smith: It is kind of a fun story. I was working on some Transformers movies for a long time, and I ended up doing a Transformers movie with [this] producer, Ron Ames, who I really worked well with. He’s got a way of working that compliments my own way of working. He’s really good at the things that I require somebody to be good at [chuckles] with me. We really worked well together on Bumblebee.
Honestly, what happened is Ron was one of the first people they sourced to do this project, and he thought of me and brought me in for a quick conversation. When I met the showrunners, and Lindsey [Weber], and the producers, it really was just coming into a room, and here’s, kind of, a family. This is a bunch of people [who] loved these books, and they loved this world as much as I did. They care about it as much as I do.
I came from a family where my Dad was really into Tolkien, and he read the books to us. We grew up watching the Rankin-Bass animated features. I came from this real Tolkien family, and here [are] these other people from different types of backgrounds, but [we] were the same. They came from the same type of background, and the minute we started talking, one of the first things we started talking about is that we have to do the orcs as practical as we can. We’ve gotta get some makeup involved, and do that as much as we can, and then we’ll use visual effects for the last little touches, just to hide little things and help our makeup friends out. It was just such a great conversation. We were all kind of feeling the same things and saying the same things and it kind of went from there.
That’s been my experience on the show, [working with] the showrunners and Lindsey. There’s an aesthetic, and there’s a way of telling stories, that keeps them rooted in a reality. Tolkien is very fantasy, right? There are giant spiders and dragons and magic, and there’s good and evil, [which] are very polarized in the way that they are in Tolkien’s writings. But we have this way of approaching things from our showrunners, and from Lindsey, that is hyper-rooted in reality. Really, everything that we’re doing.
I remember, I was working on one effect for the show, which is the Palantír effect, which is this ball and you touch it, and you might see visions. You might see things that have happened, things that might happen, or things that may not happen. It’s a little bit risky to play with these things. We had this moment where somebody was going to put their hand on it, and I had the thought that it’s revealing what it’s revealing, [and it’s] causing a little bit of strife, a little bit of division. Not to be too literal about it, but we ended up taking that idea that this crystal would actually fracture, not just the crystal, but the space around it. All the world around this thing, in a way, is crystal in the unseen world.
I only say all that just to say that this is the kind of thing that the showrunners would… if I started cracking crystal in the middle of the air, that was all fine. But they just wanted to know, ‘Hey, where’s that coming from? What are you thinking?’ I would always need to have some explanation that’s rooted in some kind of reality. I would talk to them about how the unseen world is all around us, and this thing happens, and it reveals part of the unseen world. When that transition is complete, you’re inside the vision. It’s just been a case of [having] showrunners [who] care about reality as much as I do, and care about the show being as rooted as we can in that. Of course, it’s fantasy, and so there’s a certain amount that is going to be fantasy, and we wanted it to be. It’s all just so much more fun to me when it’s well-rooted.
BTL: A little semi-caveat: I’ve only seen the first few episodes so far, but it was very exciting to see the first two episodes on the big screen at the show’s New York premiere.
Smith: When I saw the first few episodes on the big screen, it was so exciting, because that was [the] first time that I saw it with sound and music, too. For me, it was like watching it for the first time. The creatures, as you say, are very near and dear to my heart. If I was born to do one thing, it was monsters and creatures and characters.
It goes all the way back to when I was four. For Christmas, I wanted the Universal Frankenstein action figure and growing up, I started to idolize makeup artists. I knew who Rick Baker was, and I knew who Rob Bottin and Jack Pierce were. Of course, I knew Dick Smith was the godfather of makeup, and I would study these guys. I remember at the age of 14, I started gluing stuff onto my face. I didn’t know what I was doing, but I started trying to make monsters at home and kept doing it to the point that I actually really wanted to be a prosthetic makeup artist. Every prosthetic makeup artist that I work with now has to hear my sob story of how they [have] my dream job. They’re making the rubber monsters!
What ended up happening is that I was also very interested in computing and computer science, and had a thing for math, as well, and then doing figure drawing and stuff on the side. I had this weird mix of interests during college that kind of pulled me out of what I was studying, which was biology. There was one point that I realized, “Okay, what are you doing with all your time? It’s all of these things, and you’re supposed to be studying biology.” I had to finally recognize that the creatures were the thing for me.
That fed right into my career at ILM, [where] I was a creature [technical director] for many, many years, rigging, and [doing] flesh simulation, especially. I just love every detail — getting the hair right, [and] the little nuances of how the scapula ripples under the skin, and how that bunches up. All of those things really get me excited. To be given the chance to do that on Lord of the Rings and to have an influence on the design and who the designers are, it’s just been kind of a dream job, to be honest with you.
These creatures come across your desk, and you’re like, ‘Okay, a sea creature in Middle Earth.’ We probably did 200 concepts for the sea creature, and that’s not because we were being sloppy, it was because we really wanted to explore a lot of ideas that Tolkien had. He tossed out a bunch of ideas, and we wanted to explore shapes and different ideas of how they could play into the story. It really was a lot of fun. I got to handpick the crew of creature designers throughout the world. People that I know, and I know that this guy is good at this type of creature, and this guy is really good at the things that are more zany and out there. It’s just been a real blast. Every single creature, I can tell you a story.
BTL: I’m always generally interested in FX casting — when you’re picking the houses and assigning which scenes they’ll be doing. I know a little bit about the bidding process, but I have to imagine every FX house wanted to be part of this since it’s such an iconic, effects-driven show.
Smith: Yeah, and at the end of the day, it’s also quite different from a lot of the stuff that’s out there right now. There is some fantasy going on, and that’s really great to see all the other fantasy that’s going on in the industry right now. There’s plenty of room for all of it, for sure. But there’s something unique about this world. There’s something that kind of grabs onto our heartstrings. A lot of the people [who] I’ve been working with on this show are these incredibly talented supervisors from all the different houses, [and] they all say the same thing. They all say that this world is close to them, and the opportunity to step into this world is precious to them. A number of them are coming back to this world after working on the Fellowship of the Ring. Quite a few of them had already been in this world with Peter’s films and were excited for the opportunity to come back and do it again.
When we’re looking at the casting for facilities, honestly, what we’ve found is that the facilities that we’re working with are really, really good. They’re very trustworthy, and they know what they have the capacity for, and what they don’t. They all have the artistic skill to do top work, and it really comes down to, ‘do they have the right crew to pull off a big water simulation, or the right crew to pull off big creature animation?’ That’s what it comes down to. When we’re casting a big creature, for example, we’ll pick out a few houses that we know have the kind of crew that could pull off that level of work, and then decide which facility is best-suited to do that. The bidding process is a part of it, but I have to say, in general, the decision mostly comes down to who’s got the crew, who can really pull off the work, and who’s excited about it.
BTL: It’s a little different for a TV series since you’ll be working on the first two episodes while they’re shooting the third, fourth, etc. With a movie, you know you have a certain amount of time to finish since you’re aiming for an established release date. In the case of a series, you might have more time, but I imagine it’s also easy to end up with a backlog of episodes that need to be finished. How many houses were involved with completing this first season?
Smith: Oh, my goodness. I can’t call out a number offhand, but I would say it feels like eight or nine, maybe as many as 10 or more. Basically, we have specialized houses. We have the big houses that can tackle the really big work and water simulations and stuff like that, and then we’ve got smaller houses. We had one company, Cantina Creative, that handled a lot of our maps for us — this idea that we’re transitioning from one area in the world to another. They’re a company that I’ve used a lot for title-type stuff, and I’ve used them for Bumblebee‘s heads-up displays, and stuff like that. They’re very good at that animated information on the screen, so they did the maps for us. A lot of it is very specialized. Like, a company came on that only really did some of the molten metal work for us in the series. There’s definitely, like, some specialization with it.
BTL: You can watch a show like Lord of the Rings and see those interstitial maps and not even think of it as being part of the VFX work.
Smith: There is a lot involved with it, too. We have to make sure we get them all right, and we’re in the right age. Getting all that information is right, but then we also want to present it to you as something that feels real and tactile, without making you ask where it is [and] without making you feel like it’s a place in the story, so it’s an interesting job.
BTL: Since you mentioned the sea creature earlier, I was really impressed with that and all the water stuff in that sequence, and I know water is traditionally one of the toughest challenges for VFX. Can we talk about that sequence a bit?
Smith: I’ll start by saying that, for Season 1, the bulk of the water work, especially the ocean work, was accomplished by Industrial Light and Magic, [and] supervised by Nigel Sumner and others. When we started the show, the first thing that we did is I went out and shot a bunch of aerial helicopter plates of the ocean. We lined up on an empty ocean, and just flew towards the sun, flew sideways to the sun, [and] flew away from the sun. We did that at different heights [and] at different times of day, and then we went out on a boat and we did the same thing with a crane arm. The crane arm went all the way down, maybe a foot above the water if you can believe it. We’ve got all of this reference photography, knowing that some of those would be plates in the show. Sometimes, you’re gonna find one of those plates, and it just works, and we did. Some of those you will just see in the show, but a lot of them, we knew off the bat, we’re not going to be using in the show.
They were going to go to ILM, because ILM had to build a water machine, basically. We couldn’t art direct our way through hundreds of water shots like we had on the show. We had to literally make an ocean that was more of a machine [where] we could dial the size of the waves and the type of sky and the amount of foam, and that machine [needs to] be able to generate those water simulations efficiently and effectively. Of course, there’s an artistic aspect to every single one of those — the choice about the dark swath of water that runs through the frame, and who knows why that’s there? Maybe [there’s] less air in that portion of the water or something. That kind of stuff in the ocean, artists would obviously get in there and do that kind of stuff, but the system was really, really good. I will say that ILM got their water to the point that those ocean shots… they were really efficient with them. They were really able to get those ocean shots through, including some shots that were a big ocean simulation, combined with the bow crashing through waves, which you may remember. [It’s an] incredibly detailed and difficult sim to combine with the ocean itself.
It all came down to the fact that ILM [has] got great tools for this stuff, but also, they’ve got artists who just have decades of experience. They’re just really incredibly talented people. We did pick up some artists [who] I’ve worked with for many, many years myself at ILM. That’s probably what it comes down to, is that those artists are just incredibly adept at what they do. They’re very able, and it really worked out well. ILM did a really good job on that, and they did the sea creature as well. Because they were doing so much of the water, we had them do the sea creature. That sea creature is a beautiful model. It’s complex, and there’s a certain amount that we see in Season 1, and [there’s] so much more to show. We’ll see if that comes up in a future season, but it’s a very detailed and interesting model, so I hope you get to see more of it.
BTL: I spoke to the VFX Supe at WetaFX for The Batman, and he mentioned that they had a whole dial system for making scenes have more or less rain. I thought that was pretty amazing that it’s gotten to the point where you can do something like that with a mere dial.
Smith: Unreal Engine is pointing to the fact that these things that have traditionally been very manual and complicated and one-off bespoke are starting to become things that we can predict a little bit more and repeat a little bit more. It makes the future of visual effects kind of exciting because you imagine a future where the technical is still fun, but you spend more of your day really thinking about the art, really thinking about how to make this shot tell the right story. That’s what’s kind of exciting.
BTL: Earlier, you mentioned the Peter Jackson movies, which take place hundreds of years after the events of The Rings of Power, so the locations in your show are much newer and less decrepit. Are you able to use any assets from the movies when you’re using some of the same houses and artists that worked on them?
Smith: No, we definitely had to start from scratch. That was something that we did very intentionally, first out of respect, but also [because] we’re a separate entity, so we definitely couldn’t borrow or start from anything. There was also a motivation that we wouldn’t really want to. Even [for] things where we had some overlap, and we were using WetaFX, and we’re using artists there that worked on the original assets, and that did happen on our show. Even then, we were always thinking about what we could do to make this our own. We want it to be familiar to the audiences in some ways… certain aspects needed to be familiar and recognizable, and [make us] feel like [we’re] coming back to Middle Earth, of course, but we wanted to make sure that everything that we did had our own take on it.
To that point, as we were approaching a creature, for example, like the Balrog, obviously, it’s described in a certain way, and the artwork that’s out there tells you about its fiery mane, and [that] it’s a creature of smoke and flame. There are debates about the wings being there or not, but there’s something that lives out there as a version of that [that] everybody’s seen.
But when we got into it, we dove in on horn shapes and face shapes. Even knowing that we might be touching on an individual that was the same — we don’t know, but maybe it’s the same individual that does appear in Peter’s films. If that’s the case, we wanted to make sure that we had a different take on it, and not just for the sake of having a different take, but just because there’s a value in that. Tolkien does open the door for us on some of those things, like, the Maiar can change form, and the Balrog is a Maiar. It’s this kind of Archangel type of character in Tolkien’s universe. We latched on to that and said, ‘Okay, if they can change form, and maybe as they evolve in their evil or good, or whatever it is, it makes sense that they would be able to change form and shape and proportion.’
We went ahead and designed a brand-new Balrog head with Allen Williams, who was one of our creature concept artists. He did a fantastic design. The horns definitely point in a different direction. They go straight down in a longer kind of way. The face is a little bit longer and sharper. What I love about that, is knowing that even if it’s the same individual, to me, it can live in the same world, but we’re putting our own take on it. That entire body, that entire head, everything was designed with a desire to feel that we have [done] our own take.
BTL: Ramsey also mentioned the challenge for VFX to resize the actors to be the proper height for their races.
Smith: Scale was a huge thing on the show, and when we started the show, there was a moment where we said, ‘Is there some visual effects solution to this? Is there something we can do so that production doesn’t have to build two costumes, two sets, and cast size doubles and things like that?’ We quickly realized there was no way to do that without getting in the way of the performances. You really have to handle scale the right way. You have to tackle it the right way.
The truth is, we’re doing stuff the same way that it’s been done for 100 years. We’re leaning on things like forced perspective, doubles, trick cuts, split screens — we’re doing all that stuff. It’s true that we have some new tools. We’ve got some better motion control tools, so we can do a motion control move, and then do a forced perspective scale move of the same thing and do another pass. That stuff is difficult to plan and difficult to get right and difficult to handle all the handoffs and the timing and stuff, but fairly straightforward.
Technically, what really became a challenge is what you just alluded to, which is the artistic side of how [to] trick an audience into believing that the 6’2″ person is actually 4’3″ when every clue is telling you that person is 6’2″. It was really an interesting journey, looking at all the little tricks that we have up our sleeve to do that.
One of them is just camera height, relative to that person. If you’ve got a camera and you go down here [holds camera below shoulders] and look up at me, you’ve just told me that I’m taller than eye height, in a way. If you look at me from up here [holds hand above head], you’ve just told me that I’m shorter than eye height. We found little things like that. If you frame me so that my head is almost about to get a haircut, or is getting a haircut, you’re telling me that I’m tall. There’s a bunch of headroom here. What we found is that we had Harfoot women [who] were standing here at a certain height, and then we had Lenny Henry, an amazing comedian and actor, who was 6’2″. When he was standing next to these other Harfoots, it was just very clear he looked like a human and they were Harfoots, and that was not okay. In some of these shots, we thought we would go in and grab him, cut him out and bring him down in height. Of course, that would reveal the background, [and] we would have to reconstruct people walking around back there. What we ended up doing, in a way that I kind of appreciated from a sleight-of-hand perspective, is instead of moving him, we grabbed all the women around him and moved them up, so that we covered some of the background with them instead of revealing [the] background.
If you go watch those shots now, there’s no height differential. It was things like [that] across the show where it really was these little artistic things, these clues that we would leave the audience — a large flower that’s right there and a set piece that’s large right there. A lot of the time, if there’s a set piece involved in one of those scale scenes, it was no accident. We would put something there because the audience knows how big this is relative to an elephant, so let’s make a larger version and put it next to this dwarf, and then the audience is going to know that dwarf is supposed to be small. I could talk about scale for about 13 hours…
Season 1 of The Lord of the Rings: The Rings of Power is now streaming in its entirety on Prime Video.
