VR On The Lot, Interview with The Foundry CTO Jon Wadelton
The newly formed VR Society (a division of the Advanced Imaging Society) and AMD held a two-day conference called VR On The Lot that took place at Paramount Pictures Studios in Hollywood on October 13-14.
There were several keynotes and panel discussions about all aspects of this new immersive media with an impressive list of industry insiders. I enjoyed VR On The Lot because the conference really dug into production concerns, and not just the billions of dollars being invested in this new form of entertainment. I spoke with one, Jon Wadelton, CTO of The Foundry, that weekend and he was kind enough to give me his thoughts on VR and how The Foundry began experimenting with VR a few years ago, and why.
How did flying in and participating with VR On the Lot become something that was a must for The Foundry?
We are living in a special time right now: the secrets of VR are still being determined, artists are willing to share information and big companies are throwing a tremendous amount of resources around to see what will stick. With that kind of momentum, we think software companies owe it to themselves and their customers to jump in and help out. VR on the Lot was particularly appealing because of the caliber of attendees; you had everyone from the heads of VR firms to film executives all offering a different take on the artistry, economics, and roadmap of VR. Not only was it fascinating, but it created an opportunity to update ourselves on ground level issues, hear the latest and greatest and share some of what we’ve learned. Also, unexpected partnerships flourish in that kind of environment, so we knew we had to go.
How long have you (personally) and (The Foundry) been interested and invested in VR?
Personally, since the early 90s at least. Our professional R&D team has been interested for the last few years. Their whole job is to experiment and innovate, so when they saw VR coming, they immediately began working on CARA VR. That way when VR hit, we’d be ready. That was at least 3-4 years ago.
Do you think there is a clear path to monetization for VR for the B2B market? How long will that last, and what do you think will need to happen for VR to evolve to the degree the big film companies will consider worthy?
I do, the interest is there and the artists were already experimenting before companies like us started releasing tools like CARA VR. VR’s trajectory might be up for debate, but creating a better VR pipeline is not. Artists need it, and are already putting it to use. Take CARA. The idea behind that was to build a professional-level plugin into a tool production teams are already using (NUKE); that way we could satisfy recurring requests for high-level tasks like VR stitching and color correction, and give artists something they can count on.
In terms of studios, it’s clear they want to be a part of this. If they go all in remains to be seen. Their hurdle will be solving for the perspective switch of a new storytelling model. Their history is one of controlling for whatever’s in front of the camera. With VR, suddenly you have to think in 360 degrees. For VR to really take off at movie studios, they’ll have to figure out how to tell their stories in a new medium. But I believe they can do it, and I’m excited to see where they push storytelling. Our role is to make sure they have the tools they need to do it; and thankfully, a lot of what our tools do best can be directly applied to VR.
Is The Foundry working with gaming? Theme parks?
Yes, the MODO 10.1 release has a strong connection to game engines, plus tighter integration with MARI. We also have many new VR initiatives happening in R&D in the real-time space.
How can The Foundry help spread the gospel?
Number one: be a resource for the community. We are in a unique position to help since our products have widespread adoption in the M&E markets and we have great relationships with major studios, artists, and companies that are innovating in the medium. This puts us in an amazing place to collect information and share it with others. We started doing that earlier this year with the launch of ExploreVR. The idea was to create a hub for industry insights, practical tips and the latest trends in VR content production that would go deeper than our products. In the coming years, the plan is to put up articles, how-tos, artist interviews, project breakdowns and anything else we can think of that will provide helpful information for artists in the field.
Number two, I mentioned before: make tools that help artists create better content. The levels of creativity in this industry are pretty astounding, so if we can give them the right tools for the job, they’ll continue to delight us.
Might there be a cottage industry within the post production community around VR?
There definitely could be for certain markets; the talent is there, and the tools are only getting stronger. The interesting thing about VR is it has so many applications that extend beyond media & entertainment. What M&E artists have going for them is that creative piece, which could get them a lot of business as bigger brands begin to want content.
What is The Foundry’s initiative for your clients?
ExploreVR, CARA VR, building VR capabilities into our tools and championing projects that move the whole game forward. One of the things that’s going to be really important right now is to listen to what artists are saying, and not overhyping the content. A lot of people are calling this “The Wild West”, which is really telling. The monetization of VR is just getting underway now. No one really knows the best path for the medium yet. So it’s important that we don’t oversell where everything is at. So the way we look at it is: listen first, act second. Dynamism is something you roll with, and since there are sure to be a lot of changes in the coming years, we want to make sure that what we are doing is actually helping artists create better work.
Who are your clients and how do they differ?
Foundry customers are predominantly based in the film, broadcast, advertising and (now) VR industries. So everything from production studios like Pixar and Framestore to freelancers. MODO, our 3D content creation platform is also used in the design and games industries. In general, all of our users are artists or designers of some sort. The differences are more based on their roles or style of work.
Can you tell me about the documentary No Borders?
No Borders is a 360° VR documentary about the plight of refugees arriving in Italy by Radical Plans.
The Foundry created a five part behind-the-scenes documentary on filming in 360° virtual reality (VR); ‘Filming with No Borders: 360° VR truth’. The series tracks the challenges faced by independent production house as it filmed a 360° VR film, which premiered to great acclaim at this year’s Venice Film Festival.
Like other VR docs that have focused on hard news stories, e.g. Frontline’s Ebola series, No Borders attempts to bring the viewers into a specific time and place in a new way, hopefully creating an empathetic link between viewer and subject. The film premiere at Venice Days was covered in the production series (part 5).
What were some of the challenges the production faced on that documentary? Was The Foundry able to solve some of the challenges with creative software?
Being a VR project, created on-location in the midst of a refugee crisis, No Borders definitely had its share of challenges. To start, there is no narrative, only a series of locations and interviews that take you through a journey. The project was designed specifically to create an emotional connection, which is both aided by VR and changed by it (non-linear).
Finding that balance meant designing the content so viewers could ease into the experience. That meant using visual cues to guide them, as well as eye contact and dialogue that would continue to engage their interest.
Production-wise, deciding between cameras proved challenging. You need something that will give you high-quality imagery (for the VR), but also something that is highly portable and lightweight. This becomes extremely necessary when you’re in a real world environment such as a refugee shelter.
Once we chose the cameras and secured the footage, additional challenges came up in post. You find that you have less control over your VR scenes than you’d like. Stitching complex scenes with many depth layers also played a role in helping ease that tension. Here, artists focused their efforts on manual corrections that would help them deliver high-quality results on a short timeline.
For us working on this project with Radical Plans was a learning experience during the pre-release stage of our CARA VR toolset. It gave us first-hand experience of the challenges as well as the workflows needed by artists in post.
Define Creative Software.
It’s multi-faceted, really. In short: it’s a digital tool that helps you visualize what’s in your head. But it’s also a way to use the power of algorithms and technology to ease the creative process. For instance, in real life if you wanted a different paint color, you’d mix it. In our programs, you’d use a color wheel, which allows you to pick any color you want, without fear of wasting any paint. The other way I think about creative software is that it’s a way to apply digital creations to an ever-changing canvas. We’ve been talking about VR, which carries the opportunity for some artist to have an idea, create it, put it into a VR world, upload it and spread it around the world without ever having to leave their computer. Because of digital technology, artists have the ability to respond to every new medium, applying their creativity in incredible new ways. Creative software is the engine that helps them do it.
How important is planning when it comes to a VR shoot?
It depends on what your goals are. If you are just experimenting, maybe it makes sense to wing it or confine yourself to whatever you bring with you into the field. You might discover a smarter way to work. But for most things, planning tends to be your friend.
How well are the camera rigs working? Is capture hardware where it needs to be? I imagine post is very expensive with VR, without the ability to properly light. Unless you have cinema-style 4k cameras with low light sensors.
The camera rigs are evolving rapidly. Ideally you want the highest resolution, highest dynamic range rig with synchronized cameras and global shutter to shoot the best quality content. For monoscopic, you ideally want the smallest form factor possible to reduce issues in stitching, and for stereoscopic you need a ring of cameras to capture parallax. There are different rigs available, from lightweight monoscopic rigs built using GoPro cameras, the GoPro Odyssey for stereoscopic shoots, through to Nokia OZO and Jaunt One designed for production. Eventually there will be different camera rigs to suit different requirements, right now people are testing different rigs and building custom rigs for specific scenarios.
How is CARA VR helping in the post process?
CARA VR builds four main functions into NUKE, so artists can solve key compositing, editing and finishing issues while creating VR. Those functions include tools for: camera solving and stitching, plate corrections, compositing, and headset reviewing. If you saw MPC’s Goosebumps VR Adventure with Jack Black, that’s an example of CARA VR in action. What we’ve found is that a lot of post companies want tools that can blend 360° video footage with 3D elements (like creatures, environments, props, etc.) to create realistic comps. Combining NUKE and CARA VR gives them that, through production-proven technology.
What is the most challenging and often tedious process? Stitching and compositing 360° video footage?
One of the biggest challenges with live action VR content is image stitching, pulling the camera images together to produce a seamless view in 360. There are some excellent stitching solutions out there along with CARA VR. There are always challenging situations though where you need a manual workflow for an artist to correct a stitch – where the camera moves, there is lots of dynamic content or content and multiple depths. A key focus in CARA VR has been to give artists the tools to be able to break VR content apart and work with the original images to rebuild seamless views.
I appreciate your time Jon, thank you.