Subscribe
Share
Share
Embed
The Data Movement explores the people, ideas, and technologies shaping how data is created, processed and used. Through conversations with technologists, creators, and business leaders, we dive into the applications, engineering, insights, and challenges behind the breakthroughs that power our data-driven world.
Paul: [00:00:00] Welcome to The Data Movement. I'm Paul from Seagate, and on this episode I'm talking to Erik Weaver from the Entertainment Technology Center, and we're digging into AI filmmaking and the future of emerging media. Let's get into it.
Here live at NAB, this is The Data Movement. This is my good friend Erik, principal of AI and emergent media [Erik: Yes.] at the Entertainment Technology Center at USC. I know you have the most rich, interesting background, and you dabble in so many different areas, and I wanna dive into it, but maybe let's just start there.
What does one do with their days as principal of AI and emergent media?
Erik: So we get to explore what's new, what's going on, what's the cutting edge. And I've been there for a little while now, and my kind of [00:01:00] title, or what I looked at, has really changed over the years. I originally came in looking at cloud and how cloud workflows work, and then got into HDR and other things.
Virtual production — I was very big on that for a long time. But I give away an annual grant, and we do a production, and we push the boundaries of all the technology options. So we're just finishing one up called The Bends, and it's a purely generative film. And then we're also finishing one called Pathways, which is a hybrid film. [Paul: Mm-hmm.] And last we had a project before that that just got into Tribeca called Chicka Boom, which is an animation one with the stereo studios. So I get to play with the tools and have fun.
Paul: Yeah. So right on the edge of what's emerging in terms of tools and technologies, and how you can integrate those into visual storytelling.
Erik: Absolutely.
Paul: Sounds like the best job.
Erik: It's a ton of fun.
Paul: Yeah, yeah.
Erik: Yeah. It's kinda curious how it came [00:02:00] about, though, because originally I did a lot of working groups and best practices and standards, and Universal came to me and said, "Let's just try this in a film." [Paul: Uh-huh.] Because now all of a sudden we have context to what we're doing.
So getting that kind of opportunity to dip your toe in, see how it really works [Paul: Yeah.] is a great place. Generative film.
Paul: Do you find that — when you are talking about this in filmmaking, videography settings — that it's controversial?
Erik: When we do our projects, we are filmmaker creative first. [Paul: Mm-hmm.] Period. I have a DP who's analyzing every frame. I'm not doing prompt and pray. This is not a slot machine. Everything is storyboarded and animatic'd out [Paul: Mm.] that we're doing. And we also do projects that are all impossible, or near impossible, [Paul: Mm.] for a [00:03:00] traditional filmmaker to do. So The Bends is about a blobfish, which is a deep sea fish, and how it changes as it goes up the depths and becomes a very ugly fish as it comes to the surface. [Paul: Mm-hmm.] But it's chasing something beautiful and shiny [Paul: Mm-hmm.] and it changes them. So that's the basic gist of this story.
Paul: Yeah. It's like any generative experience seems to be based on the context, the data that it's been trained on, but it has access to, to generate the output. So how are you thinking about that?
Erik: In our situation, we did a couple steps here. So the writer-director actually had an animation background, so she did some of her own sketches. There are plenty of tools out there where it can create things for you, or can take very rudimentary sketches and make things, or just look at your script and come up with things.
But I think in AI, you gotta figure out where your boundaries are. And I think the first [00:04:00] boundary for us is we don't really affect story or writing. It's an individual writing. And we're creating the first frames, or designing them, in a more traditional manner. So that's a little unique to some people who might wanna go a little more AI. Not quite our flavor yet.
And so from there we go and create things like LoRA models and a series of prompts. A LoRA is a low-rank adaptation. So in this situation, a blobfish is not something you can just find on, say, VO3 [1] or any normal model, 'cause it's a very unique fish. So we had to go to oceanic institutes that had images, and we took those images and basically created three-dimensional models and cleaned them up.
And then we wrote a script that turned it around and lit it in different ways.
Paul: Hm.
Erik: And then we had to burst to mass compute, 'cause we didn't have a large enough VRAM [00:05:00] footprint to create the models, and so we created the character. When people would log in — 'cause they all had a virtual machine — they would see things like their models or the tools they could use, like VO3 and some of their base prompts.
So they had everything ready for them. In this situation, we're also running ShotGrid, so we have it completely broken down. We had 86 shots across eight different scenes, and a woman named Verena Plumb [2] actually managed the creative team, which is five people. She said, "You get these shots and these scenes," and handed them out. And then they would log into the virtual machine. They'd have access to everything, and they would work on those shots. And they would look at the animatic, and they would be designing to that. So yeah.
Paul: Interesting.
Erik: And then when they thought they had a shot, they'd raise their hand. [Paul: Yeah.] And basically the director and the DP would come over and look at the shot and say, "You know, I really like this," or, "I don't like this," and, "Can you change this or change that?" And what you'll find is [00:06:00] usually you can get about 80% of the way. Once in a while you get a little higher — a finished frame — but sometimes you still have to go in and do in-painting, out-painting, a bunch of manipulation to the shot to get it exactly as the writer-director wanted.
Paul: How different is the workflow that you're describing from a traditional one?
Erik: So it's really interesting. So if you take something like Chicka Boom, which is gonna actually premiere at Tribeca, it is very unique, 'cause it's not — again, not this concept of prompt and pray. A lot of things make it look very easy. But it is not as simple and easy as you might think. This is layers of AI being used in different things, from style transfers to procedural generation, through other components [00:07:00] of the pipeline. And then building it together, which saves a lot of time in some of the animation pieces. But it's a very sophisticated hybrid use.
Asteria finished that film — it's now theirs, with Seachrist Patterson [3]. When you get into purely generative, it's a lot like animation. You're designing things. Even though it might look live action, you're designing more of an animation workflow. When it comes to the hybrid, that's more traditional production. You're just trying to capture your performances, and what's critical there. So —
Paul: Yeah. You mentioned earlier that you're funded — you're associated with the university. The university has associations with the big studios. They're very curious and interested about how these tools and technologies — I guess they're not doing it for fun, right? They're doing it because they see opportunity, and they're curious about how these things, these projects that you're working on, can be integrated into bigger projects, right?
Erik: Yeah. When we originally started, they'd really put a pretty hardcore list of, "I wanna test this. I wanna test this." [00:08:00] And now they've kind of lightened up and said, "Go do what you do and show us what you come up with." [Paul: Mm-hmm.] So we'll go and do a roadshow afterwards to the studios and kinda give them a breakdown of how we execute this, or what was tricky here, or what we're actually finding that doesn't work.
We break a lot of systems. [Paul: Yeah.] It's pretty tricky. We really push some of these tools very hard.
Paul: Yeah.
Erik: Jetset's a really good example of this. Jetset basically allows you to map people and things into the environment. [Paul: Yeah.] And it requires a level of precision that is very, very articulate. [Paul: Yeah.] And hard to do. Unlike traditional tools that map to virtual production stages [Paul: Mm-hmm.] like a Mo-Sys or Stype or anything like that — those have a little more flexibility in exactly what's being played, but [Paul: Yeah.] this situation, when you're marrying the worlds, requires a lot [00:09:00] of nuanced detail.
Paul: Yeah. How do you feel about this as a direction for art and creativity, in general?
Erik: I absolutely think that this is gonna empower a lot more creativity. I think that it is going to bring up the value of what more independent and infinite productions [Paul: Mm.] can do. I wouldn't say — a lot of people like to say, "Well, this is time and cost savings." [Paul: Mm.] I wouldn't ever phrase it as cost savings. I would say a little bit more like your budget can stretch further [Paul: Mm-hmm.] with this. If you learn how to do this and do this appropriately, you can get a lot further with the budget you have.
You might be able to do a $30 million film on basically a $10 million budget. [Paul: Mm-hmm.] Matter of fact, we're working with one particular client who's actually looked at it, and they had an indie budget, and they had a scene where it was gonna [00:10:00] cost them a million dollars, and all they had was a $6 million budget. [Paul: Mm-hmm.] And now we could show them how to execute this scene for, you know, $100,000 — $200,000 at the most on that side. And probably savings overall on their production were about 16%, more B-roll, second unit, kinda other things, and it's about a $1.5 million savings on a $6 million film, and that makes an impact, you know?
Paul: Yeah.
Erik: And so the number one thing is creativity. [Paul: Mm-hmm.] It does push the boundaries of what you can do creative-wise. [Paul: Mm-hmm.] First thing you do normally when you get a script is you walk through it and you say, "We can't do that shot. We can't do that shot." [Paul: Mm-hmm.] They're too complex. They're too technical, [Paul: Right.] or too expensive. [Paul: Right.] Now you can do a lot of these shots. [Paul: Yeah.] So that's what's kind of exciting to me — being able to write unbound.
Paul: So you're firmly in the camp of this being [00:11:00] augmentation to the creative process rather than replacement.
Erik: I absolutely am, but I think that you really still have to keep those solid people in there — department heads. [Paul: Yeah.] You still need that structure and that organization. [Paul: Yeah.] You need a production designer. You need a DP. [Paul: Right.] You need all these people still. They work with new or different tools, [Paul: Yeah.] but their job is fundamentally their job. They have a sophisticated eye. They know what they're doing. They know what they're looking for. They have a reason they put a plant here and that there, [Paul: Yeah.] and that costume is on them and this costume isn't. They are experts at their job, and they still need to be involved in the process.
Paul: When you think about the infrastructure underneath the workflow, what are your kind of initial observations, and maybe even predictions, about how disruptive this stuff that you're doing is gonna be to the infrastructure that supports it?
Erik: So that's actually an [00:12:00] interesting point, because creatives are not computer scientists. And so there's an odd balance here coming. For example, when we were doing The Bends, we ended up — depending on how good somebody was — they'd have 50, 60, 100 shots, [Paul: Mm.] right, trying to get it right.
Paul: Mm-hmm.
Erik: And then we'd have to save so many, so there was a huge amount of data that starts [Paul: Mm.] generating, and you're like, "Well, I want this. Do I want that? Is it —" And you've gotta sort through all of this, so that's a very sophisticated thing. The other thing you're seeing is a lot more metadata across your different tools and pieces, [Paul: Mm.] which is sophisticated. And then going to be able to use things like cloud or localized storage in appropriate ways [Paul: Mm-hmm.] — that becomes very tricky, 'cause you have a lot of mediums, [Paul: Mm.] and you've gotta balance it all and make sure you don't lose content, you're not doing things wrong. [Paul: Yeah.] All [00:13:00] this stuff.
Paul: Do you see it creating new challenges?
Erik: I do, but I think with this whole generation of AI, there's a lot of tools that are gonna allow you to simplify this process.
Paul: Mm.
Erik: You know, live coding and other things — the ability to build these pipelines is gonna all of a sudden spin up much faster. And it probably won't be individuals doing it from their own production. There'll just be a couple more booths here at NAB next year that help solve these complexities, or make them simpler.
Paul: Yeah. You mentioned storage and data. Right. Do the emerging workflows and things that you're working on have kind of a compounding impact on storage requirements?
Erik: I absolutely think so, 'cause you get more and more shots, and it's more complex, and where have you stored this, and what's going on? Like, for example, when we're shooting hybrid, [00:14:00] we've got to marry the scenes and understand the lighting and keep all these different camera components and files and make sure they're connected in some way. So things like the ASC MHL — ASC did a media hash list for creating a manifest when you're on set, that makes sure people get it later, that it's all the same thing, and that it could be correlated to other files, or a whole day of production, or something like that.
Paul: Yeah.
Erik: And what's interesting is this is actually a correlation to SAG. If you're familiar with the latest SAG rules, it says that you're allowed to use a digital double, or whatever you wanna call it, for an actor, but there's different — A, pay rates, and B, at the end of the production you must delete all that data. It can't be used for anything else, and it can't go to a public model. So this is really important. So private cloud, private storage devices, or private compute to run some of these things are [00:15:00] critical.
Paul: So that's written into the license agreement?
Erik: That's written into SAG contracts for actors, but in our agreement they said the same thing.
Paul: That whole thing is just so fascinating — the digital IP, and the future of licensing, and how that gets governed.
Erik: In the future everyone's gonna have a digi-double [Paul: Right.] that understands exactly what they look like — their cryptosona, their nuances. And how licensing works is gonna completely change. How CAA or UTA deal with a client — gonna totally change. You used to have to, say, for example, you're a major actor, you might have to go to Japan to shoot the latest whiskey or cigarette commercial. Now they just license an avatar.
Paul: How far out is that, I guess?
Erik: Deals are being done today.
Paul: Yeah.
Erik: How it becomes mainstream is really probably two years — a year and a half, two years [Paul: Mm.] before it becomes really mainstream.
Paul: Yeah. Is that something they have you [00:16:00] thinking about?
Erik: So in our next production, [Paul: Yeah.] we're going to have partners that are playing with some of this stuff. We might use it in the use case of B-roll footage or dangerous scenes — things that I can't get insurance or other things on. [Paul: Yeah.] So we might play with it in these ways. So —
Paul: Yeah.
Erik: We'll see how it goes.
Paul: What are you most excited about looking forward?
Erik: So I think it's not what I'm most excited about, it's what I kinda want to tell people. Have you ever seen the movie Interstellar?
Paul: Yeah.
Erik: Do you remember where they're on the water planet, and the wave — they realize it's a wave coming at 'em. They see this thing in the distance, and they're like, "Ha, ha," and then they realize it's a tsunami coming at them.
Paul: Mm.
Erik: [00:17:00] And a lot of people have talked to me and they say, "Oh, AI has messed up Hollywood," and I'm like, "AI hasn't hit Hollywood." [Paul: Mm.] AI — yes, there might be some utility-based tools that are changing. [Paul: Mm-hmm.] But on the creative side, they're very, very few and far between [Paul: Yeah.] actual productions that have implemented this technology. [Paul: Mm.] But that wave is coming at you. It's coming at you right now, and if you don't understand that, you're gonna get hit real hard.
Paul: Mm-hmm.
Erik: It's pretty crazy. [Paul: Mm.] So, it's okay — like, if, for example, you were a physical artist and you drew things, and you didn't wanna go to digital systems, that's okay, right? But you need to know what's coming. [Paul: Yeah.] And you need to understand how it can either benefit you, or if you're just gonna be out, and put your head in a hole like an ostrich, [Paul: Mm.] that is your prerogative. So. But it's coming.
Paul: Yeah. What's your advice to — so I think that — like, they see the tsunami. Like, what do you, what do you do about that?
Erik: I think you should just go out and do a survey of the tools out [00:18:00] there. I originally used to like to suggest to people that they take one of these 31-day classes where you play with a different tool every day and just see if any of 'em are appealing to you.
Paul: Mm-hmm.
Erik: But I might actually just go to ChatGPT and ask it to put together a program of testing for you.
Paul: Mm-hmm.
Erik: And say, "These are my interests. I wanna look at tools that do these kinds of things," and ask it to put together a program of, "Let's play with this. Let's try this. Let's see what [Paul: Yeah.] this is," and so forth. And then go explore from there.
Paul: Yeah.
Erik: Again, you'll find things that you really like, and things [Paul: Yeah.] you're — "I'm not interested," or, "I don't wanna touch that at all."
Paul: Yeah. How far off are we from the first kind of generative — like, silver-screen Hollywood movie?
Erik: I think we're probably nine months to a year from the first one.
Paul: Mm-hmm.
Erik: What a lot of people don't understand are the complexities in [00:19:00] making these things. There's still a whole lot of process. [Paul: Mm-hmm.] There's a film right now that's being shot, and that — where they're replacing the backgrounds. It's live action. [Paul: Mm-hmm.] I can't remember the name of the person doing it, but — Casey Affleck is in it, and I think they've done most of the shooting already.
Paul: So it's a hybrid — it's not pure generative.
Erik: It's hybrid. It's hybrid. Yeah. And I'm honestly not gonna be a proponent of pure generative.
Paul: Yeah.
Erik: It's not really my thing, unless you're doing animation or something that's non-actor.
Paul: Yeah.
Erik: You really still wanna value the performance and the nuance of that, [Paul: Yeah.] in my personal opinion. But a purely generative one is gonna be a little longer. A few of the people who promised an AI short or animation piece [Paul: Yeah.] are realizing how hard some of the components are, so I wouldn't expect a genuine one — unless it's something [00:20:00] light and easy — to be out for another six or seven months, at least. At least.
Paul: Yeah.
Erik: There's a lot of nuanced problems that we run into when you're building out these pipelines, like locked weights or closed environments. [Paul: Mm-hmm.] So you might use the same prompts and you're getting a totally different answer the next day, 'cause of the model changing, or again, a lot of different issues there. So you gotta solve a bunch of problems behind the scenes.
Paul: Yeah. But there are funded projects today that are being worked on [Erik: Yes, absolutely.] that will hit the screen within the next year.
Erik: Yeah.
Paul: Yeah. Interesting. The comment that you made around the human performance being important —
Erik: It's critical.
Paul: Yeah.
Erik: Yeah.
Paul: I guess the emotion and —
Erik: Yeah.
Paul: — it's not replaceable from a —
Erik: And don't get me wrong, I think that there's gonna be some [00:21:00] really good versions of things [Paul: Mm-hmm.] coming out. But I personally like actors, you know.
Paul: Yeah.
Erik: So.
Paul: Yeah. Do you think we'll have pure digital actors?
Erik: Absolutely, but in my opinion that's simply a puppet. We connect with Muppets or puppets sometimes, and sometimes we don't, you know. [Paul: Mm-hmm.] It's about how does that impact you, or make you feel human.
Paul: Right. Yeah, which is, I guess — like, that's the purpose of arts and media, is to connect and to resonate.
Erik: [Paul: Mm-hmm.] And that's what I think a lot of people get lost on — this technology is not just for technology's sake. It's to serve story. If you are not serving story, [Paul: Mm.] then I'm not sure why you're doing this.
Paul: I like that. Start with a story, and then blend the technology and the [00:22:00] creativity for greater output, and then lower the barrier to entry for more people to be able to create.
Erik: Because again, there's a lot of communities who have traditionally had very hard times telling their stories — younger generations, unique regional groups or people. There's just a lot of those things where it requires so much resources to make some of these films, that that story had gotten lost [Paul: Yeah.] in storytelling. And now some of these things can be shown, can be understood, can connect with people.
Paul: Yeah. What's next?
Erik: I'm actually in the middle of winnowing down to our next script. We do our final pitches on Friday of this week.
Paul: Okay.
Erik: And so I created this award committee — it's a lot of senior people. [Paul: Yeah.] And I've got four directors, and they're gonna just pitch. They're gonna pitch their stories and say, [00:23:00] "What do we wanna make?"
Paul: Is it student directors?
Erik: So they're all people who've graduated with a master's from USC, [Paul: Yeah.] with diversity, and they get to share these things. This year is different, because we normally have them write the stories and direct. [Paul: Yeah.] But instead, we went to Mary Sweeney, who's the head of the writing department at USC [Paul: Mm-hmm.] — and David Lynch's former partner — and she found wonderful scripts out there with recent graduates. And so now they're each gonna pitch one of the stories, and then we'll pick one of them, and we'll get to making it.
Paul: Yeah, very cool. Erik, what question should I have asked you that I didn't ask you?
Erik: I don't know. I think that you were correct in the idea of data, and metadata, and complexity, and storage. [Paul: Yeah.] These worlds are just gonna continue to get more complex. [Paul: Yeah.] And it's critical to be able to figure some kind of layer [00:24:00] over that manages that complexity.
Paul: Yeah.
Erik: And so paying attention to that is absolutely important [Paul: Yeah.] as we move forward.
Paul: Yeah. It seems like just a lot of computing power, a lot of storage capacity, in order to render the files, edit the files, [Erik: Yep.] archive the files.
Erik: Understand what files you have.
Paul: Right.
Erik: Yeah, yeah, yeah. It's just gonna get bigger and bigger. It's a bigger data problem.
Paul: Kind of like a multiplying effect on the data problem.
Erik: Mm-hmm.
Paul: Which maybe AI will be able to help with too, right?
Erik: I think it eventually will. Yeah, absolutely. We're in a world where AI — if I'm quoting other things — can sort this out. It's not completely sorted out yet, but we expect tools to show up [Paul: Yeah.] very soon.
Paul: Yeah. Cool. Erik, it's been such a fun [00:25:00] conversation. Love your perspective, love that you're pushing things right at the bleeding edge of what's possible. I'm gonna be following closely to see what you guys are building over there. And I'm gonna watch The Bends.
Erik: Awesome. So we also have ETC official on Instagram — we run a lot of videos there. We publish stuff on LinkedIn as well.
Paul: Yeah.
Erik: We've traditionally done a large white paper at the end, but we're trying to do a lot more real-time media [Paul: Yeah.] out there, so that people can see things, because stuff ages really quick out here.
Paul: Yeah, yeah.
Erik: You know?
Paul: Yeah.
Erik: But, um.
Paul: So people can find you at —
Erik: Yeah, so @etc-uscofficial [4]. Try that out. It's somewhere along that line.
Paul: Very cool. Thank you, sir. Appreciate you. Thanks for joining us.
Erik: No, thank you.
Paul: That's it for this episode of The Data Movement. Thanks for listening. Find us wherever you listen to your [00:26:00] podcasts for more conversations about how data is moving the world forward.