Subscribe
Share
Share
Embed
The Future of Gaming DAO or FOGDAO is a decentralized, tokenized community exploring the future of the gaming industry.
Nico Vereecke:
GM friends and welcome to the future of gaming. You're listening to our weekly podcast. If you're on YouTube you will see that we are a lot of people today. It's a special episode. We're five. We have the usual crew. Philip Collins, Devin Becker, myself, Nico Verriken. And then we have two special guests. First and foremost we have Maria Gillies who is the new product director at Pixion Games who's
Maria Gillies:
Mmm!
Nico Vereecke:
also an advisor to Scripted. of whom we have the CEO and co-founder here, Tim Cotton.
Tim Cotten <> Scrypted:
Hello everybody.
Nico Vereecke:
There we go. I did it well, good. So goal of today is in the show notes, there will be a link to a YouTube video in which you hear the beautiful voice of Tim, which you will also hear on this podcast, explain what the hell he built. He used AI and built something really cool. It's called AutoRPG. And we're gonna dive into that today. Anyway, so watch that before you listen to this. We're gonna assume you have watched that video You know while discussing this All right Before we get recorded Phil asked Maria question that I wanted to have recorded because I want to know the answer as well And I think more of you will so Phil. What did you ask Maria?
Philip Collins:
My question was pretty simple. I was just like, Maria, so what are you up to these days?
Maria Gillies:
I've been doing walks on the beach. I discovered IKEA customer support is great today. It's been good. No, I recently joined Pixion as product director, like Nika said. I've been there for two weeks. I joined the dark side. I'm now working in blockchain gaming.
Nico Vereecke:
Hey,
Maria Gillies:
Aye.
Nico Vereecke:
welcome.
Maria Gillies:
Honestly, it's been so much fun. I think in these past two weeks, I've had so many interesting conversations and my brain just feels completely stimulated like never before so it's been super super fun. I knew I wanted to work in like an unknown territory space where there's no exact playbook and yeah it's so fun.
Nico Vereecke:
Amazing. So, maybe, so I think many of us on the call here know Pixion Games or maybe not. Can you just briefly describe what you guys are up to there?
Maria Gillies:
Yeah, so we're currently developing Fableborn, which is an async competitive game. It's like a mashup of action RPG with base building, you're raiding. I have to be careful about what I say because
Nico Vereecke:
Mmm.
Maria Gillies:
we're cooking up some announcements and I don't want to accidentally say something I shouldn't.
Devin Becker:
Leak leak leak.
Maria Gillies:
No, no, no. But it's a unique game. Like I played it when I was in the recruitment process and I felt wow. It's been a long time since I felt just a fresh mobile game experience. So I'm hyped for it. I'm excited. I'm part of bringing it to the market. I'll leave it there.
Nico Vereecke:
So I played Failableborn and it made me realize what a terrible mobile gamer I am. Like my thumbs just don't work, you know? You need some mechanics for that game. At least I had the impression, because I was playing the mobile parts more than the base building parts. Can people try it out still, Maria?
Maria Gillies:
No, enough-
Nico Vereecke:
OK.
Maria Gillies:
I have someone drilling, I'm going to go close my window.
Nico Vereecke:
Okay, good. In the meantime, I'll ask Tim, Tim, tell us a bit more about maybe what you were up to or tell us more about scripted.
Tim Cotten <> Scrypted:
Sure, let me just start with thank you for watching the video ahead of time because
Nico Vereecke:
Hmm.
Tim Cotten <> Scrypted:
everything I'm about to say will make so much more sense now that I'm sure you've watched that auto RPG video. So, you know, I founded the company kind of at that intersection of AI and web 3 to kind of figure out where these things make sense as we move the conversation forward. And two years ago, I was worried about AI. I felt like we were starting to enter another AI winter where the progress with very large language models was not going to result in any application development. And I was wrong. And I'm very happy to be wrong. Because what we've seen over the last even, I mean, just in our Fogdow channel and Discord over the last three weeks is a complete inflection point. We are seeing apps come out faster and faster than ever. pushing past just prompts. So I wanted to see how do we leverage the game development knowledge that my company has against building game dev tools, tool sets using AI. So what you see when you look at AutoRPG, it's not a procedurally built level. It's an actual AI with some ideas about what it wanted to do and how to accomplish those. And then it worked with another AI agent and in a multimodal way. And what I mean by that is not just prompts. but also visualization and recognition to build a level. And I thought it was pretty amazing for what it pulled off to the point where I look at it now, not just as an in-house tool that we can use to like build out our game faster, right? But something, this is commercially viable. I already, we put that video out, the response was overwhelming. I've had 200 people join my LinkedIn over the weekend when I was camping with my kids. dozens of VCs and angels asking, hey, how do we get into on this? I'm like, well, we need to build something next in 3D, something that shows not just tiles. So that's what we're doing right now.
Nico Vereecke:
Amazing. It must feel good, right? Everyone's struggling to raise, and you're like, you know, VCs just asking you to give you money.
Tim Cotten <> Scrypted:
It was one of those random moments, especially as you've seen with like the downward trend in web three investor interest where suddenly not only is AI hot, but it's meaningful. Like I have, we have not even completed the customer discovery process. Right. My COO is like talking with people, finding out like what, you know, valuation of this kind of tool. But I have school groups, like for the larger online private schools, uh, calling me and saying, hey, can we use this as a tool for the kids? I'd never even considered that as a possible segment, let alone what else is out there, right? So there's a lot of doors that AI is opening right now.
Nico Vereecke:
So my idea for this recording would be to maybe dig a bit more into first, like what you've done and how it works and what kind
Tim Cotten <> Scrypted:
Sure.
Nico Vereecke:
of tools or
Tim Cotten <> Scrypted:
Yeah.
Nico Vereecke:
what kind of AI advances it's leveraging and then maybe reflect on how fast all of that is moving. And then finally, I'd also like to get all of your guys' thoughts on the, the potential here, like the, the, you know, what does this look like at the rate that's advancing in maybe three years from now? Are we. Is game development being democratized? Is everyone going to play their own game essentially? Anyway, so these are the things you can already have, you know, working in the back of the backs of your minds. But first I'll kick it back over to you, Tim.
Tim Cotten <> Scrypted:
Yeah, that's great.
Nico Vereecke:
Tell us the process, like what's the starting point of what you've built and then, you know, walk us through the different steps, the different, you know, you're standing on the shoulders of giants, like who are those giants and
Tim Cotten <> Scrypted:
Mm-hmm.
Nico Vereecke:
What's your special input and how does it work?
Tim Cotten <> Scrypted:
Absolutely. So I'm going to give you just a quick overview of where we were like two months ago with AI. AI, we had reached a point with prompt systems, thanks to OpenAI and Google and others like them, where they had trained so much data that we actually got interesting and meaningful results from interactive systems with AI. It's incredible. But it didn't necessarily do anything with game design yet. or game development in general, because the output from a chat GPT-like system is naive and pretty much useless for a game developer kind of person. And if you look at my blog article related to AutoRPG, you can see an example of what I mean where it says that it's generating a forest, but in reality it spits out a tile-based system that's like a little circle of water. It's pointless. Then a fellow comes along by the name of Yohei Nakajima and others like him, but Yohei, he posted a paper about autonomous task agents. The idea being you take a GPT system, something that can handle input from a human and then compare it against its vast database of knowledge and then give you something interesting as output. And instead of manually having to sit there, and refine your stuff that you're asking it, you just say, here's the output. Now go design a better way to ask this question. Or if I need to know more, design a list of tasks for me to execute. And so it's a program, it's a methodology, that takes a GPT system and uses other GPT agents to go down the rabbit hole. Right? until it makes small solutions and then aggregates them back up to a final solution. You end up with very powerful results. Like the very first version we played with, we asked it to look up research information about the history of procedural generation in video games. I got 14,000 lines of text out of it in about an hour of API usage. That made me think. Well, what if I hooked it up to the game dev harness that we were building, where we have all of these tools and ways of thinking about putting worlds together? And so we did. We spent three weeks doing that. And as you've said, we are standing on the shoulders of giants. This is stuff that did not come from a vacuum. This comes from decades and decades of game development and AI research. Just we've reached that inflection point where we can put them together as applications instead of having to understand how to build a neural network by ourselves. That's the advance. And that's like three weeks old.
Nico Vereecke:
And so just so I understand it correctly, um, and our listeners to the autonomous task agents is essentially like a bunch of let's in this case, let's call it GPT 3.5 or fours
Tim Cotten <> Scrypted:
Yeah.
Nico Vereecke:
that each have their own role. Like one of them is maybe the task master that's, you know, decides what tasks need to be done and then each of these tasks gets sent or, um, has to be solved by one of the other GPTs, those being back results. And then. Is that the way we should think about this?
Tim Cotten <> Scrypted:
I think it's a great way to think about it, even if the implementation may vary. If you are making a game comparison and if anyone loves Mass Effect, then it's more like the geth. They're individual programs that can work together. In this case, you are right. The initial version of Baby AGI, which is what Yohei Nakajima made, has a task agent, a master agent, that makes subtask agents. They execute, return, and summarize, and then it loops back. Now our innovation was to add additional logic around game design and add additional modules such as talking to Mini-GPT-4, which is not an open AI product. What it can do is look at images and then classify them kind of on the fly. So if I show it an image of a 2D top-down role-playing game, like in the demo, and say, hey, look at the quadrant to the northeast. What's there? And it says, well, there's a river. I say, great. What's south of it? Still a river. What's west of it? An empty field on a hill. That's stuff that the AI can work with in a summarized way. Trying to throw all the map at it at the same time doesn't work. Tokenization limits, memory limits, that's what these kind of harnesses are for. It's to abstract away the hard parts and let the AI still be creative. That's what we did with AutoRPG. Now, the output for it, and this is the thing I love. is that we assigned the world builder AI a personality and said, hey, you're idealistic. So approach this from the perspective of a fantasy role playing game and make an ideal situation. Now, when I put the more anxious or the more paradoxical agents against it, I got very different results, kind of interesting, strange results, right? Additionally, we created NPCs virtually. The same way that the Stanford team did with 25 agents, with their experiment, we use those agents to say, hey, you are standing on the path that you say you created. And you're looking at the scene. What's interesting? Nothing is particularly interesting. Go create something interesting. OK, I found an art asset for a strawberry. Let's put those near the river. Now I will leave the path and want to go to the strawberries. Suddenly, there's a narrative reason for things to exist in the scene. that didn't exist before. And it's not procedural generation, it's narrative generation. It's autonomous generation. That's what's new.
Nico Vereecke:
You mentioned beyond the autonomous task agents, the term harness.
Tim Cotten <> Scrypted:
Hmm.
Nico Vereecke:
Can you, because I feel like this is something that's key to your innovation.
Tim Cotten <> Scrypted:
Yeah,
Nico Vereecke:
Can you just elaborate on what that
Tim Cotten <> Scrypted:
sure.
Nico Vereecke:
is?
Tim Cotten <> Scrypted:
Yeah, I think that's fair. I think if you look at the code for Baby AGI or Yohei's new project, Baby B AGI and Auto GPT and all the others, they're relatively short. They're a couple hundred lines of code. They're not very big. That doesn't help you build a game, right? So of course we have a larger harness instead of it being a couple hundred lines of code. It is quite large. It is things about how tiles should be related, chunking methodologies, all sorts of stuff from game design that I love tile-based games. I've worked on them since the beginning of my career that I knew was cool ways to build a world and then treat them more like brushstrokes the AI can use than individual pixels being painted on the screen. It's more like a painter working with tools. So when I talk about a harness, I'm talking about the way one agent can talk to another. the way one model can talk to a different model or get reviewed by it, and then the tools that they have at their disposal to affect the internal state. And the internal state, in this case, is a game map, a level.
Nico Vereecke:
Okay, I'm going to stop my questions here and I want to have the others. Maria, Phil, Devin. Any comments, questions, remarks, observations? Are you guys getting scared?
Devin Becker:
I
Nico Vereecke:
Go ahead,
Devin Becker:
just
Nico Vereecke:
Devin.
Devin Becker:
have a quick question. I'm curious how much, so you talk about the size of the harness and all that stuff, obviously you guys are experienced coders. Are you actually getting any code out of the AI as well that, that you're able to use like with the tooling and stuff like
Tim Cotten <> Scrypted:
Oh,
Devin Becker:
that or.
Tim Cotten <> Scrypted:
yeah. Yeah, OK. I don't want to get too secret saucy on you, but there is definitely the ability to generate useful code in a sandbox environment for further, let's call it, specialization of requests. When the AI wants to do something and the tool set says, I don't have that, we can ask the question back, can you generate a method to do that? And if yes, That sounds like a fun idea to try. Now, I've seen it kind of go both ways in our system because we don't have a perfected system yet. Things can go terribly wrong with it, but it does create interesting results. And I will say that there was some code execution that was invented by the AI for the level that you see.
Devin Becker:
One is a follow up question then for that. Like one of the things I ran into as a problem with like auto GPT type of stuff, right? Is there's like a certain amount of variance that's supposed to happen to make the responses kind of interesting or whatever in so
Tim Cotten <> Scrypted:
Hmm
Devin Becker:
you don't get like the, you know, the, the temperature or whatever, like to have the
Tim Cotten <> Scrypted:
Yes.
Devin Becker:
fluctuations, like
Tim Cotten <> Scrypted:
Yep.
Devin Becker:
has that been problematic in like the, cause when it's
Tim Cotten <> Scrypted:
Yeah.
Devin Becker:
like step by step, right? Like if any of the in-between
Tim Cotten <> Scrypted:
Yep.
Devin Becker:
steps like vary, you get pretty
Tim Cotten <> Scrypted:
Yeah.
Devin Becker:
different results, which could be like
Tim Cotten <> Scrypted:
Yeah.
Devin Becker:
good if you want creative
Tim Cotten <> Scrypted:
Yep.
Devin Becker:
results, but if you want like
Tim Cotten <> Scrypted:
Right.
Devin Becker:
consistent results, that sounds problematic. Like.
Tim Cotten <> Scrypted:
It
Devin Becker:
How
Tim Cotten <> Scrypted:
is
Devin Becker:
has
Tim Cotten <> Scrypted:
problematic.
Devin Becker:
that worked out?
Tim Cotten <> Scrypted:
That's the, that's the entire design of a harness, right? That's the goal is to figure out instead of relying on the model itself for the temperature, what level of determinism versus randomness do you actually want? Tuning a harness is the, is the key that we found was useful. Now I'm not going to give all of that up right now, but I will say that, yeah, you're on the right, right track there.
Maria Gillies:
I
Nico Vereecke:
Amazing.
Maria Gillies:
think whenever
Nico Vereecke:
Go ahead.
Maria Gillies:
I hear you, oh, Nico, what are you going to say? Whenever I hear you talk about this, Tim, you make it sound so simple and kind of obvious,
Tim Cotten <> Scrypted:
Hmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
Maria Gillies:
but my brain is screaming at me saying, why
Tim Cotten <> Scrypted:
Mm.
Maria Gillies:
am I so primitive in my thinking? Because I can't even imagine how to do this.
Tim Cotten <> Scrypted:
Let me.
Maria Gillies:
And so I think... I think there's this really unique proposition that Scripted has, which is a really good understanding of how to design games and a really good understanding of utilizing tech. And I don't think there are many companies out there where you can combine those two depths of knowledge to just create things. Like, if you're coming from a non-gaming background, you haven't designed complex games. You probably know how to use the tools, create the tools, but you can't create like these use cases to really get the value out of them to generate innovation. But when you put the two together, you can.
Tim Cotten <> Scrypted:
Right. And by the way, thank you for that. That's a very kind way to put it. I will say that when I started out in the industry and I learned how to do tricks, right, there were things I wanted to learn how to do in game design that either hadn't been invented yet or were private information in the corporate field, right? And I'll give you an example. When I worked on Ultima Online, because I always love talking about that, it's an amazing game. the they had a system for weather. It was completely empty, but the server would send a packet and say it's raining now or it's snowing now and it might do something, but the world didn't change. And for a game that was built on the idea that it was a simulation and living, that was disappointing to me. So one time I hacked the 2D client so that every tile that was leaf in the forest switched over to the appropriate snow tile and it looked terrible. because I had not understood that the original dev team, because the leaf tiles looked so similar to each other, didn't place them with transitions in a way that makes sense. They were all just random and it looked super jarring. And so I went and created a method to procedurally fix all of them in one pass as the client was rendering it. And it required me to build a table of 512 combinations. of the possible tile sets, right? And I just considered that part of my day. So fast forward that 15 years, and we have all of that kind of knowledge between myself and my co-founders that we want to apply to these kind of generative opportunities. They seem small to me, and you're right, I look at world building as the low-hanging fruit. I chose the way I did it with this demo because it was the easy way to do it. I want to take the NPC side of things and the event and puzzle design side of things. That's where the real next level is. And further, LiveOps is where this can really shine. Where we're suddenly saying, it's not a human having to design an event for 20,000 players, it's an AI that can react to the events, scale it as necessary, and add rewards when more people come in. These are the things that are going to make a difference in game development, right? Not just world building.
Maria Gillies:
Yeah, that's actually one of the things I want to talk about on this episode is how AI can solve the problem of content treadmills, not
Tim Cotten <> Scrypted:
Mmm.
Maria Gillies:
just LiveOps, but
Tim Cotten <> Scrypted:
Yep.
Maria Gillies:
building the expensive world.
Tim Cotten <> Scrypted:
Yeah. Yeah. And that's what that's what I want to talk about too. Auto RPG
Maria Gillies:
Yeah.
Tim Cotten <> Scrypted:
is what I hope you guys look at it and you say, Oh, maybe there is a solution for small teams now. Maybe there is a way that creative people can massively force, multiply their output and keep communities engaged and those building those future community engagement platforms, even in blockchain is incredibly important. I think. this kind of product has a future.
Maria Gillies:
Yeah, but I also want to talk about something else, which is I'm scared. I wear two hats. We're talking
Tim Cotten <> Scrypted:
Mm-hmm.
Maria Gillies:
about this. I'm super excited.
Tim Cotten <> Scrypted:
Yeah.
Maria Gillies:
Knowing that we can solve the content treadmill of LiveOps is just music to my ears. What problem to solve? And then my heart is scared and I want to run away from this conversation because it's uncomfortable.
Tim Cotten <> Scrypted:
And I think there are very uncomfortable things we should all talk about in this conversation. I think all of us will have input on this.
Nico Vereecke:
Let's split up these two topics and let's first maybe touch upon the content treadmill. We haven't heard much from our, I would say probably generally one of the smartest guys, if not the smartest guy on this podcast, Phil, tell us man, what are you thinking in terms of, you know, you've seen out RPG at work. What are your thoughts when it comes to its application to the game development process? Where does it add most value in your opinion? Is it during development? Is it when development is done and you're trying to just generate more content?
Philip Collins:
Yeah, I mean, it's interesting because it's kind of split here between the value that's offered to the game developers themselves and then ultimately the value that that derivatively provides to the players. And so I think the thing that I've always been excited about on the AI side and like, look, I'm a cautious optimist here. I'm not all in. This is going to fix all of our problems. But when I think of AI, I think of efficiency. And so for game devs being able to automate some of the more mechanical aspects of the creation process and hopefully allocate more time towards the more creative aspects of building and monitoring a game over time. And I think that's where my fear that Maria talked about subsides a little bit is I think that hopefully this helps us focus on what matters the most, right? And this allocates more time and more resources to creative and human functionality. And so when we see stories like IBM saying they're going to be able to lay off 7,800 people and check stock prices tanking because chat GPT is, is taking over a lot of the functionality. I think we're going to see a lot of growing pains there. And I think there are a lot of things to be concerned about that are going to need to be figured out at the human level in the impact. But in terms of AI, kind of taking over the world and taking over the industry, I think that ultimately it will kind of settle in a in a place of efficiency and productivity
Tim Cotten <> Scrypted:
Mm-hmm.
Philip Collins:
that enhances rather than takes away from the current practices of developing a game and ultimately creating new content that players love. So, I think that's kind of how I balance those two things.
Maria Gillies:
I actually want to challenge that if I can,
Philip Collins:
Okay? Yeah.
Maria Gillies:
because I hear a lot of companies and investors and people are talking about, oh, AI, and then the words used are efficiency, productivity. But if we go to the past and we look at, for example, RPD, which was you just automate with a basic robot to do certain clicks in a certain way, people were laid off. This had massive consequences. having jobs that just don't make sense anymore because you just
Philip Collins:
Yeah.
Maria Gillies:
create an RPD based system and that's it. And this is just one of the examples we can then look at like documents digitization that is now automated that laid off people. And so anytime we talk about efficiency, productivity in this business world that leads to either layoffs or you don't have to hire as many people complex games can be made with smaller teams, but then, you know, what happens with the smaller teams? What happens if people are no longer going to get hired? And so I think in these discussions, yes, it's not going to be as dramatic as we're all going to be without a job and games are going to be completely made by AI. But we have to be like we have to stop using pretty words to
Philip Collins:
Mm-hmm.
Maria Gillies:
describe like, yes, this is going to change roles. It's going to change. know potentially the number of people getting hired. Anyway, I'm going to be in circles.
Philip Collins:
No,
Tim Cotten <> Scrypted:
disruptive.
Philip Collins:
I think it makes a lot of sense. And I think the way I think about that is there will be short-term pain with technological innovation, right? And skills will have to adapt, and the jobs that people will have in 20 years might change. And I don't know why, but the first thing that came to my mind was, think about how phone networks work today, right? Back in the days, you used to have switch operators that were literally manually transmitting lines, and that job is now absolutely obsolete. And anyone
Devin Becker:
except
Philip Collins:
that...
Devin Becker:
for in the John Wick franchise.
Philip Collins:
Yeah,
Maria Gillies:
No.
Philip Collins:
but anyone that served that function will now have to go find something else to do. But it also created new roles and that takes time, right? And that will cause short-term
Maria Gillies:
Yeah,
Philip Collins:
pain and definitely, definitely don't want to write that off.
Maria Gillies:
and that's fine if you're in your 20s, in your 30s, in your 40s, but ageism is a thing. And so when we're talking about, oh, you can just go and learn a new skill and go into a new role. I mean, yes, okay. But, you know, if you're in a later stage of your life, that's not a click of a switch. And, you know, even now, if we're talking to people who are in their 60s and their 70s. You know, my dad doesn't know how to use an email and he's a CEO of a business. And he's like trying to make do with people who write his emails when he's writing physical letters. This, this is like, you can't just keep up constantly with reinventing your role. So yes, there are consequences.
Philip Collins:
100%. And I think that efficiency does come with consequences. I think the VC and me and Nico probably thinks more of the efficiency than the consequences in certain moments and in certain situations. But yeah, I think you're absolutely right there. And another element of this is maybe how fast does this actually play out where as amazing as a lot of the AI tech we're seeing today is. I think looking back, we'll see it as rather primitive in the end. And so I'm curious to see when the impacts of AI that we're currently thinking about actually come because there's inevitably going to be some lag and I don't know if that's six months, six years or 60 years.
Tim Cotten <> Scrypted:
not that
Philip Collins:
But
Tim Cotten <> Scrypted:
long.
Philip Collins:
yeah, that's a pretty long time. But I think that there's also this adaptability happens. as the technology is also growing. And like it won't necessarily be these overnight changes, but there's still a huge learning curve that absolutely has a human impact to Maria's point.
Tim Cotten <> Scrypted:
I think she has an excellent point. I have two concerns. One, the nature of game design changes. That's one issue. If it becomes more than a helpful tool to give you ideas, if AI progresses past that, suddenly we're talking about no longer just doing, and I'm going to speak in the perspective of metaverse and virtual worlds, so give me some rope. Game design no longer becomes a individualized task of creating quests. Really, I think we understand there's only four or six kinds of quests, and all the flavor writing on top of that is just relational, right? So at that point, what are you doing? Well, you're either being replaced by an AI that's doing the content and making those relationships between different areas of the game, or your game design job has changed to a meta design, right? Where you're designing the ways... that the AI works in the world, and the way that the game interacts with the AI. That's more of the Korean-Manois style, the gamer kind of thing, like, you know, you're moving towards these Moonlight Sculptor style worlds, and I think that's possible. The second, and this is a real concern across the AI spectrum as it disrupts everything, is Agecliffs. You already said it. Both of you were talking about it. But... What happens when all the youth jobs, the junior programming jobs, the junior design jobs are completely obviated by AI, and yet the AI hasn't caught up far enough to do the things that senior developers and the senior designers do? You end up with a group of senior talent without the human element of junior help. That, I think, is also a dangerous idea.
Devin Becker:
Cybersecurity ended up in a position like that to some extent because the background knowledge required a lot of times and experience required for a lot of that really in-depth cybersecurity generally skewed towards older people as younger people didn't really have that depth of knowledge yet. And so it was always interesting to see because you did end up with a problem where you lacked people like people retired, people maybe weren't looking for active jobs anymore or weren't as keen to switch jobs because it takes a lot of... you know, energy to switch jobs and retrain some stuff and things like that. So you did end up kind of in a weird position where like the sort of reverse ageism, like you're talking about.
Tim Cotten <> Scrypted:
Mm-hmm.
Devin Becker:
And I think there is an interesting aspect, you know, you bring up like having worked on Ultima online and stuff like that, right? I think there's a lack of people in the industry with that, like a depth of experience back to older MMOs and virtual worlds talking about metaverses, for example, like that's something I brought up a few times where it's like, you know, we have these conversations with like young people that like don't know the history. And then they're like talking like a little bit out of their ass about stuff that's like, this has been done
Tim Cotten <> Scrypted:
Mm-hmm.
Devin Becker:
20, 30 years ago and we already figured out why that didn't work
Tim Cotten <> Scrypted:
Right.
Devin Becker:
and why it could work better. Like stuff
Tim Cotten <> Scrypted:
Right.
Devin Becker:
like that. But it's interesting because we do, I think we need a mix regardless, right? Like you need the old people with the experience. You need the young people with the energy and new ideas. And like that's why I always like things like apprentice systems and stuff like that, because they kind of helped that sort of thing move forward. And who knows, maybe we have the AI apprenticing people, right? You work as an apprentice for the AI. and it teaches you how to do the stuff it can't do, right? You're like an intern for Skynet, right? I mean, there's lots of possibilities here. And I think if you move out of the reactive space, right? Like, sure, I'll help on the camps, why not, right? But it's, you know, if you move away from being reactive and scared and just being like, well, these are problems, okay, well, let's just solve them. Right? I think that's always the approach I've taken, right? Like I skew a bit older, but I also try and like skate to where the puck's going. So I don't usually have as much of a problem. with being aged out so much, but at the same time, I guess, if I start looking my age, maybe I will, maybe I'll get more bias against many things like that. But it's one of those things where it's like, well, you kind of have to, like that's, the aging is not gonna, well, maybe I'm wrong on that. I was gonna say aging is always gonna be a thing, but maybe not, maybe tech solves that too. But for now, aging's still a thing we have to deal with, regardless, that's like a cyclical thing. So it's like, all right, well, let's just, rather than go, let's shut down AI for six years or whatever, like six months, whatever they're trying to do. It's like, well, let's just figure out the problem. He'll ask AI how to solve the problem. Let's leverage it then to, I don't know, I'm a big fan of that. I'm like, if there's a bomb and the bomb's a problem, we'll use the bomb to do something useful then. I don't
Tim Cotten <> Scrypted:
Yeah.
Devin Becker:
know. That's my take on it is all these things are just problems to solve, not like let's just freak out kind of things. Because every generation has the freak out about the next generation's thing or two generations later thing. And that's just normal. You know, us being in the future of gaming, right? We're used to seeing everyone else freaking out over web three or it was free to play before that or there's always something to freak out about. And I think AI like, like as a, as a game designer myself, like I, I started just being like, well, cool. Well, how can I make this do game design for me? You're talking about like meta design. I'm like, awesome. Like let's do some meta design then I'll give it constraints. I'll tell it exactly what I want. I'll steer it. I'll see what it comes up with. I'll be like, Hey, ID eight for me.
Tim Cotten <> Scrypted:
Exactly.
Devin Becker:
Like if the ideas aren't as valuable anymore, because the AI could do them the cool, I'll do something else. Like. But I think if you embrace that and have fun with it, like you could start to go towards a solution a bit faster than if you go like, oh crap, I'm out of a job. If you know what I mean. I'm not saying
Maria Gillies:
Well,
Devin Becker:
that, you know,
Maria Gillies:
but...
Devin Becker:
that works for everyone, but.
Maria Gillies:
Sure, Devin, you have financial security, you know, you don't have people. Sorry, I'm saying this, I'm assuming, I'm sorry.
Devin Becker:
I
Maria Gillies:
Let's
Devin Becker:
haven't
Maria Gillies:
say
Devin Becker:
always.
Maria Gillies:
that you
Devin Becker:
I've changed careers many
Maria Gillies:
have
Devin Becker:
times.
Maria Gillies:
financial security.
Devin Becker:
Yeah.
Maria Gillies:
Yes, but if you're in a position where your job, so many people are dependent on you, you don't have the luxury to take like two years to learn how to change your job and whatnot. That's like completely
Devin Becker:
Yeah.
Maria Gillies:
different. And that's
Devin Becker:
I
Maria Gillies:
why
Devin Becker:
mean...
Maria Gillies:
we need governments to... have protections for the job market and stuff like that. So it's not as simple as, oh, it's a problem and we're going to solve it and then another problem comes and we solve it. Because not everyone has time or even mind space to solve problems because they have much more pressing problems to solve. Sorry,
Devin Becker:
Yeah,
Maria Gillies:
I'm going
Devin Becker:
I mean,
Maria Gillies:
a bit philosophical here.
Devin Becker:
I'm a little bit biased because like I've been in this situation. I've changed careers many times. I went from commentating to cybersecurity because like commentating was in a young person's game. So I was, I was going to age out of it eventually, right? Because I was like twice the age of most of the commentators already. And so I'm like, I spent the time, my off time, uh, from commentating, studying for the cybersecurity stuff. And then during the cybersecurity stuff, I started like learning the web three stuff and it's like, I'm not everyone. Obviously they're totally people that are totally different. Like you said, like I'm not. the typical example, but I'm saying it's possible. Obviously we should do something to help the people that can't do it, absolutely. We should do what we can, but regulation style, like top down, is not a great way to do it, right? When we just start putting brakes on everything, then that stops the people like myself that can handle it or want to do it that way, and then divides everyone into that kind of weird thing, but there's plenty of ways we can try and help. You know, there's always these job retraining things. There's always those help programs. Like there's plenty of ways to help solve the problem. I just mean like, and I'm not saying let's put our head in the sand either. That's what I mean is like the middle ground where it's like, let's not put our head in the sand and not discuss this topic. Let's discuss this topic. We'll let's discuss it from a problem solving thing instead of a slamming breaks mentality, I guess, and let's try and
Tim Cotten <> Scrypted:
Mm-hmm.
Devin Becker:
help people that need help for sure.
Tim Cotten <> Scrypted:
You
Nico Vereecke:
Thanks for watching!
Tim Cotten <> Scrypted:
know, here's an example. So I'm sure most of you are familiar with Scenario AI, the product that auto-generates game assets. So I have an account with them. And as I build games, I find that it's an excellent ideation platform, and that I can put in my own images, create my own model, and try to get my own result for all sorts of cool little objects and things. But. since the copyright law in the United States is quite clear that AI-generated assets, purely generated assets, are not defensible in copyright, I still pay an artist to go and use those as inspiration to make my own assets. Now, that's the first phase that we're talking about with AI in game development, where it's more of an ideation prototyping object. The issue, I think, that you guys are discussing is that As we move towards the true job replacement automation, it's adapt or perish. And so what does that mean in an example like I give an auto RPG, where in the second phase, it's not just that it built a world already. It's that you can continue telling a story, and it continues modifying the world and chops down trees to build something. And suddenly, you've offloaded a huge amount of human work that normally would have to happen in a game as you're updating it. That is disruptive, I get that. So that's where I think Devon's point about having to shift towards the meta design that we're talking about with having to design ways that the AI is actually doing things rather than the nitpicky bits that we're used to as game developers, right? I think we're going to have to pull back on the brush a little bit and talk more about the palette selection and the paints and the way we do things more than putting pixels on a canvas. That's where I think the difference is going to be in the next year of just AI development
Devin Becker:
And that actually
Tim Cotten <> Scrypted:
in
Devin Becker:
benefits
Tim Cotten <> Scrypted:
games.
Devin Becker:
the older people, right? Because they have the experience to understand steering that boat
Tim Cotten <> Scrypted:
Mm-hmm.
Devin Becker:
if they're willing to do it, right? If they're
Tim Cotten <> Scrypted:
Right.
Devin Becker:
willing to embrace like, hey, maybe I have to learn a little bit of new tricks, but my experience there is actually valuable.
Tim Cotten <> Scrypted:
Mm-hmm.
Devin Becker:
Now, obviously ageism sucks
Tim Cotten <> Scrypted:
Mm-hmm.
Devin Becker:
when it's like people just be like, well, you should retire by now and stuff like that. But if we ignore for a second the just inherent biases people have, those people are
Tim Cotten <> Scrypted:
Mm-hmm.
Devin Becker:
more valuable to do that sort of thing, to do the meta design.
Tim Cotten <> Scrypted:
Yeah.
Devin Becker:
Because if you do meta design, but you understood. design for a long period of time, like you're going to be better at the meta design than a person just coming fresh. And that's like, you know, I see people now like growing up on iPads and they have no idea how a computer works. And I'm like, well, there's value
Tim Cotten <> Scrypted:
Mmm.
Devin Becker:
in knowing how a computer works. And so the younger person is actually at a disadvantage in a lot of ways, but that doesn't always manifest itself. Right. And so like, there definitely will be, I think, a period of like rough stuff for sure. Like where some people just go through ages stuff. Some people go through just like, like you said, not having time to. retrain not having the energy or any kind of financial security to do so. And like, that's going to be hard. Like there's, there's no getting around that. Like that happens with any industrial revolution or a technology upgrade period. Like I'm sure scribes went through a real tough time when the Gutenberg press came out, but it had to happen, right? Like we had to kind of move past that manual labor and there was, you know, there was great things about scribes, you know, the calligraphy and the, you know, there was the meditative aspect. I'm sure there was like lots of benefits that we lost unfortunately, but Maybe we could find other ways to move those benefits somewhere else and take advantage of that rather than just fight
Tim Cotten <> Scrypted:
Yeah.
Devin Becker:
against it. Because you're not going to win
Tim Cotten <> Scrypted:
Yeah.
Devin Becker:
fighting against it. That's just
Tim Cotten <> Scrypted:
Yeah.
Devin Becker:
a losing battle.
Tim Cotten <> Scrypted:
Well,
Devin Becker:
You're not
Tim Cotten <> Scrypted:
I mean,
Devin Becker:
going to.
Tim Cotten <> Scrypted:
as the VCs in the channel have pointed out, the level I built represented an $0.83 investment and 22 minutes of time overall as it executed. That's quite different when I wanted to go benchmark it against some Fiverr RPG maker developers. and I paid them between $25 and $100 to generate levels that pretty much looked about the same. You know, totally different designs, but same general level of quality. Not that someone couldn't do better. A human could do much, much better than what I show you in AutoRPG, but at a much higher cost and a much higher time investment.
Maria Gillies:
Yeah, so.
Nico Vereecke:
Let's, sorry Maria, I just wanted to, you know, I think it's an important discussion, but it's a discussion that's, I would say, not unique to games because this
Tim Cotten <> Scrypted:
Mm-hmm.
Nico Vereecke:
is impacting, you know, every
Maria Gillies:
Yeah.
Nico Vereecke:
single industry in the technology world. And I also don't think, like for me at least, and we can have, you know, we can have a separate conversation around this. It's this is a responsibility of society as a whole more than industries specifically. Like as you said, Maria, like governments will have to figure this stuff out because I'm, I'm Although I get everyone's point that, you know, throughout history, we've seen new technologies overtake jobs and new jobs created. Like I think this is, this time is different, famous last words. Um, but I do think that a bunch of people, like more and more people will not be able to add value to society in a traditional sense pretty soon. Um, that's my opinion. And I think, you know, we'll need solutions for that.
Maria Gillies:
I actually wanted to shift gears into another usage of AI that I think is really important. Okay, so fun facts. In 2017, I actually did my MBA research in the assault analysis of game design. And I decided to reread it because I knew it was mostly about AI. And I actually read like how I thought AI was going to change game design. And I'm really happy because I still believe it. And so one of the opportunities was personalization. And I think that is where AI can play a massive
Tim Cotten <> Scrypted:
Mm.
Maria Gillies:
part because we're in an
Tim Cotten <> Scrypted:
Yeah.
Maria Gillies:
attention economy, post-IDFA, the Google privacy sandbox. We're all talking about,
Tim Cotten <> Scrypted:
Mm-hmm.
Maria Gillies:
oh, you have to personalize your game for different player types and purchasing behaviors. And so if you have an AI that can... When I saw AutoRPG, I was like, oh,
Tim Cotten <> Scrypted:
Mm-hmm.
Maria Gillies:
I see this. Because
Tim Cotten <> Scrypted:
Yeah,
Maria Gillies:
if
Tim Cotten <> Scrypted:
you
Maria Gillies:
you
Tim Cotten <> Scrypted:
get
Maria Gillies:
can
Tim Cotten <> Scrypted:
it.
Maria Gillies:
have the game adapting to what the player is showing preference to do. So let's say it's an exploration game. So you have this open world map and you see that this player just wants to go from place to place, quest to quest. They don't want to explore. Then the AI can start changing the worlds and the level design to, you know, create what the player seems to have the most fun doing. Or if the player is like, oh, I want to explore all the nooks and crannies and all the side quests and I want to know every detail of the lore, the game can adapt to do that. So we're currently talking about personalization of what starter offer is shown, what's the fatui that you go through, but AI can bring actual game personalisation in
Tim Cotten <> Scrypted:
Mm-hmm.
Maria Gillies:
terms of the deep end of game design.
Philip Collins:
Yeah, and I think that's part of something that you actually brought up probably almost a year ago, Maria, and it was an idea that I loved in the Discord, talking about how something that NFTs could do for games is almost creating a choose your own adventure where the way that you play impacts how the rest of the game unfolds. And so creating that at scale where the game is changing and all five of us have different outcomes and all five million people that play also have different outcomes because it's constantly generating new things based on how we've... historically acted. I think that's an interesting way to kind of do what you're talking about at scale. But yeah, that idea stuck with me for some reason, 10, 12 months later.
Maria Gillies:
I forgot about it. I'm glad you remembered.
Tim Cotten <> Scrypted:
Well, I think it's spot on, right? Because it's not just a game dev tool. Auto RPG represents a three week project that can become something that takes user generated content to a level where it's personalizing those experiences for the players. It's new. It's a little scary that there's so much power that can be done. And there have to be careful rails and protections to make sure that things don't go terribly wrong. But it can unlock. so much gameplay that previously would have required 100 people on a dev team and 100 million dollars to do and won't now.
Maria Gillies:
Yeah, and even from a narrative perspective, like before, if you, you know, we talked about Mass Effect, where actually is
Tim Cotten <> Scrypted:
Yeah.
Maria Gillies:
the narrative didn't really change the end game. But if we're talking about you can personalize the
Tim Cotten <> Scrypted:
Mm-hmm.
Maria Gillies:
story of this arch enemy that
Tim Cotten <> Scrypted:
Mm-hmm.
Maria Gillies:
touches like what you find personally
Tim Cotten <> Scrypted:
Yeah.
Maria Gillies:
to be enraging. I don't know if they killed a red panda, I'd be furious and I wanted to defeat them and like, yeah, narrative personalization as well of the storytelling.
Tim Cotten <> Scrypted:
You know, when I talk to people about how a game can make an emotional connection, right? The number one example I use is Earthbound or Mother, the Japanese version. And if anyone hasn't played Mother or Earthbound US in this channel, I urge you to play it. Because as you play that game, which doesn't have AI, doesn't have any of that stuff, but with its story, the way it forges a connection, if you could imagine that applied to what Maria is talking about, you have a future of gaming. That is truly... Not just not an addiction. It is something that makes you feel involved. It's where game transcends into art. That's new. I would love to see that. Be part of it.
Devin Becker:
Yeah, smart reactivity, I think, is really a big unlock. Because it's like, you know, you take TTRPGs, like tabletop RPGs, and that reactivity is like an essential element to it, like being able to adapt to what the players want to do, being able to generate new content.
Tim Cotten <> Scrypted:
Yeah.
Devin Becker:
And I think you can extend that much beyond narrative stuff. It can get into gameplay stuff. It can get
Tim Cotten <> Scrypted:
All
Devin Becker:
into
Tim Cotten <> Scrypted:
right,
Devin Becker:
all kinds of things, where
Tim Cotten <> Scrypted:
absolutely.
Devin Becker:
being able to react smartly in a way that at least acts like it comprehends is something much,
Tim Cotten <> Scrypted:
Mm-hmm.
Devin Becker:
much like higher than this procedural algorithmic style of responses that we've had to deal with so far. The hard part actually is probably gonna be more controlling it and corralling it and figuring out like how to like shape it in a way that is doing what you want it to do for players
Tim Cotten <> Scrypted:
Correct.
Devin Becker:
more than just like allowing it to do stuff because it's just gonna do crazy stuff if you just open
Tim Cotten <> Scrypted:
Oh
Devin Becker:
up
Tim Cotten <> Scrypted:
yeah.
Devin Becker:
the box, right? Like it's very Pandora's
Tim Cotten <> Scrypted:
Well,
Devin Becker:
boxy like that, you know?
Tim Cotten <> Scrypted:
yeah, you're
Devin Becker:
And
Tim Cotten <> Scrypted:
describing.
Devin Becker:
so like...
Tim Cotten <> Scrypted:
Go ahead. I'm sorry, I didn't mean to interrupt you.
Devin Becker:
Yeah, it's a tricky thing and I'm sure you've dealt with that a bit and I was gonna ask about that earlier, like how you deal with like, you start getting it to generate all these things for the player, but you've got to like kind of shape a little bit, right? You're trying to play this like light hand, like touch of like, well, we've got to steer this but I can't like shove it in a direction, right? And it's like, that's something game designers have always kind of aspired to in a way, right? Like you want to have players have the experiences you want them to have, but you also like want them to feel freedom and autonomy. So you're always trying to find that balance between how much do I steer my players without them knowing it in the right direction. And it's like now that can actually be part of the system itself rather than just rules design or like hidden tricks and things like that. It can actually be like a smarter response to that. And I think that's a big value unlock, but that's also going to take time to figure out how to actually do that steering.
Tim Cotten <> Scrypted:
Yeah. And that's, I think, what you're talking about is probably one of the most important things to the AI, the autonomous AI conversation in game design. And what I mean by that is procedural generation has been with us for a very long time. Procedural generation has been used to maximum effect in great games like Dwarf Fortress, which give excellent emergent behavior situations where things happen with elephants and miasmas that you don't expect, right? And you end up with communities creating huge engagement platforms, sharing save files so that they can go the keep moving on the legend like boat murdering. Boat murder. That was an amazing story if you've never heard of it in Dwarf Fortress. But Autonomous AI lets you leave the mathematics and the combinatorics behind and have some sort of recursive intent. where you can have a goal, even if the goal is self-generated, and find logical paths to reach that goal. And if you're worried about going down the rabbit hole too much, you can limit the recursion. You can force it to come back up. That's what my hardness has to do. I have to limit the recursive depth right off the bat, because otherwise it would spin out happily for hours, digging down rabbit holes that aren't useful to the end product. And so I think. As we explore different expressions, different ways of doing output, different ways of like creating things with it that are not just world building or putting 3D skins on avatars and hooking them up to a GPT system, right? As we move towards what you're talking about where the puzzles and the inner game mechanics start getting designed, that's where I'm taking auto RPG. I want it to be able to do more than just show you a game. I want you to be able to play something and react to it.
Devin Becker:
Maybe in
Nico Vereecke:
Maybe
Devin Becker:
the future
Nico Vereecke:
final,
Devin Becker:
we start talking about...
Nico Vereecke:
final
Devin Becker:
Go
Nico Vereecke:
question,
Devin Becker:
ahead. Yeah.
Nico Vereecke:
Tim. What is, like... What will be the main problem that your tools will be solving? Should we think about this in terms of rapidly iterating on game designs? Or should we think about this as solving the content treadmill problem?
Tim Cotten <> Scrypted:
I have two goals right now. The first is the easy low-hanging fruit, which is content treadmill, which is just quick game dev prototyping. That's the one where you're replacing a junior salary, and you're getting three people out of it instead of one. So instead of paying $60,000, $70,000 a year, a studio can pay that for an AI over a monthly subscription, but they get four developers out of it. That's a force multiplier. The second is the content treadmill for live ops. where the game can continue to modify itself under your rules, like Devin's been talking about, right? Under the things with the idea that Maria had. That stuff is, I suppose we call it the holy grail, right? It's like the holy grail of like MMO design, for instance, is a game that can design itself and execute. But with autonomous agents, that's actually becoming very possible, right? that's not that far away anymore, and I see a clear path for what my tech can do to get there, I think that's extremely valuable to blockchain, to Web3, and to MMOs and virtual worlds for the metaverse in general.
Maria Gillies:
I actually think there's another holy grail.
Tim Cotten <> Scrypted:
I love holy grills. Tos.
Maria Gillies:
We're talking AR, VR, and
Tim Cotten <> Scrypted:
Mm-hmm.
Maria Gillies:
we look at what Niantic is doing about, we have location-based games. I think AI can just create a revolution of like AR or superimposed gaming with your environment.
Tim Cotten <> Scrypted:
Mm. Uh-oh.
Maria Gillies:
That might be another innovation where you have like an AI agent who's playing with you, so the character is an AI agent, and they're discovering the world that you're discovering, so they only know what you know. And as you're going through the streets... that you're playing a game with it could be like a murder mystery. Uh, we can expand past gaming where it could be, you know, it's integrated into fitness apps that creates a story and interacts with your environments to motivate that. And yeah, because it's like, you can, you can sort of try to personalize a controlled world and narrative within a game that you've created. Like being able to create a game wherever you are, where you live, your environment, your surroundings. I find that so exciting.
Nico Vereecke:
Maria, you know you can make these posts on LinkedIn and Twitter that start with the years 2030 and you describe ridiculous future. People really seem to dig it. So I highly recommend giving that a shot with what you just
Tim Cotten <> Scrypted:
I
Nico Vereecke:
described. It sounds
Devin Becker:
Goes
Tim Cotten <> Scrypted:
love
Nico Vereecke:
great.
Devin Becker:
over
Tim Cotten <> Scrypted:
it.
Devin Becker:
very well.
Nico Vereecke:
Yeah.
Devin Becker:
You'll go viral fast.
Nico Vereecke:
Yeah.
Maria Gillies:
I'm a shy, I'm a shy person. You can, you can write it, Nika.
Tim Cotten <> Scrypted:
Hahaha
Nico Vereecke:
Good. All right. With that, I'm going to have to cut this. We're nearing the hour mark and I feel like we keep chatting and so perhaps we need to do a part two. Tim, you know, at the speed you're advancing here, it seems like, you know, in a few weeks we can do a follow-up and you'll be further
Tim Cotten <> Scrypted:
Yeah,
Nico Vereecke:
and maybe you're
Tim Cotten <> Scrypted:
well you'll see the
Nico Vereecke:
you've
Tim Cotten <> Scrypted:
3D
Nico Vereecke:
built.
Tim Cotten <> Scrypted:
version.
Nico Vereecke:
Exactly. So, you know, that's going to be excited. So with that, Tim and Maria, thank you guys so much for joining. great guests as expected to be fair. Maria is probably the greatest web like gaming podcast host out there. So
Tim Cotten <> Scrypted:
That's right.
Nico Vereecke:
what was, what was to be expected. And then Tim has recently gone viral. So we had to have him. Come on.
Tim Cotten <> Scrypted:
Thanks for having me on guys.
Nico Vereecke:
It was,
Maria Gillies:
Gosh.
Nico Vereecke:
um, it was great. Um,
Maria Gillies:
Thank
Devin Becker:
Definitely.
Nico Vereecke:
Devin,
Maria Gillies:
you.
Nico Vereecke:
Devin Phil, thank you as usual for joining and sharing your thoughts and questions and most of all, listener, thank you. If you made it till here, really appreciate it. Um, if you want to ask Tim. Maria, Devin, Phil or me some questions. Join the discord if you're not already there. And if you did enjoy it, please let us know by liking this video, wherever you're listening or watching. And with that, we are out and we look forward to speaking to you in the next episode. Ciao.
Maria Gillies:
Bye!