Subscribe
Share
Share
Embed
Join host Jamin Warren on conversations with someone of the most unique and experimental artists, designers, and thinkers in the worlds of games, play and culture
Jamin Warren founded Killscreen and has produced events such as the Versions conference for VR arts and creativity, in partnership with NEW INC. Warren also programmed the first Tribeca Games Festival, the groundbreaking Arcade at the Museum of Modern Art, and the Kill Screen Festival, which Mashable called "the TED of videogames." Additionally, he has served as an advisor for the Museum of Modern Art's design department, acted as cluster chair for the Gaming category for the Webbys, and hosted Game/Show for PBS Digital Studios.
This is a verbatim transcript of the video:
Interviewer: What does it mean to really listen to a game? Vadim Nickel is a researcher and game developer at Concordia University who studies exactly that question. His recent survey of sound-first games, titles where music and sound drive the action rather than just accompany it, turns up only 43 examples across four decades of game history. That scarcity is itself the story. We talk about why the tools to make these games have only recently caught up to the ambition, what film sound theory can teach us about how players hear, and why the most interesting territory in game audio might not come from games at all, but from soundwalks, acoustic ecology, and the experimental music practices of artists like R. Murray Schafer and Pauline Oliveros. Hope you enjoy it.
All right, cool. Thank you so much for making time, I really appreciate you chatting with me today.
Vadim Nickel: No, thanks for having me.
Interviewer: Yeah, of course. What drew you to like music and games as a subject matter? Like how did you make your way in there?
Vadim Nickel: I was—I got into making games pretty early on, I was like in my early teens. There's this software called RPG Maker and that was making some games with that and eventually I just—I always had like music and melodies in my head and I wanted to manifest them and put them in the games that I was making. And that's how I got into, yeah, music production. I also had some piano lessons as a kid, which I was not a big fan of, but music production was really fun for me and to the point that I got so deep into it that I abandoned making games for some years. But yeah, eventually I got back into game development and yeah, ever since I've been combining the two: music and games.
Interviewer: Was there like a particular moment for you that's like—oh, this is the aspect of games, because there are so many ways into it: architecture, design, narrative, like music as we'll talk about is one of these areas that maybe doesn't get as much love as it obviously should. Was there a game moment for you that's like—oh, there’s something really interesting happening here with games?
Vadim Nickel: With games specifically? Yeah. I definitely had a moment like that. I think I played a game—I was playing a game called Fract OSC and yeah, it’s—I’m generally a big adventure games fan and games were a big influence for me, and so that was basically that, but it was also sound and music because you explore the synthesizer world and all your actions in the game, they’re somehow reflected in the sounds that you can hear. Yeah, that really fascinated me to the point that got me thinking and that made me want to actually make my own games again.
Interviewer: Yeah, that's really interesting. I think for me, Super Mario World many years ago, I remember playing that game and being like really enamored with the sound and I know that—many years later I interviewed Koji Kondo and I did not know that there's an extra when you get on Yoshi, there's an extra little beat of syncopation and it's this like marker for you that you are—like a little audio marker for you to know that like when you are—just to engage with the world in that particular way. Even the sound, the sound like really pulls—pulls me through a lot of experiences. How did you and Gabriel come to work on this particular project? What was like the origin story there?
Vadim Nickel: Well, what was the origin story? I got into Concordia University in Montreal and my research topic is using games to affect emotion and through that also to affect behavior. The idea is to help players make progress in a game but in a way that they might not actually necessarily realize. And so my thought was to use music for that. And I explored these interactions with music and sound and Gabriel at the time was not yet my supervisor, but then my—one of my other supervisors said, hey, there's this music person who's also dealing with things like AI and like interactive media and all of that, and we got talking and it was a good match.
Interviewer: Yeah, that's really cool. Your advisor was thinking about games across a variety of media is—is helpful in some way.
Vadim Nickel: Yeah, that's true because my other supervisors, they're more rooted in more specifically music. Yeah, so they might not be very familiar with maybe the games aspect and since he works more at the intersection of interactivity—he’s not necessarily computation arts in general, but it still makes sense for us to work together because interactivity is what—what we have in common in that sense.
Interviewer: Yeah, that's very cool. Well, jumping into the paper more specifically, the paper documents, I believe 43, is that right, 43 titles, is that right in terms of the total number?
Vadim Nickel: I think, yeah. Yeah.
Interviewer: All things considered given the long history of games, that seems like a pretty thin slice and I was curious what you think explains that scarcity? Is it like a design problem, is it a market problem or is it like something more fundamental about what it is that we're looking for when we play games?
Vadim Nickel: So I think that there's actually many things that figure into that. First of all, as I mentioned, the number 43 games, it's more that—I mean, this is also maybe more of a curated selection. For example, what reduces this number is things like there are many sound-led games that are essentially the same in terms of gameplay. We have five, six, seven Guitar Hero games and we counted only one of those. But when it comes to actually finding these games, that's one issue. It's actually quite tricky because there seems to be not that big of an awareness of this type of mechanic where the music and the sound is the leading element beyond music games. So if you actually want to find a game, like I talked about before, that counts as a first-person 3D adventure game, you might not find it under the music games category, right, and you don't find it under the rhythm games category. There's another good example, it's called Unheard, and it's this game where there's a crime that takes place and you basically try to trace back what happened by listening to an audio recording that shows what transpired there. So you listen to conversations, you listen to maybe sounds that happen, but again, if you want to find this game, there's no category. If you go on Steam, yeah, there's a category for rhythm games, but there is no category for sound-focused games. There's not even a common term that people would use to—to address that. I think if even if you were looking for it, perhaps you wouldn't necessarily even know where to start.
Interviewer: So it sounds like it's a combination. Is there anything that you feel like is preventing this avenue for games to—using music in that way if you're just like a designer that wants to express feeling to players?
Vadim Nickel: Oh, there definitely are, as a game developer myself, I've experienced that and so there are technical barriers, for example. Whenever you want to implement sound in the way where timing is important, the milliseconds engines for the longest time did not have anything built in to develop these interactions. And to give you an example, I've been a Unity developer for like a decade and what ended up for me being the reason to switch over to Unreal is that 2022 or 23 they released the system called MetaSounds. And this was the first time that you had a system that allows you to develop anything sound-related integrated with the engine. Okay. And now with that, it's very easy actually to connect game actions to the sound engine. And if you look at something like Fract OSC, the way the sound was implemented there was using a system called Pure Data.
Interviewer: Okay.
Vadim Nickel: Which is a digital signal processing system, it’s been around for ages. It allows you to create your own synthesizers and musical instruments and all of that. And what they did is they essentially offloaded all of the sound of the game to this system. So even in terms of what's called spatialization, so when you turn around and the sounds come from the direction that you can see, even that, they did all of that with—with Pure Data. But the problem is when you use external systems like that and you want to deploy on different devices, then you'd have to deal with that, because I tried using Pure Data and there's a lot of technical stuff you'd have to do when you want to integrate with a—with a game engine.
Interviewer: Yeah.
Vadim Nickel: Yeah, so if you look at Unreal, that’s not the case, although I heard of some other frameworks and technologies that might be now easier to implement. But even still, if we look at the games landscape, it's not like there has been an extreme influx of sound-led games in the past five years or so. Yeah, if anything, I think the Golden Era of music games, like in the mid-2000s, when we had all the plastic guitars with the Guitar Hero, Rock Band.
Interviewer: Yeah. Do you feel like that moment—obviously there's an exhaustion of the popularity of those types of games in the late 2000s. Do you feel like that correction maybe set music games back in some ways? Did it maybe close off design possibilities that might have otherwise been pursued, this perception that oh, you’re making a game, even if you’re doing something entirely different, some of that baggage from that era maybe carries over?
Vadim Nickel: Um, I'm not sure because it seems like these games are not even necessarily perceived in that context.
Interviewer: Okay. I want to talk—as I do want to talk about peripherals. How do you think about the relationship between the physical form—?
Vadim Nickel: I think that the relationship between movement and sound in games is perceived very intuitively. If we think about, you know, listening to music and how we sometimes just automatically start to nod our head with the beat. They know how to move an electronic music party, they know how to move. So there is some quality in the music that moves us.
Interviewer: Yeah, the point about visceral reaction to the music that is really different. Music by definition just encourages engagement. Do you feel like the periphery—the peripherals side—?
Vadim Nickel: I think peripherals suggest a certain way of movement. If we think about the plastic guitars, um, there's a certain way to hold it, and that influences how I move. But if we look at maybe more recent examples, something like Beat Saber and other VR rhythm games, we have a lot more freedom, right? We have the two sticks that we hold and that gives us actually quite a lot of freedom. I feel like in that case, it's not just the peripheral itself and of course something like the Dance Dance Revolution also has, you know, very specific movements that it encourages, but it's not just the peripheral itself, it's also the gameplay. Because when you play something like Beat Saber, you have a certain type of movement and if you played a boxing game maybe you would move your hands differently. So it was both the peripherals but also the gameplay that you're dealing with.
Interviewer: Cool. All right, that's it. If you liked this conversation, would like to keep—continuing to listen to it, you can head on over to our members podcast, you can sign up for it at killscreen.com. If it piques your fancy, please check out our website, I’m always publishing new things, follow us on Instagram, like this episode on your favorite podcast app, you know, all the normal stuff. All right, see you soon.