[00:00:00] ​

[00:01:00] Jacob Haimes (Aside): This interview was recorded on May 14th, 2026, and asides were recorded on June 10th, 2026

[00:01:08] Jacob Haimes (Aside): In the current landscape of AI safety, most of the ways people make AI safe is done after the models are trained. Guardrails, red teaming, refusal training, et cetera. While these are all well and good, I've always been a fan of interventions that are baked into the creation of the systems in the first place, which are also known as pre-training time safety methods, and they can provide much stronger statements on the safety of the created systems than their post-training counterparts.

[00:01:37] Jacob Haimes (Aside): I met today's guest at a conference earlier this year, and we quickly got to talking about these kinds of methods because it turns out we both think they're particularly promising. At that time, he had been working on a paper in his role as an alignment researcher with AE Studio in collaboration with some researchers from Anthropic on gradient routing mixture of experts or GRMoE.

[00:01:59] Jacob Haimes (Aside): Back then, the research and paper were still being finalized, but fast-forward to today and Anthropic has just announced the line of work on their own website and the corresponding paper, which he is first author on, has received a spotlight at ICML 2026. For context, ICML is one of the top-tier publication venues for machine learning work, and for 2026, only 2.2% of the submissions were designated as spotlights.

[00:02:27] Jacob Haimes (Aside): One thing I brought up during that first conversation was that purely technical solutions can't be everything. For example, the GRMoE method he presents enables frontier model developers to serve different users, effectively different models without needing to retrain each one from scratch, functionally enabling access control for specific capabilities.

[00:02:52] Jacob Haimes (Aside): But who gets to decide who gets access to the best models? In addition to digging into these questions, we also discuss China, the public mood on AI, and I even get his advice for breaking into the AI safety space when you don't have the traditional background. Before getting into it, I do want to mention that as part of my effort to make this whole podcasting thing a little bit more sustainable, I have created extended versions of the episode, and this one is particularly suited to the more technically inclined listeners.

[00:03:24] Jacob Haimes (Aside): If you'd like, you can head over to the Kairos FM Patreon to check it out. With that, I am excited to introduce Ethan Rowland.

[00:03:32] Ethan Roland: So my name is Ethan Roland. I am a senior alignment researcher at AE Studio, where I focus on pre-training time methods to improve alignment-relevant capabilities in frontier models 

[00:03:48] Jacob Haimes: Awesome. And in one sentence, what are pre-training time methods for AI safety, and why are they important?

[00:03:55] Ethan Roland: Yeah. So pre-training time methods are, I would say, defined in contrast to the traditional way that people think about modifying models to be more aligned, which is typically purely in a post hoc setting where you're doing like, you know, post-training. 

[00:04:13] Jacob Haimes: Mm-hmm.

[00:04:13] Ethan Roland: and so the idea is that there are more powerful and more reliable and, uh, higher levels of theoretical guarantees alignment techniques that are implemented on during training, particularly in the pre-training phase. 

[00:04:31] Jacob Haimes: Mm-hmm.

[00:04:31] Ethan Roland: this can look like just-- Oh, sorry, that was like definitely more than one sentence. But yeah, this is, this is the general fam- general family of types of things that you would, you would do, and it's good because it, it works better generally. 

[00:04:44] Jacob Haimes: Awesome. So let's start then with how you got to where you are now. You're at AE Studio working on this line of research in particular, uh, but that's not where you started. So tell us about how you got here, what sort of events led to you being in this position, and when sort of AI safety was like, "Oh, that's something I, I wanna work on, I wanna, I wanna be dedicated to."

[00:05:10] Ethan Roland: Yeah, definitely. So it's always been something that's been on my mind. I would say that early as 2016 or so, I was definitely a, a Less Wrong lurker, and reading the similar blogs and tomes in the, in the field. but I didn't really consider it as something that was worthy or not even worthy, but just like something that would be possible for me to even do as a career until relatively recently, maybe 

[00:05:39] Jacob Haimes: Mm-hmm.

[00:05:39] Ethan Roland: or three years ago. Um, I was pretty involved, I would say, starting three-ish years ago with the larger like effective altruism community and, and led some of the works in that out of the, the local, uh, Atlanta community where I'm based.

[00:05:54] Jacob Haimes: Um, and then also writing,

[00:05:57] Ethan Roland: a variety of different industries, data 

[00:05:59] Jacob Haimes: also doing a lot of, you

[00:05:59] Ethan Roland: corporate 

[00:06:00] Jacob Haimes: game. And then, and then also doing a lot of like testing and stuff like

[00:06:05] Ethan Roland: and being good buddies with lots of EA folks led

[00:06:10] Jacob Haimes: that.

[00:06:15] Ethan Roland: things, uh, for a real career, and you don't have to just purely look and think about, you know, making money and putting, putting food on the table, and you can do both at the same time and also do things that seem to be good for the world. And, uh, I went out to a EA Global conference, I guess it would've been maybe like a year and a half ago, 

[00:06:33] Jacob Haimes: Mm-hmm.

[00:06:34] Ethan Roland: and met some folks from a studio in the Bay, hit it off. They liked me, I liked them. Um, joined shortly thereafter, and the rest is history.


[00:06:44] Inside AE Studio
---

[00:06:44] Jacob Haimes: Gotcha. And what is working at AE Studio like? Um, like what-- maybe what does AE Studio do, and, uh, how do you fit into that?

[00:06:58] Ethan Roland: Yeah. So they're a little bit unique in the sense that rather than being a lab, they really do two somewhat separate things. Um, on one side, they do more traditional data science, corporate consulting. 

[00:07:15] Jacob Haimes: Mm-hmm.

[00:07:15] Ethan Roland: then on the other side, they're doing really just pure technical alignment research. And they have a kind of unique business model, uh, in the sense that a lot of the for-profit work, uh, pays for a significant portion of the non-for-profit, uh, more kind of like pro-social alignment, uh, funded work. I think this is really interesting, uh, in, in one sense because, you know, speaking as, as myself and not for the company, I, I think it gives them a lot more flexibility to the types of things that they do and a lot more kind of freedom as to the particular research agendas they want to pursue. And I've definitely seen that in terms of just the, you know, intellectual freedom that researchers often have at AE. 

[00:08:01] Jacob Haimes: Mm-hmm.

[00:08:02] Ethan Roland: for myself, I am working purely on the alignment, uh, wing, which is myself and, and quite a few other colleagues, uh, on that team. And, um, we work on a huge variety of different things. Uh, myself, I'm working on pre-training time methods for improving safety relevant characteristics of AI models. But I have other colleagues that are working on things from interpretability, things from alternative model architectures, things from, you know, game theoretical, uh, type analyses of model behavior.

[00:08:34] Ethan Roland: So there, there's a really big variety of, uh, research, would say, questions or agendas that are being pursued, uh, within AE as a, as a whole. 

[00:08:43] Jacob Haimes: Gotcha. Okay. And so does that mean that you are mostly, I guess like, yeah, working on... It's almost like a collaborative of independent researchers more than, uh, all working on one sort of agenda together? Or how does that sort of play out in practice?

[00:09:04] Ethan Roland: Yeah. I would say it's a collaboration of lots of different research agendas, but I wouldn't necessarily characterize the employees of AE as independent researchers. I'd more so think of them as partnering with other external researchers, uh, whether those be like known or seasoned, uh, alignment researchers and, and other labs or like academics outside of alignment writ large. I think the way that we typically see ourselves as researchers at AE within the alignment team, we see ourselves more so as like accelerators of external collaborators. And so in that way it's a bit unique where, you know, if you're doing a client-facing project for XYZ Corporation, uh, you would be having some stakeholder, they would say, "I want this thing built," and then you would have a bunch of engineers go and build the thing. 

[00:10:06] Jacob Haimes: Mm-hmm.

[00:10:08] Ethan Roland: to lead to that, and somewhat uniquely with AE's alignment research division, the way that we typically set up those projects is we'll find someone that is, who we might call like an external vision-visionary, who has lots of kind of bona fide, uh, demonstrated competence in alignment. And we'll come to them and say, "Hey, we really like the type of research that you're doing. We wanna give you tons of resources. We wanna pair you with a larger research team. We want to give you lots of compute, et cetera, and let's work together and have a big collaboration." So, so typically speaking, it's, it's more formatted like that, and that's at least the case for, for my own particular project. 

[00:10:45] Jacob Haimes: Gotcha. Okay. And then again, just like t-trying to sort of structure how this works, um, because AE Studio isn't a traditional home for, for this kind of research, right? You said, as you said, the, uh, sort of product slash, uh, customer-facing side, um, i-is distinct, uh, ish. Um, but I guess I'm curious about how does, how does the...

[00:11:17] Jacob Haimes: Or rather, do the findings and, uh, maybe connections, like, what are the more standard, like, business, uh, benefits, uh, or, or company benefits of having this alignment research, uh, division?

[00:11:36] Ethan Roland: Yeah. I mean, and I don't wanna characterize it as totally distinct because, like, certainly there are overlaps between the two, uh, and increasingly so over time. I'd say that there's lots of different benefits. So first off, like the type of work that's done commercially, I actually suspect increasingly just from like corporate clients, there is gonna be increasingly large appetite for alignment relevant things.

[00:12:03] Ethan Roland: So for instance, like we're doing kind of frontier alignment research, but at the same time, if we can have kind of commercial clients that want us to do alignment relevant things for increasing the, the like or just like reliability of their commercial products, that would be an awesome thing too.

[00:12:20] Ethan Roland: And so I think that's like the competencies that are built in doing research are not mutually exclusive with the competencies that are useful for doing more commercial facing products. 

[00:12:30] Jacob Haimes: Mm-hmm. 

[00:12:31] Ethan Roland: and then I think at the same time, there are particular distinct advantages for the way in which we do research in the sense that like when you're doing client facing consulting, um, you have like very strict deadlines.

[00:12:46] Ethan Roland: You have very strict like ways in which you deliver things, ways in which you like measure and metric success. I think one failure mode that you often see in traditional academia that without that forcing function of dollars and cents with a particular corporate client, easy to, I think, continue doing research in a way that's not necessarily maximizing the amount of impact you have per unit of time. And I think taking this more corporate client consulting philosophy and applying it in the context of doing frontier research makes us a lot faster and more efficient than I think we may otherwise be if we didn't have that, that approach to alignment research. 

[00:13:38] Jacob Haimes: Gotcha.

[00:13:40] Ethan Roland: In a way, it's kind of analogous to, oh, I forget the term.

[00:13:43] Ethan Roland: I think it's called export discipline. There's a, there's a, um, an economics term for this, and I read it in some book some while ago. But basically, the idea is that, uh, in developing economies, when they're trying to, uh, ensure that their internal industries are, are really effective, uh, what they'll do is they'll say, "Hey, you know, we will subsidize you, uh, for making steel or making boats or whatever, but under the condition that you are able to successfully export and sell on international markets X percentage of your product." And what this does is it says like, "Yeah, we'll subsidize internal industries, but we will also, like, force these to have some, like, contact with reality." And I think to make a kind of an a- analog there, I think that in some sense, us doing alignment research while also doing kind of more commercial-facing, development is almost an analog of export discipline in a sense that, like, we as an organization get a lot of contact with reality by delivering real corporate products that people will pay for. 

[00:14:49] Jacob Haimes: Mm-hmm.

[00:14:50] Ethan Roland: I think that type of discipline into a good culture that produces high effective research at the same time 

[00:14:59] Jacob Haimes: Mm, okay. I mean, it's also convenient to some extent, right? Like, because you get to say, "Oh, like we need more, more money, uh, funding, uh, to do like the safety research," but then that work can also be, like to some extent, sort of leveraged in the commercial side, right?

[00:15:24] Ethan Roland: Yeah. Yeah, definitely. So these two things are, are definitely kind of together, I would say


[00:15:31] China & the Alignment vs. Controllability Framing
---

[00:15:31] Jacob Haimes: okay, cool. And then another thing that I just like-- when I was reviewing your, uh, you know, public information, I saw that you, you studied Chinese and spent some time in, uh, Shanghai, if I am correct. And I just-- What I'm curious about, about that aspect is, um, how has that shaped your perspective and thinking when it comes to AI development and AI safety, or has it?

[00:16:00] Ethan Roland: I think maybe a bit. You know, I think I've always been really interested in Chinese culture and, and, you know, society more generally. Um, probably one of the motivations for, for why I studied the language for so long. But yeah, I'd say that I have great respect for Chinese researchers, and I think that perhaps compared to, you know, the average person that hasn't, uh, engaged with like Chinese culture as much, that I maybe am like more willing to, to like believe in the competence and believe in the, uh, you know, like capabilities of, of Chinese labs. Uh, and so I would say that like, yeah, I think oftentimes, uh, the like endogenous research abilities of, of, of Chinese researchers are many times like underestimated. 

[00:16:52] Jacob Haimes: Mm.

[00:16:53] Ethan Roland: and yeah, maybe that, maybe that's like an influencer on, on me is like, yeah, I think they're, they're really legit. They do a great work. They don't do very much, uh, alignment research in general. Uh, but I think that's more of just like a kind of, uh, what's popular versus not popular. But in terms 

[00:17:07] Jacob Haimes: yeah, 

[00:17:07] Ethan Roland: pure, pure research, uh, yeah, they, they have their stuff together. 

[00:17:11] Jacob Haimes: And there is also an argument that they actually do a good bit of alignment research. It's just not sort of alignment flavored, um, because, you know, they're, they're working on actually problems similar to, um, like the data filtering problem, right? Of, uh, they do a lot of work on unlearning, if I remember correctly.

[00:17:31] Jacob Haimes: And there's, um... So, like, there are, there are things that are relevant. They're just not sort of marketed in that way maybe.

[00:17:40] Ethan Roland: Yeah. Yeah, I think that's true. I mean, it's perhaps better thought of as like techniques for AI controllability, and I think that's usually the framing in, in China or Chinese public- publications that I've read. 

[00:17:51] Jacob Haimes: Mm-hmm 

[00:17:51] Ethan Roland: Uh, which when you think about like AI controllability techniques versus like AI alignment techniques, for many things these are very much the same thing, just like a, a marketing decision as to how you talk about it. 

[00:18:07] Jacob Haimes: Yeah.

[00:18:08] Ethan Roland: that's certainly not always the case, but I think for a very large percentage of research it's, you could call them controllability, you could control it, alignment or safety. It's just like a decision of what word you use 

[00:18:18] Jacob Haimes: Yeah. All right. So I'm, I'm really excited to get into the technical stuff. Uh, so to make sure that I've got it correct, it's called gradient routing, correct?

[00:18:28] Ethan Roland: That is correct?


[00:18:28] Data Filtering & Gradient Routing (Aside)
---

[00:18:29] Jacob Haimes (Aside): As I just mentioned, this does get a bit technical, but it's quite intuitive at a high level. Before we get into gradient routing, it helps a lot to talk about one of my personal favorite pre-training time safety methods, data filtering. A recent paper that really explored this space was called Deep Ignorance, which isn't important other than that I mention that name a bit later.

[00:18:52] Jacob Haimes (Aside): Anyways, to do this, you first need to have all the data which you would normally use to train your AI model and label it as either relevant to the kind of information you want to remove or not relevant to the kind of information you want to remove. The typical real-world use case is something along the lines of, I want to prevent this model from being able to provide any coherent advice on how to make drugs.

[00:19:15] Jacob Haimes (Aside): Then you simply don't train the model on the data that's flagged as related to, for example, making drugs, which results in a model which can't generate reliable text when asked questions about making drugs. Pretty straightforward. Gradient routing extends this idea, and instead of not training on the making drugs data, you set aside a specific known portion of the neural network which only learns from the making drugs data, and the rest of the network is prevented from learning from the making drugs data.

[00:19:52] Jacob Haimes (Aside): Then, once you've completed a single training run, you effectively have two variants of the same model: the complete one and the one that doesn't know anything about making drugs. This has a bunch of potential use cases and implications, which we're going to get into. One reasonable question to ask is, wait, but what's the difference between these two approaches?

[00:20:15] Jacob Haimes (Aside): And also, what's the point? If both data filtering and gradient routing require the same costly labeling of training data, then why go through this extra step?

[00:20:26] Ethan Roland: Yeah, I, I would say that gradient routing is really intimately connected to data filtering, and you would expect that the end result of a model is trained via gradient routing will oftentimes be very similar to a result of a model trained via, data filtering. And the reason that you would use gradient routing rather than data filtering, is for a couple different reasons.

[00:20:54] Ethan Roland: So one is flexibility the sense that data filtering, if you're just filtering the data but you're otherwise still training a normal model, um, means that these capabilities just never get produced, ever. In, in some sense, this can be good for things that you just, like, wouldn't ever have any use for.

[00:21:13] Ethan Roland: For instance, like, you know, certain forms of, like, bigotry or, or, or prejudice. in other senses, uh, particularly for, for capabilities that you might consider to be dual use, for instance, like knowledge of advanced virology, in which case you, like, would want vaccine researchers to have uplifts from advanced models. Um, would seem to totally get rid of the possibility being accelerated via advanced AI for these researchers that are using these dual use capabilities for good. so you get in gradient routing is that, yeah, you can delete the parameters associated with, for instance, advanced virology, but you also can add them back. And so what this means is that you delete those parameters, you yield a model that is to as if you had done data filtering and never trained on anything related to virology. But if you add them back, then you suddenly get a model that's very, uh, adept and, like, knowledgeable about virology at the same time. And so you could set up a, uh, a kind of like governance regime where, for instance, uh, kind of analogous to, like, KYC in banking, you could say, "Hey, if I know this person is, like, a trusted virology researcher, they've, like, signed a form. They've, like, said, 'I promise not to be evil.' We have their ID," et cetera. Then I'm gonna add these parameters back to the model that I serve to them, and only for that particular trusted user, I will have this, like, access control regime that will enable them to have this, like, more advanced capability. And that's something that you just would not be able to implement for data filtering.

[00:22:53] Ethan Roland: And so that's, like, one of the most obvious examples for why this is. There's a lot of other kind of, like, more niche technical reasons why I think it's excel-- like, uh, exciting, and I'm happy to get into those as well, but that's, like, the most, uh, obvious, I think, governance unlock that you wouldn't get with, with other techniques, just like data filtering. 

[00:23:10] Jacob Haimes: Gotcha. Okay. And then also maybe to just contextualize this, this work and, and sort of how you, uh, and AE Studio became involved and, and, like, knowledgeable about it. Um, there was previous work, I believe, from, like, a 2024 paper, um, that was originally about gradient routing, and then I think there may have been-- there was one last year, I think in December, uh, or at least a preprint from December, um, that was, uh, a follow-up to that.

[00:23:38] Jacob Haimes: And, um, my understanding is you've been working on this at least since, uh, around December, if not a little bit before then. Um, w- what did these prior work sort of get right that the field maybe hadn't appreciated to their full extent? And what was left on the table for you to really take and, and bring forward into the work that, that you're doing now?

[00:24:06] Ethan Roland: Yeah. So I think that, you know, I, I've spoken to the authors of, of those two other papers that you mentioned. Um, both have done fantastic work. I think that they set a lot of good foundation work for the types of things that we've been exploring since.

[00:24:27] Jacob Haimes: Mm-hmm. 

[00:24:28] Ethan Roland: what we are trying to do differently is more so prove out the, I would say two different aspects. So first, prove out that gradient routing can be used in a setting where you can simultaneously multiple concepts at once. So rather than just, uh, for instance, you split your model into two subsets, uh, general plus virology or general plus mathematics, what if you set it, split it into five subsets all at once?

[00:24:58] Ethan Roland: So you have, uh, general purpose knowledge, you have virology knowledge, you have nuclear knowledge, you have cyber knowledge, et cetera. Um, so we have this multi-way gradient routing. So that's the first, uh, major unique, uh, kind of adding to this. And then the other thing that we have shown that I think is, is particularly useful and convincing for frontier researchers and those that are working at frontier labs, or at least that is part of my hope, uh, is the presence of scaling laws.

[00:25:26] Ethan Roland: And so we've done a lot of work in order to try to have convincing evidence that, uh, our work scales, works consistently and predictably over a variety of different scales, and works with like fairly large scale. So we, we go up to, uh, five billion parameters, which of course is like not as large as like modern, uh, modern frontier models, but for, you know, an academic paper is, is actually very, very large and significantly larger than, than, uh, most, most people would be able to scale up towards.

[00:25:56] Ethan Roland: But I think it's because we are really aiming towards being maximally convincing to, uh, actual pre-training teams that this is a technique worthy of attention. And so those are the two I would say like most, most significant things is just multi, multi-way gradient routing for simultaneous, uh, simultaneous modularization of multiple things, especially like super and relevant things, and then a lot of effort into proving out that this works as a, as a scalable technique. 

[00:26:24] Jacob Haimes: Gotcha. And then that has sort of, uh, culminated, I guess, in, uh, your team specifically. I mean, you're the first author, uh, on getting a paper accepted to ICML as a spotlight, uh, which is fantastic. Congratulations. Uh, but, uh, it's called Modular Pre-Training Enables Access Control. Uh, and first of all, just like how does that feel?

[00:26:49] Jacob Haimes: Uh, was that like a goal for you? Um, is this-- Yeah, how long has that been a goal for you? I'm just curious about, you know, how you f- how you're thinking about that 

[00:26:59] Ethan Roland: Yeah, it's a big honor. I, I haven't properly absorbed that yet, uh, especially, like, not coming from academia research and certainly, yeah, not having a real background in, in, uh, ML research, or at least not in a formal sense. Yeah, it's a bit shocking to kind of, uh, get into this and then within a year have that as a spotlight paper to ICML.

[00:27:20] Ethan Roland: So yeah, that's, uh, that's certainly been weird, uh, but a good, a good kind of word, weird. So I, yeah, I feel thankful. Um, certainly it's not just my own contributions. It's, it's, uh, a huge percentage, uh, dedicated to the rest of the team and, and the other authors there. They've, they've done a, the awesome jobs, particularly Eric and, and Murat.

[00:27:40] Ethan Roland: They've, they've both been really, really influential in, in producing some awesome work. and yeah, I'd say that it was never kind of an explicit goal. I mean, the, the goal was always just to produce good research, and, I think my theory of change has always been try to produce stuff that is convincing to frontier lab researchers, and the publication and, and journals and conferences have always been more of an afterthought. but just like a nice, uh, checkpoint along the way. So it was surprising, uh, a big honor, not an explicit goal, but, uh, one that I'm certainly very happy about 

[00:28:13] Jacob Haimes: nice, So before we go into that even further, I do think, uh, like I wanna step sort of aside and address the framing shift, uh, between the prior work Um, and, and your, your paper. Um, because I read it as a pretty significant shift in the approach to how you're presenting things, um, compared to this other work.

[00:28:42] Jacob Haimes: And so the, the two prior papers in particular had this flavor of like, we want to remove a dangerous capability from a model, or maybe not even dangerous, just a capability or a, you know, thing, uh, behavior maybe from a model. Um, but your paper focuses on we want to enable access control. I mean, it's in the name, right?

[00:29:05] Jacob Haimes: Um, so that would mean different users get different capability profiles of a model using essentially the same training run. Uh, and that's-- I, uh, like, I, I think that's a very important shift to be like-- to notice and be purposeful about. And so I'm curious, when and why did your thinking move towards access control as the framing?

[00:29:33] Ethan Roland: Yeah, I think it observed or, or rather like emerged fairly organically. I would say it was motivated mostly by thinking about some of the more unique obvious advantages associated with gradient routing. And, and, and kind of like what you were saying earlier, where it's like, why would you use gradient routing versus data filtering?

[00:29:55] Ethan Roland: And assuming that you have high quality labels, and it's a thing that you would never want to as a capability in the model, then like, yeah, you actually probably would just use data filtering many of the times. so I think part of our motivation for thinking about access control and thinking about these like dual use capabilities and things that potentially you would want to have, more so just saying, "Hey, you know, what, what are some of the unique advantages that you wouldn't get with other things?"

[00:30:22] Ethan Roland: And then we wanna drill really hard into what makes gradient routing potentially unique in terms of its governance solution for the world. And so, yeah, I think that's, that's the, the primary motivation is just, um, it's kind of this like more parsimonious fit for the types of unique advantages of gradient routing, uh, and trying to dis- distinguish and differentiate itself versus data filtering.


[00:30:44] Mixture of Experts Explained (Aside)
---

[00:30:44] Jacob Haimes: Okay. Okay. And then, uh, a more technical but still very foundational aspect of your work in comparison to the other works is, um, the mixture of experts portion. Um, so mixture of experts, experts we don't need to get into in depth. I might do an aside about it just as a clarifier. 

[00:31:04] Jacob Haimes (Aside): Damn, I really shouldn't invoke my future self like that, but now I'm here, I guess I might as well finish the aside. Mixture of Experts, or MoE, is an approach that is widely used for the open weights and open source large language models, and is very, very likely an architecture used by the less transparent ones as well.

[00:31:27] Jacob Haimes (Aside): I won't get into too much detail here, but basically each layer of an MoE model is made up of a bunch of smaller expert models and one router which decides which expert to use in a given instance. By keeping most of these sub-models turned off at any given time, you save a lot of energy without sacrificing the size of the model.

[00:31:55] Jacob Haimes (Aside): In standard mixture of experts, there is no difference between one expert and another from an architectural sense. The specialization of each expert, which doesn't get more complex than specifying syntactic information, for example, is learned during the training process. Ethan's approach is almost identical, but instead makes a small portion of this routing deliberate, requiring that all information regarding a given topic goes through some specific set of experts.

[00:32:26] Jacob Haimes (Aside): If you're interested, I found a solid visual guide from Maarten Grootendorst, which is a great name, and that is linked in the show notes

[00:32:35] Jacob Haimes: the shift from how prior works did it to the mixture of expert approach that you guys are working with, um, I guess first, how or what intuitions Or, um, understandings of, of how these models are, are trained and, and function led you to that?

[00:33:00] Ethan Roland: Yeah. I, I mean, I would say in some sense, architectural choice, and we have some research showing this, um, that 

[00:33:11] Jacob Haimes: Hmm.

[00:33:15] Ethan Roland: adapters seem mostly arbitrary. And so there are lots of different ways that you could implement this architecturally. and you could just have multiple LoRA adapters, and you just like, add LoRA adapters being like small little components of, of, of, uh, parameters that are added to every single layer. Should we do that, or should we do this slightly fancier method of doing interventions from scratch from when we initially start pre-training, which is what our method is, which is GRMoE. And, um, yeah, I think that the story is like a little complicated, um, it somewhat depends. So under some circumstances, these two techniques can achieve similar end results in terms of when you want to be good at a particular auxiliary category, they can be both equally good, and when you want to be bad at a given auxiliary category and, and remove parameters, they both can be equally bad. Uh, where the story becomes better for the case of GRMoE, uh, which is our method, is usually in the case of, for instance, like If you are attempting to train on data that is particularly noisy or partial, we have some, some, I would say like somewhat preliminary, but I would still say like convincing to myself, evidence that there's this effect that we call absorption, uh, which was the primary focus of the Shiloh paper. basically, it's the idea that if you have really partial labels, that modularity that's set up, say, for instance, by enforcing what parameters get updated a certain percentage of time, say only like twenty-five percent of the time, seventy-five percent of the time you're training on things that you don't know whether or not it's virology or general purpose data, 

[00:35:09] Jacob Haimes: Um. 

[00:35:13] Ethan Roland: that you can expect that parameters that have already been set up to be responsible to certain types of functionalities when trained in this unrestricted way to continue learning the things that they've already been set up to, to be responsible for. And so in that sense, modularity has this momentum to it, it allows this kind of sponge-like absorption of all of those capabilities into the module that you've already set up. And this is something that, yeah, we've observed in our experiments as well, and certainly other, other work has observed. so in my mind, one of the, the, the clear reasons why you would use pure pre-training from scratch interventions rather than LoRA fine-tuning is under the circumstance where you don't trust the quality of your labels, you have this very powerful effect that allows you to still achieve modularity in a way that is significantly better than you would get if you were using or restricted exclusively to data filtering or kind of this like ordered post-hoc fine-tuning via LoRA adapters.


[00:36:30] Why Pre-Training Interventions Are Rare
---

[00:36:30] Jacob Haimes: Okay. And then, yeah, I, I do think that the ab-absorption effect is definitely, like, the strongest, uh, maybe, like, single, uh, point within, within this work. Uh, and I think I, I think that, uh, I'd like to, to get into that a little bit. Um, the one thing that I, I also want to maybe flag and, and push back on is this idea of like, well, we would need to have essentially better data, right?

[00:37:01] Jacob Haimes: We would need to have data that's, uh, labeled more accurately that we can trust. Uh, and if we, if we had that data, then, you know, maybe we could do the, you know, deep ignorance style, a full removal, uh, of the-- of this information. Um, and that would, I assume, based on, on my reading of the paper and, and understanding of this, be more robust to preventing external actors from, uh, I guess, reconstituting the, um, intended, uh, ablated capabilities.

[00:37:53] Jacob Haimes: Um, so it's not that it's like-- that it's not possible, it's just that it would be more costly. Is that correct?

[00:38:03] Ethan Roland: Yeah. Yeah, this is correct. Um, so generally speaking, the reason that you care about pre-training time interventions, whether that be data filtering or gradient routing, is because the types of safety that you induce in the sense of being ignorant of a given topic is much more robust than, for instance, interventions to induce refusal to talk about how to build a bomb or how to make a virus or et cetera. the sense that when you, when you have removed that capability, or, or rather when you induce, uh, refusals, um, you have two, two points of vulnerability. First, there are jailbreaks, which can just, like, ask the question in a weird way and then, you know, kind of elicits this knowledge that still is fundamentally in the model, and so it just tells you anyways. Or if it's, for instance, like an open source release, it's pretty trivially easy to fine-tune these models to no longer do refusals, and then it elicits or, or unlocks all of these, uh, latent capabilities of knowledge about potentially dangerous things. And if these types of capabilities at first, or their types of knowledge totally absent from the model, then in order to yield a model that is good about those topics, what you would have to do is you would have to fine-tune that model in some way in order to imbue that model with knowledge related to the potentially dangerous topic. So you would have to teach it from scratch about virology or bomb making or et cetera. And this is, especially at frontier scale, very expensive to do, way more expensive than just fine-tuning to get rid of refusals. And so we get this, like, much stronger and more robust guarantee of safety relevant characteristics when we do pre-training time interventions. Uh, and that is a characteristic of, of, of any pre-training time intervention, generally speaking, both data filtering as well as, uh, gradient routing 

[00:40:10] Jacob Haimes: Mm-hmm. Why do you think it's taken so long for these to start-- these pre-training methods to start being adopted then? Like, if you're, you're saying, "Oh, well, it's, it's actually much stronger. We get better guarantees." Uh, but in practice, that hasn't been what people have been using.

[00:40:29] Ethan Roland: Yeah, I think it's mostly a practical question. So, I think there's a lot of appetite for this in the research and, and development community in general. 

[00:40:38] Jacob Haimes: Mm-hmm. 

[00:40:38] Ethan Roland: Um, I'd say there's a few reasons. So one reason is that when you're spending, you know, however much money a frontier run costs, let's say like a billion dollars, generally speaking, uh, I think people are very conservative about messing with a formula for how to spend a billion dollars. And, and reasonably and understandably so. And so I think that like oftentimes, at least my impression, is that folks don't want to change that setup in order to have guarantees that they're gonna have at least a reasonably good and performant model. And so there's like worries that, you know, if I do something that's exotic, then this is going to jeopardize a huge amount of spend, and this will be very bad for, for, you know, my organization. Um, so I think that's, that's like one clear reason. then I think a lot of the other things is that like, uh, if you just have post-training methods, these are pretty cheap to implement and experiment with, 'cause it's just a thing that's like, you know, you do a little bit of post-training on a, on a model, and so you can just like iterate through a lot of different experiments. 

[00:41:47] Jacob Haimes: Mm-hmm. 

[00:41:47] Ethan Roland: but getting signal on pre-training time interventions and the quality and how well they work, takes a lot more effort because every time single data point is itself an entire model training run. You know, so it's like we're, we're training, you know, tons and tons of multi-billion parameter models. Uh, and so this is like, you know, much longer in terms of your research iteration cycle.

[00:42:10] Ethan Roland: So just like the field tends to move a little bit slower, 'cause it's a more difficult type of research. Uh, and it's also just much more expensive. So yeah, it's like if you're, uh, running and doing full model pre-training times or full model for every data point that you get about how well your, your intervention works, uh, yeah, this is like very easy to, to rack up very large compute bills, uh, very quickly.

[00:42:34] Ethan Roland: So I think there's, there's like these kind of structural reasons related to the research is just like difficult and expensive to do. Uh, but also just like general conservativis- conservativeness when it comes to like willingness to implement pre-training time interventions.


[00:42:48] Ethan's Theory of Change
---

[00:42:48] Jacob Haimes: For sure. And so I think that ties in, if I, if I understand correctly, to your theory of change here, right? Your core motivation, especially in how it's framed and, and how you've been talking about it, is get other model developers in particular, basically get other people who can engage with this problem to be doing so.

[00:43:10] Jacob Haimes: Is, is that correct? Maybe-- And can you add like a little bit more, uh, color to that as well?

[00:43:16] Ethan Roland: Yeah. I mean, I have a lot of different motivations, but I'd say that I want to have other researchers working on this because I do think that relative to its expected impact, that pre-training time interventions and gradient routing in particular is and under-pursued, uh, yeah, again, relative to, to the expected impact.

[00:43:40] Ethan Roland: And so I think more people should be working on it. Um, I think that because of some of these research cost constraints in that pre-training time interventions tend to be a little bit less accessible to like general labs, uh, that I differentially want to drum up excitement amongst well-resourced researchers, which tend to be concentrated within frontier labs or similar organizations. so, yeah, I think I'd be very excited to see that type of interest happen within those labs. Uh, a-a-and in general, like even outside the like pure research part, like one thing I would love to see pre-training time interventions actually implemented in frontier training runs. You know, like I think that's, that's the ultimate theory of change here, right?

[00:44:31] Ethan Roland: Where it's like, what's the purpose of research if it's not actually implemented in frontier model training? And like you can deliver these really wonderful points, but if it, it's not actually implemented by the people that matter, then you haven't actually done anything useful. And so in that sense, like I think the actual goal here is to get people that are producing these frontier systems to use these type of techniques.

[00:44:54] Ethan Roland: And I, I think that generally should be the goal of most, uh, training style technical research and, and alignment is that, you know, unless you are yourself a model frontier developer, you should probably be doing your research in such a way that is going to be very convincing to frontier developers because that is like the path through which your research actually makes impact. Uh, and so yeah, I would, I would hope that, uh, in addition to just like encouraging frontier researchers to do more like pre-training research, uh, also to like implement them within their actual pre-training processes, uh, which would be particularly exciting for me to see. 

[00:45:33] Jacob Haimes: Mm-hmm I guess I, I do agree to a certain extent, but then also that, that narrative's, like, very convenient for frontier model developers, right? Uh, like, the best path forward for the little guy is to support the big guy, which, like... I don't know. It seems, it seems a little-- I feel like maybe there's, uh, so-something else to, to pull on there.

[00:45:57] Jacob Haimes: But yeah, it just, like... I, I agree to a certain extent, but then at the same time, it just doesn't wholly sit right, you know?

[00:46:06] Ethan Roland: Oh, yeah. I mean, I, I'm not certainly claiming that, like, this is necessarily a desirable, uh, uh, like, order of events. Like, I think that if every single individual could have huge, gigantic impact of, uh, of research, regardless of whether or not it has up-uptake by frontier labs or not, this would be a good circumstance for the world to be in. 

[00:46:27] Jacob Haimes: Sure.

[00:46:29] Ethan Roland: more of a matter of the happens of-- the happenstance of, like, frontier model development happens to be very expensive, and so correspondingly, it's, like, concentrated into a very small number of players. And frontier model development is where, like, all of the risk profile lives, and so therefore, if you're going to make impact on, via doing research that actually changes real re-- like risk profiles, your research must be convincing to the small handful of people that actually have the concentrated resources. 

[00:47:02] Jacob Haimes: Well, I guess it also depends on, on what your-- what the risk profile is that you care about,

[00:47:08] Ethan Roland: Yeah, 

[00:47:08] Jacob Haimes: right? 'Cause there-- I mean, there's considerable risk, uh, s- you know, happening already w- like, s- around, you know, uh, character AI, uh, you know, style, um, like dependency, uh, and mental health, uh, difficulties that are, uh, sort of augmented.

[00:47:29] Jacob Haimes: And that's not-- I mean, those-- That was a flourishing, uh, sub-industry with models in 2023, you know?

[00:47:41] Ethan Roland: Yeah, that's very true. I think it's-- I'm like showing my bias here as someone that's mostly focused on like CBRNE type risks, where I do think most of the risk profile exists in the frontier. But that's also not entirely true, right? Especially with open source releases becoming incredibly, increasingly more powerful. Um, you know, like how far away are we from a Mythos level open source model? You know, a year, two years? if that is the case, then I think that I do see a place where there could be a lot more impact in producing techniques, even if those techniques are not necessarily adapted by the frontier labs themselves. yeah, I guess I, I'm willing to kind of motte-and-bailey that just a little bit. But, uh, yeah, generally speaking, like I think it's-- really nice if frontier labs adopt your things.

[00:48:43] Jacob Haimes: Oh, absolutely.

[00:48:44] Ethan Roland: yeah, 

[00:48:45] Jacob Haimes: it's-- it would be great, right?

[00:48:47] Jacob Haimes: Um, and I guess that, that's also related to, uh... So I don't know if it's technically like a collaboration or like they also worked on the paper. I don't know what, what, what I can say there, but, um, uh, some of the authors on, on your paper, uh, are working from Anthropic, right?

[00:49:06] Jacob Haimes: Um, and so-- And, and way back at the beginning, you talked about, well, AE Studio finds these sort of enfranchised, uh, high impact, uh, and, uh, sort of people who have, uh, demonstrated that they know how to do interesting research. And you say, "Hey, we're-- like, we wanna support you," right? Um, so what was that like here?

[00:49:35] Jacob Haimes: And, and what was working with, um, developers, uh, who are, you know, working in, in the frontier?

[00:49:42] Ethan Roland: Yeah. Um, you know, it's a great honor. I think that it's lovely to have the experience and, and insight and research taste of, of folks that are themselves working at the frontier. yeah, it's-- I, I think it's useful. Uh, it's useful for the purposes of what I was saying earlier of, of trying to produce things that are, are convincing to frontier researchers. Um, I think it's, it's nice. It, uh, adds hopefully like a, a... I would say like gives me more confidence that the research is of good quality I'm able to work side by side with these researchers that are themselves at like particularly, uh, high reputation or like high output organizations such as Anthropic or other, other frontier labs. 

[00:50:34] Jacob Haimes: Gotcha. And- Do-- What, what does this collaboration, like the, the fact that they were willing to take a, uh, like ra- they... Anthropic was willing to allow, uh, these researchers to take time that they're getting paid for to be working on, you know, this research effort. Um, what do you think that says and, and maybe also what are, like what their focuses were, uh, or what they were particularly interested in pushing on, uh, says about how Anthropic or other frontier labs might actually be wanting to pursue this moving forward?

[00:51:16] Ethan Roland: Yeah, I, I can't speculate on, like, the relationship between researchers and the labs for which they, uh, work at. But 

[00:51:25] Jacob Haimes: Sure.

[00:51:27] Ethan Roland: I have seen a lot of enthusiasm about pre-training time techniques from a variety of 

[00:51:35] Jacob Haimes: Gotcha.

[00:51:39] Ethan Roland: probably folks from four different of the frontier labs at this point, um, and various conferences and social events.

[00:51:46] Ethan Roland: And generally speaking, the enthusiasm for, for, for pre-training time techniques seem to be very high. so I think it's a very ripe field for making impact because as soon as we have kind of a formula that one should follow, uh, I think that you're going to see a lot of adoption for this throughout the industry, or at least that's my, my strong impression. 

[00:52:12] Jacob Haimes: Okay. And yeah, no, I, I think that makes, that makes sense. Um, and part of that is the, the access control framing as well, right? It-- There's almost like a, I don't know, seductive cleanness to it, right? It's just like, oh, oh, we're, you know, we're just saying that we, we wanna give different, uh, users different model variants and, and we're s- you know, making it easier to, to do that.

[00:52:38] Jacob Haimes: Um, but then at least for me, and maybe this is because I'm like, you know, skeptical slash, uh, untrusting of, uh, larger corporations in general, uh, but like who decides who counts as a user that gets access to version A versus version B? Um, I mean, uh, understandably, your, your paper doesn't really engage with that, uh, but it's still something that, uh, I'm curious what, what your thoughts are.

[00:53:07] Ethan Roland: Yeah. I mean, I think It highlights a good and important thing to keep in mind whenever you're doing technical research that, like, the technical component is only one component. And what I view this as is the enablement of new types of governance solutions. And so we can make access control and differential serving of capabilities a thing that's possible to do that does not answer the question of, okay, now you have this new option for governance, how do you want to implement the governance piece? Uh, and that is like its whole new thing that, you know, definitely governance people should be thinking about. Um, generally speaking, I do think that it's not too exotic of a problem in the sense that, like, how does society at large, in the US at least, already handle and access to potentially dangerous things? for instance, when we have, like, gain-of-function research labs, which, you know, maybe should or should not exist depending on your opinion, like, there already are very strong regulations about how bio research labs work, and there must be, for instance, um, you know, certifications of the researchers, certifications of the way in which, uh, that research occurs. Uh, and, and, and similarly certainly for, like, nuclear cases, right? Where it's like, you know, not everyone can just have access to, uh, nuclear materials for doing nuclear research. Like, you have to work in a specific, specific facility with, like, the right type of governmentally approved, um, you know, and, like, guarantees that you're not creating some type of ecological disaster. And so I could imagine a similar type of licensing regime, and probably that regime would get more stringent over time capabilities become more concerning and more potentially harmful, where initially maybe it is just a case of, um, okay, I have their, like, name, email, photo, contacts, et cetera, and I have like, you know, again, kind of like KYC style banking.

[00:55:26] Ethan Roland: I've, like, ran a background check on them. I know that they're not, like, a foreign national. I know that they don't have a criminal record, and so therefore they have access to this. Uh, and then as the capabilities become more and more dangerous, then it may be more of like, oh, well, uh, I, like, know who this person is, and they have some, like, government license to do bio research, for instance, and they are associated with a, uh, government licensed, uh, research facility doing virology research.

[00:55:53] Ethan Roland: And, like, if those conditions are true, then we yield that model that has virology. So I think it's, it's a question that doesn't seem to be extraordinarily exotic, and I think we have the types of paradigms that have already existed for just how we do potentially dangerous research already that we can to the question of who does or does not get access to these particular types of dual use capabilities


[00:56:22] Access Control Governance and KYC (Aside)
---

[00:56:22] Jacob Haimes (Aside): This one's been getting increasing attention in the AI policy space, so let's take a second to unpack it. KYC or know your customer requirements refer to an existing regulatory framework used in banking and other financial services. The regulator requires institutions to know their customer by collecting identification, running background checks, and making sure that the individuals aren't on sanctions lists.

[00:56:48] Jacob Haimes (Aside): So similarly to how banks don't let anonymous parties open accounts, a model provider could be required to collect similar information in order to provide a given user access to certain models. It's worth noting that KYC in banking represents a significant cost to financial institutions, especially global ones, and the infrastructure which supports it has been developed since the US Bank Secrecy Act of 1970.

[00:57:15] Jacob Haimes (Aside): In other words, implementing KYC isn't something that could be done simply, and even if it was, whether we'd actually want it here is an open question. 

[00:57:24] Jacob Haimes: Mm-hmm. But at the same time, you specifically in, in the framing for this, this paper, it, it's for, you know, developers, not for, um, the governance side as much. do you see

[00:57:44] Jacob Haimes: this difference as meaningful? Uh, the, the fact that you were, you were much more focused on the, uh, maybe way to turn this technique into something that is good for maximizing profits for the companies as opposed to, um, you know, i-if we have this, it enables, uh, this sort of access regime that, um, you know, is realistic and, uh, sort of pitching to policymakers instead.

[00:58:22] Ethan Roland: Yeah. I mean, there's a couple different angles. Like, one angle here is that, you know, I'm a technical researcher. I have certain, like advantages or like and like my particular niche and thing I'm good at is producing technical research. I am not like a policy expert, and so therefore, if I'm going to spend time, I should probably like mostly focus on technical research. 

[00:58:43] Jacob Haimes: sure.

[00:58:43] Ethan Roland: not to say that like the policy part does not exist, it's just like not within my, uh, like comparative advantage to speculate as much about. Um, and so I like hope and have talked to some policy people already to hopefully get this as like a, an angle that they're thinking about more. Uh, but I just like think it as like being somewhat outside of my purview of my own personal comparative advantage. Um, I would say that like the angle of kind of like profit motive- motivated things, I think that This is one thing that I, I hear at times, and I would disagree with in the sense that think that that make companies a lot of money and things that produce really good safety results are not at all exclusive. what I mean by that is that like, if you're, if you're purely going from this from like a cynical, like, "I want to make this much money," a thing that makes you a lot of... Like, it costs you a lot of money, is being liable for really bad results of things that are used for your products. And, you know, this is one of the things that policy type people have, have speculated on as like a mechanism for encouraging uptake of safety relevant techniques in labs, which is, uh, you know, maybe this occurs just de facto via, litigation or occurs, you know, via, via policy action. Uh, if there is existence of like liability for misuse, uh, then like cov- companies have a huge, uh, financial incentive to, to, uh, pursue these types of, uh, types of techniques that, that greatly reduce the chances of misuse. And so, you know, it's like, you know, in, in some sense you would hope that like companies and, and, and developers are, um, implementing these like purely for prosocial, like we wanna make the world a better place and like not hurt people.

[01:00:32] Ethan Roland: But also like, I think that these like motivations are not mutually exclusive of like, uh, causing harm is also really expensive. And to the extent that you exclusively care about like making money, you also care about not making harm. And so, yeah, I think it's like, uh, the, the most robust systems in the world tend to be ones in which there is a, a confluence of multiple motivations that all point in the same direction.

[01:00:55] Ethan Roland: And I think in this particular case, uh, both the like prosocial and like pro-capitalistic elements tend to point towards similar actions. 

[01:01:05] Jacob Haimes: Yeah. I, I, I do see what you're saying. I, I think I, I disagree in that they do. I, I believe that they can. Um, but like you said, without sort of stronger enforcement slash, uh, mechanism to ensure liability, um, I, I don't think that that's necessarily the case. So if we were able to obtain that, right, if we were able to, to get there, um- I think that, yeah, things, things become much more aligned, uh, again.

[01:01:40] Jacob Haimes: But while we are not in that regime, uh, regardless of how much we want to hope that a, a company might have, uh, pro-social, uh, sort of motives, I, I think that the realistic model is to assume that companies are profit maximizers and nothing else, um, just based on, on historical data. Um, and so we need to, to ad-adjust the incentive structure if we want the behavior to align with, with those values.

[01:02:12] Jacob Haimes: At least that's, that's sort of how I think about it.

[01:02:14] Ethan Roland: Yeah. I mean, I certainly will never argue that policy action and policy advocacy is unnecessary. I mean, I strongly think it is. I think if anything, I will just, like, present a case that there is some reason for optimism from a purely, like, profit-maximizing company, why they would still want to do this, 

[01:02:38] Jacob Haimes: Mm-hmm.

[01:02:38] Ethan Roland: why they still may be caring about

[01:02:41] Jacob Haimes: Yeah.

[01:02:45] Ethan Roland: governance. I think there, there's, you know, strong reasons why you would expect this still occur, to occur. But that is not to say that, like, you know, I think the, the best systems are ones in which there are many different things that are, like, motivating you and pointing in the same direction. And so I think it's, like, good if you also have the motivation of, like, policy requirements at the same time. Uh, and if all of these things are, like, pointing in the same direction, like, that does not hurt, uh, at all. And, like, I would say, you know, in many, in many possible futures, like, may be strictly necessary in order to achieve good, good circumstances. I'm, like, uncertain what world we live in and whether or not, like, pure, uh, profit-maximizing motivations are sufficient to incentivize developers to produce and implement robust safety techniques.

[01:03:37] Ethan Roland: Probably not the world in which we live in, but, you know, it's, it's possible that, that we live in this, like, relatively optimistic world. Uh, but Yeah.

[01:03:43] Ethan Roland: I would, I would certainly not argue that, like, policy advocacy is bad. It, it 

[01:03:48] Jacob Haimes: Oh,

[01:03:48] Ethan Roland: wonderful thing to do 

[01:03:49] Jacob Haimes: for sure. I, I guess that also just based on, on the framing that you just said and, and kind of adding some, some of my thoughts to it, does this, in, in your eyes, change the sort of what's most important or most useful to be working on, um, at, at all? Um, whether that's necessarily what you're going to do or, or not.

[01:04:15] Jacob Haimes: Like, does this change the, we don't have the technical tools style questions, uh, to be much more in line with the, we don't-- Uh, like, that's sort of checked off now, at least to some extent. Uh, the More pressing issue is we don't have the political or institutional structures that make sure that those tools are used appropriately.

[01:04:41] Jacob Haimes: Um, does that shift your framing? Does that pull you towards different work? Uh, what is your take on that?


[01:04:52] The Researcher's Role in Policy Advocacy
---

[01:04:52] Ethan Roland: Yeah. I think that for alignment in general, argument could be made that An increasing amount of resources should be put towards policy advocacy rather than pure technical alignment research as a consequence of all of the progress that has been made for technical alignment research. Now, I think if you'd asked me in twenty sixteen, would be very clear that, like, most of ninety-nine percent of all the resources should be towards technical research because what can you advocate for as policy if you don't have the things or the tools to be implemented? But as these tools have become more developed, then the impact surface shifts more towards policy. So I think that is very true. Um, as to, like, what's the optimal distribution of resources between policy and technical research right now, I really don't know. I mean, I, I-- it's 

[01:05:54] Jacob Haimes: Yeah, solve our problems for us, please.

[01:05:56] Ethan Roland: not getting it right, but I don't know what the right solution is unfortunately. Um, as for myself, like, I think I'm, you know, this is like one of a, a question of comparative advantage. And so, like, it could still be that we, we live in a world where, the majority of resources should be towards, put toward policy research or policy advocacy, and I should still personally be working on technical research just because I predict that I'm, like, much better at, you know, writing research code than I am at talking to lawmakers. 

[01:06:29] Jacob Haimes: Sure.

[01:06:30] Ethan Roland: think it's like, as a question of the community in general, um, I think this is something that definitely is shifting more in favor of advocacy over time. Um, though I think it's still an open question of, like, what's the optimal distribution of resources? 

[01:06:47] Jacob Haimes: For sure. Okay. And then I also do just wanna, I guess, maybe like register my pushback again on, um, you know, this idea almost of the like, uh, the purely technical researcher. Um, you know, like, "Oh, I can just, you know, put my head down, do technical research, and other people can handle, uh, the advocacy kind of thing."

[01:07:12] Jacob Haimes: Because I think it's really important actually to bring the expertise that you've cultivated to some of those conversations. Because for a lot of people, um, that are in these policymaker positions, they- The thing-- One of the things that they lack, uh, or, or feel that they lack at the very least, um, is having more confidence in, like, the opinion, the stance that I'm taking is backed by, uh, evidence, I guess.

[01:07:52] Jacob Haimes: Uh, and, and so people who have that rich context and can still communicate, um, al-almost have like a, uh... I mean, in, in my eyes, I see almost like an obligation. Um, but like, uh, maybe a...

[01:08:13] Jacob Haimes: It would, it would make me happy uh, if, if, um, people were, were more willing to be that like, you know, subject matter expert kind of, uh, person to call on and, and making that more visible, I guess. I, I, I'm not totally sure how the best way to go about it is, but I, I think that's an important thing to mention when you're so, like, steeped in this, this area that you think is going to be important for policymaking.

[01:08:46] Ethan Roland: Yeah. I mean, I, I thi- certainly think that's true. And, you know, to, to certain small extents, I feel like I've some opportunities to do that, uh, when it comes to, like, communication to less technical audiences to try to advocate for technical alignment or, like, alignment relevant subjects in general. yeah, I think this is true, definitely. I mean, I don't, I don't see a, foresee a future in which than 50% of my time is done doing technical advocacy again, 'cause it's just, like, 

[01:09:22] Jacob Haimes: Yeah, I wouldn't expect that either.

[01:09:23] Ethan Roland: a misallocation of resources, uh, most likely. But yeah, I mean, I, I certainly think in general, if you are in a position where have the ability to advocate general like pro-alignment policies and you have that technical expertise, that's something that people should be eager to do. yeah, I think oftentimes in the policies field, there is a lack of people that have good bona fide hands-on experience doing technical work, to inform some of those decisions. And so yeah, I, like, I, I want our policy to be like intelligent and well-made, uh, in general, and I think oftentimes policy, you know, just i- in general across all politics is like not this. And so, uh, yeah, I think that this, this should definitely a goal and that technical researchers can, uh, help, help kind of reduce this problem. I would say another angle on this is that kind of the converse, that when you're doing technical alignment research, one failure mode of making your research impactful is not thinking about policy impacts from the start. And so one of the things that I think motivated our paper framing is kind of what you were saying, where it's this like very straightforward, almost seductive way of thinking about our impact of, you know, here is this technical thing and the way that you could use this technical thing is through like, access control as a new governance mechanism. But I think that we had our sights laser-focused on, can I do the research that enables this type of governance? I think that it's really important in order to produce a coherent story of why your research is doing-- your technical research is doing something good in the world, to think about what does the governance piece look like when that piece of thing is actually implemented. And, uh, this is something I think is like underappreciated and under factored in a lot of the times, but I, I think oftentimes people should be thinking more about what the governance relationship to their technical research is. 

[01:11:34] Jacob Haimes: yeah, How, how do I enable the thing that I would like to happen very robustly?

[01:11:39] Ethan Roland: Yeah. Or I mean, or just like, what does the thing I actually want to happen actually look like?


[01:11:43] Speed Round
---

[01:11:43] Jacob Haimes: Mm-hmm. Okay. Yeah, no, that, that makes a lot of sense. Um, so I guess I, I do have a couple, like, sort of speed round questions I like to do before the end, uh, and then, uh, a couple at the end as well. Um, so if, if you're good with that, um, we'll just head into those.

[01:12:03] Ethan Roland: Yeah, let's do it 

[01:12:04] Jacob Haimes: Awesome. So what is your hottest take on AI safety, like the field or, or AI development or, you know, something along those lines?

[01:12:14] Jacob Haimes: What's your hottest take?

[01:12:16] Ethan Roland: Hmm. My hottest take?

[01:12:20] Jacob Haimes: Yes.

[01:12:22] Ethan Roland: I think I put a 20% chance that new paradigms in how one trains models, whether this be like 

[01:12:34] Jacob Haimes: Interesting.

[01:12:38] Ethan Roland: paradigms, or very complicated forms of in-context learning, could accidentally invalidate maybe 80% of existing technical alignment research. this is the thing that keeps me up at night in the sense that, yeah, I worry that, uh, a lot of our research historically will be for naught because there will be some large paradigm shift on the way in which we train and have intelligent models 

[01:13:05] Jacob Haimes: Okay. Yeah, I, I think that's re-- I, I think that's reasonable. I think maybe the, the hot part is that it's not thought about as much. Um, but yeah, no, that's interesting. Okay. What's-- what, like, grinds your gears about the work you do? Uh, it doesn't matter if this is, like, technical or social or structural, it's just like, what annoys you about your work?

[01:13:30] Ethan Roland: Well, this is an easy question to answer for pre-training research because it's, uh, slow and expensive and difficult. So, like every single data point, like I said earlier, takes, you know, hours and hours and hours to get, and like a quite large quantity of money. Uh, and so yeah, I think structurally, uh, the thing that grinds my gears is that we can't, like, move faster and be at, like, 10X larger scale because, uh, you know, we have to, like, spend not infinite amounts of dollars.

[01:13:58] Ethan Roland: And, uh, yeah, compute is only like, you know, finitely fast at training very large models. Uh, and I wish I could move faster, uh, if it weren't for, for the fact that pre-training is itself kind of a ordeal 

[01:14:10] Jacob Haimes: Gotcha. Yeah. Uh, yeah, that totally makes sense. Uh, and then, like, at least from my perspective, the public-facing technology portion of, of, like, AI is, is going through almost like a vibe shift right now. I think, uh, over the past, I mean, over a year, honestly, um, sort of public sentiment has been increasingly negative.

[01:14:34] Jacob Haimes: Um, and yeah, I guess I'm, I'm just curious, uh, if, like, this shift impacts, uh, how you think about the work you're doing, or if there's something that you're seeing, um, maybe behind the curtain that you think, uh, makes that not as, uh, concerning or, yeah, the things like that.

[01:15:03] Ethan Roland: Yeah. Well, it's kind of funny. So when people-- meet people and they're like, "What do you do for work?" And I say, "Well, I work in AI." Sometimes I see them kind of scowl, and they're like, "Oh gosh, you... That nasty thing that, I hear all this bad stuff that's destroying the world." And I said, "No, I promise, I promise I'm doing the good version of that.

[01:15:20] Ethan Roland: I'm trying to make everything better." Um, and sometimes they understand. But yeah, it is, it is interesting. It's been like, something I've had to be more cagey about when meeting strangers recently, 'cause I, I have to say that like, I work in the good version of this and not the version that seems, uh, to be making everything worse." I do think that, um, there is some opportunity if it is properly captured, that some of the public concern related to AI can be used to result either, either in policy or just like funding towards more pro-alignment, um, efforts or ventures. And so this is something that I think is potentially a good, a good cause. But, uh, yeah, I do, I do have some concerns 'cause I think that like concern about, uh, AI that is, um, you know, generously not always super technically well-founded, uh, but it's just like reasonably related to like a lot of anxiety and economic and social anxiety that, um, yeah, I, I worry that like populist concerns are not always like super productive.

[01:16:27] Ethan Roland: Um, but my, my hope is that this like general body of concern can be turned into something that, uh, produces good results through, you know, better support for, for technical alignment research. 

[01:16:38] Jacob Haimes: just, to, clarify that ambiguity, which, um, which concerns, or like which bodies or groups of concerns are you referring to?

[01:16:47] Ethan Roland: Like, uh, like, like concerns about, for instance, like data center water usage, 

[01:16:53] Jacob Haimes: Mm-hmm.

[01:16:54] Ethan Roland: concerns about like job loss. Uh, ma- many of these are like very valid, though oftentimes they're like very unnuanced in terms of public discussion about this 

[01:17:02] Jacob Haimes: Mm-hmm. Yeah. I, I, I will say I'm a-- I was-- To me, the, the water usage I think, you know, can be a compelling, I guess like slightly compelling thing i-in, in a certain... Uh, basically like if there is a drought, right? If there is a, if there is a water, uh, control, then in some cases it's, it's easier to frame it in a way that is, uh, that looks particularly bad, uh, I think because of, uh, how the water...

[01:17:32] Jacob Haimes: Like a lot of the water is recycled, um, and it is groundwater that couldn't be used anyways. And so-- But like the thing that's more important about, to me about the data centers is just like the energy usage, uh, 'cause I think that is actually like a, a thing that we should be, be tracking. But yeah, I, I totally agree.

[01:17:48] Jacob Haimes: Uh, there is definitely ways that we can leverage sort of unrest almost, uh, to, to get towards solutions that are going to be more intentional and, uh, helpful for people at large.

[01:18:09] Ethan Roland: Yeah. I mean, I think, you know, it's like a really nuanced and tricky problem of how you deal with that general, quote unquote, "unrest." 'Cause in some sense, you know, is a open question, but there was, like, a data center moratorium, and maybe this would be good for the environment, and in some sense this would be like a pause, but it would be like a US exclusive pause.

[01:18:31] Ethan Roland: And people have, have argued that, like, do you actually want a pause that's only in the US when there's not gonna be pau- uh, pauses in, like, competitor countries like China? is this a bad thing because then you kind of, uh, accelerate the, like, relative control of, of, of, like, the Chinese government? you know, maybe that's actually bad on net. so I think these are, like, things that are not very clear that have, like, a very positive impact of, like, data center moratoriums. Uh, whereas there is, like, other policies that are unambiguously more positive, for instance, uh, just like, know, maybe even, like, liability regimes or, like, very explicitly, like, mandating certain forms of, of, like, testing for, like, CBRN risks at, before, like, frontier model deployments. These things are, like, significantly less likely to accidentally foot gun ourselves in terms of the future of how things go. Um, but there's a question of when you have this kind of, like, populous rage, does that concern be captured or is that concern captured and used unto productive ends? Or is it used in policies that actually end up foot gunning ourselves, either in, like, reducing the US competitive position or just, like, capturing, uh, public attention and, like, it towards distractor issues that aren't actually relevant for, like, the things that matter, which I would say are, like, existential risks 

[01:19:57] Jacob Haimes: Okay. And since, since you brought it up, I'm curious, you know, you mentioned you spent time in China, um, and you have this, uh, appreciation for, uh, Chinese culture and, and that, um, in a way that I, I would say a lot of people don't. Um, how does that impact your perspective on, um, this sort of, uh, I don't know, in my opinion, like war hawkish, uh, framing almost, uh, that often goes around the like AI, uh, US versus China stuff?

[01:20:38] Ethan Roland: There's a certain contradiction in the discourse around this because simultaneously you hear a lot of policy advocates, advocates thinking Chinese researchers are in some sense incompetent and that China can't end-- innovate, they can only copy. then simultaneously you hear oftentimes the same advocates say, "Hey, we can't lose the China, uh, the US-China race, and we must, like, dominate them in all, all forms."

[01:21:07] Ethan Roland: And I, I think that, like, these are obviously somewhat contradictory and in many ways I think I have opposite opinions that I think that the Chinese research apparatus for frontier research is much more adept than many people in the US give it credit for. 

[01:21:28] Jacob Haimes: Mm-hmm.

[01:21:30] Ethan Roland: I also think that, like, um, generally speaking, I, I actually don't think that there is, like, much appetite at all within the Chinese government for, like, world domination.

[01:21:42] Ethan Roland: Like that, that, that's not consistent with their, uh, generally exhibited, uh, behavior, right? I think that they like, want strong regional influence. Uh, but I, like, I don't think that they are, you know, wanting to be like a world hegemon and like I don't think that they're-- uh, like in, in the same sense that the war hawks, uh, fear that they are. And so, Yeah. I think that like the kind of ideological threat of the Chinese these like world dominating, uh, cartoonish vis-- villains that we must defeat is vastly overstated. I definitely think that's true. But I also simultaneously think that we in general underestimate the, the technical prowess and chops of, of Chinese researchers.

[01:22:28] Jacob Haimes: Yeah. Awesome. All right. So you, you came into AI safety without the standard credentials, and, uh, now you are the first author on an ICML spotlight. For someone who's listening in industry or doing applied AI work or, uh, who j-- wants to contribute to research like this but doesn't see an obvious path, what worked for you and what would you tell them?

[01:22:53] Ethan Roland: Yeah, I'd say that It's not as difficult as I would've expected. I do think that there's some advantages to going into alignment in the sense that it's kind of a weird niche subfield still, you typically can have outsized impacts as a bit of an outsider When getting into more niche fields because there's more low-hanging fruit, uh, there are not these like super rigid hierarchical or bureaucratic structures that you have to navigate.

[01:23:23] Ethan Roland: It's kind of just people are doing whatever. so there is a easier path of climbing the ladder, I think, given how niche the field is. Uh, that's been, I guess, to my own personal advantage. I think one of the things that has enabled me to get onboarded as a, you know, hopefully reasonably competent researcher over the last year or so, or, you know, two, two years or so, um, has been that, you know, even before I was doing more bonafide research, I certainly still was doing a lot of, uh, personal research.

[01:23:56] Ethan Roland: Uh, you know, I was spending late nights writing PyTorch and, uh, watching Karpathy videos and whatnot. I think that this personal, almost hobbyist type desire to work on a thing, even outside of it being a job or outside of even dream of being published or, uh, recognized externally, but just doing it for its own sake, actually ended up being something that delivered or, uh, inculcated within me a lot of the fundamental skills that I, I use now.

[01:24:32] Ethan Roland: And so I think that like, uh, yeah, to, to cultivate like a, a desire and passion of a hobbyist is, is a good thing, uh, because it actually will end up being really, really useful. And then I-- the other thing I would say is that, yeah, be underambitious when it comes to trying to do things. and I think for a long time, I viewed myself as, oh, you know, I'm just a normal corporate data scientist.

[01:25:00] Ethan Roland: I do like my own little kind of thing and the little subset of the world that I do, but that's always for like exotic lab people, and that's not the world I live in. And, um, I think being involved with like the larger effective altruism community and like seeing some people that were more tangential to, to the labs, gave me this like evidence that like such a life was possible. And so then I said, "Oh, well, you know, I guess I can do this, and I guess I'll just like kinda try to stumble my way into it and talk to people that'll, you know, give me a position at doing some type of research." Um, and, you know, it worked out. But I think that, uh, one must be willing to, to like take risks and not think that those types of styles of work or doing that research is a thing that someone else with more credentials and more like bonafide whatever does.

[01:25:52] Ethan Roland: Uh, if you're smart and creative and have like worked on this yourself, uh, then there certainly is a chance that you can do it yourself. Just like don't, don't be the limiter on your own success, which I think is a, a thing that I, I often have forgotten historically, but am maybe getting a little bit better with over time. 

[01:26:08] Jacob Haimes: Awesome. And where, where do listeners go to follow your work, AE Studio's work? Uh, yeah, h-how do they stay in tune?

[01:26:18] Ethan Roland: Yeah. Um, I would say follow AE Studio on Twitter. Um, you can find me on Twitter, uh, @RolandTweats. Tweets with a T-W-E-A-T, uh, or at my website ethanroland.com 

[01:26:34] Jacob Haimes: Awesome. And then the standard final question that I ask is, uh, what is your favorite part about the work that you do 

[01:26:44] Ethan Roland: Um, my favorite part is seeing gigantic GPUs that cost millions of dollars and having the responsibility to run them properly. I have this, uh, this one thing I, I, I did a calculation over recently, and I said, "Oh my goodness, how many GPUs am I controlling right now, and what's their market value?" And I think that the number was like five million. And so I, I think I feel so incredibly privileged to be able to yield these or wield these enormously powerful machines and gain this type of experience that probably only a couple thousands of people in the world have the opportunity for. Uh, and just getting that kind of experience with these giant models and these giant computers and these giant clusters doing this really interesting and cutting-edge research, I think is just such an enormous privilege, and it's one that I feel very thankful for. And so I would say that that's my, my favorite part of the work, is just able to do something that I would have no, no possibility to do if I weren't in the position that I am in now


[01:28:00] Outro
---

[01:28:00] Jacob Haimes: Awesome. Well, yeah, Ethan, it was, it was great talking to you. Thank you so much for joining me.

[01:28:05] Jacob Haimes (Aside): Again, I'd like to give a huge shout-out to Ethan for taking the time to join me on the show. I really enjoyed the conversation. After we wrapped recording, Ethan mentioned that his favorite part wasn't as much getting into the deeply technical stuff, but discussing the impacts of technical solutions, and I would tend to agree with him.

[01:28:25] Jacob Haimes (Aside): If you got something out of this one, the best thing that you can do for the show is to share it with someone who you think would also get something out of it. So I guess, like, do that or something. Thanks. And lastly, although I don't have a Twitter account for Chiris FM or myself for that matter, you can follow Ethan @RolandTweets.

[01:28:47] Jacob Haimes (Aside): Anyways, guess I'll see you next time