[00:00:00] ​

[00:00:55] Jacob Haimes (ASIDE): This episode was recorded on May 27th, 2026, and asides were recorded on July 14th, 2026

[00:01:03] Jacob Haimes (ASIDE): So this one starts with a little bit of a humble brag, I guess. I was just heading back to my much less expensive hotel from a hotel conference venue when I passed someone who said, "I love your podcast." And I was like, "Oh my God, it's happening." No. I mean, I mean, just o- only a little bit. But, like, I definitely wasn't expecting it, and it did kinda make my day.

[00:01:29] Jacob Haimes (ASIDE): Anyways, that person was Suket Chan, who does a bunch of cool stuff. But when I later reached out asking if he'd like to be a guest, he instead recommended I have someone else on, and that's how I got in touch with today's guest, and how could I say no with the path that he's taken? It started out with a PhD at Caltech building hardware for the quantum internet, then moved on to Protocol Labs, a place he jokingly refers to as the biggest blockchain company you've never heard of, where he ran multiple research programs and dabbled in experimental macroeconomics via incentive structure design for cryptocurrencies.

[00:02:08] Jacob Haimes (ASIDE): Again, his words. After doing that for a few years, a fairly persuasive dinner convinced him to start Atlas Computing, which initially focused on formal methods, not like how you wear a tux, but math stuff. And more recently has decided to pivot to what he calls field strategy based on his belief that there's a bunch of important work which isn't moving forward, not because it's too difficult, but because it's not the right person's job yet.

[00:02:37] Jacob Haimes (ASIDE): In addition to a sidetrack into what formal methods even are, we talk about the motivations of Atlas's pivot, the resilience gap map, a tool that they've been working on to identify organization-shaped holes in developing societal resilience, and we even get into value alignment. And if you don't know what that means, don't worry.

[00:02:59] Jacob Haimes (ASIDE): We'll cover it. Before we start, I wanted to flag a couple things. First, we drop a ton of organization names throughout this one, so I didn't add all the chimes in, but rest assured they're in the show notes. Second, we talked for two and a half hours, so rather than subject you all to that, anyone who is interested can head over to the Kairos FM Patreon to get more of that sweet, sweet content.

[00:03:27] Jacob Haimes (ASIDE): Anyways, I'd now like to introduce Dr. Evan Miyazono

[00:03:31] Jacob Haimes: in one sentence, if Atlas Computing does exactly what it's trying to do over the next, say, two years, what will exist that doesn't currently exist?

[00:03:45] Evan Miyazono: My best case success scenario would be that there is a pretty reliable and replicated path to go from, is a thing that we want to see in the world," "Here is the fastest, cheapest, most reliable intervention that could be funded create that public good," when those problems that we're starting with are coordination failures or market failures or things that don't look like that one can sell or research solutions that one could discover, but are rather just people doing the things that you would expect them to do if people perfect coordination and awareness of what everyone else were doing and their skills and more trust, et cetera.

[00:04:34] Jacob Haimes: Awesome. Okay, cool.

[00:04:36] Evan Miyazono: Probably a bad one sentence.

[00:04:38] Jacob Haimes: well, you know,

[00:04:39] Evan Miyazono: Maybe rambly two sentences?

[00:04:41] Jacob Haimes: everyone's really bad at the one sentence thing, but it does make it more like a paragraph, which is nice.

[00:04:47] Jacob Haimes: Um

[00:04:47] Evan Miyazono: the truest one sentence is, like even shorter than a sentence. I want to manifest revolutionary coordination systems.

[00:04:55] Jacob Haimes: Okay. Okay. I think that's

[00:04:58] Evan Miyazono: to be helpful

[00:04:59] Jacob Haimes: too, well, too vague, but also, like, in the context of the last thing does help.


[00:05:03] From Quantum Internet to Blockchain
---

[00:05:03] Jacob Haimes: So b- before we get into, like, more about Atlas Computing, what you're currently doing, I want you to contextualize for me just a bit what your journey to that looked like. Because, um, based on just, you know, my research into, to what you've done, you know, your PhD finished up in 2017, and you were building hardware for concrete applications of quantum computing, which is just so far away from what you've just described.

[00:05:31] Jacob Haimes: So please just give me a little bit of context on how you got here

[00:05:36] Evan Miyazono: Uh, so I was-- always loved math and science, ended up in materials engineering for undergrad because I wanted to live in a Star Trek future and thought that materials were like a reasonable, It was-- It's a reasonable argument to have made that materials were the blocker on that because you can just look and see that there were so many new things that science had and technology could do in Star Trek. um, at the end of undergrad, became convinced I needed more fundamentals, applied to ex-exclusively physics and applied physics PhD programs, which I did not have sufficient background for. Still kind of hadn't understood what a prerequisite was or why they existed by that point. Um, and started working on a project with, uh, the-- The research I did specifically for my PhD was trying to build an optical quantum memory for a provably secure quantum internet.

[00:06:35] Evan Miyazono: I was very excited about doing something that would matter in the world, and I think there is reasonable argu- argument to be made that we got from, for a given set of parameters, 0.01% efficiency to like p- .2% efficiency, a 20X gain, and you only need five nines efficiency for this to be a useful engineering material.

[00:06:53] Evan Miyazono: Um, and so, uh, of cool physics, lots of experiments. Um, there's a version of this w- that says I spent the PhD doing simulations and data analysis and nanofabrication and optical characterization, but there's also a version of this has a much clearer story. That is, I was my advisor's first grad student.

[00:07:15] Evan Miyazono: I helped set up of protocols for how new students would join a lab and what the infrastructure looked like and all the relationships with other labs so that if we needed a piece that we didn't have an extra of, but every reasonable lab would have an extra of, I knew who to go ask and could get us that piece with zero shipping time, we would just give them the new one when ours, when the one I ordered arrived. So,

[00:07:38] Jacob Haimes: Hmm.

[00:07:39] Evan Miyazono: if y- from that lens, going and joining a startup that was also working on that was very value and ideologically based, trying to democratize access to internet and data storage, becoming m- a much cleaner story than being the first PhD hire at a company called Protocol Labs, which I often joke to people unfamiliar with blockchain as the biggest blockchain company you've never heard of. Um, and I set up basically all of the research infrastructure. This included, um, we did a lot of the interviewing, promoting, hiring of researchers, grant sponsorships, advisorships, and also running a special projects team and building out the team that built-- that ran the grants program. The CEO would consistently give incredibly ambitious, uh, tasks to me like, "Evan, build me the best grants program ever." And that meant that I had to go and figure out what that meant. And to me,

[00:08:41] Jacob Haimes: Mm-hmm.

[00:08:41] Evan Miyazono: I understood that to mean more, better science, faster and cheaper.

[00:08:45] Jacob Haimes: Okay

[00:08:46] Evan Miyazono: faster and cheaper, measurable. Um, more better science is incredibly subjective. You get into some, like,

[00:08:53] Jacob Haimes: Yeah

[00:08:54] Evan Miyazono: weird social choice theory or, like, economics-y type questions pretty quickly. And it wasn't long before we ended up with, like, are we sure grants are the right mechanism? And so for people who are less familiar with blockchain and companies and that ether-- like, that ecosystem, the part that got a lot of visibility and publicity was the part that was very, uh Well, I will claim that blockchains are a generally technology. There are many applications where you want a decentralized permissionless append-only ledger. However, they are more useful for speculation and fraud than anything else we've come up with so far

[00:09:40] Jacob Haimes: Yeah

[00:09:42] Evan Miyazono: And that kind of draws all the capital out of the room. But there are-- there is a corner of the blockchain space that is highly ideological, building good technology for all of the right reasons. uh, it's interesting because a lot of those relationships are showing back up with a lot of democratization and security around AI, ideological, technologically inclined people are less rare than-- or are more rare than you or I might like. And as a result, when there are relevant problems that are interesting, the same people do seem to keep showing up again and again.

[00:10:19] Jacob Haimes: Mm-hmm.

[00:10:20] Evan Miyazono: Anyway, um, so I did a lot of meta-science. I occasionally joke that I change fields roughly every, like, three to five years, and, um, I think that actually still kind of holding up. But while I was at Protocol Labs, there was a lot of supporting of, um, cryptography and decentralized systems and networking, and the ethos was very much one of how do we design the incentives to create the outcomes that we want from players in the system? So it was very much applied mechanism design. I also sometimes joke that I, uh, had a brief stint in experimental macroeconomics, because I don't know how else to describe designing the monetary flows and basically taxation structures for a cryptocurrency, which they call gas fees. Um,

[00:11:16] Jacob Haimes: Okay. Yeah, I honestly have no clue of anything about cry- like, the thing that I know about crypto is that I don't, I shouldn't invest in it personally

[00:11:25] Evan Miyazono: Generally a good, good rule of thumb. Um,

[00:11:28] Jacob Haimes: Yeah

[00:11:30] Evan Miyazono: yeah, I think there, there are some I would definitely recommend the 3Blue1Brown video on what does it mean to own a Bitcoin. I think

[00:11:42] Jacob Haimes: Okay

[00:11:42] Evan Miyazono: one of Grant's most viewed videos, and it does a really great job of explaining how it works, why it's secure, and it, it really is a novel innovation. the, uh... that, there was definitely a time where stock markets were probably equally rife with speculation and fraud before they were reasonably regulated and, um

[00:12:08] Jacob Haimes: one could argue

[00:12:11] Evan Miyazono: yes

[00:12:11] Jacob Haimes: That maybe, maybe things, well, it's who, who and what has shifted, but not wholly a different story.

[00:12:20] Evan Miyazono: very much.

[00:12:21] Jacob Haimes: Um

[00:12:22] Evan Miyazono: so anyway, I spent a lot of time doing mechanism design. The last two years at Protocol Labs ended up being, uh, chasing down the task of building tools that could fund public goods and commons like research or open source software, and reward in proportion to how impactful those were, the people who worked on them. And that led to five teams, each pursuing a different piece of the solution, and we spun out all of them when I announced in 2023, actually, like leading up to parental leave with my second child, um, starting a new org. Or that, that

[00:13:13] Jacob Haimes: That sounds

[00:13:14] Evan Miyazono: Atlas Computing zero stars to starting a new org on parental leave. Would not

[00:13:18] Jacob Haimes: Yeah, that sounds like take, take one of the most like hectic times in your life and then, uh, add another one.

[00:13:27] Evan Miyazono: the other one, yeah.

[00:13:28] Jacob Haimes: Yeah.


[00:13:29] Founding Atlas Computing
---

[00:13:29] Jacob Haimes: Um, and like how did that, how did that happen? What was the catalyst? I, I know that it's, it, at least how you're describing it here is like it's a continuation of the work that you were doing and you were sort of, uh, breaking up these.

[00:13:42] Jacob Haimes: But there are also, at least I'm assuming, that there's some sort of catalyst there as to like, "Oh, now I real- now I understand that w- you know, we need to split this, and, and this portion in particular is the part that I can play well."

[00:13:57] Evan Miyazono: Yeah. And I think that it is definitely the case that I have an easier time telling a story after the fact in hindsight than coming up with that or feeling like I have a coherent narrative in the moment.

[00:14:14] Evan Miyazono: Um, anyway, the... To answer your question, the In the moment, it felt like a massive split. wild running that post-facto venture studio type thing, um, called Network Goods. Uh, I was simultaneously supporting a, uh, special projects team which had, I think it ranged between three and seven researchers who were working on things that the CEO thought were...

[00:14:43] Evan Miyazono: The CEO and I thought were very important, but where there wasn't otherwise a home or a place for them, those things to get funded. one of those projects was, uh, David Dalrymple or Dava Dhad working on what was at the time the Open Agency Architecture, which became the ARIA Safeguarded AI program.

[00:15:05] Jacob Haimes: Okay

[00:15:06] Evan Miyazono: he and I agreed that it would make a lot of sense for him to run this as a research program, and in deciding between whether that should be a DARPA program or an ARIA program, landed on... We both agree he should go to ARIA, but that there should be a nonprofit that does a bunch of other stuff that he can't...

[00:15:26] Evan Miyazono: That he wouldn't be able to do reasonably in a science, like a government funding body. after a few months of looking, I was unable to find a person to run that and raised my hand of, "Well, I'm not qualified to in AI, but is this, like, if you think I could do it, um, like I would... I'm, I'm willing to quit this job and do that because this feels like the most important thing that could be done." you could imagine Dava Dhad in 2023 was pretty persuasive to the right person about the risks of AI. And, um, he and Andrew Critch actually sat me down over a dinner and said, "We think you should do this." I came back with, "I think, I think I can... only makes sense if I can, like, write checks backed by your reputation." And they were like, Um, that

[00:16:24] Jacob Haimes: that's gotta feel pretty good though too, right? Like, you're

[00:16:27] Evan Miyazono: Yeah

[00:16:28] Jacob Haimes: two people that, you know, at least really are part of the space and have become well-established and they're saying, you know, like, "No, you're the person that should be leading this effort." And what, what was that? What was your purpose?

[00:16:45] Jacob Haimes: What were those things that, uh, Davidad couldn't do necessarily, and, and like what was your focus with Atlas Computing to start?

[00:16:55] Evan Miyazono: Yeah, the starting focus was to take... To complement Davor D's approach of what is the world we want to build toward with safeguarded AI, where we have evidence-backed quantitative safety, uh, margins with AI systems backed by formal methods. He was building for a future that could exist and trying to work backwards to today, and the idea of Atlas was to start from today and work into the future.

[00:17:26] Evan Miyazono: What things would have to be true in order for safeguarded AI to be very successful? And the hypothesis that seemed untested at the time was, if safeguarded AI works, then there are a lot of people who have gotten very good at doing formal methods, because that was kind of a prerequisite in the tech tree for safeguard AI, and that wasn't the technology side of it.

[00:17:58] Evan Miyazono: It was really the user willingness and understanding of why formal methods was useful and, uh, why

[00:18:08] Jacob Haimes: Maybe this is a great time then to give me a lesson on why formal methods is useful, just to make sure, uh, 'cause I totally know it really, really well. Um, but, you know, just in case,


[00:18:25] Formal Methods Explained (ASIDE)
---

[00:18:25] Jacob Haimes (ASIDE): Evan did explain things, but as is the case when you're asked to explain a highly technical concept off the cuff, it was a bit windy and there was a tangent or two. So here's the streamlined version. Formal methods are a family of mathematical tools which allow us to reason about how correct a system is.

[00:18:45] Jacob Haimes (ASIDE): Formal verification is essentially an application of these tools to check whether a given program is correct. Before getting into formal verification, I think it's helpful to first establish the three translation points where things can go wrong on the road from having an intention in your head to having the thing in the real world.

[00:19:06] Jacob Haimes (ASIDE): Let's say I want a naturally lit place to work in my home. This is the intent or the thing in my head. After thinking about it a bit, I hire some contractors, and I tell them I want a two meter by three meter window on my south-facing wall. In providing these instructions, I've translated my intent to a specification.

[00:19:26] Jacob Haimes (ASIDE): The contractors get to work, and some indeterminate amount of time later, I have a window. They have translated my specification into an implementation. Finally, I can begin working next to my new window, translating the implementation into what actually happens in the world or reality. As I said earlier, each of these translation points represents a place where things could go wrong.

[00:19:54] Jacob Haimes (ASIDE): Formal verification closes the gap between specification and implementation by introducing a mathematical proof which demonstrates based on some fundamental model of the underlying hardware or programming language that you're working with, that it must meet all the characteristics in your specification always.

[00:20:17] Jacob Haimes (ASIDE): So if I had some sort of formal verification in this case, I would be able to say with extreme confidence that there was a two meter by three meter window on my south-facing wall. However, this doesn't address the gap between my intent and what I asked for. For example, in my specification, I assumed that my interpretation of window was the same as the contractors.

[00:20:43] Jacob Haimes (ASIDE): So it could just be a hole in the wall or a solid piece of glass that I can't open. This is the specification gap. It also doesn't address the gap between the implementation and what actually happens in the world. Perhaps my neighbor put in a giant billboard just outside my window, or I live in the North Pole and it's December.

[00:21:06] Jacob Haimes (ASIDE): This is the world model gap.

[00:21:09] Evan Miyazono: And so the, the first things we did at Atlas were trying to prototype, first, of this formal verification process, what does improvements to artificial intelligence make very easy? What remains very hard?

[00:21:24] Jacob Haimes: Okay

[00:21:24] Evan Miyazono: should people be paying attention? And so there aren't a lot of, um... I think that we had some of the earliest artifacts of could this look like. Um, and what I found initially surprising, but now find very unsurprising, is that a lot of the formal methods community is very skeptical of AI progress. Not just like normal person skeptical, but the way this should make sense is that the bitter lesson is kind of this, um, this now famous statement that, you know, if you just throw more data, it works better than hard coding in human understanding. And if you look at what happened to the field of AI, there are people who stuck with the symbolic reasoning approach, and that has high overlap with formal methods because that includes all of the automated theorem provers and SAT solvers and things like this, you're hard coding those things in. This is getting kind of in the weeds, but should be, um, there's a little bit of a selection bias of if you reject the bitter lesson upon coming across it in a very viscerally strong way, you are more likely to go into formal methods than basically any other field of computer science, you would be trying to take the symbolic approach to artificial intelligence, which looks like

[00:23:00] Jacob Haimes: Interesting

[00:23:00] Evan Miyazono: say what an AI should do, and then making the AI system do those things and prove to you that it is doing those things. So met with a, a non-trivial amount of resistance, have been... I, I think also did a surprisingly good job opening or, like, the more open minds and convincing more people. We ran a couple of events. Um, this included, like, a guaranteed safe AI summit, with a bunch of really high-profile folks. Um, and from that, actually, a, uh, a library for Lean that tries to formalize the under- undergraduate computer science curriculum, not just, like, variables, but also all of the, like, notion of complexity and things like this. Um,

[00:23:47] Jacob Haimes: Hmm

[00:23:47] Evan Miyazono: that traces back to that GSAI workshop. have, um... I guess by the time this airs, we'll, we'll, we'll have run two FMAI conferences. That's Formal Methods and AI, um, which really should've been FML. But, um, I think the FML joke would've probably been, uh, too tempting, and we, we named it once. FMAI is, like, rolls off the tongue. Um,

[00:24:13] Jacob Haimes: Yeah. It's not a NeurIPS situation

[00:24:17] Evan Miyazono: Yes. Um, or at least we landed where NeRFs ended up, uh, or like close, metaphorically to. Yeah

[00:24:25] Jacob Haimes: Yeah. Um

[00:24:28] Evan Miyazono: Uh, so that, that will be i- at SRI, um, next week for us, um,

[00:24:34] Jacob Haimes: Mm-hmm.

[00:24:36] Evan Miyazono: in the

[00:24:36] Jacob Haimes: But

[00:24:36] Evan Miyazono: for

[00:24:37] Jacob Haimes: in the past for others. Okay.

[00:24:38] Evan Miyazono: Yeah


[00:24:39] Narrower AI & Domain-Specific Safety
---

[00:24:39] Jacob Haimes: Um, and then I guess just, like, the last thing about sort of past Atlas that I want to touch on is this prior framing you had at least, uh... Like, I saw it, I think, in a different, um, uh, video or, or episode or some sort of recording of you. I think it was a talk actually.

[00:24:57] Jacob Haimes: But, um, trustworthy AI through domain-specific formal oversight. So, uh, breaking that down, that's like w- the way that we're going to get systems that we can actually be confident in the outputs and how they're performing is by defining domains and then using these, uh, formal methods to guarantee certain behaviors, uh, of w- within those domains.

[00:25:23] Jacob Haimes: And to me, that seems very... Like, my intuition, I guess, when I first heard about all these AI safety problems, um, was like, "Wait a second. Just build narrower systems, and then you can actually specify the constraints," right? And it sounds like it's very aligned with sort of what you're saying. So, um, do you still think that's sort of what AI safety needs?

[00:25:48] Jacob Haimes: Um, is that what you're thinking and this is just a different way to approach it? Or, like, how, how have priorities changed for you in the past, uh, you know, couple of years?

[00:26:01] Evan Miyazono: I think there are two interesting elements of that. One is that I do tend to think that it is probably safer to build narrower systems, and I think that this is something that people are speaking a lot about with tool AI. I think this is something that people wanted more, uh, before it seemed less I don't know. I don't, I don't wanna be too dark, but I, I somewhat give enough hope that people will truly limit general systems. I think that it would be much safer, and I do think that there's a version of this that looks very compelling where... and maybe it's worth cutting to this point. I think there's a version of this that's looks really compelling where we have siloed off various, like, extra strong capabilities, like knowledge of bio. very hard to draw a clean line between what is teaching biology or advancing biomedical research and what is, uh, creating bioweapons. And, like, I think that this is kind of just a fundamental aspect of The state of the world at the moment and, like, what technology can do, and this isn't maybe necessarily always going to be true, one could argue that the CDC should definitely have access to that without hitting annoying safeguards all the time. also, they probably shouldn't have access to also the ability to do super persuasion and cyber, uh, like, hardcore cyber red teaming. Siloing those thing, those kinds of capabilities off might be much more plausible with classifiers, but I do think that you do get a really nice safety property of not giving anyone the whole, uh, the whole incredibly powerful general system. I also think that, there's different notion of defining domains and narrowing to domains where you can define what safety means or what bad or, like, what disallowed would mean in different domains. And even if you don't constrain the general system, could still constrain d- have constrained definitions of safety in different, uh, applications or different regimes. And I think that this actually looks a lot like how we control or how, how we define, like, what is legal or what is illegal in different domains. Like, there are legal specialties around, like, cyber fraud, versus, like, IP theft versus, like,

[00:29:05] Jacob Haimes: Mm-hmm.

[00:29:06] Evan Miyazono: uh, property law and, like... I think that having... If we follow the model of how we've gotten security with people, we don't pick a person and say, like, "Everyone, this is the moral person. Be like that person." Um, we say, "Here are some fundamental ground rules. should follow those ground rules, and if you do, then you can do whatever you want. Like, optimize for whatever you want on top of that constraint." Um,

[00:29:34] Jacob Haimes: Mm-hmm.

[00:29:35] Evan Miyazono: there was a... Someone pointed out to me at some point, like, you can only ever optimize for one thing.

[00:29:40] Evan Miyazono: If you think you're optimizing for two things, then are optimizing for one function of two things. And-

[00:29:47] Jacob Haimes: As someone who've, who did a lot of work on multi-objective optimization, I fundamentally disagree. Uh, however, I understand what the intent is

[00:29:58] Evan Miyazono: Yeah. I, I think the I'd-- I would be curious to hear more of the disagreement, but maybe that's a topic for another time

[00:30:06] Jacob Haimes: you can... I- it's just you change what you're, you're looking for. You're no longer looking for, um, a single... Like, you're no longer optimizing one function. You're saying, "What's the best on this spectrum?" Like, what's the best in terms of, or what are the best potential options along this trade-off? Um, and then you, you can still optimize for that, what, it, what is called, like, the Pareto, uh, frontier, um, can be optimized for And I guess like When you pick a final, um, uh, like candidate, you have to be optimizing for one thing, but in the exploration phase, you can do a lot more

[00:30:58] Evan Miyazono: Yeah. Yeah. Okay. I think, I think that that's, that is the intent I wished to convey. Um,

[00:31:04] Jacob Haimes: Okay. I'm glad I could do that for you.

[00:31:06] Evan Miyazono: Yeah. Thank you. Um, I... The, the caveat to that, the, um, to, like, all of that is that if you... Like, you can pile on as many constraints as you want before you do that optimization, that's fine. And so, um, alignment feels a little bit...

[00:31:25] Evan Miyazono: I, I have many issues with the notion of alignment, but one of them is that, uh, I think we point to it as an optimization pressure that should just be the thing that we trust, and we don't do that with people at all. Um, we just have lots of constraints, and we have, like, very nuanced constraint boundaries that take the form of laws for people, and we could do that for AI systems.

[00:31:51] Evan Miyazono: We could do that with, um... Like, I think the thing we should try to have is Something like, uh, you could imagine for medicine, um, if we had a super intelligent system, there should be something that has, like, basically a... There should be a subsystem that has no agency and merely simulates a human given all of the information that might be relevant, you could have a different system separate from that, uses that to try to design therapies for a given goal, like treating a disease, longevity, um, diagnosing something. the, like, standalone simulation system, if it were good enough to be able to say, "We've customized one for each of these thousand, uh, in the trial, and not only will this, um, this therapy work on, like, 60% of them, but it will be this 60%. And these are the side effects, and these are who will experience those side effects." Like, this doesn't have to have any agency to be able to do that, uh, but it would be a huge breakthrough, and it doesn't necessarily improve our understanding with, like, in the same way that a human being able to do that would necessarily improve our understanding. you could have in that system a definition for what, uh, or it would be a, non-trivial exercise, but in that simulation, you could define what alive or dead looked like. Um, you could probably start carving out, uh, regimes of the state space that look like, this person healthy or not? And you don't have to define those perfectly. But

[00:33:55] Jacob Haimes: Mm-hmm

[00:33:56] Evan Miyazono: if you can kind of get a conservative bound on like, "As long as these numbers have these properties, we'll consider this to be a model of a heil- healthy person."

[00:34:08] Jacob Haimes: Mm-hmm.

[00:34:08] Evan Miyazono: And you can interrogate it to see, like, okay, well, like, here's something. does

[00:34:14] Jacob Haimes: What does that actually mean in this case?

[00:34:16] Evan Miyazono: Yeah. Does that look healthy to you? Um, what does it mean in this case? Yeah. And so, like, I think that there are things like this that would be very useful even for the, like, mid, like, mid-intermediate term centaur era for AI-accelerated science, like, the AI systems are answering all of the questions, it's still not clear how much humans can or should trust these systems. And, like, it falls kind of back to this question of how do you specify the properties and how do you get the AI systems to give you sufficient evidence to believe that their actions or decisions or outputs have those properties? And so it... Formal methods has ended up being, like, surprisingly fundamental, and this, like, the specification problem of how do you know you said what you intend to say, I mean, it shows up with legal contracts. It shows up with, um, like, building codes. It shows up with definitions of safety, uh, across, like, software and hardware and medicine and all these things. Um, it shows, like... It seems to be a surprisingly universal concept, and, I think that there's, like, two interpretations of the fact that a lot of cultures seem to have myths where people are cursed with exactly what they wish for. And one of them is, don't be greedy, don't ask for more things, and the other one is, you should be very, very careful what you ask for because it is very hard to specify that.

[00:35:53] Jacob Haimes: Yes.


[00:35:54] The Pivot to Field Strategy
---

[00:35:54] Jacob Haimes: Uh, the story that I've heard from, uh, like a conversation with Tsu, uh, Kit Yan, who is, uh, working with you now at Atlas Computing is, you know, he was talking with you and you were talking with some other people, and it essentially came up this idea of like, well, I can, I can spin up these, uh, AI safety organizations like pretty well.

[00:36:17] Jacob Haimes: I've done, I've done so a number of times. I think the word, the, the, the wording that I remembered was like, "In my sleep." I think you've confirmed that

[00:36:25] Evan Miyazono: I think,

[00:36:25] Jacob Haimes: you did not say this. Oh, okay

[00:36:28] Evan Miyazono: I think, I think this is, this is a thing that Zhu has said. Um, it is, it is flattering. It is definitely more effort than that. I would not characterize it as that. But it, it has been a thing where, um, I think I like on some level just have this, um, have like a strong inferiority complex and imposter syndrome since, um, like basically college, and like just want to be helpful, want to be useful.

[00:36:55] Evan Miyazono: And so, like people would consistently say like, "No one is... Like, someone needs to be on this because no-- like here is an important thing, and no one is doing it." um, like between trying to help drive product, uh, prototyping within Atlas when that was what we were doing, and like trying to like find a person to go own that thing, like the latter was far closer to what I had been doing for the last couple of years.

[00:37:28] Jacob Haimes: Gotcha. Okay. And so, uh, just

[00:37:31] Evan Miyazono: short.

[00:37:32] Jacob Haimes: through this a little bit more about like why this... Why, what this pivot is, uh, to you and why this pivot came about. Uh, 'cause I assume it wasn't only talking to Tsu, regardless of how charismatic he is

[00:37:49] Evan Miyazono: yeah

[00:37:50] Evan Miyazono: So about, um, Q3-ish last year, talking to Zhu, who kept prompting things along the lines of, "Evan, what is your unfair advantage?" And I realized that of the things that we had been doing around, um, prototyping four method stuff, that it was a lot easier to help other people with deep technical background and no founder experience go and do the thing, rather than me try to build up that experience and build a team and lead it. It... There was just some amount of strategy that needed to be done first and some amount of, like, logistics and, kind of support, really just activation energy barrier lowering.

[00:38:41] Jacob Haimes: Yeah. Yeah, for sure

[00:38:42] Evan Miyazono: is, uh, Jason Gross, uh, who now runs Theorem, was doing mech interp research on, uh, small neural networks to see if he could prove safety properties about them. And I told him, "I think that this has a lot less, uh- Well, first I found him, because he has basically the perfect background to run Theorem, uh, and do what Theorem is doing now. And I was like, "I think you are squandering your perfect background for doing AI-based like AI-first program synthesis, um, auto-formalization work. I think that now is the time. Will you join Atlas for six months? I can pay you to, like, some time doing whatever experiments you think are necessary to determine whether or not now is the time, and whether or not this is a viable direction, and also help out with these other things." After three months, he, um, and I had r- applied to, Coefficient for, uh, f- for funding extend that work, and he decided that it was better done as a for-profit, and I was supportive of this decision.

[00:39:46] Evan Miyazono: He resigned on Thursday to incorporate Friday to get the non-dilutive first check on Monday. Um,

[00:39:52] Jacob Haimes: Gotcha

[00:39:53] Evan Miyazono: so I... A lot of people point to Theorem as like, this is a great team. This was like... It was really the first company doing AI-first auto-formalization. A lot of people were doing like, "Oh, yeah, we'll, um, like, be humans using AI tools to do formal verification." And they were like, "No, we'll, we'll make the models do it." Um, I feel like I played a non-trivial role in that. And so, like, things like Theorem, things like, um, Mehmet and Arundyl and the tamper-responsive heat sinks, um, I, I realized that I should... Like, my superpower is much closer to helping accelerate other people, lower the barrier to entry, and get people who otherwise hadn't and maybe wouldn't found a thing go found a thing because it looks much more like a job description.

[00:40:40] Evan Miyazono: And so the,

[00:40:42] Jacob Haimes: Mm-hmm

[00:40:42] Evan Miyazono: the pivot was an explicit decision of move all of the formal verification prototyping out of Atlas. And so that meant, um, sending Alex Rademacher to Renaissance Philanthropy. He's now the director of CSLib there. Um, it involved, uh, my CTO at the time departing and, um, the new direction has been trying to figure out we can use this approach and ideally building a team of people that I can train to take on this approach of making it easier for other people to take ownership for an intervention that has been well specified. kind of generalized from start new organizations to actually just figure out what the best thing is. I think there are many instances where The thing that is missing is actually some fairly minimal coordination. Maybe this is a contract, maybe this is a collaboration, this is a 501 which a lot of people seem to have not come across as a legal entity. C6s are like what the car companies use to coordinate around seatbelt safety standards. it helps avoid the, like, collusion, antitrust regulatory concerns.

[00:42:08] Jacob Haimes: Okay

[00:42:08] Evan Miyazono: think that there are, uh, many applications where, like... the Frontier Model Forum is an example of a 501c6, and that's much more on... Uh, that seems to be fairly narrowly scoped, but I think that there are C6s in the AI space that would be very useful for other applications as well.

[00:42:27] Evan Miyazono: So the question becomes, can we go from here is a concern to, um... And, like, where I've landed specifically is there are foreseeable AI capabilities that break things out in the world, um, or should be expected to break things out in the world. there things that we can do now that would make the world robust to those capabilities showing up?

[00:42:54] Jacob Haimes: Right

[00:42:54] Evan Miyazono: is the smallest possible thing we can do now that will quickly and deterministically get us closer to that world?

[00:43:04] Jacob Haimes: Yeah. And so-- And then you're saying, okay, so now someone has specified this. Um, they, uh, in theory then would, would come to you, uh, or your organization and be like, "Okay." Uh, or maybe your organization would, would do this yourselves, right? Uh, l-leave the how, how we got this idea, uh, as a black box for now.

[00:43:26] Jacob Haimes: You have an idea now. You believe that it is going to do that, is going to get you closer to the robust, uh, version of the world to some AI, uh, centric, uh, AI-related harm. Um, or maybe not even AI-related necessarily. Um, but you further scope it, like define a couple things, recruit the right leader, give them some resources, and then send them off.

[00:44:02] Jacob Haimes: Is that correct?

[00:44:06] Evan Miyazono: I think I, I will add a few things.

[00:44:09] Jacob Haimes: Okay. Yeah, please do

[00:44:10] Evan Miyazono: Um, yeah. So, m- the, the process that I am arguing for is that if you have the ability to say, is what the future should look like," I'm... a lot of places where, uh, it's very ambiguous what looks like. Uh, alignment would be one of this, adversarial robustness would be one of this, online learning is another phrase that people use to describe a problem that falls in this category. There are a lot of things where what should we be driving towards is a big open question. I think there are other things where, nope, we, we, like, kinda know that, like, there should be a tool where someone can, like, sit down at a computer and run it on their software systems, and it says, "Hey, here are some things that are, like, unintuitive about what this software that's running here seems to do. this right?" And just elicits a specification from that user, because what should the system do is probably in the person's head more than it is on the computer. If it were in the computer, then there would be no bugs. the bug is in fact the difference between what the person thinks it should do and what it actually does. So elicit the spec from the person. a tool that does that. This is, like, a very clear thing that would be very useful. It is a very hard problem, very hard to get funding for this problem. You'd

[00:45:36] Jacob Haimes: Yeah

[00:45:36] Evan Miyazono: a lot of, like, weird expertise. am helping a, um, a f- FRO founder at Convergent build this as an FRO. Um, I, like- The-- I think the right process is if you have a clear vision for what it should do Go and iterate with all of the relevant experts that you can talk to of, is this vision possible? And then go and also iterate, maybe simultaneously, with a bunch of potential stakeholders and say, like, "Hey, if this existed, would you use it? Does this actually solve a problem for you?" And this is basically establishing product market fit for a thing that you

[00:46:23] Jacob Haimes: Yeah

[00:46:23] Evan Miyazono: becoming increasingly convinced is real. Um, and I claim that if you can do those things and you can get the experts to point to one paper and say, "This is possible," and the stakeholder u- potential user customers point to the same paper and say, "Yes, this would solve a problem for me," then you have massively de-risked this. And then you can go and figure out a lot of the, like, kind of normal strategic aspects of like, okay, what do the six-month milestones look like? What, what does the team structure look like to do this? what are the job descriptions for all those members of the team in order to do this? And the more you can structure that, the more recruiting someone to take the leadership roles in that like applying for a job. And so I'm trying to pitch to various, uh, philanthropic funders is, hey, you worry about topics. can generate a bunch of ideas of things that should be done there for any one of those ideas, go through, de-risk it with experts and potential stakeholders, and then bring a team and have a funding-ready proposal with people conditionally committed to, yes, if the money shows up, we will run on, run with this for six months and demonstrate by hitting these milestones that we can do this, and you re-up and you get the impact on track.

[00:47:55] Evan Miyazono: And so there's... There-- It is very easy, would say, make it look like you are doing this and do it badly don't get the right experts, or you keep things too ambiguous, or you don't, like, really push to kill the idea if it is in fact a bad idea. Or y- the six-month milestones don't quite correspond with what success actually looks like. But I think that if this is done well, then you can really deterministically get to something that should, with very high confidence, move the needle. I don't know how useful it is for me to keep giving examples of

[00:48:33] Jacob Haimes: Okay.

[00:48:34] Evan Miyazono: things or, like, potential future

[00:48:35] Jacob Haimes: No, I think that's, that's helpful. So then just to like, I guess, move it to the next spot though. So you say doing this work, um, gets you to a point where you now essentially have, uh, either one or a series of job descriptions basically. Uh, and maybe these are more task-oriented than your traditional job descriptions as like, uh, do this, um, like get to this milestone, then get to this milestone as opposed to, um, you know, you're doing this every day kind of thing.

[00:49:08] Jacob Haimes: But still you have this like, this is the task, um, and now you're recruiting for that task. Um What does that look like? How are you doing that? Are you getting those people involved in that first portion as well, like when you were talking with the experts? And, um, what... Yeah, I guess maybe what is special or different about how you approach this?

[00:49:38] Evan Miyazono: Yeah, I would say that one of-- topic-- one good area of skepticism people will raise when they have a lot of experience trying to recruit founders, maybe because they run an incubator or they're a VC or something, is that People will want really high ownership. You will need people who can navigate uncertainty. Things along these lines that I think are very true if you don't have product market fit yet. Or if you are working Excuse me. Uh, or it's also true if you are looking for something needs a total addressable market that is massive or needs to be able to return the fund. I think there are a lot of things that are much more narrowly scoped, um, smaller, that are still really impactful, that just don't get a lot of attention or airtime because they're not the sexy research problem, or they're not the bullet solution. Um, I'm a big advocate for defense in depth or, like, the Swiss cheese effect, where, like, you have lots of different defenses you hope... You know that each one has holes, um, but you hope that anything gets through one layer won't get through the next. Um, in practice, if I have a job description for a founder or, like, a profile, be iterating that with the experts that I talk to.

[00:51:16] Evan Miyazono: Is it possible to find a person? Who would be the person that would most increase your confidence that this effort will succeed if they signed on?

[00:51:24] Jacob Haimes: Mm-hmm.

[00:51:25] Evan Miyazono: And then when you go talk to the-- When I go talk to those people, I will say things like, "Through pure serendipity through s-some combination of serendipity and your interests, you have become one of the top five people in the world to be most qualified to do this thing. And N-- Like, I don't, I don't say this to everyone. I say this to basically no one because it is almost never true, it is possible that you doing this thing could be critical for the stability of civilization, and it looks shockingly close to the things that you've been already doing and probably already thinking about and caring about. And there is this potential intervention. There are these advisors. There are these stakeholders. If you can deliver on this thing for these stakeholders, it will leave the world in a much better place. it take for you to say yes to this?


[00:52:27] Value Alignment vs. Skill
---

[00:52:27] Jacob Haimes: Okay. And then, uh, that, that leads to another question that I, I do wanna to spend some time on 'cause it, uh, I think it's really relevant to maybe some of the practices or, or, or, or, uh,

[00:52:46] Jacob Haimes: like leanings, I guess, of the, of some of the AI safety space that, like, I don't like as much. Um, and it's about, um, sort of like what your expectation is of these new hires. Um, not just in terms of, you know, essentially they're, they're the best in those things that have been specified and they have that mix of those, um, traits and those skills, um, 'cause that's obviously important.

[00:53:19] Jacob Haimes: Um, but then also, like, is there anything else, things that are more, um... I, I think the phrase that's often used is, like, value alignment, for example, or, um, personality traits or, or things like that, that you are filtering for as well, um, either implicitly or explicitly in, like, how it's framed and, and how this process comes about

[00:53:48] Evan Miyazono: I think that I'm definitely implicitly filtering for No, maybe, maybe not that strong. I, I think on some level, I think that this works best if people do believe that it is very worth planning for contingency where AI makes things go really weird really soon.

[00:54:09] Jacob Haimes: Hmm

[00:54:10] Evan Miyazono: Um, but also, I think it's fine if they're just contingency planning.

[00:54:15] Evan Miyazono: Like, I tend to disagree most with people who are highly certain about how things will go,

[00:54:21] Jacob Haimes: Mm-hmm

[00:54:22] Evan Miyazono: and I basically never disagree that it is worth planning for the possibility that it might go that particular way. And so, um, I am, I'm very happy to think about, like, what governance structures should look like if AI systems end up aligned by default and for, like, a reasonable definition of alignment, and they decide to continue to serve humans, um, and not ever have, like, a treacherous twist moment or anything like that.

[00:54:57] Evan Miyazono: But also, it would be really good to be able to detect, um, and predict and, like, something like a treacherous twist, assuming that it is going to happen, and also prepare for concentration of power and epistemic decay and a possible, um, wildfire pandemic and, or stealth pandemic or, like, all of these things. And so I, I tend to be, um... I tend to be less opinionated about value alignment. think that there are a lot of things that one can believe, and it is very easy, I think, uh, I think- Trying to remember. It's like there are Yeah, fewer than two to the 23rd people in the, um... Is that right? No.

[00:55:50] Jacob Haimes: I don't know. That sounds like a big number

[00:55:51] Evan Miyazono: 32. Um there are fewer than two to the 33 people, and so you can only ask 22 yes or no questions where there are some people on both sides before some of the buckets start having no people. That is not a lot of questions.

[00:56:09] Jacob Haimes: Okay. Okay

[00:56:10] Evan Miyazono: if you're, if you're looking for value alignment, and you want a specific flavor of value alignment, and also certain skills and availability and, um, a bunch of other things, I think you just carve down your space way too quickly. And I think it's, I think it's unnecessary because if, if I have a user who's saying, "I want-" Uh, so concrete example, think that we, instead of reaching for like OAuth-based identification of people and using this for cryptographic identity and security measures anything from like-- We'll use a simple example of like who's ordering these DNA sequences?

[00:56:56] Evan Miyazono: Be really good to know, like link that to a person. Um- One path is to use OAuth. One path is to use mobile driver's licenses or passports. we should be using banking KYC. And I think that if you could say, if you could get to people at operating systems or browsers and financial, financial service providers and AI providers to say, like, "Okay, we'll all agree to use this standard, expose these APIs.

[00:57:25] Evan Miyazono: This is how it will work." I think that if you have amount of that sketched out and it's like, okay, I need someone to deliver this for these people because it will make things better. I don't care what those people believe.

[00:57:39] Jacob Haimes: Yeah

[00:57:40] Evan Miyazono: Like, they, they don't need to be able to navi... Having value alignment is really good if you're pushing through uncertainty on really long timescales, and you can't define a priori what success would look like.

[00:57:52] Evan Miyazono: If we have users who are already, like, basically conditionally committed to a thing existing, just say, like, "Nope, that's not what I want." And you just need someone who will be like, "Okay, what do you want?"

[00:58:02] Jacob Haimes: Yeah. Yeah

[00:58:03] Evan Miyazono: a lot, of different ideologies will lead you to, okay, what do you want? Um, people who are strongly financially incentivized would be one group where, like, if there is money in it, then, like, I, I don't think that I would balk from hiring, like, a true mercenary to solve a problem if that were the

[00:58:26] Jacob Haimes: The incentives, yeah.

[00:58:27] Evan Miyazono: Yeah.

[00:58:29] Jacob Haimes: Okay, that makes sense

[00:58:30] Evan Miyazono: I think that that doesn't work for my role in this, and I'm trying to hire other people to do my role, which I'm calling field strategist. And those folks, I think, need to be able to make the claim that when we start from, like, okay, here's a problem area people care about. Epistemic risk. How good is our ability to receive trustworthy information and make good decisions with it, like, based on it? Lots of AI capabilities you could foresee would make that wor- worse or harder. Lots of technologies that have been proposed that could be useful. I would like to, for any one of those, be able to take a proposal, proposed intervention and a proposed conditionally committed team and say, "This is, by construction, the best possible intervention we have come across in all of our conversations. Anytime anyone expressed skepticism, we asked, 'Can you think of a better way to solve this problem?' Because we would love a better way to solve this problem. We agree with your skepticism. This does sound unlikely. What's a better way to do it?" And If we end up with something at the end of this process have confidence in, will be we ditched every other alternative. But if there's some incentive at the end, whatever that is, it kind of muddies our ability, my ability to stand with confidence and say, "This is the right thing." If the field strategist... If, if we get a kickback based on how many of these we create, then that's an incentive to make these smaller or quicker or, um, if it's proportional to the size, we take like 2% of the intervention, then it's an incentive to have more expensive ones. If the field strategist wants to go found the thing, are going to be subliminally tracking toward a thing that they would most enjoy or most be able to run. And I think this is a problem with a lot of the, like, we'll find a generalist founder to define the problem and go solve it, or we'll fund an incubator or an accelerator to go build solution to this. They're already pre-assuming the form of the solution as a startup or a product or something that the, the founder in-investigating this problem can run. I think by, like, creating this abstraction barrier of like,

[01:00:54] Jacob Haimes: Mm-hmm

[01:00:55] Evan Miyazono: we will do the strategy, we'll figure out what the best solution is, and we will have no ties to what that solution is until we've settled on this is the one that we think is best, and then we will find the people to do it. I think this dramatically opens the number of things that you can do that suddenly would never make sense for a research team or, um, a, an incubator or a VC or anything like that.

[01:01:23] Jacob Haimes: Yes. Okay. So just to re- go back to where we were, um, the, the flip side of this, um, like maybe idealized version that you've presented is, okay, well, we, we don't have these incentives, uh, in that like we, we need to be able to run it, uh, or, or we want to be able to be, um, you know, meeting these, these targets.

[01:01:49] Jacob Haimes: And so it allows for more exploration and to find the theoretically best, um, way to execute on this. And that's like one framing, but then the other framing could, uh, and I think it like is reasonable to consider, okay, but that like the other sort of version is that the specs don't survive contact with reality.

[01:02:16] Jacob Haimes: Um, so the, the strategist hands over the specs to the operators. The operators are then the ones that are going to have to live with the consequences of those specs, and you don't ever see the, uh, joining of those. And so that can lead to, um- You know, like a, a waste of the scoping work or, or to that extent.

[01:02:42] Jacob Haimes: Does that make sense? It's sort of like

[01:02:44] Evan Miyazono: Yeah.

[01:02:44] Jacob Haimes: consulting trap

[01:02:45] Evan Miyazono: Is that the consulting trap? I feel like when I think of the consulting trap, it's like also the fact that often there is no incentive to like implement all the things that McKinsey gave you because the board wanted you to hire McKinsey to ask, like to answer some questions for you. Um, I, I don't know this with very high confidence, so This is more like my My hermeneutics of words around, like, challenges related to consulting. But, um, I think, I think that you're highlighting a true concern, it is very much the case that as the world starts moving faster and faster, the ability to go from, "Here's a predicted c- AI capability. Here's an intervention. We built consensus around this intervention. We have someone who will execute on it. Is this still relevant? How long did this whole thing take?" I think trying to make this whole process take as little time as possible is actually very important, and also trying to scope the interventions to be very narrow things. not like multi-year research projects, but something closer to, like, an advanced market commitment or, um, up some API access to various things across various applications. Um, ideally, things that would take no more than a few months to land. I think that something... I, well, maybe either land or to start providing some value.

[01:04:20] Evan Miyazono: If it is not useful, then you can scrap this. The goal should be to have something where the incentives are aligned what each person in this has and also that with what is better for scale coordination. Um, like what improves the like... How do we... The question becomes, how do you take people with their, uh, like with their jobs and their incentives and their OKRs and connect those to the most useful thing that could be done for humanity to be prepared for powerful AI capabilities? And, like, of the things that I expect, like, I expect field strategists to come from a lot of different places a lot of different backgrounds, one thing that I plan to make sure everyone develops a really strong understanding of is how What mechanisms exist out there to adjust incentives or to bypass bureaucracies or, um, better model what it looks like for a large organization that you may not have familiarity with to respond to an opportunity or a challenge or something like this, where this expertise all exists in people, but it rarely all exists in the same person across, like, government and, um, which would include, like, Congress and defense and the intelligence community, and also, like, Fortune fi-- like, Fortune 50 companies and, like, Fortune 200 lower-end companies and, like, um, tech companies who fancy themselves different and VC um, the, like, philanthropic safety space and the international, like, governance space. There are lots of different Another thing that I like flagging is that there are lots of different quora who might, or quorums, I don't actually know which one is the correct plural, but, um, you can often identify a quorum that would be sufficient to, if they agreed, solve the problem or at least adopt a solution. The one you are first to identify may not be the easiest one to convince. So how do you find all of the other ones then sort them based on your ability to reach them and convince them to do this thing? I expect it'll be a lot of... I expect field strategy work to do a lot of, uh, convincing of people, but I also frequently repeat that as intelligence gets expensive, agreements and relation...

[01:07:25] Evan Miyazono: Sorry, as intelligence gets cheap, or relationships or consensus get expensive. And so this is something that I think will get increasingly hard, increasingly valuable, um, and I guess this is a very long-winded answer to happens if the intervention itself doesn't survive contact with reality Probably should have just stopped after the intervention as quick as possible and Uh,

[01:07:57] Jacob Haimes: And s- and scoped as possible as well. So you're s- it's, it's

[01:08:01] Evan Miyazono: And try and make it provide value as quickly as possible so that at least people...

[01:08:06] Jacob Haimes: steps

[01:08:07] Evan Miyazono: Yeah. If, if your users aren't saying like, "Great, this is helpful, but I also want this," um

[01:08:21] Jacob Haimes: Okay

[01:08:21] Evan Miyazono: Like, I, I think that, like, having that kind of a pull gets very easy you talk to founders who are like, "Yes, we found pr- product market fit, and we know we did because suddenly, like, the number of feature requests was far longer than the number of things that we had on our roadmap." This is the kind of, uh, the kind of phenomena that I would hope for for everyone executing on of these interventions.

[01:08:50] Jacob Haimes: Okay. And then the next sort of large area that I... is really worth spending time on, I think, is how you got to and how you get to those interventions. Um, so, like, from what I understand, uh, the blueprint that you're working from, um, and, like, h- the thing that defines the problems that you will help build organizations around is a resilience gap map.

[01:09:16] 


[01:09:16] The Resilience Gap Map
---

[01:09:17] Jacob Haimes (ASIDE): All right, so real quick, let's talk about what this is since Evan and I really just jump right into it. Atlas has a publicly accessible Google sheet called the AI Resilience Gap Map, which lists a collection of what Evan calls organization-shaped holes, which would be good to address in the near future.

[01:09:35] Jacob Haimes (ASIDE): In a post on the Atlas blog, he explicitly flags that the list isn't a blueprint, rigorous research, or a finalized output. It's the to-do list that Atlas has put together. To build this, he asked a bunch of AI safety organization leaders, what's something that someone should do that no one is? And then he organized it into categorizations seen in the sheet today, which led to additional gaps which are currently seen in the sheet as well.

[01:10:05] Jacob Haimes (ASIDE): The gaps in the list, although interesting, aren't what actually matters here. Instead, it's the process of filling out the other columns on the table, like theory of change, solution direction, initial de-risking, and minimum necessary skills that make up the real value of this effort. The idea is that by filling out a row, you've developed a scoped, de-risked, hand-off ready organization that can go and, you know, make societal resilience better.

[01:10:39] Jacob Haimes: And, um, the idea is this has, like, a list of high leverage, underserved opportunities, um, that you think are worth having, conducting this approach on, right? Um, how did you cr- how did you go about creating that? Uh, like, who decided what gets on the lists, and how?

[01:11:04] Evan Miyazono: I would say that the resilience gap map is definitely more of a starting point than kind of a blueprint. I tend to be pretty, pretty precise. well, maybe, maybe inconsistently precise, but I try to be precise and, um, would probably say that instead of a blueprint being this thing where all of the pieces need to, need to happen and they all fit together neatly. It feels more like a Feels more like a, a maintenance backlog, which is actually the, the title of the, um, the blog post where I introduced that. Um, I called it Civilization's Maintenance Backlog, and I think that there are, like, a lot of things that... There are a lot of things on there that aren't a good fit for field strategy, admittedly. I think that... Well, that, that list actually was a list that I curated from talking to experts for about a quarter of, like, random conversations with, of people, l- heads of most of the, um, the AI safety organizations that, uh, you and listeners would've come across, saying, are things that someone should do no one is doing?" And wrote all those down. Uh, I think that there might be a, a private version somewhere where I actually have, like, who said which one, but did not want to, uh, want to leak, leak that information because some of the things, uh, up being a little outside the Overton window. Um, and it was really meant to just be, like, I would put down a lot of these, and I am in the process of iterating on something that looks more like how...

[01:12:58] Evan Miyazono: That also includes how good a fit for field strategy a given direction is and why.

[01:13:05] Jacob Haimes: Okay

[01:13:05] Evan Miyazono: There are kind of three directions that I'm pushing on currently. One is an advanced market commitment for a formally verified AI inference and training stack from cloud service providers. I think that there will be demand for that, and pulling that demand signal earlier in time will be very useful. And as I'm working my way through conversations with people in those organizations, um, the... Another one is this notion of, um, exposing KYC for a bunch of applications. I think that as we have more agents, it'll be really useful for your agents to know it's talking to my agent. We don't really have a good way of doing that right now. Um, the, uh... There are a couple of other things. One of them is a, um, trying to figure out exactly how much value a new C6 for coordination on model specs, uh, would be

[01:14:13] Jacob Haimes: Mm-hmm

[01:14:14] Evan Miyazono: I've flirted with some possible options around, like one around economics, around, um a couple around ep- epistemics, and most of the questions Most of what has put things higher on my list personally are, does it feel like the incentives should be aligned?

[01:14:37] Evan Miyazono: And there is, like, one quick trick that seems like it just makes people go from, "This is not my job. This does not make sense for me to help with, but I like that you're trying to do it," to like, "Oh, yeah, I could totally do that. sounds like a great idea." Um, if I, if my intuition says that, like, the intervention m- shifts from the former category to the latter, then it's on my shortlist and, like, tie to some catastrophic harm reduction might be a little bit more tenuous.

[01:15:14] Evan Miyazono: The advance market commitment on cloud inference actually started back with the FlexHag work because it w- the question became, does this get adopted because of policy requirement, or does it get adopted because is financially useful to the companies? I thought the latter was more high certainty and therefore finding the right incentive to drive a lot of the investment in the security properties you would want for treaty verification would be really good. Turns out that, that, those, um, those assurances are also kind of the assurances that an operator wants against agents running amok on their hardware. So, um, this f- this felt like a similar alignment of incentives, but is a, like, good question of who's deciding what's on the list and how. I would be happy to put anything on the list.

[01:16:09] Evan Miyazono: It's actually, um, anyone on the web can comment, so if people just, like, me with like, "Hey, add these 10 things to the list," I will add them. There's structure on the list. that structure was entirely created post facto for me to try and sort these things. I do think it is useful to try to, um, organize in a way that makes it more, uh, more comprehensive, so you can elicit some number of things that would be gaps. there were... Like, as I was adding structure, there were, I think, like some-somewhere b- probably between five and 10 things where I was like, "Oh, well, if people are talking about, like, these things, then it would also make sense to have these other scenarios also listed where we try to mitigate those."

[01:16:59] Jacob Haimes: Mm-hmm.

[01:17:00] Evan Miyazono: I have a few other, like, deeper dives. is into cybersecurity, and so there's, like, another cybersecurity gap map with, like, two dozen, I think, uh, like, specific interventions. I have another one that is, like, in progress for epistemic stuff. But in terms of... That, that's

[01:17:18] Jacob Haimes: a

[01:17:18] Evan Miyazono: the list. That's not which ones am I working on.

[01:17:21] Jacob Haimes: Right. Uh, I guess just like sort of digging deeper into that, you, you say on the, um, one of the tabs I think of, of the gap map that like we're striving for progress, not consensus. Um, and while that may be like a... Uh, there are some things that are good about that, but then the other thing is that gives you and also the people who are funding you and the people who you talk to a lot of, of power, right?

[01:17:49] Jacob Haimes: 'Cause it's not ever-- I mean, even the people listening to this, um, conversation, I mean, that's a small, small subset of, uh, everyone. So even like majority of people don't know that this exists, uh, to contribute to it, let alone have, you know, the time and, and, and whatever to do so. So maybe, you know, like some people are okay with this idea, but others probably aren't.

[01:18:15] Jacob Haimes: And I don't know, the, the framing that it reminded me of was Zuckerberg's like, "Move fast and break things." It's like, it's not, it's not as gross, uh, like by any means. It's much more tactful. Uh, but it still does have this sort of like, we, we wanna get things done and it, uh, doesn't matter who agrees with us as much.

[01:18:39] Evan Miyazono: Yeah, and I think that this also comes up a little bit with the fact that I, at least in, like, field strategy job descriptions, have said that I am focusing on, uh, either candidates or... Like, both candidates and potential interventions in the United States,

[01:18:58] Jacob Haimes: Mm-hmm.

[01:18:58] Evan Miyazono: with the expectation being that, that that is higher leverage solutions that work in the US are more likely to propagate, um, things like this. I also feel, like, a little bit icky about this. Um, I think that there's, like, a lot of really hard, like, fundamental, um, like ethical philosophic, like, questions related to to weigh in on this. Um, I guess defense in depth is, like, a, um, a reasonable... Like, multi-layered solutions, um, is a reasonable approach.

[01:19:34] Evan Miyazono: I think that the The very, like, raw, straightforward answer of like How does this not just prioritize the people who to me, who like come across this list, who comment on it? It does prioritize them. Uh, I think that this is not the failure mode it may initially seem to be. Um, I want to I try to take a perspective that the kinds of interventions should be globally useful to nearly all humans. I think that There is no way to be confident that... I, I think it would be insane to try to ma- like, only do things that are useful to all humans or that everyone would agree with. I don't know... I'm, I'm pretty sure there isn't a good way to define consensus over all of the people. I, I have this thought experiment that I love of, like, assume technical alignment got solved tomorrow. We have a box. You could put a person in it, and, when they come out, the AI system that was connected to that box will come up aligned to them. What do you do with this box? And, like, get a lot of answers. Um, well, I think one of the better answers was, well, put, you put every human into the box and you have the aligned AI systems that come out and debate each other. It's like, okay, well, what if it takes 80 hours for the person... Like, what if the person has to spend 80 hours in the box or, like, 800 hours in the box and it costs $10 million to do it?

[01:21:15] Evan Miyazono: Then what do you do? Like, there are a lot of different consensus mechanisms and, or, like, voting preference elicitation mechanisms. as soon as you go from, "I have preference over these K options," to, like, "These N people have preference over these K options," if you wanna say now, like, the group of N people have these preferences, you have just done a lossy compression by a factor of

[01:21:42] Jacob Haimes: Sure

[01:21:43] Evan Miyazono: y- there are many ways you could do that.

[01:21:44] Evan Miyazono: You could weight by... could weight everyone equally. You could weight everyone in proportion to how much they care about the thing and give them voice points. You could give them, um, like, the ability to rank things, arrows and possibility theorem, blah, blah, blah. Um, you could, um,

[01:22:01] Jacob Haimes: I think the one that I'm most excited about personally, and this, this has come from a couple different, um, things like, uh, that I, that I've done work in. But, um, the, the way it was described to me best was actually, uh, from an advisor from one of the other groups that I work with, um, named Jan Krawczuk.

[01:22:22] Jacob Haimes: And he said, um, "We're not aiming for consensus, we're aiming for consent." And this is operationalized by, um, consent-based voting, um, in some ways, where instead of voting for who you prefer, you say, "What am I okay with acce- like, w- who, who are the people that I am okay with being, uh, in charge?" For example, if you were voting for something.

[01:22:57] Jacob Haimes: And I think that having... introducing this, um, it's, it's not about what I would prefer, it's, it's about, you know, what I'm okay with, and then finding the thing that most, the most people are okay with, um,

[01:23:17] Evan Miyazono: This is

[01:23:17] Jacob Haimes: is really helpful. Sorry, go ahead.

[01:23:20] Evan Miyazono: The, the, the technical term in, like, voting, uh, I guess voting theory, I guess, um, is called

[01:23:26] Jacob Haimes: Yeah.

[01:23:27] Evan Miyazono: voting.

[01:23:28] Jacob Haimes: Yes, that's what

[01:23:28] Evan Miyazono: this.

[01:23:29] Jacob Haimes: I was forgetting the word

[01:23:29] Evan Miyazono: Uh, yeah. Um, spent a non-trivial amount of time looking at, like, different voting structures. Um, because, like, if you're trying to define, it's like going way back to, uh, like, the best science, uh, like more better science, faster and cheaper.

[01:23:44] Evan Miyazono: Like, you basically have to elicit preferences over that.

[01:23:48] Jacob Haimes: Yeah

[01:23:49] Evan Miyazono: and like, you have to combine those preferences. Um,

[01:23:52] Jacob Haimes: So what are your thoughts about approval voting versus the preference vo- I mean, just

[01:23:56] Evan Miyazono: like a, I like approval voting. I, I also have, like, a random, a, a random soft spot in my heart for sortition, which is election by lottery. Um, I- I think Of the problems that I'm looking at, I'm really trying to find things where, like AI will be able to do X. The... It is easy to imagine X resulting in Y would be something that nearly all humans would point to and say, "I was negatively infact- impacted by Y." Mm-hmm. Um, that an example from, like, the, um... Like, an example from the past, I think it's, like, very easy for everyone to say, like, "I was negatively infacted- impacted by, uh, like, the COVID-19 pandemic." Like, would be really great to avoid things like that that are, like, as universally loathed as that. I think that, um, like, I don't know anyone who, like, community notes. I know a lot of people who love it. I know a lot of people are ambivalent to community notes, having that in more places, if there was, like, an easy thing to do to make that more common.

[01:25:13] Evan Miyazono: I think this looks a lot different than trying to, Trying to have interventions that are more about defining or shaping good rather than preventing catastrophes or mitigating harms or things like that. Um, there are some, like, really... There are some harms that are in direct tension with each other. I think that, um, privacy and epistemic security and, uh, and, like, concentration of power sit somewhat antithetically to biosecurity and cybersecurity and, um, some, uh, aspects of loss of control. And, like, it is unclear what the Pareto surface is there, but I think-- or Pareto frontier is there, but I think it's worth trying to find things that are, like, reasonable Pareto improvements over the current technological trajectory and do some kind of, like, differential technological development where relevant or, like, differential coordination, things like that.


[01:26:32] Guarding Against Funder Capture
---

[01:26:32] Jacob Haimes: Okay. And then so you, you talked a little bit about like how you prioritize them, which is, at least from what I understand, and correct me if I'm wrong, essentially like what is the thing that is the most tractable out of these? Um, but then there's also got to be an aspect that's about, you know, like, I mean, there, there has to be preferences somewhere that, uh, are not as objective.

[01:27:00] Jacob Haimes: Um, and I, I don't know, it m- this is probably just partially my, my worldview, uh, coming out here, but, like, I, I do see a significant concern being that, well, the, the grantor, those who provide the money, are just going to say, "Oh, well, we actually, you know, only care about this subset of the issues that you've presented, so that's what we're gonna fund."

[01:27:26] Jacob Haimes: Then you do that, and then you become, uh, really this like, I, I don't know, uh, like it, it, it becomes consulting that has this like altruistic flavoring, uh, but is not actually that.

[01:27:44] Evan Miyazono: Interestingly, I think that even in that failure mode, which I would consider a failure mode,

[01:27:50] Jacob Haimes: Mm-hmm

[01:27:50] Evan Miyazono: the people who work at that funding agency not going to consider it a failure.

[01:27:56] Jacob Haimes: No, they are not

[01:27:58] Evan Miyazono: Um, this is one of the reasons that I think... I will, like, briefly point out as I walk by the fact that I don't know if any preferences are non-objective. Um, I... Yeah. I'm always the guy referencing, uh, Hume's guillotine or the is-ought chasm in conversations like this. I think that there... So there was at least one opportunity where I was, like, invited/asked to apply to be a grant maker. Um, and, like, when I said, "I think, like, I want a team to do this," was told, like, "You could, you could have a team." And I think that there's a... The hard answer for, like, why I didn't say yes to that is exactly this, like, I don't want to actually become captive to a particular funder's interests, because I think that this approach is compelling enough that it should be applied across a lot of things. And if I can make it work, then I would expect a lot of other people to mirror it. Or, like, if we automate all the jobs, then I would expect other organizations to adopt this type signature. Um, but also, like, easier answers include things like, well, what if- It turns out that funder wanted an exploration of the interventions, but the in- best interventions don't suit the kinds of things that they would support for whatever reason, but they still want it done. I can go and shop that to different people might be able to fund that end state intervention. And so is kind of, like, a weird consensus mechanism of, like, someone needs to care enough about both the problem and believe enough in the inter- intervention for this to work, and if that ends up being two different funders, that's extra validation. I'm not gonna argue that that's a good consensus mechanism, but I think that the fact that that will require a second approval is, like, a good surfacing of, like, Evan, is Atlas doing good work?

[01:30:21] Jacob Haimes: Yeah

[01:30:23] Evan Miyazono: I think that there are I'm in the process of trying to come up with better ways of sorting or, um, identifying than purely like, is it tractable? Did Evan come up with a cute solution that he likes? Um, I, love coming up with cute solutions. Um-

[01:30:43] Jacob Haimes: too. Uh, like cute solutions are great

[01:30:47] Jacob Haimes: the sort of idea that you were bringing up with, um, getting, I guess, a buy-in. Like, someone says, "I, I want you to determine what the best solutions are to, to do X," and then you go do that.

[01:31:05] Jacob Haimes: And so now you're establishing yourself as doing a good job at conducting this sort of field research, uh, prioritization, uh, uh, for interventions. And then they probably fund that because they think the things that they want to do, uh, either, like, they really care about the output, or the things that they wanna do are gonna be aligned with, um, the interventions that you surface.

[01:31:33] Jacob Haimes: Um, and if that isn't the case, if the interventions you surface are not what they were thinking, you say, "Okay, that's fine. I'll go share this with everyone else and say, 'Here are these cool interventions that I think will address this problem. Do you wanna do it?'" And by having that second mechanism, you're able to secure this.

[01:31:54] Jacob Haimes: Uh, that's all great. The other just sort of wrench that's thrown in here is you are a nonprofit organization that's not taking any cuts of, uh, the organizational interventions that you develop, and you need funding to exist. How are you building in resilience to, to this plan? Because it sounds like, again, in theory, uh, great, but you also sort of need to have that resilience.

[01:32:28] Evan Miyazono: Yeah

[01:32:29] Evan Miyazono: Uh, I think The incentive as Atlas would be to get enough funding upfront to interventions and, like, on some level, we only get funded to generate interventions, then there's no incentive to actually try to make the inve- interventions happen directly.

[01:32:50] Jacob Haimes: Right

[01:32:51] Evan Miyazono: strong indirect incentive in that the only way we are going to fundraise successfully for a second set of interventions is if some of the things actually landed. Um, so I, I think that I tend-- I, I think that there is... I should probably worry more about this. I feel like doing useful things, particularly addressing this problem where I think there are a lot of funders who say, "I wish there were more highly qualified teams with clear plans that'd really seem to mitigate the risks that I care about or, like, unblock the potential capabilities that I care about."

[01:33:33] Jacob Haimes: Mm-hmm.

[01:33:33] Evan Miyazono: And I think that this is I think this is the thing that people who will be trying to deploy philanthropic capital want. It's at least one of the things that they'll want. I think that funding research is incredibly valuable and complementary. I think funding products is incredibly valuable and complementary. It's... I think that there will be some products that will never return the fund, that just don't make sense for venture investment, that, like, we might do a better job highlighting. But if we found some of those companies, then, like, it feels like it'd be pretty easy to just send the founders to an incubator or an accelerator or something.

[01:34:11] Jacob Haimes: Mm-hmm.

[01:34:12] Evan Miyazono: but, like, I don't have plans to run that, like build or run that incubator accelerator. Um, I think that taking some, uh Just trying to have some amount of longevity baked into the intervention where like, yes, we will spend like to nine months building out interventions for things in this topic with the understanding being that we're asking for six to 12 months of funding for it. Um, and if people wanted to provide additional funding or runway to support, uh, the team generally, like I think that that works well as an-- Like, did good work.

[01:35:00] Evan Miyazono: Here's extra funding," um, might not fit a lot of funders, but would be right

[01:35:08] Jacob Haimes: But do you really think they'll do that? 'Cause like, I don't know, I, I understand what you're saying,

[01:35:13] Evan Miyazono: I mean

[01:35:14] Jacob Haimes: uh, in principle, but I just don't... That to me, like the, the incentives just aren't there

[01:35:20] Evan Miyazono: right, yeah. The, a more realistic version is um, like one for us, one for you kind of model, where it's like, "Hey, you have come up with great interventions for the things that we thought were important. What do you think are important? Why don't you have some field strategists chase those things down, because we're interested to see what you come up with?" Um, I also am on some level preparing for a, uh, recoverable catastrophe, AKA the warning shot. I think that the warning shots look different in different domains, but I would expect that the thing that you'll want to do afterward is, um, be able to deploy a lot of government funding to improve security, uh, in different aspects and identify the fastest, most high confidence intervention. like the inter- the types of interventions probably change, uh, because it's now much easier to like pass a law to do a thing maybe. Um, but, uh, of the things that I think about, one of them is the fact that like it's not clear to me what the warning shots look like for epistemic risks. Bio, cyber, those look pretty clear, but like I think that, uh It's not clear what kind of an event that is foreseeable would increase to pay costs to improve the epistemic environment. And so

[01:36:48] Jacob Haimes: I think it's much easier to think of the kinds of in-- uh, catastrophes or even not catastrophes, just like states of the world, uh, which would, uh, de-incentivize even further, uh, doing that for the people who can. Um,

[01:37:08] Evan Miyazono: Yeah.

[01:37:09] Jacob Haimes: because, uh, power wants to entrench itself. That's just how it works. Um, and

[01:37:18] Jacob Haimes: Yeah. Um, I don't know what the solution is to that. I think it's r- but that, that's like the one that keeps me...

[01:37:25] Evan Miyazono: and balances is the canonical one.

[01:37:27] Jacob Haimes: Checks and balances i- is what should in theory work, and there are other, there are other like, you know, more interesting mechanisms as well, but you then need to get buy-in from the different parties, and you need to not have those undermined, uh, systemically for, you know, half a century.

[01:37:48] Jacob Haimes: Um, yeah. So

[01:37:52] Evan Miyazono: And you can even point to, uh, I think George Washington's farewell address included, "Don't start political parties."

[01:38:01] Jacob Haimes: Yeah

[01:38:02] Evan Miyazono: Oops

[01:38:05] Jacob Haimes: Yeah. Um, real quick, I know that we're already five-- like four minutes over. Do you have a, like a little bit longer?

[01:38:12] Evan Miyazono: I do

[01:38:13] Jacob Haimes: Okay, cool. Um, so I have-- I want to hit one last thing here, and then we'll skip to like the, the closing stuff. Um, so in terms of funders, I assume, uh, and I guess I don't know for sure, and I don't know if y-you can fully disclose either, but like, uh, I think you, you say on some of the promotional material, like Open Philanthropy/Coefficient Giving, uh, Survival and Flourishing Fund.

[01:38:37] Jacob Haimes: So I, I sort of think of these as the usual suspects for this kind of work. But then you also have DARPA on there, and that I don't think is as common. And so I'd like to know a little bit more about, uh, why, why include them, and also what interventions or, or, um, security like, uh, watches you're putting in place, uh, to prevent becoming just like a sort of like selling sovereignty kind of organization where you are just conducting the interventions, uh, for whatever, um, government wants, uh, to, to buy your, your services.

[01:39:27] Evan Miyazono: I don't know if I've come across the, selling sovereignty paradigm.

[01:39:32] Jacob Haimes: Uh, so I think that... So may- maybe the, the term that I'm, I'm pulling from is, like, um, from m- I think it was, it was discussed in, like, a book about Muskism. So it's using similar to, like, Fordism, um, being, uh, a paradigm from, like, you know, last, the 1900s. Um, Muskism is, is this sort of turn to, um... But, like, they, they use Elon Musk as a, as a, uh, case study.

[01:40:04] Jacob Haimes: Um, but then they sort of demonstrate this, like SpaceX being, uh, you know, replacing a lot of, um, the Like, functionality of, of like NASA and, and things like that or, or, um, you know, Starlink.

[01:40:28] Evan Miyazono: Yeah

[01:40:28] Jacob Haimes: Yeah. Uh, uh, Starlink being, uh, you know, like a critical aspect, uh, for certain people's, uh, national security now. And so, um, it, it, it then allows, you know, these

[01:40:42] Evan Miyazono: Yeah

[01:40:42] Evan Miyazono: these are like

[01:40:43] Jacob Haimes: organizations...

[01:40:45] Jacob Haimes: Yes

[01:40:46] Evan Miyazono: Yeah. Um, have you come across the administration markets, uh, blog post?

[01:40:51] Jacob Haimes: I have not

[01:40:53] Evan Miyazono: I'll, let me share it with you as a really interesting proposal of, like, why does each, like, small town each state have to do its own, like, administration on, like, like, property taxes or, like, school management, things like this.

[01:41:15] Evan Miyazono: And, like, what would it look like to actually have markets for this where the groups that do well, can offer to more, but if it's still a government service and, like... Then going in, like, if one were privatizing that, that creates a lot of, uh, a lot of hazards.

[01:41:33] Jacob Haimes: Yes, exactly. And then that happening,

[01:41:36] Evan Miyazono: Yeah.

[01:41:36] Jacob Haimes: I guess

[01:41:37] Evan Miyazono: And it, it is happening, yeah, in certain things.

[01:41:39] Evan Miyazono: So I would say, on the one hand, um, noting that while I would say Atlas is collaborating with DARPA, there's lots of conversations, um, lots of like, "Oh, hey," like, "Talk to these people," um, things like that. Also, ARIA in the UK is an ARPA-like entity. It is more focused on science than defense, um, as the main distinguishing factor, but the current CEO of, uh, ARIA was shortly before that the head of I2O at DARPA. Um- I think that there... Like, if you look at what Atlas is trying to do and you said, like, can't be new. Who has done th- like, whose responsibility was this before now?" was government. Like, find problems, predict problems, try to prevent those problems from causing catas- like, large, like, society scale catastrophes or, like, decreases in, um, like, citizen welfare or things like... Th- this is a government service. It fits reasonably well as a nonprofit as we have government, uh, becoming, uh, less... Right? As we see state capacity decrease, it makes more sense to do this as a nonprofit. I think that we are seeing a current social trend towards increasing state capacity with, like, the abundance movement and the progress movement and things like this.

[01:43:12] Evan Miyazono: So it hopefully makes increasing amounts of sense for government to put money into this to, to think about this. I think one of the, one of the risks of selling sovereignty comes from the notion that, like, a corporation, a for-profit corporation must be profit-seeking or ends up being profit-seeking. I think that there isn't as much in... Like, the incentive is much, much smaller for Atlas to expand globally. Like, I'm not going to see a big markup in my equity because it's a nonprofit and there is no equity. Um, it would be nice to have this. Like, there is-- To me, there is virtually no difference between hiring someone to be head of, uh, like, the London office or, like, for UK Atlas or me helping someone start something Atlas shaped in the UK

[01:44:19] Jacob Haimes: Right. Okay. Yeah, that makes sense.

[01:44:22] Evan Miyazono: like,

[01:44:23] Jacob Haimes: There's no reason for you to care. It's--

[01:44:26] Evan Miyazono: yeah

[01:44:28] Jacob Haimes: Yeah, because they should want to work with you regardless because it's helpful and

[01:44:33] Evan Miyazono: to create a... Like, I'm thinking through what it would mean to, uh, have an adjunct field strategist role, where if someone basically could be a field strategist, is doing the work, would like to coordinate and collaborate with us on a bunch of things, but for whatever reason, like, take a paycheck or, like, want to take a paycheck from Atlas, that, like, we could expose them to all of our, like, conversation notes and all of our, um, like, plans and our crazy ideas and, like, who we're talking to next, so they can get questions in front of the right expert if we're, able to make the right introduction. This feels like a very useful thing to do, and I... it, it wouldn't if we were for-profit because there'd be, like, IP in this somewhere that we need to claim and, like, fight with their current employer about it or things like this. So I don't think... I, I think that this works weirdly well a nonprofit and for, like, a lot of small, narrow, um, and, like, kind of need to get into it to think about it, uh, reason. I also think that, like, a non-zero chance that, like... you get into this because you like public service, you wanna fix things, um, maybe you're worried about stuff. I gotta say, I love meta-science still. I think if I weren't worried about AI, if, like, we-- If all of the GPUs got bricked and we found out that, like, current approaches are hitting a wall, I'd be like, "Great, time to go do meta-science over, uh, at Convergent Research," where I'm, like, to start one FRO.

[01:46:18] Evan Miyazono: Would love to help start a bunch of other FROs. There's, like, a call for the NSF X Labs, which are basically FROs, but, like, very expensive FROs. Um, NSF wants to fund a bunch of these. Um, some of them include, like, exactly what my PhD was on, quantum networking and, um, photonic interconnects. And, like, it's hilarious how relevant my background is to exactly this, but I'm too worried about AI to work on that. So, um, yeah, I, like... I think that the, uh- I I worry about a lot of things, including my own incentives, I don't particularly worry about a world where I am, like, exercising international control over, uh, states' abilities to solve their own problems, because I think exactly...

[01:47:16] Evan Miyazono: Like, uh, maybe a, a good, uh, example is that someone leaves my... Anytime someone has left my team, I think they've always been someone where, um, like if they left to go do something else, I've always been excited for them to go and do those things, I hired them because I wanted to work with them to advance goals that w- I thought were good. And if it made sense for them to go do this other thing, then they're doing it because it, they believe it will help them better achieve goals that I think are good.

[01:47:53] Jacob Haimes: Yeah

[01:47:54] Evan Miyazono: think this is ver- this is not something that I think would be, like, true, um, all, like, potential employers or managers.

[01:48:03] Evan Miyazono: But I think that the fact that this is something I hold reasonably strongly is, like, a, a decently good indicator that like, if the, if we started a UK branch, and then the UK branch wanted to declare independence, I'd be like,

[01:48:18] Jacob Haimes: As long as you're...

[01:48:19] Evan Miyazono: me."

[01:48:19] Jacob Haimes: Yeah, yeah. That, that makes sense.

[01:48:23] Evan Miyazono: I would also say that if someone, like someone said, "Evan, I like what you're doing with Atlas. I wanna take over. Here are all the reasons why, like, you should just leave and no longer be a part of this. got it from here." I'd be like, "Thank you."

[01:48:39] Jacob Haimes: Yeah, that is what we all want, uh, isn't it? Although I, I guess the other thing is just like, isn't, isn't that what someone who wanted to, you know, take over, uh, be controlling all the different nations was saying? No, I'm just kidding. Um, the

[01:48:56] Evan Miyazono: know. I think that, uh, at least one person with that aspiration would say, um, what was the

[01:49:02] Jacob Haimes: Exactly

[01:49:02] Evan Miyazono: that's, that's loser mentality, and I'm not a loser


[01:49:09] Rapid Fire Questions
---

[01:49:09] Jacob Haimes: Yeah. You know, that's a, that's a great point. Um, and maybe, uh, that's a good, like, shift towards the, the final sort of like more fun, uh, l- quicker things that I, I like to ask towards the end. The first one is just like what, uh, sort of what grinds your gears? What makes you, like, irritated or, or things that you don't as much enjoy about what it is that you do?

[01:49:31] Evan Miyazono: I can probably get like painfully specific, but the category is I like doing things that are clearly trying to solve the problem and like doing the thing. And I think that when there are barriers in front of me that look like, "Well, you can't do the thing we need you to do this unrelated thing that will not actually help you do the thing." yeah, uh,

[01:50:06] Jacob Haimes: That's irritating.

[01:50:07] Evan Miyazono: That

[01:50:08] Jacob Haimes: It's

[01:50:08] Evan Miyazono: Yeah, 

[01:50:09] Jacob Haimes: Um, yeah. Um, okay, next one is, what's something, uh, like, within the past year or so that y- you used to believe about AI, uh, or AI safety, uh, and you've, like, updated, uh, on it, it, something has changed?

[01:50:26] Jacob Haimes: Uh, not something... Yes, something that you essentially, you, you would be like, "Yeah, I was wrong," uh, kind of thing

[01:50:38] Evan Miyazono: Well, first one that comes to mind is That, uh, I had a, a nice long chat with Dava Dhad about his, kind of like, uh, change of assumptions from, like, safe- the safeguard AI framework to where he's currently at. Um, and, uh, I think that there was a combination of both, like, trying to get safety properties over all AI systems over a certain level of capability is, like, I now think is probably in- infeasible, um, unless there is also, like, regulatory wi- regulatory willingness on a lot more things. Regulatory willingness might be a thing for a lot more. Um, I think that even the but China arguments have gotten weaker as a lot of the language out of China comes down on the, like, "This is a real risk. We need to not, uh, be in a race to the bottom."

[01:51:41] Jacob Haimes: Okay. And then, uh, let's say a listener is like, "All right, Atlas sounds great. This is what I want. The, the... Like, I, I wanted something like this, uh, for so long. Uh, what should I do?" What would your advice to them be?

[01:52:03] Evan Miyazono: I think there are probably two... There are people who think that or have said that, that fall into roughly two categories. there is one category of people who, like, basically are already doing this and just didn't have the term for it. And for those people, I'm, like, roughly trying to assemble this team.

[01:52:25] Evan Miyazono: I'm in the, currently in the process of fundraising for, um, assembling a team of, like, I think to 10-ish, um, initially, trying to grow quickly thereafter. Um, would love to talk to people who, uh, done this work in some way. Um, I am very findable online, evan@atlascomputing.org is my email. to people who want to do this and maybe don't have experience, I am trying to build up a team to, uh, be able to take on mentees soon train people who could do this.

[01:53:04] Evan Miyazono: I want real explosion in the number of people doing this. I think that it has capacity to be, like, roughly on the same scale as, like, impact investing or, like, uh, impact venture, um, or, like, the number, the number of, uh... It seems like the number of incubators focused on, like, AI safety or AI security probably...

[01:53:27] Evan Miyazono: Like, the number of... Sorry. The number of startups, uh, doing AI safety and, AI security products should probably be about the same size as the number of field strategists identifying interventions that are not, uh, products that could return the fund. Just as, like, rough order of magnitude, I would probably argue that most of the correction terms make the number of field strategists probably higher.

[01:53:53] Evan Miyazono: Um, unless you... Well, neglectedness makes the field strategy number go higher. The time needed to deploy an intervention makes the field strategy number, like, carrying capacity go lower. I think starting at the same order of magnitude would roughly be a reasonable estimate. And I think starting by, um Finding a thing that... I guess the specific advice would be to do this. I have, like, a blog post up that's like, "Here is my general process." I have of things. the thing that fit- seems like the best fit, and then really go through and try to identify Who are the relevant experts? Get to those people and get them to sign off on this being a good intervention.

[01:54:53] Evan Miyazono: Use the models to emulate them before you do that and, like, really red team. And similarly, try to find paths to the potential a- adopters. I think people who are already in the AI security or AI safety space already have access to a surprising number of possible stakeholders through relevant events. Like, very easy to catch, like, CISO of a frontier lab, um, after a conference, or ac- after their talk at a conference, um, compared to a lot of other fields.

[01:55:27] Jacob Haimes: Okay. And then last but not least, the question I ask everyone, what is your favorite part about what you do?

[01:55:36] Evan Miyazono: I think probably the split between, uh People saying, "What you described is exactly what I've been doing or been trying to do. Like, this is a perfect fit for me." Or like, um, I think having people be very excited to do what is basically work, especially if they're like qualitatively more excited than they thought they could be excited about a job. Um, that is good. I also really like I get buy-in on an idea, especially if it's like, especially if it's a particularly cute idea. Like, I think that there... I don't know who gets trained where to design incentive mechanisms and do better coordination, but I feel like humanity does not do it nearly enough, and like there, there is so much room for improvement. And, like this, this is... I don't know how much of this is intentional or latent or just like an echo that keeps reverberating through, um, my work history, but the, the motto for Network Goods was revolutionary coordination systems, and it feels like it's what I'm trying to do now, lots of mechanisms to support public goods. I think that, like lots... the, maybe the thing I enjoy most, the univer- the overarching theme is that I really like getting evidence, concrete evidence that things can actually just be better in a pretty, like universal and like, um Like, yeah, in a pretty universal way where it seems unobjectionable like, yeah, if that exists, it's hard to imagine someone saying that would be a bad thing they were saying so in, in bad faith. like, I think there are a lot of those, and I think trying to Either making progress toward that or helping more people see the distance between where we are and where we could be. Probably my two favorite parts


[01:58:01] Outro
---

[01:58:01] Jacob Haimes: Awesome. Well, Evan, thank you so much for, for joining me. It's been a ton of fun to talk with you, um, and

[01:58:09] Evan Miyazono: Thanks for having 

[01:58:09] Jacob Haimes: I'm really excited to see where Atlas Computing is going

[01:58:14] Evan Miyazono: Appreciate it. Uh, this was a delight chatting and, uh, look forward to chatting more at some point

[01:58:19] Jacob Haimes (ASIDE): If you couldn't tell from the interview, I had a fantastic time speaking with Evan. One thing that I really liked about his perspective, which also just resonates with me strongly, is that a lot of very important work is stalled because it requires someone deciding it's their problem, which is maybe both a bit uncomfy and reassuring at the same time.

[01:58:48] Jacob Haimes (ASIDE): Also, I literally just checked, and while I greatly appreciate all the star-based reviews I've received, I still don't have a single testimonial review, at least as of recording this. Like, seriously, those, like, really drive the platform suggestions. So if you think this is worth other people hearing, please consider leaving your thoughts.

[01:59:11] Jacob Haimes (ASIDE): That's gonna be it from me. Take care of yourselves, and I'll see you next time.