Oxide and Friends

LLMs have had a dramatic impact on education. There are obvious reasons for concern, but what about the less obvious opportunities afforded by LLMs? Bryan and Adam were joined by Michael Littman, professor at Brown University and Associate Provost for AI, to talk about his role advising the university on productive, innovative, creative uses for AI in higher education.

In addition to Bryan Cantrill and Adam Leventhal, our special guest was Michael Littman.

Previous, on Oxide and Friends:
Some of the topics we hit on, in the order that we hit them:
If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Creators and Guests

Host
Adam Leventhal
Host
Bryan Cantrill

What is Oxide and Friends?

Oxide hosts a weekly Discord show where we discuss a wide range of topics: computer history, startups, Oxide hardware bringup, and other topics du jour. These are the recordings in podcast form.
Join us live (usually Mondays at 5pm PT) https://discord.gg/gcQxNHAKCB
Subscribe to our calendar: https://calendar.google.com/calendar/ical/c_318925f4185aa71c4524d0d6127f31058c9e21f29f017d48a0fca6f564969cd0%40group.calendar.google.com/public/basic.ics

Adam Leventhal:

Bryan's here. I'll invite him to the stage.

Bryan Cantrill:

Oh, just invited himself. Just invited himself. You know? He can figure it out.

Adam Leventhal:

Good. Now we need to scan the audience for Michael.

Bryan Cantrill:

I can do things, Adam.

Adam Leventhal:

Yes. No. You're you're doing great.

Bryan Cantrill:

Thank you.

Adam Leventhal:

Somebody had mentioned in chat. I was like, oh, yeah. We used to just cheat by having our answers in our t I 82. I'm sure you're like, TI82 back in my day.

Bryan Cantrill:

Was TI85.

Adam Leventhal:

Right.

Bryan Cantrill:

Are you are are you doing this to are you doing this to test out some hypothesis that I'm actually an LLM? That I'll I'll and you've actually have got written on your you must obviously you have HP 42 s or HP 48 g or HP 48 g x on your oxide and friends. Go through

Adam Leventhal:

And then like you

Bryan Cantrill:

could all of those.

Adam Leventhal:

Right. That's the thing

Bryan Cantrill:

about bingo. I'm gonna get bingo. We're not even gonna be we won't even kick off the recording. Right. Just don't even even gonna have joined yet and I love bingo.

Bryan Cantrill:

I I am you know, this is this power corrupts and absolute power corrupts absolutely, Adam. And to give us bingo cards is really very dangerous. It's just to invite. I mean, ever since you and I have been doing Oxnard and Friends bingo for money, I think it has really distorted things a lot. Think it's

Adam Leventhal:

nice to know that.

Bryan Cantrill:

High stakes. Cheers. It pained me to give you bingo on on the HP calculator line from the the nineteen eighties and nineties, but you know what? It was worth it. Oh, Michael was Michael's like, I am in the wrong room.

Bryan Cantrill:

I It's like eight w's, can you? Right there. I seem to be in some sort of HP nostalgia. The bingo card today looks amazing.

Adam Leventhal:

Have you seen this? Yeah. It's crazy.

Bryan Cantrill:

We this is what we do now. We have, like we customize it for guests. This is great.

Adam Leventhal:

We we don't. J h seventy seven does. Okay. We're clear.

Bryan Cantrill:

When when I say we, I mean I mean we in the And I'm looking on my own bagel card. Why is HP why are HP calculators not on my own bagel card? Damn it. Well, it's just tough. I also I I gotta I mean, I love the my HP 42 s.

Bryan Cantrill:

Just absolutely amazing. What? Oh, oh, sorry. I mean The question

Adam Leventhal:

caught everyone's mind.

Bryan Cantrill:

Sorry. Exactly. It's like, could you please please elaborate. I'm sorry.

Adam Leventhal:

You mentioned you're you mentioned your high school calculator without going into really sufficient detail. It's sufficient detail.

Bryan Cantrill:

Oh, there

Adam Leventhal:

you go, Michael. Thank God I found you. Yes. There

Bryan Cantrill:

you go. There you go. The so, you know, I actually use a so one, they they re someone remade the HP 42 s. So you can that's like a kickstarter, which is amazing. And so I have one of those.

Bryan Cantrill:

I have a I have my original 42 s still works. I've got my which again, I mean, and I I and purists are gonna disagree. I'm here. I'm having the debate that no one's asking for about the 42 s versus the 48, but I'm but the I I truly love my 42 s. The and then I also use the there is an app all for the phone, that is a 42 s emulator that I still prefer for my calculator.

Bryan Cantrill:

So there you go. It is it is God's own calculator. That's all I gotta

Adam Leventhal:

So Michael, thank God you're here. Michael,

Bryan Cantrill:

I don't know if you think you're like I'm either in the wrong room or the right room. I'm not sure where but I oh, actually, you know, because Michael actually opening salvo sorry, question. Do you consider yourself an xer? You wait. You must be you consider yourself a very old xer, I assume.

Bryan Cantrill:

This is this is essential. I'm sorry. At this podcast. This is like this is podcast is all about generational divide.

Michael Littman:

Oh, well, I was born in 1966. So you can do with that

Bryan Cantrill:

what you hundredth x Yeah. Yeah. A hundredth x r. No. With and I I mean, I just wanted to actually just wanted to hear you hear it out of your own.

Bryan Cantrill:

Oh, I guess you didn't really but you maybe you're an x r. No. Because you grew up on basic. This is like my if if basic is your first pro street basic as to make a reference to an earlier episode. You know Dykstra had disdain for basic, Michael.

Bryan Cantrill:

Did know this? I mean, Dijkstra had disdain for many things.

Michael Littman:

Well, he definitely didn't like goto , and BASIC was pretty go to heavy. So, yeah, I

Bryan Cantrill:

can see that. You know, Dijkstra arguably we we did this years ago, Dijkstra on basic. Dijkstra arguably thought about basic. What arguably some people might think about LLMs today in terms of corrupting the minds of youth. I mean, Dijkstra believed that basic was would destroy an entire generation.

Bryan Cantrill:

We'll just say our generation. So

Adam Leventhal:

You know, Brian, this is not the sixth the the first time this morning that Michael has suffered through a description of us as actors because we as we were trying to navigate Discord, I was like, look, I will help you, but I am a Gen Xer, so my help on getting navigating through Discord may be limited.

Bryan Cantrill:

Yeah. Well, you you definitely have to just close every dialogue box offering you because you get these dialogue boxes where you're like, am I it feels like I'm I should are adults in this room? Am I this is only made for children? It's like, no. This is actually this is you get up.

Michael Littman:

I am I am finding the interface extremely overwhelming. And I I worry like, I do think I've only used this was the preferred kind of interface, but, like, maybe I'm just old and slow now. But I'm like, oh my gosh. I am my circuits are completely overloaded.

Bryan Cantrill:

Yeah. Yeah.

Adam Leventhal:

Yeah. If you like being offered Nitro, definitely turn that down. Whatever if anyone's offering you Nitro, just say no.

Bryan Cantrill:

If anyone's offering listen, Michael, you're gonna be at some parties where where you're gonna be at some parties where Nitro is gonna be going around, and we just want you to close those dialogue boxes. You know, you're gonna make your own your own decision in the moment, but, you know, Adam and I would hope you make the right decision, and don't don't adopt any Nitro. Yeah. It it's a little bonkers. But, know, this may maybe the the you can you know, the the youngs use Discord, Michael.

Bryan Cantrill:

This is this can help you connect with this this younger generation.

Michael Littman:

Well, sure. I mean, the thing that I try to do to connect to the younger generation is I actually listen to the Billboard Hot 100 Top 10 every week. And because I was noticing that I would I like to make parody songs of the whatever it is that I'm working on. And all my parody songs are from the eighties. And I was realizing that I was very rapidly losing the younger audience.

Michael Littman:

So now it turns out I actually know too much about pop music and it goes way over the heads of the college students.

Bryan Cantrill:

Yeah. Well, because I was gonna ask them because your kids my kids do not really listen. I mean, my 13 year old does, but I don't think my I don't know about my 18 and 21 year old. I'm not sure if they but do these have you found that it's been useful or do you have to you gotta go like a do you need to do like a TikTok dance or something like that? I don't know.

Bryan Cantrill:

That that's even too old now. That's like a geriatric zeter. Right? I mean

Michael Littman:

I It's I can't speak to that.

Adam Leventhal:

There we go. Brian, I don't I don't mean to, like, get us on track too early or anything.

Bryan Cantrill:

And yet. And yet. And yet.

Adam Leventhal:

I I was gonna introduce Michael.

Bryan Cantrill:

Yes, please. Yes. Absolutely. So

Adam Leventhal:

I know Michael as professor of computer science at Brown. Did you know Michael as a PhD candidate at Brown? Because I think you guys were there at the same time.

Bryan Cantrill:

I did. And I actually know Michael because Michael, your I believe oldest child was born at when you were Brown, finished up your PhD. And I was trying and that was a wild ride. Hope it sounds like too much to say it's a medical Marvel. No.

Michael Littman:

No. I'm pretty open about it. Yeah. No. That's that's that's fair.

Bryan Cantrill:

And it was in I mean, it was and and Michael, where were you publishing kind of the updates? I mean, the the because there were updates that you were kind of publishing. And I should remember if that was not was that 1994 maybe? When was that? '95?

Michael Littman:

'23. That's exactly right. So it would have

Bryan Cantrill:

been '94.

Michael Littman:

Yeah. All of '94. And I basically made a blog, but the word didn't exist yet.

Bryan Cantrill:

That's right. Yeah. Yeah. Yeah.

Michael Littman:

Where did

Bryan Cantrill:

you post that? Was that

Michael Littman:

It was just on the Internet. I just I just created a URL, like a webpage. And I just kept, and I would, it had kept like a diary, but then I also put in pictures, which was very new then the idea of like digital, like I had a digital camera, which was crazy. It had a big floppy disc that you could slide into the side of it to get the media in and out. And I also even got video in which required me going to the graphics lab in the computer science department and borrowing their digitization equipment because like you don't take digital video, you take like normal video, you just have to make it digital later.

Michael Littman:

So it was really quite an interesting little excursion back in '94.

Bryan Cantrill:

Well, will tell you that it had a real effect on me because I you know, I just you know, you're a kid when you go away to school. Right? And you and I were not Peterson, you were a PhD student when I was an undergrad, but you know, it's like like you can kind of see across a chasm a little bit. Yep. And watching you, I mean, and the I I guess on the internet, this is I mean, he was what was his weight at birth?

Bryan Cantrill:

Three. I remember that it was like a pound three ounces. Is my remembering that correctly? Half

Michael Littman:

an ounce. Yeah. Very close.

Bryan Cantrill:

One pound half an ounce.

Michael Littman:

Yeah.

Bryan Cantrill:

And it I was hanging on every word of that thing. And because it was I mean, obviously, just as, know, kind of a you you you were approximate to all of us.

Michael Littman:

Yeah.

Bryan Cantrill:

But just the story was so extraordinary. I remember thinking like, man, fifteen years ago or twenty years ago, this was impossible. And not just like that mean focusing on the medical marvel aspect of it, but also like the kind of the socialization of that and you having that writing that was because I it was I I as I was recalling this, I'm like, I think that was actually on the internet, but I don't know how in 1994 was it like up on like an FTP store for example. Was an online gopher, right? Right.

Bryan Cantrill:

Yeah. No. Right. It just existed. Yeah.

Michael Littman:

So much of it was new, right? Like the the even, you know, the notion of having a website was new. I mean email wasn't that new, but it was new that lots of people had it. And so yeah, it was the beginning of a new age for sure. I feel like I should give you like an update, right?

Michael Littman:

My son is 31 now. He's married for a year. He is an elementary school music teacher, which is just a cute one. Oh, wow. And so he lives in Massachusetts and he teaches elementary school music.

Michael Littman:

And he's just the he's the cutest, nicest man. But he's like a man now. He's all furry. He's got like a beard. We come from hairy people, but like I've never been able to grow a beard.

Michael Littman:

Like that's not been a skill of mine, but he's mastered that. Like he's doing the thing. It's it's quite lovely. And he's, yeah, like I said, he's just the sweetest person. Like not me, I'm not but that he's just a good guy and he does good stuff.

Bryan Cantrill:

Well, congratulations to him. That is great. And you and and, again, that that was a really affecting story. And it was funny because as I was thinking about it, for whatever reason, this is sad. I'm so sorry to just like take us right into the gutter.

Bryan Cantrill:

But I always think of the foreign rectal bodies page as my real awakening with respect to the Internet. Do you remember the foreign rectal bodies page, Michael, where the foreign rectal bodies page was like a I mean, it's yeah. You can imagine it. It pretty much is what it says on the tin. It was a basically a collection of like actual like medical case studies of foreign record bodies.

Bryan Cantrill:

And I remember thinking to myself, wow, the Internet's got legs. Like this is

Adam Leventhal:

This is gonna go

Bryan Cantrill:

fundamentally change everything. This is gonna change everything. And as I was gonna reflect you back on it, like the former retrobodies page was at least like six months after your the the description of your of your son being born. And I'm like, that should have been much more affecting actually. It really did have a an effect on me.

Bryan Cantrill:

I really should have thought about I mean, it gets maybe, you know, maybe there was the entire we saw the entire Internet before us in this span of like, we saw all of the the the the this extraordinary birth, and then we saw just the the just the ditch that is the foreign rectobodies page. Although, really interesting page, gotta say. Super fascinating.

Michael Littman:

So I left Brown. I got got my PhD and I became a professor at Duke University. And it turned out that Duke University was famous not for foreign rectal bodies, but for but for people who had swallowed toothbrushes by accident.

Bryan Cantrill:

Oh. That was like

Michael Littman:

That was an awakening for me of a sort.

Bryan Cantrill:

It feels like a neighboring story. It feels like actually just like the it's kind of, you know, the more dignified half quite literally of that problem.

Michael Littman:

You know, of the of the various orifices like that's The

Bryan Cantrill:

various orifices. Yeah. The the so and so you you go to Duke and then came back to Brown as a professor, which is, I guess so Adam, you knew Michael when he was at he was already back at Brown by the time. Michael, what did you were you

Adam Leventhal:

back at Brown in, like, o one or o two? When when were you back?

Michael Littman:

No. I didn't get to I didn't become a professor at Brown until 02/2012.

Adam Leventhal:

Okay. Got it.

Michael Littman:

Period when I was in the wilderness.

Adam Leventhal:

Got it. So, yeah, I think it was when I was on the Brown CS alumni advisory board.

Bryan Cantrill:

Oh, there you go.

Adam Leventhal:

Yeah. That that that I remember you from that.

Michael Littman:

Yeah. Because should I be

Bryan Cantrill:

yeah. Go ahead, Michael.

Michael Littman:

I was just gonna say that when I got when I got back, so I had I was in between, I was a professor at Rutgers. And at the very end at Rutgers, I was the department chair at Rutgers. And so when I got to Brown, they're like, oh, you should do like department running things. And I'm like, I left Rutgers because I didn't wanna do that. It's okay.

Adam Leventhal:

So put

Michael Littman:

me on the advisory board right away. And but that was actually fascinating because I I did get to meet some some really interesting alums.

Adam Leventhal:

Yeah. And me too. Yeah. I feel like that is it's like a classic move in the Brown CS department where it's like, no. Come come paint this fence.

Adam Leventhal:

Come come run the department. Like, it's really fun. Right? Super fun. It's super fun.

Bryan Cantrill:

Super fun.

Michael Littman:

Left. Left Rutgers specifically to stop being department chair. Please don't do actually said to the to one of the associate deans, I'm like, I don't think I will survive a term as chair. She's like, but let's talk about this a little bit. I think you're being very cavalier with my life.

Bryan Cantrill:

Right. Exactly. You you might not survive, but actually more more importantly, we need a chair and you seem to be the right candidate.

Michael Littman:

One term is three years? Like, can survive that long. Right? I'm like, no. I'm not sure that I can.

Adam Leventhal:

Okay. But a year and a half, fine. Well, then we'll find someone new then when you're dead.

Michael Littman:

Right. Right. Exactly. And down the road.

Bryan Cantrill:

That's a future Us problem. Yeah. The but it should also be said, so you were machine learning is your domain, and you were in machine learn I mean, you because I kinda feel like that that mid nineties for AI. I mean, is it too reductive to say that machine learning deliberately rebranded itself to something that was not AI to avoid some of the stigma of the time of kind of strong AI. Because that was that was my inference from the outside looking in at that No.

Bryan Cantrill:

That's not

Michael Littman:

far from the target, I think. I think what happened was in the so the late eighties when I was first getting into this stuff, it was the case that neural nets were kind of They're having their second rebirth, their second And AI people actually didn't like it cause it was so different from the way that they wanted to think about the world that it was very clear that there was machine learning people and there was AI people and never the twain shall meet. So the conferences, for example, the AI conferences would not really have machine learning people and the machine learning conference didn't really have AI people. And so it was like, let's just keep things separate. So I actually was working at a lab in New Jersey, a phone company lab in that period.

Michael Littman:

And I considered myself a machine learning person and they were pretty anti AI. Like they didn't use the word AI. So I got to Brown and my advisor, Leslie Kellbling, I said to her, like, I'm not an AI person. She's like, you need to get over that. This is AI.

Michael Littman:

I was like, Oh really? Oh, interesting. Yeah. And so she kind of just kept pushing me and pushing me until it was like, okay, fine. This is also AI.

Michael Littman:

So it was definitely the case in the field as a whole at that point. That was when we started to see machine learning work showing up in the main AI conferences and people in the machine learning conferences asking explicit AI type questions. And so the merger kind of happened then. And I think it was more that I think than rebranding to avoid the stigma of AI. Though I think that was in the beginning part of it.

Michael Littman:

Like it was definitely the case that in the group that I was in at Bellcore in New Jersey. Yeah, AI was a bad word. And so we needed a word that wasn't that.

Bryan Cantrill:

Yeah, interesting. When I was you, it's funny because I viewed Leslie as being kind of on the vanguard of that rebranding, but I guess not. If she was telling you to get over it and call it AI. Yes.

Michael Littman:

Like, embrace it. Own it. Like, this is us now.

Bryan Cantrill:

Yeah. That's that's wild. So and then so you've been doing this for your entire career. And wow, have things gone in a did you anticipate this timeline? I mean, we are in a very a very a spot that that would be tough to anticipate in terms of in so many different dimensions.

Bryan Cantrill:

And we've got, wow, do we have so many different dimensions to talk about. So you are not just a professor of computer science, not just an an ML researcher, but you are also Brown's first AI provost. Right? You are the you have been designated by Brown to really take on the AI problem writ large and the I mean, fair to say both the problems and the opportunities that it poses for higher education. Is that a fair a fair sentence?

Michael Littman:

That's really well said. I sometimes brisk a little bit about that notion of being an AI provost because I always click I am not a robot, but nonetheless.

Bryan Cantrill:

I I I know you do that, but we just need to you to click and hold a button here just for a second. Just say there we go. Very good. Excellent. You don't know.

Adam Leventhal:

And and so, Michael, one of the reasons we got here was I think that we've had a bunch of conversations about AI and ML and the implications. And we've had some guests on, and and we've talked about, you know, some of the what seemed like kind of facile observations about the impact of AI in education. You know, someone saying, oh, you know, the the high school teacher I know, you know, hates it because for all these reasons. And as we're we were thinking about this, literally on my desk was the Brown alumni monthly with an article about you folded over to this page as I'm thinking, what would be a better conversation to have, and who could we have it with? And And it's literally a picture of you staring up at me.

Adam Leventhal:

I'm like, oh, idiot. Of course. So I'm so delighted that you joined us.

Michael Littman:

Oh, no. I'm really this is this is this is a kick. For sure. Yeah. No.

Michael Littman:

I this it is it is a very big deal. I don't I don't have all the answers. I may not have any of the answers, but I'm not sure that anybody else does either. Like people who at this point claim that they know the answer, it's kind of like ChatuchPetee claiming that it knows the answer. Sure.

Michael Littman:

It may turn out that you were right, but you didn't know that. Right? You didn't know whether you'd be right or not. So I feel

Bryan Cantrill:

I mean, it's I was trying to think of, like, similar revolutions in higher education. And it's I I feel I mean, I always hesitate because I feel like there's so much there's so many revolutions and so much change in the past. Right? So I I I I kind of bristle when people say that the current times are unprecedented because you you can find so much chaos and so much tumult in in history. But boy, does this feel like it's really changing pedagogy.

Bryan Cantrill:

I I I mean, it feels like this is a is a very broad based change. And by this, I I think for the moment, I would we should maybe stick to LLMs, I would say. I mean, I would love to get your guidance on this because I think we've got another whole I mean, I've got other whole questions about the the the degree to which I mean and you had Rich Sutton on your podcast and he is is very outspoken about LLMs are actually sorry. This is actually not artificial intelligence. Because they don't they they they don't I mean, I have often believed that that AI creates some of its own misconceptions because it it calls on us to anthropomorphize it.

Bryan Cantrill:

And so you must have kind of at once be like, hey, these LLMs are actually like limited about what they can actually go do. But they can also go they are also indisputably really an extraordinarily powerful tool that will change kind of it feels like every domain at the university. But I would love to and it feels like it's very domain specific as well. But is that I mean, are you seeing this? This must be very must have a very broad impact, one.

Bryan Cantrill:

And two, is it unprecedented? Are there other similar changes you can look to in history to help guide this?

Michael Littman:

Yeah. I mean, it's a it's a great framing. It it is definitely the case that it is extraordinarily disruptive. Right? I feel like for a while we really liked that word disruptive.

Michael Littman:

We're like, yeah, we gotta make this kind disruptive. They're like the things that we had that were working don't work anymore and we don't have great ideas about how to replace them. So this is kind of a mess. It's impacting professors all across the university, right? So there's, I talked to someone in the English department.

Michael Littman:

The person was like, basically, I hate you. Your field is destroying my field. And I'm like, no, no, no, no. Don't say that at all. We're destroying our field too.

Bryan Cantrill:

Computer science, I wanna have that conversation too because I think computer science, there are a I mean, I in some ways actually like computer science, actually, was funny to go back to that English professor. You're like, actually, my field is ground zero for this. Like, you think you you struggle. At least you know that when you see an Emdash, it's it's a bullshit chat GBT Emdash. Like, we you know, if it's correct code, it's very hard to know if it's machine generated.

Michael Littman:

I'm such a big M dash user that this issue just really, really is hard for me, but, yeah. Actually, first of

Bryan Cantrill:

all, we are, and I I've been saying this repeatedly, we're going do an entire episode on the M dash. And I know that sounds like a threat the way I'm phrasing it,

Michael Littman:

but It should be three it should be three episodes, one right after the other with no spaces around.

Bryan Cantrill:

Oh, played. The but the m dash

Michael Littman:

yeah. Go ahead. Sorry. Go ahead.

Bryan Cantrill:

So well well, one of the things that I have really learned is that like the way I use an Emdash and the way that an LLM uses them in Emdash are actually fundamentally different. And the the well, because I use Emdash though and I think broadly, think good writers use an Emdash. But I definitely use an Emdash because I'm trying to give you guidance about how the sentence should be spoken. I want you to hear me speaking the sentence and the Emdash is is guidance about how I would read the sentence. And that's not in an LM will use an m dash to actually even a sentence.

Bryan Cantrill:

So what I will use an m dash to actually stop the pacing. I wanna actually stop you and then hit you with a punch line. Sometimes literally, Adam would probably say. The versus an LLM will use it to even the pacing. And I've been reading way too much LLM generated content.

Bryan Cantrill:

We have a very writing intensive hiring process and I've gotten good at detecting detecting LLM author content, but it there's a there's a real difference in the way the ambush is used. Anyway, look forward. Yeah.

Michael Littman:

That's fascinating. I didn't yes. I hadn't I don't think I've seen it so much in the chatbot stuff, but maybe because I'm not reading so much chatbot stuff because it seems, I don't know, painful to read.

Bryan Cantrill:

It is. Yeah. It is. It is pain it is did you know Mike Cafferall? Michael Brown.

Bryan Cantrill:

Did

Adam Leventhal:

the Very

Michael Littman:

well. Yeah. Yeah. Yeah. We actually did some research together.

Bryan Cantrill:

Yeah. That's great. Yeah. So so the Mike's been on the podcast before, and Mike refers to as norm core, that the ChatGPT is very norm core. It's giving you this very, like, normal, you know, kind of the top of the bell curve kind of an answer.

Bryan Cantrill:

And, that that becomes its own kind of grading after a short while as it turns out. So are me about your conversation with the English professor because that's super interesting. So the English professor is like, finally, my nemesis has arrived and only one of us is gonna leave here alive. And you explained to the English professor like, no, I'm actually I've I've come here as a kindred spirit. What how does that conversation go?

Bryan Cantrill:

What what what is their thinking about how their domain is affected by this?

Michael Littman:

So I've tended not to to follow with the, no, no, no. My field is also getting disrupted maybe even more than yours. Because like that doesn't that doesn't the discussion doesn't go anywhere productive. But it is definitely the case that they have very strong feelings and it's, I think it's been evolving, right? I think in the beginning of the post chat GPT world, it was very much like we need to catch people.

Michael Littman:

Then it evolved into, at the moment, I think really the main thing that people are concerned about is this is making people not learn, right? It's just making that interaction so effortless and the creation so effortless that there's no gear enmeshing. There's no friction at all. And so the brain doesn't engage. And so the students are not learning.

Michael Littman:

And that's way scarier. I think it pisses professors off. They're like, Oh wait, we can't do things the way we used to do things. But it's an existential threat to think, Oh, this technology is now blocking people from learning. That really is my job as a professor is to try to help people learn.

Michael Littman:

And to the extent that I can't do that anymore, well, the heck am I even doing? And I think that's, it's really evolved. The fear has evolved into that form now. Like what the heck are we doing to these people's brains?

Bryan Cantrill:

And what's the that's certainly a quandary. And then what is because I think that also there's a great potential, great potential, which I I think I've seen in my own kids. Certainly my my 18 year old is a freshman in college and is uses LLMs a lot, but not to but to actually aid in his, you know, he has LLMs, you know, he's for his math course, he was having an LLM generate practice exams. Know, the things that so he was feeding it the practice problems that he'd been given and say, hey, can you know make 10 more like these? And then he was doing them out.

Bryan Cantrill:

So we there are lots of ways in which I'm seeing I know Adam, you're seeing it in your own son as well Yeah. About the the kind of the way like once you get past that like no, but how can you these things are actually extremely powerful and you can put them to good honest use that actually helps learning. I know I've got this like super controversial I take. It could actually help you learn better. It's possible.

Bryan Cantrill:

I don't wanna

Michael Littman:

No. I absolutely agree, but it's but the thing is that it doesn't come with the, like the button that you can push that says, now it's gonna be good for learning. Now it's gonna be bad for learning. There's this fractal boundary between those two things that we haven't really navigated yet. And so I think that's the concern is more, you know, here are these college students, maybe their frontal lobes are not fully developed yet.

Michael Littman:

And what they're given is the option of like, here's a thing that can help you learn, but it can also make your life really, really easy. And that's so, so, so, so tempting, right? So I think But you can

Bryan Cantrill:

also, okay, so the counter to that would be you can also cut class. I mean, this is like part of the of the prefrontal cortex development that we have in from 18 to 22 is like, yeah, no one's taking attendance. You can cut every class if you want. It it you are short changing yourself by doing that. I and I mean, isn't it part of that education is like learning?

Bryan Cantrill:

I mean, that's

Michael Littman:

No. You're yeah. Absolutely. Absolutely. I mean, the hope is that it doesn't it doesn't block things and also leave you in a position where it's like too late to start learning things.

Michael Littman:

Yeah, mean, of the things that I've been trying to emphasize to the faculty when they talk to me about this is they kind of want, they want me to give them a fix and I don't think there is a fix. If there is, I certainly have it. So what I'd, one of the things that I've been emphasizing is like, because at that point you can be like, I don't know. We'll just see how this all plays out. But wanna give them something productive to do.

Michael Littman:

And what I've been saying is have conversations with your class about this. Like get it out the shadows because to the extent that this is a thing that students like sneak off and do in their dorm room, that'll, it's never going to turn into anything positive. But if we can just shine the sunlight on it and have these conversations and say, Hey, did you notice when you did it this way? Did it actually, like, did you feel smarter afterwards or did you feel dumber? And so I've been doing that in my classes.

Michael Littman:

This is maybe not a representative class, but this is my, right now this semester, I'm teaching my first class since ChatGPPD came out because the last three years

Bryan Cantrill:

I've teaching

Michael Littman:

medical science foundation. And so this was this was very new to me and I've been trying to be very open with the students. It's a 2,000 level course. So it's got PhD students and master's students and undergrads as well, but it's pretty advanced undergrad. So I'm not talking to freshmen yet.

Michael Littman:

I don't know how those conversations go, but at least with my, with my, you know, highly mixed advanced class, they're really, really very smart about, or at least thoughtful, right? About what it is that they're experiencing and how they're engaging with it. And sometimes they'll say, look, I needed to get through this assignment. And I wasn't like, I don't think I'm really blocking my learning on this because I really didn't need to know this anyway. But this other thing that I really think I do need to know, I've been buckling down, I've been avoiding using the chatbot where I use the chatbot in ways to kind of support my understanding, not to supplant it.

Michael Littman:

And so at least what I'm getting from the students show up to class, all of them, but the ones who show up to class is this really interesting analysis where we're working on this together. None of us know the answer to this. I don't view what they're doing as being immoral. Think they're, they're facing some real challenges and some choices. And to the extent that they've got good answers, I want the other students in the class to know them.

Michael Littman:

And so it's, that's been really, I think that's been very productive so far, but again, it's, it's early days. So I can't really say what the long term impact has been.

Bryan Cantrill:

I think that's a great idea. Because I think you've got I mean, because you have young people that are themselves trying to grapple with this. And this this technology has been around long enough that they've all used it quite a bit. Right? And and they've also like, they've

Michael Littman:

all used it to cheat

Bryan Cantrill:

at some point. At this point, like, by the time you have like an 18 year old, they this again has been around long enough that, you know, we we we feel it's been around for a very short period of time. It's actually been really usable.

Michael Littman:

It's been usable to cheat

Bryan Cantrill:

in high school for at least the time they've been in school.

Adam Leventhal:

So Yeah. For like a significant part of their education. Right? Like

Bryan Cantrill:

That's right. And so they themselves are, I think, trying to grapple with what does it mean to be educated? And I also can learning that if I just if if if I don't learn it myself, like, I am, you know, my own I'm I'm denting my own education. Right. I actually one thing I do wonder is, like, is it is it challenging at all the purpose of higher level education?

Bryan Cantrill:

Because I do feel that that there's a regrettable but understandable trend where the emphasis on higher education has really been on this kind of result at the end of it, namely a job. And you know, I and I think that's part of what has obviously fed our own domain. Computer science has become grossly oversubscribed and everywhere, but very much including Brown. And I I think that I mean that calculus, you you probably do wanna shift that calculus to like the purpose of higher education maybe to get educated. I'm just gonna throw over versus a a pathway to work.

Bryan Cantrill:

I don't know. Is any of that kind of discussion happening, Michael?

Michael Littman:

Oh yes, absolutely. A 100. And so it's earlier in this discussion just now you were saying, hey, this feels like is this really like an unprecedented moment in the history of education? And I feel like you do have to go back probably at least to the the fifties before you hit anything that's even in the same ballpark. But I think this is bigger than that.

Michael Littman:

And I think you maybe have to go back to the 1800s before you see like such a big realignment of what it is that higher education is trying to do and who it's trying And to so I really do think it's a very big deal. And so these discussions are happening. I think a lot of faculty very much didn't, I don't know how to say this. I don't wanna say that they didn't embrace it because that seems disingenuous, but it is the case that I think in their heart of hearts, most faculty feel like their role is to educate in the sense of helping people learn to be better thinkers, right? That's not just, I want to scram facts into you, but like, I want to help you.

Michael Littman:

I want to help guide you to the point where you can, you can engage intellectually with things that you'd never seen before.

Bryan Cantrill:

Yes.

Michael Littman:

I think most of my colleagues, that's what they think their job is. Yes, you know, what do we call it? Workforce development, right? So to the notion of getting people jobs, like that's an easy story to tell. That's, you know, it has value in lots of different ways, but I don't think that's what compels people to become faculty, right?

Michael Littman:

What's drawing people to become faculty is this notion of like, Oh my gosh, this intellectual world is so rich and so interesting. Let me help you learn how to navigate this world because you're gonna be faced with all kinds of ideas that I can't even tell you about yet because I don't know about yet, and you should be ready. I do that could it could really fundamentally change the the the structure of what these universities are trying to do and how they're doing them. Do you find

Bryan Cantrill:

yourself also explaining the limitations of LLMs? Because LLMs are such persuasive bullshit artists that people don't realize that like there's actually there's it it feels a lot like intelligence. But ultimately, like I absolutely agree with Sutton that this is this is not that. This is it's extraordinary and it's important, but it is not actually thinking for itself. Do you find yourself having to explain some of the technical limitations to someone who may be, you know, an English professor who may write I mean or or maybe the English professor is is is the the the first to realize that it's actually limited in that capacity.

Michael Littman:

It's really interesting because I find myself constantly arguing both sides, right? To sort of point people. This actually has some utility on the one side and the other side is like, this stuff is fundamentally flawed and we have to be really, really careful with the way that we engage with it. I think my biggest success story so far is I've been working with folks at the Sheridan Center, which is Brown's teaching and learning branch.

Bryan Cantrill:

They

Michael Littman:

support education across the university and they're like education experts. So the main AI person there who I've been interacting with, I think it took me a while to convince them that these things weren't as good as they seem to be from interacting with them, but also from what you read in the newspaper. These are remarkable things and they're changing. They're gonna solve climate change. Like they're gonna do it.

Michael Littman:

They're so smart. They're so smart. They're smarter than any one of us and maybe all of us put together. And I was like, yeah, they're weird though because they have all that, but they also like make really dumb mistakes. And there's something about them that doesn't distinguish between those two cases, right?

Michael Littman:

Confidently give an answer that happens to be right or confidently give an answer that happens to be wrong. And from its perspective, those are completely equivalent. From our perspective, they're fundamentally different in a very, very important way. And so the more he gave him another way of looking at the stuff that he was seeing. And I think now he's completely on board with like, Oh yeah, it does these things really, really well.

Michael Littman:

But like, you know, don't depend on it. Like don't risk anything. Don't risk your life. Don't risk your education because you can't really trust these systems to consistently deliver what it is that they promise.

Bryan Cantrill:

Well, and I get it, I think of their own lack of agency. I think that you're much better off thinking of this thing as closer to Wikipedia than not. And like, how do you think of Wikipedia? How did Wikipedia I mean, because I don't know, Michael, were you I you must have been an encyclopedia kid. I was such an encyclopedia kid.

Bryan Cantrill:

I I I I I I love the encyclopedia.

Michael Littman:

In physical. Yeah.

Bryan Cantrill:

It the the the I just and I loved the physicality of the encyclopedia. And I I I love just reading the encyclopedia. It was great. And then, you know, yeah, maybe and I'd be in some conversation and drop some knowledge that, was obtained by the encyclopedia. Yeah.

Bryan Cantrill:

I would get would get disparaged for that now. Would and then ask sarcastically if I'd gotten that out of just reading the encyclopedia and I just, you know, would kind of, maybe become a little bit quiet.

Adam Leventhal:

I I Brian, I'm gonna one up to I love the smell of the encyclopedia. I feel like

Bryan Cantrill:

I love this encyclopedia. I didn't wanna say that because I thought it was gonna sound too weird, but, you know, thank God. I'm I'm at the you know, I've got the right cohost here. No. I love the smell of it.

Bryan Cantrill:

I loved the I I mean, there's were you a world book guy or it's like a botanical guy? This is almost a

Adam Leventhal:

World book at my nana's house. And I would

Bryan Cantrill:

I I knew it. Go in and I I

Adam Leventhal:

knew it. Beeline to it.

Bryan Cantrill:

No. Me too. So look, listen, TI and HP divides us and it's a division that we can never actually breach because we can never cross. But no, absolutely World And the whole like, you know, there was like this encyclopedia Britannica. Like this idea that in an encyclopedia should be insufferable and should be it should be unapproachable.

Bryan Cantrill:

And I loved World Book because it was so approachable. And Yes. And what do you yes. Michael, were you a Michael, meanwhile, is a seething Encyclopedia Britannica person being like, I didn't you had a podcast with World Book people. I cannot believe it.

Michael Littman:

I had World Book in my bedroom. It was it was like, basically, the two main things in my bedroom other than the bed were my t r s 80 and my World Book. And that was yeah. Was go.

Bryan Cantrill:

Let's to the Jetta JetX Central. Oh god. That's so great. And I could see it. And so you had a also, first of you had a trash 80 in your bedroom.

Bryan Cantrill:

Like, you were did you have the GI Joe aircraft carrier as well? I mean, you got hit because that's like the I mean

Adam Leventhal:

This is what Brian says whenever I'm like, oh, I had a Macintosh in 1984. It's like, wow. You must have been one of those kids.

Bryan Cantrill:

They're one of those kids. One of the J. J. Aircraft Carrier kids. Like, listen.

Bryan Cantrill:

We know the the those kids just got they were just given a little too much. But the

Michael Littman:

computer I got, it was it was my parents said, look, you're turning 13. You can either have a bar mitzvah, which is gonna be a big party and cost us a lot of money, or you can get a computer. And I was like, oh my gosh, that's really

Bryan Cantrill:

easy. Okay. You earned it. That's impressive. That's impressive.

Adam Leventhal:

A bunch of Hebrew or get a computer like is this an intelligence test?

Bryan Cantrill:

Like, what

Adam Leventhal:

are we doing here?

Bryan Cantrill:

Right. We will cancel your entire thirteenth year. You'll go straight from 12 to 14, but you can get a computer. You're like, okay. Yep.

Bryan Cantrill:

That's I that that's a deal I'm willing to cut. I think I that that's awesome. That is extraordinary. But the you know, and I think that then when, you know, like and someone in the chat had mentioned Encarta, you know, when that came out on CD ROM, you're like, oh my god. It's a it's an inside like, get the entire encyclopedia.

Bryan Cantrill:

And Wikipedia is great too. Like, I I I you know, I'm I miss to a certain degree browsing encyclopedia, but I've spent plenty of time browsing Wikipedia and going down very strange Wikipedia holes. And it's great. Like, Wikipedia is net positive for us all. And but there's some negative too, obviously.

Bryan Cantrill:

Like you I I don't know, Michael, what do you think about, like, thinking about it as closer to Wikipedia than a than than thinking of it as kind of a a doppelganger?

Michael Littman:

So there's a professor at the University of California, Berkeley, a cognitive scientist who gives analogy that I really, really like, which is basically, these are You can create kind of chain of developments in terms of information delivery that goes from books to libraries, to the internet and Wikipedia to LLMs. And so viewing it as an information access device or medium is I think it really puts it in an interesting frame, right? It's not in some ways not really even designed for that, right? It's really designed to just like pick up on patterns of words. And we've had models like that for a really long time in the field.

Michael Littman:

But somehow when they were trained to use a lot of context, the one way to pick up on the patterns of words is to kind of understand where the thread is going and to be able to complete sentences and to know stuff about the world. It kind of takes you to this, okay, well now it's just a world of information and you're exploring it, but you're exploring it in this probabilistic way. There's no books. There's no barriers between ideas. It's all just an amorphous blob.

Michael Littman:

So, yeah, I think that's not a terrible way to think about it. It's Wikipedia, but kind of the next step, right? Because we're not interacting with the words as they were written by whoever authored it. We're interacting with these statistical patterns that have emerged by looking at how people talk about things and it can interpolate, right? It can actually give you combinations of ideas that were never in its training data because it's picked up on how those idea combinations can actually be made to work.

Michael Littman:

So it it it fills in a lot of the gaps in some ways of traditional writing and publishing. That's it's so, yeah, I think that's actually a cool way to look at it.

Bryan Cantrill:

Well, I mean, and I just think that the the information comprehension of these things is just off the charts amazing. And I think that it's just like pure net positive. Like the ability to have something and to to I mean, and just uploading my own writing. Because I and I absolutely use LLMs as an editor. It's kind of a final pass editor.

Bryan Cantrill:

I I kind of amusingly, they they tell me what I already kind of know. Like, hey, this transition sentence is a little rough or and I think it'd be interesting to actually keep track of this, but I think I have actually deleted more based on the feedback of LLMs. Or it's like, you know what, this paragraph is not working. I'm actually gonna delete the whole thing. So it's like, yeah, did an LLM help me write this?

Bryan Cantrill:

I mean, kind of it told me to chuck a paragraph that wasn't really working. So I guess it helped me.

Michael Littman:

I love that. So I've been trying to push. So I don't think we have a lot of evidence to support the idea that that kind of editing is beneficial to people, but my intuition is strongly It's leading. Yours, which is No. Come on.

Michael Littman:

Yeah.

Bryan Cantrill:

No way. What? I think that's beneficial? That's crazy to me.

Adam Leventhal:

Evidence meaning beneficial. Michael, you mean, like, people have not researched this and Not

Bryan Cantrill:

that I mean by demonstrated. Oh, yeah. Evidence. Evidence. Yeah.

Bryan Cantrill:

Yeah. Yeah.

Adam Leventhal:

Not anecdote, Brian.

Bryan Cantrill:

Oh, I mean, are really just two different words to the same thing. Are they not? Yeah. Fine. Fine.

Bryan Cantrill:

Fine. Fine. Yeah. We have not actually studied it.

Michael Littman:

Right. But I think but I've been trying to push on the people in the public health school at Brown. Like, can you guys study this? Because I think actually there's

Bryan Cantrill:

Please.

Michael Littman:

Because what I learned, so I wrote a book, it's called Code to Why Everyone Should Learn a Little Programming. And it was a really fun experience to do the writing. People told me how horrible it was to write, but I actually had a blast. And I recommend it if people wanna pick up my book. Is, one of the things I discovered was at the end I hired what's called a developmental editor, like an editor to actually go through and read my book and then to give me feedback and to help make things better.

Michael Littman:

And not only did he do that, he helped make the book better, but like the way that he felt, the things that he focused on and the suggestions that he made helped make me a better writer, right? So I left this not a worse writer, but a better writer. Whereas I think a lot of things, a lot of times people use these chatbot tools to write for them, which causes them, human to become less good writers. I think this, you know, editing is actually a way to help make someone a better writer. But again, So, I don't have documentation for that, but that is my sense is that these tools could be really, really helpful at that stage.

Michael Littman:

I've, I got hurt in anecdote. I actually visited the anthropology department. I've been trying to do the rounds at Brown to, to learn about what people are seeing in different parts of the university. And one of the professors in anthropology told me that he had had a student who was using chatbots like this, that he would ask the chatbot to write his assignment for him. Then he, the person would rewrite it at a sentence level.

Michael Littman:

So it would be in quote unquote, his own words. And this is maybe like the worst possible combination of the skills of the two because now the person doesn't, the human didn't do any of the thinking. But then the one thing that the chatbot is really good at, which is making prose that just like slides through. It's like effortless prose has become like totally torqued. And the the way that he got was the professor read this and like, this is so weird.

Michael Littman:

It feels like the sentence structure of a chatbot, but not the word choice.

Bryan Cantrill:

Amazing. It's like it feels like the sense of a chart, but you're using bra and myth and And

Adam Leventhal:

a bunch of misspellings and

Bryan Cantrill:

A bunch of stuff. Yeah. I mean, that's an okay. But that's that's an is an interesting idea because I I mean, like, obviously, a bad idea for the student. Also though, like, by forcing them to go through that sentence by sentence, are they I mean, to me, it's a little reductive to say that you don't learn anything in that exercise.

Bryan Cantrill:

It's not I I wouldn't recommend it. Wouldn't wouldn't recommend it, but the you know, I've always thought that, like, certainly in high school, like, have people read things aloud because the LLMs are gonna write things that they wouldn't they wouldn't write. And having them read it aloud, you're at least are gonna get to some writing that is actually probably better or certainly better than than what they're doing in high school. And I mean, now we want to be propelled beyond that. Obviously, we wanna learn how to truly think.

Bryan Cantrill:

But I'm not sure that being exposed to a bunch of really good writing or really good. A bunch of like Really good writing. Well, in the norm core writing. Yeah, exactly. I don't know that that is I mean it's I don't know, I'm mixed on it.

Bryan Cantrill:

It's not as clear to me that this this is not that that has zero pedagogical value. Although admittedly, in that case, pedagogical value seems low.

Michael Littman:

That's

Bryan Cantrill:

fair. Actually, a question for you because that's coming from the anthropology department. Are you I mean, you must be it must be fascinating for you to see the different reactions from across different departments. I imagine there's at least one surprise of a domain that's like or may maybe even anthropology where it's like, hey. I don't know if you know what we do, but we actually study the history of humanity.

Bryan Cantrill:

We actually we have some things to say about this that you might find interesting. I mean, are you finding any of that anywhere?

Michael Littman:

I I had I I met with a history professor in my office, and it he was he's a very he's very strongly and vocally AI skeptical. I think that's a nice way to say And it didn't take very long for the conversation to devolve into a lecture from him to me on capitalism and Marxism and how basically we're screwed. And so

Bryan Cantrill:

Okay. Well, this is going well. I'm the AI provost and well, anyway, let's stay in touch.

Michael Littman:

So that was really, really interesting, especially because it came right on the heels of a meeting that I had with Brown's chief investor who is not only embracing AI, but she's also really digging very deeply into, well, let me understand the entire AI value chain because I wanna know where to invest. I wanna know where there's a place where if we put some money down, this is gonna redound to us. Right? And so she was so grounded and she's like, whenever she uses AI, on her team, they can say, well, this is actually saving us like X minutes investment decision or something like that. It's just so grounded and so empirical, right?

Michael Littman:

In a way that the history professor was just became very philosophical and sort of high minded. And these are all campus members, right? The campus has everything in between there. All the possible nooks and crannies, all the possible perspectives are represented someplace on campus. And yes, it is absolutely fascinating.

Michael Littman:

And I wanna support everybody. Like I'm not gonna say, Oh, history professor, I don't like what you're saying. It's confusing to me. So please don't talk to me anymore. Because, you know, they're part of the game too, and they're being impacted and they have they have interesting perspectives on it.

Michael Littman:

And and I think we I think we all need to be talking to each other.

Bryan Cantrill:

Totally. And then and so do you kinda come to them? Because I I imagine, god, must be some of these first meetings must have been like, oh, the AI provost, I've been waiting for this meeting. You're like, oh god. Do you kind of offer yourself up as a resource for them?

Bryan Cantrill:

How do you kinda structure your relationship to them?

Michael Littman:

Yeah. I mean, but mostly I try to I think, okay. One of the things I like to say is that my title is associate provost for artificial intelligence. Aside from the word for, nobody knows what any of those words mean, right? Like what the heck's a provost?

Michael Littman:

And is it maybe people could say, well, it's not a provost, it's an associate provost. Okay, now you've really lost me.

Adam Leventhal:

Can I talk to your boss please?

Michael Littman:

We don't have a great definition of that. So like it's, yeah, it's all very, very murky. So I usually start these conversations off saying, look, this is what we think we mean. When we say that, this is what I think my job is. And it very much is to support the campus.

Michael Littman:

I like to say that navigating the opportunities and the risks that are suddenly thrust upon us. I don't want people to feel like this is something that I'm doing to them because I'm really not like, this was not my thing. I mean, was my thing.

Bryan Cantrill:

You're like, this is not my thing. Aren't you an ML researcher? It's like, listen, okay. Not a complicated answer, but I feel like you're trying to drive me here a little bit.

Michael Littman:

That's right. It's like, this is not my fault, except that's kind of my fault, but it's also kind of not my fault. But let's let's let's put that aside for a second and talk talk about you. How is this impacting you?

Adam Leventhal:

Right. Let's talk about you, not me.

Michael Littman:

That's right.

Bryan Cantrill:

Right.

Adam Leventhal:

It's you. Michael, you talked about, you know, some of these discussions with other departments approaching it. Talk about your your two excuse me. I almost said 200. I'm I'm a I'm a order of magnitude off, you know, because we used to have 200 classes, not 2,000 classes.

Michael Littman:

But Yes.

Bryan Cantrill:

It's for Adam. Exactly.

Adam Leventhal:

Right. But for and I can imagine for grad students, master students, advanced undergrads, they see, like, I'm only in this class because I wanted to be. Nobody forced me to take this class. So presumably, I wanna learn something from it. What are you seeing in particular with introductory or intermediate computer science, and how how are LLMs being embraced or or what guardrails are put on there, or or how are students being educated about appropriate uses?

Michael Littman:

Yeah. That's a great question. I I don't have a lot of visibility into that yet, but I do know that so CS 15, which is probably the biggest intro. Okay. It's not it would be what is it really?

Michael Littman:

A 150?

Adam Leventhal:

A 150?

Michael Littman:

Yeah. I guess you gotta stick a zero on the end now. But I'll see s 15. Everybody calls it c s 15. So this is a this is a long standing introduction to computer science class focused in now it's on Java.

Michael Littman:

I think in past instantiations, was Pascal. And before that it was like, I don't know, rocks. Like it's a really, really old class. And when ChatGPT came out, it was very clear that the students were just like, Oh, let me just prompt it to generate the Java code for exactly what the thing that I was just asked to. I mean, chatbots are so perfectly designed to take as input homework assignments and to produce as output.

Adam Leventhal:

Large corpus, It's

Michael Littman:

so perfectly tuned to that as, and so, okay, so students realize that the class has gone on to, first of all, embrace the use of chatbots in the sense that it's always been a challenge in that class. There's sometimes there's 400 some odd students in there and there's a lot of handholding. The TA crowd is like, it's like 40 people I think that actually are just the TAs in the class than your typical non computer science class is just the TAs in the class. So the, and the lines waiting to get help from the TAs are interminably long. And so the, they, they decided to adopt a kind of a TA, a chatbot TA and, and which they try to very carefully prompt not to give away answers, which by the way, have to do with the human TAs too, because the human TAs also are really easy to tempt into, Oh, open.

Michael Littman:

Like if you just explain this to me, like just once it's concrete, then I'll understand it. Just tell me how to solve this one problem that happens to be the homework problem. So it was already a challenge getting people not to be too revealing. They're, they're also trying

Bryan Cantrill:

to make

Michael Littman:

chatbots not revealing of the answers. And I think that they have found it to be actually really valuable. That's great. I think they tell the students only use our chatbot. Do not go out of band with this because then you're gonna, yeah, then you really are cheating.

Michael Littman:

So I don't know how they're enforcing that. I don't know if they're finding that people are going out of bound, out of band, but that is a thing that we're seeing in the intro classes is that the students actually kind of really appreciate the opportunity to learn about the technology a little bit more. So this is their first attempt to really be exposed to what these things are and what they can be used for. But also they're taking steps in the class to make sure that this isn't going to subvert their ability to actually learn how to program.

Bryan Cantrill:

Yeah. Interesting. And and they're finding it because I mean, I also imagine that there was, especially in those intro courses, TA bandwidth was a real issue, you know, that you would end up with a lie a line at the TA Room, you know. And this is a way to actually really eliminate the line line at the TA Room when you've got a chatbot that you can go and you can also ask a dumb question too, which I also think is one of the the very pedagogically valuable things about a about LLMs is that you can ask them anything. You can ask them moronic questions, and it's not gonna you don't have to feel like you're being judged.

Adam Leventhal:

What do think? You don't have you don't have a nineteen year old who's looking down as you as an 18 year old for the dumb question you just asked?

Bryan Cantrill:

Oh, absolutely. And I mean, how many times I mean, did you like, are you as a TA, like, you're holding hours and you've got somebody who's kind of it has the courage to ask a question that they think is a dumb question, but it's actually I mean, I've always found that people ask are asking questions that they think are dumb questions, they're actually they're asking basic questions that are often very insightful. Those questions are actually very important often. And I always found it was like the older students, especially grad students. Grad students just like don't give a fuck about what you think of them.

Bryan Cantrill:

So they would always ask like, oh, I'm not just gonna ask the questions I got. Like, actually, don't I'm sorry. I'm too old to worry about how I'm being perceived. And there was this kind of phenomenon where, like, when someone would ask a question that everybody has, but they're all afraid to ask, they're all, like, kinda throw the round for the answer. Like, oh, yeah.

Bryan Cantrill:

What is the answer to that actually by the way? Now now that this this moron has asked this question, may I please get the answer to that as well? But I feel like that is a a really I mean, that's a powerful aspect of these things that they are that they kind of, that you have this privacy around what you're asking it. And that to me, like that's, that has potential great pedagogical value.

Michael Littman:

Yes. I think, I think this is in fact documented. So this is turning out really well. And I totally believe it because I've been noticing that I will not only will I turn to a chatbot and ask dumb questions, but if I have an ongoing conversation with the chatbot and then in the middle of that, I have a dumb question, I will actually go to a different chatbot because I don't wanna embarrass myself in front of the one that I'm just And so like, this is very clearly a thing that is motivating for

Bryan Cantrill:

humans. God, that'd be great. And also if they just, you know, the the the sycophancy of these things is obviously turned up a little high. It'd be great. You know, I'm I'm waiting for them to start turning that down a little bit.

Bryan Cantrill:

Be like, that is a stupid question. Mean, on my computer, I'm obviously gonna answer it, but holy God, that is just dumb as a sack of rocks. I know. I thought you were like a pretty smart guy. Like, you're asking good questions before, but now like, woof.

Bryan Cantrill:

Did you kids get up with you or something? What's going on? I need

Michael Littman:

to reassess my my my belief about you for sure. Yeah, no, totally. It's, I know that some people have been finding it really useful to install a system prompt that says, stop sucking up to me and just like, just please pretend. Like, I think they literally say, will say this, pretend you're a robot. Which is just so funny because it like absolutely is a robot.

Michael Littman:

And so the idea that it it's a robot that needs to to act like

Bryan Cantrill:

a robot needs to pretend.

Michael Littman:

It's pretty funny.

Bryan Cantrill:

But has very good Eastern ideas. I mean, you have to acknowledge that, you know, some of this praise is really well placed.

Adam Leventhal:

I mean,

Bryan Cantrill:

it's like it's not all wrong about and I know. I've I've been the the second fence date could be a little bit a little bit much, but also it's like, well, okay. Well, let's leave it on. Let's leave it on for now. Right.

Michael Littman:

Right. Right. But I really am smart. It's just, you know, I just don't want it to be pointed out constantly. Constantly.

Michael Littman:

That's That's right. Could you

Bryan Cantrill:

be slightly more subtle about it? I just it just feels a little bit disingenuous the way you're saying it all the time. Just some of the time. Yeah. That is that is really interesting.

Bryan Cantrill:

And then are are you in terms of of working with are you working with individual professors? Are you gonna taking templates for what work and then and then taking that elsewhere? Are you

Michael Littman:

Yeah. This is this that's great question. So so I am meeting with individual professors, but I think that's mostly me trying to get up to speed and get kind of get the pulse of what's going on and how people are thinking about things. I think moving forward, we do have some programs run by the Sheridan Center where they work with individual faculty to help redo their curriculum or kind of rejigger their course structure so that it's, you know, chat bot friendly or supportive or not gonna be undermined by chat bots. So a lot of that one on one handholding to really dig in and like change a course is being handled by the teaching and learning center.

Michael Littman:

But I'm still really interested in talking to individual faculty just to get a sense of like, what are they actually concerned about and what are they doing? Mean, they're real, they're all very, very smart people, right? So they've had to figure out something and what did they figure out? And so it's been, that's been really amazing just to get a window into that.

Bryan Cantrill:

And are you, I mean, I assume you're asking all of them to actually use this stuff Because if you don't use it, I don't think you have an intuition for it. It it may be extremely distasteful for some professors to I mean, you're that the history professor that was taking this opportunity to to lecture you. But I think you've got to to understand both the the the peril. I mean, this is what, you know, when we had Simon Wilson on, Adam, and he was describing the importance of running LLMs on your own laptop where they will be so underpowered that they will they will make really obvious mistakes. And so you will learn that actually this thing that you're running that is when you're going to Claude or Chachi Petit is actually just much more powerful but has all the same flaws.

Bryan Cantrill:

I think it's I mean, you've got to use these things to learn how they're actually flawed and the potential that they actually have.

Michael Littman:

Yeah. So I've definitely been encouraging people to play with it. I don't know that everybody really needs to adopt it and like become a power user, But I think it's really hard to even be engaged in the conversation and appreciate what the issues are, unless you've kind of played with these things and seen what they can do and what they can't do. So this spring before I formally started, but you know, but I knew where I was headed. We announced that Gemini was available on campus and it was supported in a sense, right?

Michael Littman:

So that it was sort of cleared for use with sensitive data. So we've been telling, we had been telling people like, be really careful when you use chatbot because anything you say to it like can and will be used against you in a database. And so it should be the case that if you're gonna, if the university says, Oh, this is okay to use, we should have an assurance that that data isn't going any place that it shouldn't. And once we got that assurance from Google, we then announced to the campus, Hey, it's now safe, quote unquote to use Gemini, at least with as far as data leaks is concerned. And so when I talk to people about AI on campus, I say, Hey, look, this is a thing that we actually now have officially been made available.

Michael Littman:

If you haven't played with it yet, you should totally play with it because it's so, it's so interesting what it can do, what it can't do. And you should, you're not gonna understand that unless you actually engage.

Bryan Cantrill:

I'm actually I'm measuring

Adam Leventhal:

the baseline at least saying when when they make assignments, ask that assignment to Gemini and see what it comes back with.

Michael Littman:

Yeah. Have more nuance on that now because this is, again, this is my first class that I've been teaching since ChatGPT happened. And I took my first homework assignment and I entered it into Gemini and chat GPT and it got everything wrong. And I'm like, knew it. Knew knew I was cat cockroof.

Michael Littman:

And then I told this to my TAs and my TAs said, But did you first upload the course notes? And I was like, No. So then I uploaded the course notes, like the relevant lecture for the homework. And it got, not only did it get everything right, but it had insightful comments about two of the questions that actually revealed some of the mathematical structure that I hadn't previously realized. I was like, I hate this.

Bryan Cantrill:

This is like,

Michael Littman:

this is really something and I don't know what to make of it. And It's extraordinary though.

Bryan Cantrill:

I mean, fact that like I mean, again, that they it's like it's the information processing aspect of this thing that I find extraordinary. And one thing, by the way, I would recommend for every professor, because I mean, the nice thing you've got there are certain commonalities across all the professors. All the pressors have published. They've all published work. And I don't know if they've got access to Notebook LM.

Bryan Cantrill:

If you use Notebook LM's podcasting thing, that is like mean

Michael Littman:

I mean, it's not as charming as the two of you, obviously.

Bryan Cantrill:

It's it's close. I mean, I I mean Sounds like an LLM. Sounds LLM. Like like an LLM would say. The I mean, I and I was kind of like, oh, know, kinda play around with this and just like uploaded, you know, a blog entry to it.

Bryan Cantrill:

And to listen to an entire podcast about a blog entry, I mean, it was it was jaw dropping. And I think that like and I think that there's, you know, a natural like narcissism of of academia. Sorry, academia. Guess that was not meant to be pejorative, but I guess it was. But the know, listening to a discussion of of a paper that like you put a lot of time into, but probably, you know, may not have been broadly read or brought it's like to listen to something not just seemingly care about it, but actually like understand it and comprehend it in a really deep way.

Bryan Cantrill:

I think it's like a a watershed moment for people. Like listening to work that you know was not LLM assisted. This this you know, that this paper that you wrote, you know you didn't use LLM. And to listen to it actually or read it's it able to to I mean, I think it's extraordinary. And I think that that that is something that every professor at the university should do at least once to understand the power of these things.

Michael Littman:

It's it's yeah. It really is kind of charming and fascinating and jaw dropping. I did bring it up in front of, let's say, I spoke to the group that the Carney Center for, I don't think, I think it's neuroscience or neuroscience and cognitive neuroscience, but, but yeah, a bunch of them were like, Oh yeah, we've definitely done that. It's funny because I actually have a, my reaction to it was well, to be super impressed because it's like, it was weirdly fluid. It really did kind of get the structure of podcasts and conversations about technical topics.

Michael Littman:

It felt very sort of NPR in a lot of ways. Like it was really, it was really remarkable. But it didn't answer the questions that I actually had. And I discovered at least about myself, I think I'm a very mean paper reader. Like I'm very nice when I talk to people, but I think when I'm reading a paper, I wanna like find the jugular and rip it out.

Michael Littman:

Like I wanna know like what's the main claim being made here and do I buy it? Like, I, is this, how fragile is this idea? Or can I incorporate it into my own understanding of the world? Or can I kind of dismiss this as, okay, well you did a thing, but it's like, it doesn't really change my underlying views? And the LLM, the notebook LLM

Bryan Cantrill:

I think I've gotten the feedback verbatim actually on a paper I submitted to a conference actually. I'm pretty sure.

Michael Littman:

Fair enough. But just the notion that I wanted to get at like what's the fundamental claim and how strong is the evidence for that claim? But the bot, the LLM bot as it were, very much takes on the voice of the author and really represents that well. Which is I think why you said it's really fun to have it read your own papers because like, well, yeah, I'm totally on board with that. But when I have it read other people's paper I haven't done it with my own papers, though.

Michael Littman:

Now I'm now I'm very tempted. But but I have it None of the

Bryan Cantrill:

people do it with your own papers. Yeah. And then I would ask them, then ask it to be more critical. I mean, you wanna ask to be like, hey, by the way, you know, you are now reviewing this for

Michael Littman:

a conference, rip this That's thing really interesting. Yeah.

Bryan Cantrill:

The notion

Michael Littman:

that you can actually ask it explicit to be critical. I hadn't tried that, but I, so recently there was a paper that was written where the press descriptions of it said, this AI discovered a new law of nature. And I was like, that's a strong claim. I would like to understand what that, the form of that claim is because I think a lot of, like if you look at AI and science, a lot of people are excited about what can AI be doing to help science? Most of the things that I've seen so far fall, well, I say everything I've seen so far falls into like three categories.

Michael Littman:

They're all really helpful, but they're not like doing science, right? They're actually just supporting the things that scientists need them to do. And so this paper was like, no, no, no. We've actually discovered a new law of science. Okay.

Michael Littman:

Then this is you, you know how the universe works better. Let me, let me dig into that. And so I've loaded it up into, I tried to read it myself and it's like, okay, can't read chemistry papers. But I put it in Notebook LLM and I said like, what's the law of nature that this found? And I was like, well, and it came very much the party line of what the paper described.

Michael Littman:

I'm like, yeah, but did it discover the new law of nature? And there's like, and again, gave very much the party line. And I was like, okay, but that's not a law of nature. That's just an association, right? And it's like, oh yes, of course.

Michael Littman:

You're so right. That was what an insightful comment. I'm like, yeah, I know. And the more I pushed on it, the more that I felt like I was getting to the limits of what the papers justifications were. And since it didn't really have a stronger justification than that, I think, I think it was not supported.

Michael Littman:

So it never came out and told me, yeah, yeah, yeah. This is kind of an overstatement, but that, but I came away with that as an impression from my interaction and that I'd like the tool to be more helpful about that.

Bryan Cantrill:

Yeah, but I think when you were kind of getting that back and forth about a written corpus, it is actually So when we had our, the AI materials and fraud episode with Ben and Schindle talking about this fraudulent economics paper that MIT retracted. Michael, if you remember this from last year. And the the paper is a paper that's too neat basically is the problem with the paper. And it was it was you may recall this was material scientists that were supposedly using AI to discover new materials. It turns out like the thing is just totally made up.

Bryan Cantrill:

It was really interesting to iterate on that with an LLM and with ChatGPT. Because ChatGPT versus like, I uploaded it. It's like, yeah, this seems like wow. It seems like great paper and yeah, amazing stuff. And I'm like, know, there's some kind of concerns about this and some concerns about kind of like giving it some things to look into.

Bryan Cantrill:

And man, once there was blood in the water, it was just finding all sorts of things that I had also missed. You're like, wow, that is that is and of course, you know, praising me along the way for being such an expert. It was really interesting to have this kind of to take a corpus and to have a iterative discussion. Then you like discover things that are like, well, is like emphatically true about the corpus. And like, yes, this is a discovery that was facilitated by a discussion with an LLM, but we've actually made a discovery about the corpus is actually important.

Bryan Cantrill:

If I were certainly if I were reviewing papers, I mean, what fraction of a program committee is using LMs review to at least aid in the reviewing of a paper? It must be a very high fraction. It's gotta be. Right?

Michael Littman:

Mean, a lot of them are saying don't do it. There's some conferences that have started to say, we're gonna have an explicit LLM reviewer, but then the other reviewer should not be doing that. But, I don't know.

Bryan Cantrill:

It's hard to know. That's a mistake. That is a I think that no. I really do. I think that is a mistake because no.

Bryan Cantrill:

That is a that is a huge mistake because that is anthropomorphizing LLM. That's saying, no. The LLM is this, strange robotic peer of ours. It's gonna do a review and then we're gonna, I don't know, look at that or not. But it's like, no, no, everybody should be using this as a tool to like, you're reading the chemistry paper being like, I don't know though, this doesn't sit right.

Bryan Cantrill:

Like, help me explore this a little bit. I'm not looking for the LLM to give me an answer. I'm looking for its power to absorb a huge amount of text quickly and and leveraging that and its generative capacity to go iterate on this paper to help me understand the core ideas faster. Like, this doesn't that that it feels like that just feels like in a a very straightforward and appropriate use, but that's amazing that conferences are not doing that.

Michael Littman:

At least not explicitly. Yeah.

Adam Leventhal:

Yeah. Wow.

Bryan Cantrill:

Okay. One of the question for you. The because I know at least at the high school level, one of the things that there's there can be an asymmetry where teachers say, hey, nobody can use an LLM. And then they very clearly use LLMs to grade assignments. And Mhmm.

Bryan Cantrill:

And that hypocrisy is something that I I think that speaking as a parent of of a former high schooler, that hypocrisy is seen by the students. And they're like, wait a minute. So hold on. LLMs for for the but not for me. Do you have some of the same challenges at the university level?

Michael Littman:

That hasn't come up yet. It is definitely the case that no faculty member has said to me with a with a straight face, I I would really like to be using this to just help with grading. So so that hasn't come up yet. It's almost certainly needs to be addressed. But I you know, there is an asymmetry.

Michael Littman:

Like, I get it. I get it. And we need to be really careful about it. But at the same time, if the goal of the class is for the students to learn, then why does it need to be the faculty member who can't use the LLM? If it's with the student learning, it's not necessarily a bad thing.

Michael Littman:

I think it smells wrong. It's a bad look, right? The sort of idea of like, well, this, shouldn't have any support, but I have all the support I want. But I don't think it's necessary. And it does seem problematic, but I I don't think it's necessarily like a a bright line saying that this there's no way you should never do this because again, there there is a fundamental asymmetry between teaching and learning.

Bryan Cantrill:

Yeah. Well, and so it sounds like your fundamental guidance is this kind of conversation. I'll also say, I think it's great that you are currently teaching a course. I think that is because it means that you are experiencing this for yourself. I mean, obviously, have taught a lot, but you're you're you are are experiencing this real change yourself.

Michael Littman:

Yeah. Yeah. Absolutely. I think that's fair. I'm hating it because it's it's a lot of work on top of a new job, which is a lot of work.

Michael Littman:

But you're absolutely right that this is I I do feel like I have more credibility because I'm I'm I'm I'm I'm I'm not just reporting on other people's experiences. I'm having these experiences myself.

Bryan Cantrill:

And then do you think it's it's Brown, I think was certainly what drew me to the new curriculum and draws many people to Brown. So Brown had a a revolution in curriculum in 1969, ditched any idea of a core curriculum and and allows students to take what they want. I mean, they basically trust students with their own education, which I think is was tremendous. That was certainly what the the the big attraction for me to Brown. And then of course, took an SCB in computer science, dictates like, you know, '29

Adam Leventhal:

classes or whatever it was.

Bryan Cantrill:

But you know, yeah yeah yeah yeah came in for the right reasons. Know, I I had the the do you view this as an opportunity for real leadership again where Brown can really lead the way in terms of how do we trust 18 year olds in this, which I know even as I say it aloud, I'm like, why are we doing It's a

Adam Leventhal:

dumb sentence. Exactly.

Michael Littman:

So one of the first things that I was tasked with when I took this new position is to head up a committee that will generate a report for the provost, basically talking about the use of generative AI in teaching and learning. So we call the committee GADL, generative AI in teaching and learning, which I feel like rhymes with dreidel. So I thought that was fun. And we've been meeting monthly. We're on track to have a document to hand to the provost in December.

Michael Littman:

So that's sort of how we're trying to progress on this. And I do think that one of our messages is gonna be that we, that first of all, that we have to make sure that whatever rules we institute are harmonized with that kind of open curriculum. This is sort of Brown's take on higher education because it's just so integrated into everything that we do. And so the idea that like AI is gonna be an exception to that is just gonna make it untenable. So the committee is very much like rereading some of this stuff about the open curriculum because we want to make sure that we're interpreting this moment through that lens.

Michael Littman:

So I think that's a positive and we're seeing like, well, and another issue is, okay, every university has had a committee like this. Like, why are we also doing this? Why can't we just take somebody else's like Cornell's report and adopt it as our own? And what we're finding is no, actually this particular teaching philosophy and this philosophy of education really is very relevant within the context of AI. And we have to make sure that we speak to that explicitly in a way that, you know, Cornell wouldn't do for us and Harvard wouldn't do for us, but like we should do for ourselves.

Michael Littman:

So yeah, I'm completely on board with this idea that we have to keep that front and center and have an opportunity to lead. Now, of course, not every university will be relevant to like Columbia, right? Where they have a very, very strong core curriculum. Core curriculum. Right.

Michael Littman:

If we, if we say, oh, well, is, this is the kind of flexibility you should have. Columbia can just say that doesn't apply to us. And that's fine because at the end of the day, we're not responsible for making decisions for them. But I do think that we're going to have a lot to say that will pieces of which I think will be helpful to to folks all over.

Bryan Cantrill:

Yeah. If we're reverse with my thought would be and I I always love when people are applying to both Brown and Columbia. It's like pick a lane. These are mean, because these are so different. These are such different pedagogical approaches and they each have value.

Bryan Cantrill:

And for me obviously, the Brown's approach. But you the new curriculum and I I think it's great Michael that you're going back to some of the things that were written about it in 1969 because I mean talk about another period of great tumult and you had young people not trusted an idea that that society I mean, you had, I mean, honestly in a much more violent fashion than we see even today. You had it's society kind of coming apart at the seams in a way that would not be totally unfamiliar. And the idea that we're going to actually like trust young people with their own education. I mean, I I think it's still a radical idea.

Bryan Cantrill:

It was a very radical idea in 1969. I think that like the the kind of the current age calls for a new kind of radicalism in that regard and really trusting 18 year olds and putting their own fate in their hands. And I think that like a part of the of the challenge that we are coming through is this idea that like the purpose of a higher education is to get a job at Google. And it's really not. And I I think that like we've got an opportunity and I think LLMs like kind of sandblast that off and and show that opportunity for what it is.

Bryan Cantrill:

Like this is what it means to really trust an 18 year old with their own education. Not every institution is is willing to do that and this of course now that I say all that, not my 18 year old by the way. Of course, this guy in all honesty actually this part of part of my my conviction of this comes from my own 18 year old and watching the degree, you know, he either mean, God bless this kid. He's he's just he knows the value of a buck. And at one point he's like, dad, I did the math and like every class here is like $71.

Bryan Cantrill:

Every single class period is $71. And And I'm like, oh god, I hadn't quite done that math, but okay. It seems plausible. I was like, why would anyone ever skip class? Doesn't make sense to me.

Adam Leventhal:

God bless that kid.

Bryan Cantrill:

Right? And I'm like, am I hallucinating this conversation? Like, what do I need to say to not like I feel like I'm like in like a China shop with like a wobbly Chinese vase that like, okay, how do I like back away from this thing and not break it? It's like, okay. Wow.

Bryan Cantrill:

Amazing. And like listening to the way that he's taking agency over his own education in a way that is it's pretty wild. And I think that, you know, I I think that there we've got opportunities across higher education to give kids that same agency, young adults.

Michael Littman:

I think that was very well said. And and I've long been saying that so my daughter my my youngest is now 24 or so, 24, 25. And it it was the case that when social media was kind of really kind of running rampant through that age group and really like wrecking relationships and wrecking people's ability to even think about their own futures. Like I thought, okay, we need to fix this for them. And I came to the conclusion that we like, what were we calling us on this call?

Michael Littman:

Xers? We can't, like we broke it, but we can't They fix have to fix it. They have to decide for themselves what is reasonable ways of interacting. I don't think they've solved the problem yet, but I have long believed that really it's kind of in their hands because I don't think they'll trust us to make any significant changes because of what we've already done that is problematic.

Bryan Cantrill:

That's a great analogy, and that did change over time. Like, you look at the, know, kids have a very teenagers have a very different disposition towards social media than teenagers did ten and fifteen years ago because they know that, you know, that there's a different disposition and I mean it's kind of a return to text messaging where they can actually hide from a lot of the stuff and they can do the things they wanna do in private. But the that's a very good analogy and yeah, we we kind of like overshot and then figured it out to a degree. Not that we've, you know, with plenty of of societal problems along the way, but as always, it's the adults more than the kids that are problematic.

Michael Littman:

Fair enough.

Bryan Cantrill:

Well, this has been great. I Michael, I this thank you so much for being willing to join us and take this apart. And this is so important, I think. And higher education is right at the crosshairs of the problem that I wouldn't say you created. That's that's a little reductive,

Michael Littman:

know. But

Bryan Cantrill:

I I actually on it in all honesty, I thank you for taking this on. I know I speak that not as someone who, you know, as as an alum who wants to see Brown do well in the world, but I also think it's very valuable for a not just a computer scientist, but someone who really is an ML researcher to really take this on. And, you know, you were, I guess suckered into taking on the department chair role yet again, but at a much larger but in all honesty, I think that this is so important.

Michael Littman:

No. I feel very different about this role than I did when I was when I was arm twisted into becoming department chair because it's a mission that I really, really strongly believe in. It's kind of at the whole university level. So I'm not kind of like tucked into a little corner. I really am trying to help all of us, all of us that are part of this organization.

Michael Littman:

And so I'm on board. It's been a lot of work and I don't have my nights and weekends anymore, and I don't have any time during the day, but I do feel like that every day I'm engaged with something that I think is really important. And so I'm just very grateful that I'm in a position that I can work on things that I think are important.

Bryan Cantrill:

Absolutely. Critically important and I would love to have you back at some point to reflect back on us because I'm sure you're gonna make a lot of interesting discoveries along the way. I'm sure there are gonna be a lot of surprises. It is more changed than we've seen in higher education. I I I agree with you since like, you gotta go back to the nineteenth century and becoming at the transition from an agrarian society to an industrial one.

Bryan Cantrill:

I mean, it really is that kind of a change.

Michael Littman:

Yeah. Well, wonderful. This is this is a very fun conversation with the two of you and, yeah, I'm glad to have been a part of it. Thanks.

Bryan Cantrill:

Awesome. Thank you. And thanks everyone for joining us. We've got a special European friendly time. I don't know what Adam, we'll have to come back at at at it's been fun to do this Yeah.

Bryan Cantrill:

In the morning, you know. So I'll have to

Adam Leventhal:

I'll have to

Bryan Cantrill:

do it again sometime. But I don't know if we're getting maybe out again next week unfortunately. This is just a rocky fall here. Yeah. We got some good episodes coming up.

Bryan Cantrill:

So I'm I'm excited for some of the the folks we've got on deck. So stay tuned. It's gonna be great. Alright. Thanks everyone.

Bryan Cantrill:

Thanks, Michael. Talk to you next time.