TalkRL: The Reinforcement Learning Podcast

Finale Doshi-Velez is a Professor at the Harvard Paulson School of Engineering and Applied Sciences. 

This off-the-cuff interview was recorded at UMass Amherst during the workshop day of RL Conference on August 9th 2024.   

Host notes: I've been a fan of some of Prof Doshi-Velez' past work on clinical RL and hoped to feature her for some time now, so I jumped at the chance to get a few minutes of her thoughts -- even though you can tell I was not prepared and a bit flustered tbh.  Thanks to Prof Doshi-Velez for taking a moment for this, and I hope to cross paths in future for a more in depth interview.

References  

Creators & Guests

Host
Robin Ranjit Singh Chauhan
🌱 Head of Eng @AgFunder 🧠 AI:Reinforcement Learning/ML/DL/NLP🎙️Host @TalkRLPodcast 💳 ex-@Microsoft ecomm PgmMgr 🤖 @UWaterloo CompEng 🇨🇦 🇮🇳

What is TalkRL: The Reinforcement Learning Podcast?

TalkRL podcast is All Reinforcement Learning, All the Time.
In-depth interviews with brilliant people at the forefront of RL research and practice.
Guests from places like MILA, OpenAI, MIT, DeepMind, Berkeley, Amii, Oxford, Google Research, Brown, Waymo, Caltech, and Vector Institute.
Hosted by Robin Ranjit Singh Chauhan.

Robin:

I'm here with, professor Finale Doshi Velez. What are you working on lately?

Finale:

Working on a number of interesting problems. I'll name one thing that I'm excited about, which is many times when we're thinking about reinforcement learning, we're often trying to optimize at the macro level, like, what action should the agent take in the world? I I work in health care, so what sort of treatment for what patient? And I'm realizing that there's a lot of really interesting problems in how to optimize the human AI interaction. So if a agent is providing, sorry, decision support advice to a human, what's the best way for the agent to provide the information such that the final human plus AI decision come out the best as possible.

Finale:

So that's like a micro RL problem. So that's one flavor of interesting problem that I'm excited about these days.

Robin:

Can you mention anything else besides your own work that you find interesting that's going on in in the RL space ?

Finale:

Okay. So maybe you mean, like, related to the interpretability workshop?

Robin:

Sure. We could start with that.

Finale:

Sure. Yeah. Sure. I I'm excited that lots of more attention is being given to human interaction with agents. And so there's a lot of work here at the interpretable policies workshop and just many of the other workshop presentations as well, thinking about how agents behave, in the context of people, making realistic models of people so that the agents can behave not against, like, perfectly rational or similar clones, but actually humans, how to, model human cognitive limitations and strengths and agents that can pair with those, agents that can expose weight things that such that humans can edit and adapt, the agents to align them.

Finale:

So these sorts of questions I find really exciting.

Robin:

How close are we to getting RL that actually is helpful to clinicians in the real world?

Finale:

Oh, that's a I don't know if that's a good question to be asking. Okay. Because I feel like I feel like the limitations currently are less about the r l and more about how our health care systems are set up that we're really not set up for push. We're set up for pull. We're set up to take out data, and do offline analysis.

Finale:

But to actually be able to push information to humans, in a way that we can adapt it nimbly. Because, again, as I was saying, the design of the interaction needs work. Right? So we're not gonna have it right exactly the first time. I that that's some that's infrastructure, and that's, legal and that's every compliance.

Finale:

That's something that's kinda, you know, again, like, if you're talking about health specifically, I think that that's a major aspect that holds us back.

Robin:

And, of course, that's like a, nation dependent thing. Are there some countries where that are way ahead that are ready or or already doing this or are gonna be first to doing

Finale:

that? There are different places certainly. Like, there's certain locations in the US that have better infrastructure than others. I spent a year in Singapore, where they have some really great infrastructure set up to be able to set up these sort of push outputs. So I agree that I think there's significant variation in there.

Finale:

And the fact that there's some places that are able to do it gives us a model, of what other places could be adopting.

Robin:

Anything else we we should add right now?

Finale:

Okay. Sure. Well, I I thought you were gonna be talk talking more about, like, interpretability when you caught me.

Robin:

I'm just a long term fan, so it's just I I'll just anything anything you wanna share with the audience? Yeah.

Finale:

Yeah. Yeah. Well, the the thing about interpretability that I have that this kind of my thing at the moment is that I think we're doing a great job coming up with methods to explore, like, understand what agents are doing, like, inspect them, etcetera, etcetera. We're not doing a great job at generalization, which is, you know, I have a new task, and then I have a zillion different ways of potentially doing interpretability. How am I gonna do it?

Finale:

So my big thing these days has been that we really need to define our abstractions saying that, there are certain properties. So if you have a task like this, and the explanation has this, this, and this, then, you know, a human will be able to do their task well with that agent because that agent satisfies these specific checkboxes. And if it was a different task, maybe it needs a different set of check boxes. And we really need to do the human user studies to figure out what are the check boxes needed for each task. And then as we establish those check boxes, then on the computational side, we can, design agents that are sufficiently compact, sufficiently, high performing under other sorts of constraints, where the explanation is sufficiently, robust or faithful to some other neural network policy or whatever it is.

Finale:

But we'll know what the check boxes are, and we can check our methods against those check boxes. So that's kind of my thing right now about how we can take the really great work that people are doing on specific methods for interpretability. And then, like, take that from, okay, did that work on this one particular task that you applied it on to, okay, now we have this entire class of methods, and now we know which one to apply to whichever problems come up.

Robin:

So it strikes me that in, in in when it comes to clinical work, the distance between the observation and the underlying reality is is is is really vast. Like, the the process inside the patient that's causing this Mhmm. Very far from the things that you're observing. And I wonder, is that or, like, is are we I guess we're just getting the low hanging fruit of optimizing on this on the surface right now and then ultimately the long term goal being, methods that can can, that can use more of the

Finale:

You're not seeing enough of the patient. Is that what you mean? That we we we get a few sparse observations.

Robin:

Yeah.

Finale:

Yeah. So this is where, again, I think that human AI interaction comes in because the observations we get, might be, long over time. So they might be longitudinal. They might be dense in the sense that, if the patient is hooked up to a particular monitor, we might actually have a lot of data points, but they're always gonna be along relatively few dimensions. Right?

Finale:

Whereas the human provider may have, information across many more dimensions, but not with maybe that level of granularity either in terms of sampling frequency or just how much time longitudinal record. Right? So this is where I think we really need methods that can combine the information that both of these human and AI agents have. I don't think we're ever going to get to a point where the AI agent has everything. But maybe maybe for some routine things.

Finale:

Right? Where you're like, okay. Do you have a fever? Do you have a scratchy throat? What etcetera.

Finale:

Okay. Go go get some Tylenol or no. You gotta take a COVID test. Yeah. But, beyond these kind of basic problems, for more interesting problems, like, we are going to need that human AI teaming.

Robin:

Awesome, thank you so much.

Robin:

That was wonderful.

Finale:

Alright. Yeah. Hopefully, that was useful.

Robin:

It was fantastic.