Subscribe
Share
Share
Embed
Brains, Black Holes, and Beyond (B Cubed) is a collaborative project between The Daily Princetonian and Princeton Insights. The show releases 3 episodes monthly: one longer episode as part of the Insights partnership, and two shorter episodes independently created by the 'Prince.' This show is produced by Senna Aldoubosh '25 under the 147th Board of the 'Prince.' Insights producers are Crystal Lee, Addie Minerva, and Thiago Tarraf Varella. This show is a reimagined version of the show formerly produced as Princeton Insights: The Highlights under the 145th Board of the 'Prince.'
Please direct pitches and questions to podcast@dailyprincetonian.com, and any corrections to corrections@dailyprincetonian.com.
Senna Aldoubosh: Hi everyone, welcome to Brains, Black Holes, and Beyond, a collaboration podcast between the Princeton Insights newsletter and the Daily Princetonian. From the Prince, my name is Senna Aldoubosh. Today's guest on the show is Dr. Ruth Fung, a lecturer in the computer science department. Dr. Fung got her Bachelor's in Computer Science at Harvard University, before getting her Master's and PhD in Engineering Science from Oxford University. At Princeton, she now conducts research in computer vision, machine learning, deep learning and Explainable AI, in her own lab, the Looking Glass Lab, as well as collaborating with the visual AI Lab. She also teaches intro and AIML computer science courses. Dr. Fong, welcome to the show.
Dr. Ruth Fong: Thanks for having me.
SA: My first question for you is what initially got you interested in computer science? More specifically, computer vision interoperability?
RF: Yeah, so I'll answer this in two parts. For what got me interested in computer vision. My parents are both biologists by training. And I always loved science and STEM. So I actually thought I was going to be a biologist up until maybe midway through college, but in high school, I was fortunate to have a fantastic computer science teacher, Mrs. Wendy Gall. And I think the thing that really drew me to computer science was this ability to create. In biology and a lot of the science, you have to run experiments, these take long times, sometimes things like, you know, flies dying, or you know, something being the wrong temperature can throw off an experiment, and you have to repeat. Whereas, what I liked about computer science is, you know, if the failures are there, they're probably because of your code, but you can fix them. And then there aren't any other variables that can contribute to problems. So, I think that ability to kind of build something quickly and prototype and as some of you guys may have done like COS126 final projects, that type of creativity that you get very quickly in computer science compared to other STEM fields, is what really, ultimately drew me to computer science over other types of sciences I was considering. In college kind of once I decided that, you know, on computer science, and I guess another fun fact I like to share is, I loved biology, but I actually realized I was really rubbish at it, because I was bad at memorizing things. But I didn't learn this until maybe I still have a master's degree in something related to Bioscience. So it's okay for you to do things that you love, but aren't good at. But, in terms of what drew me to computer vision, and machine learning interpretability I think similarly, and I was just meeting with a female student and sharing about this. Oftentimes, you want to find something that you're passionate about, and where you have unique skill sets in and where, you know, you think there are interesting problems in the world. And that is the sweet spot. In college, I actually took a lot of classes in kind of the more mathy side of computer science, what's known as theory. And again, I was really solidly mediocre, I love these classes. But it was very clear to me like it would be a bad idea trying to make a career out of this. So I kind of did a optimization where I first tried the software engineering route in my summers. And then I kind of compared it to really my senior thesis experiment, experience, I was in a really unique lab that was half neuroscience and half computer science. And it was studying vision from kind of mammals as smallest rats to monkeys all the way up to kind of artificial vision, in what are known as deep neural networks, kind of the way these deep neural networks, understand and interpret images and videos. And it was really kind of that experience that made me realize, Oh, I really like this, I'm arguably way better at this than, kind of the math side of CS that I've spent most of my time in college on. And I think for me, it was really important to work on something that I could explain to my friends, most of whom weren't STEM majors. And kind of what computer vision is all about is trying to understand the visual world. So typically, this means training, or kind of running algorithms on images and videos. And it's very easy to explain, oh, like I'm trying to make AI models do better when they're using image images as inputs compared to maybe some other branches of computer science, it was just a lot harder to kind of explain. And I think kind of that desire to want to explain my work also directly influenced my specific choice of subfield and Explainable AI where they're, we're really interested in explaining kind of the state of the art AI models, which now are made up of millions, billions and even trillions of parameters. And trying to explain to humans, who are going to be using these models and are affected by these models. Why did these models make these predictions? And kind of the main premise is as AI is increasingly being applied to high impact, but also high risk scenarios, like precision medicine, self driving cars, you know, we can't just trust that, you know, someone gives us these models and say they're good enough. We want to be able to audit them. We want to be able to understand them. And that's really what motivates did install motivates me today into why I'm interested in computer vision and Explainable AI?
Senna Aldoubosh 5:06
When I was looking at some of your research that you did for this episode, I saw your paper that used fossils segmentation as a case study for improving segmentation using interpretable modifications. And I found it really cool. And I was wondering if you could talk more about that paper and where the idea came from, you know, what you found out? And then also, if there's any other paper you'd love to talk about, I'd love to hear about it.
RF: Yeah, so I'm glad you picked that paper. That is a paper that started off as a junior paper by Indu Panigrahi. She's a COS BSE major, and she's continuing on in a senior thesis topic very similar to this. And really, she brought the idea to me she had done her fall JP, with kind of a similar problem in collaboration with a professor Adam Moloof in the geosciences department, and kind of came to me saying, you know, I see that you've done interesting work in computer vision. I want to do this collaboration with geosciences, where it's very hard to collect data. And that's one of the main challenges in machine learning, you need large scale data sets to train these complex models, is just like, you know, most of machine learning is focused on large scale data sets. Here's a domain where it's very hard to collect data, it takes maybe several hours to annotate one image, and we're trying to segment coral reef fossils. I should have said a beginning the project is we're trying to segment coral reef fossils, because the better we can understand the structure of ancient coral reef fossils, the better we can understand climate in those previous eras. And that can help us understand, you know, how does coral reef fault coral reef structures impact biodiversity and kind of contribute this larger conversation of what are the different factors for climate change and things like that. So, I had no background in geosciences or climate change. And she kind of said, you know, here's the problem, like you can provide some of the advising on the computer science and computer vision side. And it's been really neat to kind of see how this project in collaboration with geosciences, and Professor Maloof’s lab has developed. And I would say, a large part of why this project has been so interesting and successful is really, because Indu kind of came with this idea brought the two of us together, and we never would have met otherwise, probably. And yea has been doing amazing job in, you know, her new senior thesis project is a slight pivot off of the JP. But his focus is on this problem of how do we do better with segmenting coral reef fossils?
SA: Gotcha. And I was kind of curious to know, like, I know right now you're working, focusing on like things like computer vision, but I'm curious to know like, how do you see your research evolving? Is there any project that you're working on that you're really excited will come out? Or you're really excited about the results and things like that?
RF: Yeah, so there's, there's a number of projects, it's it's hard to pick one others I think since coming in at Princeton. As you mentioned earlier, I've been collaborating with Professor Olga Russakovsky’s lab and that's been a really amazing collaboration. I kind of bring my expertise in Explainable AI and she's done some amazing work in machine learning fairness. So one of the first machine learning fairness papers I've gotten a chance to work with her on was co led by a Nicole Meister and Dora, both former Princeton undergrads, this was Nicole senior thesis project while Dora was a master's student here. And their project was on understanding kind of gender artifacts in computer vision data sets. The kind of one of the big problems in machine learning fairness is they're trained on these large scale data sets and then deployed in the real world. But often the performance in the real world, one doesn't match maybe the model, initial performance on the dataset it was trained on. But too often we see disparate performance for different kinds of populations that may have been underrepresented in the data distribution. So you may have seen, you know, in the news, or the New York Times, kind of a pattern of AI being released. And then some articles come out saying like, it doesn't work well on some subsets of the population, for instance, some face recognition did really poorly on women of color. And then, you know, companies retracted that pattern of let's release without thinking about the implications realize this bad retract. So ML fairness is really focused on kind of that problem space. Nicole and Dora’s project and you know, is trying to understand, we know that there are kind of differences in our image data sets, in the images where women appear to present more feminine and in images where people tend to present more male and the reason I'm using presentation is the way we kind of label or get kind of gender annotations for this project isn't by asking people, you know, what's their gender identity? It's kind of perceived gender expression. So it's not the same as gender identity. And we want to be very clear about that. But we know kind of perceived gender expression is going to look different in these visual data sets. But no one had really explored, how exactly do they present differently in these large scale data sets that most computer vision models are trained on. So Nicole and Dora kind of did a number of different experiments, I think the most kind of surprising one was they took images from these popular data sets, reduce it down to its mean, pixel value, the mean color, so mean RGB value, and then train a model and ask, can it predict these perceived gender expression labels. And still, with just the main color, you're able to predict gender, and they did all sorts of other experiments along these lines, and every time we thought, okay, this is going to be too hard for the model. I think it was only until we made the model made the images black and white, and then reduce the down to very small images, that now the model couldn't predict gender at all, or above random chance. And some of the more interesting findings we found was that images that contained women tended to have women more in the center and larger in the images and in less, quote, unquote, active poses, whereas the images containing men tended to be smaller, not as center focused. And in more active poses, when we kind of dug through these image data sets, we realized that a lot of the sports images contained people that were presenting as men. Um, so this just kind of is kind of a one small, you know, interesting quirk. But we found kind of many different patterns like this, and no one has ever really studied, studied this and as depth as Nicole and Dora did. And kind of one of the big implications for the ML fairness world is just like in maybe analogous conversation and politics, there's this kind of idea of maybe if we're just colorblind, or gender blind in our approaches, and hiring, etc, that will solve the problem. But what we found in Nicole and Dora’s work is that these gender artifacts are everywhere, they may have nothing, sometimes they are, you know, substantive, substantive, and, you know, meaningfully correlated with, you know, certain kinds of gender expressions. But other times, it was, you know, women can play sports too, like, there's no reason why, you know, active position, you know, should encode, lean, more male presenting. So, what we found is, basically, these gender artifacts are everywhere in visual data, even in the mean pixel color of an image. But a lot of the kind of techniques for trying to make models less bias is to take a quote, unquote, bias blind approach, let's try to erase or forget gender in our models, let's try to erase all notions of our race, the ability to predict gender in our models, and then we should have a fair model, right. But what the work showed is, if these cues and artifacts are everywhere, there's no way you can want to completely remove all the artifacts for gender. And to it, even if you're able to successfully do this, this would probably severely hamper your model. So kind of the big main takeaway for the larger community is we need to move away from these bias blind approaches to these more context aware approaches, you know, we need to be as aware as possible, and encode that technically, in our models, that there are these statistical differences in these different demographics that we care about in terms of fairness, and bias. And let's try to make sure that we ensure, you know, fair performance as much as possible with these different demographic groups without trying to, you know, destroy our model to the point where it can no longer detect gender artifacts, or kind of predict gender, quote, unquote. So that's one work that, you know, also highlights, you can do really neat things in your senior thesis, in JP’s. But I think has pretty big implications for, you know, the broader machine learning fairness and broader AIML community.
SA: That's really cool. On this podcast, we've talked a little bit about AI ethics and like how they can be discriminatory. So it's really interesting to kind of look at it from the perspective of like, you know, rather than having AI not be able to detect these differences, it's important for AI to detect these differences, but also make sure that each group is treated fairly. Before I get on to my last question, I was wondering if there was anything else that you wanted to include in the podcast that I hadn't asked you yet? Or maybe that we didn't get to touch on as much?
RF: Yeah, so I think we focus primarily on research. But I think a lot of my job also includes teaching, right now I'm teaching COS324 for the intro to machine learning class. And I think that the piece of advice I keep giving Princeton undergrads over and over again is you know, it's okay not to be perfect. And it's also okay to take breaks. I had a lovely conversation with a student who, you know, took a gap year and you know, came back much better for it. And I think I was also sharing in another kind of women and CS discussion that in my own kind of academic career, I, you know, went straight through college and grad school, and then burned out by the end of my PhD. So I took a gap year, right after my PhD before coming to Princeton, and taught middle school and high school computer science for a year, it is really what I needed to kind of, you know, get back on my feet. And just, I wish I had told my younger self like, it's not a race, you don't have to, you know, be the first one to finish your PhD or, you know, graduate with a 4.0, that you want to look at your life, including your academics and your career holistically. And you know, the best way to do great work in your career and professionally, is if you're taking care of yourself first. And I know from experience that that is still a lesson, I'm still actively trying to learn and teach myself,
SA: As many listeners may know, fields like computer science and engineering are very male dominated fields. So for any listeners, who are women in STEM trying to navigate those obstacles. What What advice do you have, like, I know you're a super successful researcher, and professor and so like, it'd be really great to get your insight.
RF: Yeah, so I think the traditional answer to this is like find mentors. I think my spin on this is, find your village. In college, when I was a freshman, or first year student, a group of women founded the women in CS group at Harvard, and that group is still going strong. I had a friend who went back as a keynote speaker there. And the women that I graduated with, we still keep in touch, we're still encouraging each other celebrating both professional and personal highs and lows. It's been a great group, not only in college, but I've I've been really surprised how much this group has continued to support one another, even after college. And I think kind of throughout my academic journey and career, I think, really, you want to find your village, it doesn't have to be other women. Some of my best collaborators are p-set buddies and partners were men, and they were great. You know, sometimes they made comments, but they were teachable. And you know, I was like, hey, like, I don't think that was like a really fair comment you made about someone and they're like, You are right, I'm sorry. So you know, your village can look like a lot of different things. Often it will look like women or people like you, but it doesn't have to just look like that. And even here, like I think I have n number of people in the department who I know, like have my back not just professionally, personally, I think Professor Russakovsky and I have been on a few like panels and talks together and will also always be like, Oh yeah, we love, you know, working together. And, you know, that's been a research collaboration that has flourished into much more. And to remember that, you know, our professional lives are a really big part of our identity. It's, you know, one of the main parts of my identity. But, you know, I'm not just doing this alone, there's a lot of people who, you know, want me to do well and succeed and want to support me holistically. And I think finding those people and, you know, realizing that, I think particularly in academia, it can feel lonely sometimes and realizing that it doesn't have to be that way and trying to kind of create structures and supports that work for you to support you along the journey is really important.
SA: I love that advice. I think that's amazing. Well, thank you so much. Dr. Fong, it was great talking to you and learning about your research and getting a lot of really cool insight.
RF: Great, thanks so much for having me.
SA: Yeah, of course.