Subscribe
Share
Share
Embed
Business leaders and policy makers often fail to consider how their policies, products, services and processes can have impacts on inequality. To think about how to unlock innovative opportunities using an intersectional gender lens, GATE joined with our Rotman partner, the TD Management and Data Analytics lab, to host a one-day conference in the Spring of 2023 called “Gender Analytics: Possibilities” (or GAP). This podcast shares exciting panel conversations from the conference on topics like decolonizing data and design, inclusive government policy, responsible AI, and more.
Dilip Soman: While I firmly believe that every group is different, every situation is different, every context is different, we can develop a general approach to embrace those differences. Systematically studying heterogeneity or differences. We never do that in policy making. We focus on which is the one that's on average going to be the best. But I think we need to start developing the sort of the rigor of looking for differences as opposed to looking at averages. But I think again we can create a process that embraces those differences.
Sarah Kaplan: You’ve probably heard about “nudging.” So called, “nudge theory” as popularized by Richard Thaler and Cass Sunstein in their bestselling 2008 book—not surprisingly called “Nudge,”—is about how to use behavioural interventions to get people to act in their best self interest. But, what does this look like when we think about this in the context of government policy? And, how, in particular, can this concept be mobilized to achieve more equitable policy outcomes?
Welcome to episode 2 of Designing for Everyone, a podcast by the Institute for Gender and the Economy (or GATE). I’m Sarah Kaplan (she/her pronouns), and a Professor of Strategic Management at the University of Toronto’s Rotman School of Management, Founding Director of GATE, and your podcast host. In this 7-part limited series, we’re featuring a high-impact set of conversations we had in April 2023 at our Gender Analytics: Possibilities conference.
In this panel session, we’ll hear first from my very own colleague, Dilip Soman, who is Canada Research Chair in Behavioural Science and Economics, and Director of Behavioural Economics in Action at Rotman (otherwise known as BEAR). He is also the author or editor of many books, including Managing Customer Value, The Last Mile, The Behaviorally Informed Organization, and Behavioral Science in the Wild.
He was joined by Alia Kamlani who is a Partner at Deloitte in Government & Public Services. She is a seasoned public sector transformation partner and has spent her career working alongside senior executives and political leaders to develop business cases, engage stakeholders, develop strategic options, and manage numerous large-scale transformations across North America.
Their conversation was moderated by Kate Bezanson who is Special advisor in the Office of the Prime Minister of Canada and Professor of Sociology and Associate Dean in the Faculty of Social Sciences at Brock University—and she’s also a Faculty Research Fellow at GATE. She specializes and advises in the areas of social and family policy, gender, carework, constitutional law, political economy, and federalism.
Their conversation was provocative and got me thinking about all of the ways that policies can have unintended consequences for inequality in our society, and how even some small policy changes could have big impacts.
So, let’s get right to it.
Kate Bezanson: So I'm really looking forward to our discussion today, so maybe I'll just start by turning to Dilip, if that's alright? And I will I I'd like to talk to you about some of the studies that you've been experiments that you've been involved in attempting to change behavior through nudges. I read your paper on vaccination, which was fascinating. And today we're discussing achieving greater equity and inclusion and policies. Can you talk a bit about your work, some examples from policy? Tell us about nudging.
Dilip Soman: Gosh, OK. So there's a lot going on in that question. There's nudges. There's policy, whose policies that government policy, is it organization. So let me try and unpack that a little bit. I'm going to do that by making references to stuff that was said in the first panel. There were three things in particular that resonated with me. It turns out Dori said all those three things. One of them, she made a comment about society being built by white men and others having to fit in. And I'm just going to extend that. And Sarah knows I call a spade a spade. Our organizations are built by white men for white men, and I think we need to change that. So that's one. She did say that we should go away from human centered design. I disagree with that a little bit. I think the important question is which humans. It still needs to be human centric. But I'm gonna make the point about the fact that there are differences across people. Economists love to call it heterogeneity and we need to think about that. And then finally, she spoke about incentives for dismantling colonial structures, aligning with the white people. I slightly disagree with that in an organizational context. The thing about privilege is people that have it don't know they have it. And so, you can tell them as much as you want to that it's their job to dismantle systematic barriers to equity and inclusion. It's not going to happen. So I think it's up to all of us to try and do that. And I think the bottom line of all of these points is that, in order to achieve successful behavioral change, you have to change the system and not so much the people. People take a long time to change. Systems are easier to change. And I know those of you who work with systems will tell me that's not true. But it is easier to change systems than it is easier to change people.
So, this whole notion of nudging or choice architecture relies on the premise that there's a lot of power in things that we don't think there is power in. The way in which you ask a question changes the way in which people answer the question. The way in which you design a website changes the information that people seek. The way in which you design a questionnaire change how they respond to that. And so that's the general idea of choice architecture. It can create environments to steer people to do things that we want them to do. So, a couple of quick examples. There is Sonia Kang. She’s in the room. She's clearly the world leader in this space. One of her students, Joyce He, who's now a faculty member at UCLA. In her thesis asked the question, why is it that women don't like. even if organizations hire women, why is it that they don't get promoted? Why don't they rise to the top? And one of the frictions is that in many organizations, including in academia, the onus is on the employee to ask to be promoted. Or guess who doesn't put their hand up and say that I want to be promoted? It's women. It's ethnic minorities, right? And so the question that Joyce in her thesis asked was, well, what if we changed the default? What if we made it such that, at the end of every four years, you will get promoted unless you choose not to. And of course, their work was done experimentally. They find that in, you know, in situations where you change the default. In fact, people are, or women are, or minorities are more likely to want to get promoted. Those are the kinds of things we need to be thinking about more. So, folks in the audience. If your company wants to try some of these behavioral interventions, Sonia is the person to speak to or me. But I think we need to institutionalize some of these ideas, change the system and not just try and educate people.
When it gets down to other things, you talk about nudge. Nudge has an evil cousin called sludge. Sludge makes it harder to do things. Sludge imposes frictions on people. So, for example, several years back you might remember we had in this country a lovely welfare program called the Canada Learning bond. $2000 of quote and quote free money, I'm quoting from the government advertising, to low-income Canadians as long as the money is used to educate their kids. Free money. I remember being in the room when this program was announced. I remember somebody saying who won't take it. Right? Guess what? 84% of eligible Canadians did not access the free money and it wasn't because they weren't aware of it. It was because they had to fill up forms and go to banks and, you know, ask for welfare and a lot of people who are eligible for it came were recent immigrants. They came from cultures where it wasn't kosher to go and ask for help. And so, that was the friction. It wasn't the fact that the program was fully designed. So, sludge affects different people differently. And I think we need to be sensitive to that.
So where does all of these go? I think there are a couple of takeaways from at least my work in in applying behavioral interventions. One is the devil is in the detail. Situations matter. The specifics of the situation matter. That's why it's a hard policy challenge. Government policy works well when you want everything, or everyone to do the same thing, and they're all in the same situation. Here we know that situations matter. And then the second point that I made was heterogeneity. Different people react differently to different situations. So, I think those are two things to keep in mind.
So let me just say two big things and then I will shut up for now. What sort of policies have been successful? I think about policies as either outcome focused policies or process or resource focused policies. Outcome focus, things like quotas, right? So for example, Norway a couple years back talked about 40% reservation for women on corporate boards. That's an outcome focused policy. India has long had this tradition of reserving seats in colleges and jobs for so-called Scheduled Castes, Scheduled Tribes, quota policy, right? But then there's resource allocation policies. There's a lot of great examples from Africa. Rwanda, for example, a few years back decided that the way we're going to get rid of inequality is to channel resources to people where we feel that those people need development. So, they look at gender. They look at specific ethnic tribes. They've basically made policy redirecting resources. So, maternal health, contraception, you know, training women to be in the workforce, things like that. And I think those are successful. Because I think they tackle the root problem of inequality as opposed to the outcome basically.
So, just to wrap up my remarks for now. One, I think policies that address the root causes of inequality are way better than the ones that just impose quotas. Two, policies that match institutional structures with social structures are better. What I mean by that is, in parts of India and Africa, banking is financial management, which is done at the level of the community and not the individual. All of our banking structures only allow individual accounts. And so, South African Republics, for example, has started experimenting with allowing community bank accounts. Because then that matches the institutional structure with the social structure. Policy that is done in the context or the domain is better than policy that's just an overarching one. So, figure out whether this is really an education problem. And incorporate gender and equity into education, and into finance, and into government, as opposed to having something overarching. And then finally, policy based on understanding humans as humans, as opposed to just citizens who are nameless and pieces of data, is clearly better than policy that doesn't do that. Long answer to your short question. But hopefully, there are other perspectives for that.
Kate Bezanson: Thank you so much. In listening to you reflect on the policy outcomes and the policy challenges, it really puts in stark relief the challenge of understanding the kind of systematic structural barriers that people come to a policy problem with, and the kinds of ways that those can be both conceptualized and approached. And I think, for example, I'm just going to say this for I really enjoyed your paper on vaccines. The goal is greater vaccination, but the reasons that people are apprehensive about vaccination are socioeconomic. They're historical. They're based on exclusion. They're based on colonization, and so on. And it's a very different project than simply saying we want to increase the rate of vaccination. So, I I really appreciate that.
Let me turn to Alia. So, you were co-author on a report that I really enjoyed. I recommend everyone read it. That looked at how behavioral interventions could shape policy and I wonder, maybe if you could walk us through that report. What were the biggest takeaways from your analysis? And also maybe, were there any surprises?
Alia Kamlani: Sure. And I have to say I'm so thrilled to be back on UT campus. I'm an alumni. So very happy to be back in the building, and in in the vicinity. So, I think, you know that at the starting point, I would say governments and policymakers are really at the heart of transforming society, our lives, etc. It's an obvious point, right? But I think what's interesting and about this moment in time is it feels like there's been a lot of increased emphasis on wanting to address systemic bias, wanting to take a different approach and deal with the systemic racism and bias in our institutions. So, I think that's a positive. I think though, at the same time, there's a need to marry the intention, the desire, with action and implementation. And there's a real moment in time, as governments are rapidly transforming. We're in a real shift and transformation of a lot of our institutions, given technology and the push towards digitization and these other drivers that are that are underpinning some of the big transformation. We talked about in the paper. To meet that moment is really about bringing an equity centered approach to the design or redesign of government programs and policies. And what that really requires and what we talk about is a fundamental rethink in the mindset, the tools, the relationships you have with stakeholders, maybe not even calling them stakeholders, your end users, your participants, the individuals and the humans at the end of the day, who are going to benefit from a policy or service. So, it requires a different approach to how you include them in that transformation to really achieve those broader goals, to achieve systemic change. And so, equities centred design really requires acknowledgement of biases that are built in institutions. So, those are hundreds and hundreds of years old. The speakers previous to us spoke a lot about that and gave us a lot to think about in that regard. But at the very beginning, equity center design takes some acknowledgment of that and intentional identification of what those biases are at the beginning of any problem definition or policy definition. Because to those who are in those institutions or benefit from those institutions, those are kind of invisible to us. So, it's really taking that time to to name them and to identify them at the very beginning. I think it also requires a lot of self-reflection. And so again, I think some of our speakers earlier today talked about positionality. That's at the core of the whole process. And so, making sure that there's an opportunity for ongoing self-reflection throughout the policy definition all the way through implementation of something new. That has to be part of the process in terms of understanding the role you play, how you're viewed by others, how others view you, and how that needs to inform how you engage in that definition of the problem or the solutioning that follows from there.
So, a couple of key takeaways from the paper. There's a shift. When I talk about a shift, it is really about shifting the way in which you engage. So, it's not about stakeholder engagement. It's about co de-design. We talk a lot about co-design. And it requires direct engagement with your end users. Your design targets. So, who are you designing for and what role do they play in and how can you bring them way up into the process. So, they're helping you understand the problem or what you think is the problem that you're solving for in the first place. And you're engaging them all the way through. So in problem definition, but then also, as reviewers of drafts, perhaps, or as testers of a final policy or program before you roll it out. So, you can kind of continue to make sure those perspectives are baked in all the way through. It's really about how you do it and how you do it differently as well as who you include in the process. And a couple of surprises, maybe. We talked about a couple of examples. The potential positive unintended consequences of that. So, close captioning is a good example on Netflix. Netflix rolled out closed captioning to deal with those who have hearing impairment, et cetera. But last year they reported 40% of their viewers use closed captioning for a variety of other reasons. So it's beneficial for those who also have behavioral and cognitive issues in addition to kind of the obvious reasons why someone might be using that. Curb cuts are another example. Curb cuts kind of modify the crosswalks at intersections. That was really designed for those who are in wheelchairs. But that has additional benefits for those who use a stroller or have additional mobility requirements. So, those are the surprising outcomes and the surprising benefits that come from when you are really designing not just from the margins but with the margins. It can really be a benefit for all.
Kate Bezanson: Thank you. Let me turn back to you to Dilip for a moment. Access to gender disaggregated data is essential in design and effective behavioral interventions for more inclusive policy. And governments collect this data through the census or labor force survey. As an expert in behavioral intervention, why are ethical and legal collection processes critical for effective policy design? And, maybe what would be your recommendations to facilitate the process?
Dilip Soman: The data piece is interesting because as you know, I spent a year in the federal system working on behavioral challenges. I've never found the StatsCan data useful for any behavioral intervention. Because it doesn't measure the right things. And we spoke about this in the earlier panel as well. It's all about sort of just counts and numbers and frequencies. Whereas as a behavioral scientist or anyone that's interested in design, we're interested in how people interact with the product, what are the frictions, where do they stumble, where do they get information from. That's not collected by government. So, I think governments need to think about a completely way of doing collections. Let’s go back to the example I spoke about the Canada learning bond. I remember when the take up rates were 16%, somebody said it must be that people aren't aware. And my rule of thumb is, every time people look at data and say it must be, that means we need more data, and we probably need different forms of data. Of course somebody said it must be that people aren't aware, so, a lot of money was spent and trying to get people to be aware and that pushed the doll from 16 to 16 and 1/2 or something like that. No, it wasn't that they weren't aware. You had to go and spend time at the centers in the communities to try and figure out why people weren't claiming this benefit. This was back in 2014. Banks would shut at 6:00. And people who were eligible for this benefit were working two jobs, they had kids to look after. They couldn't get to the bank before 6:00. So that's one. The second thing is you made them go to a fancy bank where people dress like you and I in suits. We're going to talk to them and ask them what they wanted. and they didn't want to talk about it. And so, once you understood that those were the barriers and not awareness, you could then design the right interventions. We came up with a QR code system where the person could just show a QR code at the bank teller. The bank teller would scan it. They would then know, and there was no need for conversation. So, things like that, those are the kinds of solutions we need to be thinking about.
So back to the data piece. One of the challenges with data and the biases in collecting data is we collect data as governments, as academic, we never feed it back to people who actually are the source of the data. So, one of the things we started doing in the impact unit as well as what we do here at the center is, every time we collect data, we feed it back. We said, well, here's the patterns we're seeing. Does that resonate with you? Do you want? Does this make sense? Why do you think this is going? And the stuff we learn in just that process of feeding it back to the people that we collected the data from is amazing. So, I think that's the one thing that we need to do a lot more of.
But other than that, I'll go back to what I said before. It's important to gather baselines. It's important to look at trends over time. But it is important to look at heterogeneity. Different people do things differently. During COVID vaccination, we learned that. At the end of the day, the goal was to encourage people to get vaccinations. But how we did it was different across communities, across ethnic groups, across language groups. Because different communities exchanged ideas and word of mouth flow differently and there were different influences. Unless you understood that, just changing the information and translating that into 27 languages, which is great by the way, didn't help because you need to know who should do the talking and that's the kind of data we want more.
Kate Bezanson: Alia, that segues quite nicely to your work, especially on modernizing service delivery. And the real challenge of this moment of knowing that we need to modernize service delivery across a whole range of areas, especially in government services, but thinking about it, not just as a moment for efficiency, but as a profound rethinking of how we do services and what our underlying assumptions. So, I wonder, maybe if you could elaborate a little bit along the same thing.
Alia Kamlani: Yeah. Maybe I'll talk about the 10 key tenants of the strategies that we talked about deploying in the paper and then maybe some of the misses or the missed opportunities when you were undertaking that. So, we talked about contextualizing the problem at the very beginning. I touched on this earlier. So really at the outset, making sure you're questioning where your data came from, what assumptions you're making, when you even define a problem and how do you again embed perspectives of those who have lived experience, who have actual interactions with a particular challenge or problem into how you define it in the first place. And one thing that can be a bit uncomfortable at times is being less rigid in your process and approach to that. And so especially when it comes to research, I'm not an academic. I think that's probably fairly obvious to this group. So, others in the room will have much more to say around research methodology. But, opening that up for challenge and testing with a range of individuals or stakeholders is a relatively new idea. But I would say making sure you're not being overly rigid in your approach when you're at the very outset. I'll give an example. We did some work with Eva's initiatives for homeless youth, which is an organization works with young people in Toronto that are experiencing homelessness. We did some research and collaboration with the lead researcher in the organization to better understand that the journey of youth through those programs. The priority through that process was making sure that all the participants in that research has psychological safety. And how we did that was making sure that the research plan was reviewed and tested by a range of different lenses. So, we had a peer researchers, academics, frontline staff, case workers. And research participants themselves actually weighed in on how we were doing the research and to test some of those assumptions. And when actually conducting the research, making sure we had peer researchers who were embedded in the organization, had the trust, had knowledge of the community to do the research with us was another key element. So, what that required though is iterating and refining the approach all the way through, which is a slightly different model. So, that is one element.
Another key tenant in terms of how you can do things differently is take a different approach that we call co-design. But I think, embedded in that, or common mistake, or potential misstep, is not fully addressing the unique needs of your participants and underestimating the value of trust. So, for example, if your research is going to involve vulnerable populations, those who have experienced violence, mental health issues or other barriers, you want to be thinking about who's leading the engagement with them, and how is that engagement happening, and whether you as an actor outside of that community or outside of that stakeholder group, are there other person to have the conversation. And so, things to think about in that regard would be who is the right leader of those conversations and how do you again continue to assess your role in that dynamic and then question and perhaps give some space to others to help you via proxies and others who have those trusted relationships to lead those, to get input to, to learn from the experiences, etcetera. And those might require specific tools and methodologies, too. So, being broad in your thinking in terms of when you do co-design, it's not one approach fits all necessarily. You kind of have to think about how you modulate your approach, depending on who you're engaging with and then in terms of redefining impact. So, it's one step to definitely state kind of an intention to want to take an equity centered approach and how that kind of fits into the work you're doing. I think though it's important at the same time to make sure you build an accountability into that intention, so that it can guide you through the process. If you're kind of starting to say, “ok, we want to make sure we engage with a diverse group of perspectives. We want to engage those with lived experiences. We want to create a particular kind of environment for those engage.” Hence, how do you stay true to that through the whole process. It's a journey. I think someone said earlier it's a journey, right? We're not going to get it necessarily all right every single time. But making sure you do those checks and balances through the process, it gives you a sense of where are you being successful and how are you getting feedback from those who you're working with about how it's going. Are there blind spots that we still need to be checking throughout the process? So that's one element. And one example of how that can be done is you can, at the beginning of a project or a policy or program, initiative, really be working with your participants or those you're going to engage, to align on, what are all the other principles that are going to guide this collective group? And then have you met those principles through the process? And then the other way to do is also to think about how you're defining outcomes. So, the outcomes of a policy or program should be ideally defined and co-created with those who are going to be impacted by the outcomes. So, you're engaging again the end user, your design target, in helping you to define and measure success. That's another way to build in some accountability. Not just at the traditional metrics level, but at an outcome person-centered level as well. And you're taking that into consideration for how you're measuring the ultimate product or service.
Dilip Soman: The point you made about the metrics I think is key and I think this came up in the first panel when they spoke about happiness and well-being versus financial well-being. We tend to impose our sense of well-being, we as government, we as policymakers, we as academics, onto the world. People will be happier if they have more money in the bank. We know that's not true. And then actually I was reminded of this several years ago, I was doing research in in Chennai in South India on helping people save more. I interviewed an old 82-year old woman who is a flower seller in the markets of Chennai, and she's been doing this for 40 years of her life. She really knows how to sell flowers. But she has the same daily routine for 40 years. She wake up in the morning, get a bath, go to the temple, offer a prayers, go to the money lender, get ₹500 on a 10% per day loan. So ₹50 she has to pay back. Go to the wholesale market, buy flowers, sell for stalls, sell them, go back to the money lender, pay off the ₹50. And then if she'd have ₹100 left, she would actually spend the money on buying a dosa for her grand grandkids. And I would try, as she was doing this day after day for 40 years, and I said, you know if you just set aside those ₹100 for five successive days, you don't need to go to the money lender on the 6th day. But the whole concept of saving money today for tomorrow just didn't exist in her world view. As long as she went to bed at the end of the day not in the red, she was happy. And that was a metric for happiness. So, this whole notion of balancing money today, holding it for tomorrow. And I think we need to truly understand what makes people happy in order to design policies. We don't spend time on that at all. We go into assumptions and we then design policies and then they don't work. And surprise, surprise, why not? Because we've measured the wrong thing.
Kate Bezanson: We do have some audience questions that I'd like to turn to that are really in this vein. One is how might we counter biases that favor quantitative over more qualitative relational forms of data collection and analysis and who gets to decide what constitutes evidence based decision making.
Alia Kamlani: Do you want to take that?
Dilip Soman: Sure. So I'll answer. So, one of the things that's actually worked really well for me when I worked with organizations is to ask people to make predictions on what's going to work based on the data that they have. Ex-post like when things work and things don't work, we have something called the hindsight bias. So I knew that was gonna work. They knew that was gonna be the best thing. So, the vaccination paper that you spoke about. We essentially had 10 teams of researchers that designed interventions. But before we deployed them, we actually asked people to predict which one is going to work and by how much. And gosh, how wrong we were. And I've learned that the number of times that I get it wrong is so high that’s taught me not to trust a certain kind of taken, that's taught me to go out and validate. So, remember Ronald Reagan always used to say trust but verify? I think it's a nice sort of rule of thumb to use with quantitative data. Trust it, but go and check. And if it turns out that the ground reality is different, then we need different forms of data. And I think that's the approach that we should be doing.
Alia Kamlani: I might also add. I think it's opening up like what we think the evidence actually is and who's telling us what the evidence should be. And then also what should we measure against that evidence. So part of it is opening up that process to where we are inviting other perspectives on what we should be measuring and also what the source data should be. And sometimes we don't have the right source data I would also say. So it's about creating a way to then start gathering new data and being open to that.
Kate Bezanson: I think we all agree on the importance of co-design and policy interventions, bringing equity and inclusion for stakeholders as part of the process from the very beginning. And I'd like to take a moment, maybe a few minutes, if we have time at the end, to talk about the term stakeholders. But governments do tend to have councils and advise reports, and are bound by confidentiality agreements. And the ability to bring in others opinions might actually be severely limited. Can we talk a bit about how we address challenge, mix up these stalling points?
Alia Kamlani: Yeah, I can start and in the middle you can jump in. I think there's sort of like a macro view through all the discussions we've already had this morning around whether those are the right structures to be engaging at all. So there's a bigger question around like is that the right construct to be soliciting feedback from stakeholders. And I'd love to hear your views on stakeholders. I have some, too. But recognizing of course there are requirements and just the nature of government work and the nature of how this all operates and a need for confidentiality sometimes that's just part of it, right? But I don't think that mitigates, or should mitigate, the ability to create feedback loops. I think part of what we need to do is make sure that those conversations are not in isolation with perspectives again from those who are most impacted by a policy change or a program change, et cetera. So I think there are ways to create and integrate that information through whatever existing channels need to be kind of bifurcated or separated from that at the bare minimum. So, I think that those perspectives can be layered and if you structure the process in a different way and how those councils or those tables get information and how that information is flowed down. I think also recognizing that ,as you're developing a policy or program, sometimes there are moments in that value chain that need to be kept confidential, so that the business of government can continue and you can move through right from ideation to the end. But I think it doesn't mitigate your ability to conduct even the base case for data gathering and including the perspectives of lived experience in those conversations. So, I think there's a way. from a process level, in terms of how do you create these feedback loops across the different streams of discussion that are happening. But then how do you ground and make sure that the conversation in the first place is grounded in the right way.
Dilip Soman: So I think there's an optimal way of doing things. We can think about the right data and the right people and the right methods. We're not going to get to the optimal like it's not happening tomorrow, right? And so what's next best? I think the challenge is oftentimes we say, oh, we can't go there, then let's just do what we do. Can we at least start the process of getting people who are making decisions on policies that affect subgroup X to go to Sub Group X. I mean when I remember when I was in Ottawa there was somebody making decisions on birth registrations and there was a committee of people, none of whom were mothers. And I'm like that doesn't seem right. And they hadn't even spoken to mothers or recent mothers. So, I'm like, you know, go out like, what do mothers think about this? We don't know. But we think they should think this. And have you spoken to the service Canada reps or they don't? No, we don't do that. We just tell them what to do. So just the culture of going out, I think, needs to be more prevalent in government and that's an easier thing to do. We're starting to see some changes on that. You might have observations based on your time in Ottawa. I don't know. But I think we should stop trying to be the best, but at least let's go for second best. Anything better than where we are.
Alia Kamlani: And then the only thing I would add to that is just sort of rethinking to like who sits at those tables.
Dilip Soman: Ya!
Alia Kamlani: So those tables may need to exist, but who's sitting at the table?
Kate Bezanson: How do people get to the table?
Alia Kamlani: And how people get to the table for sure.
Kate Bezanson: And there's some of the work that I think has been really challenging, especially working for, let's just say, for example in gender based violence, is there's a fatigue from whatever we would call, I really don't like the term stakeholders, I just haven't come up with a better one, so I would welcome.
Alia Kamlani: I try to avoid using the word, but I also don't have the wording, so, if anyone has a great idea, let us know what it is.
Kate Bezanson: I don't think it's the right term. I think we're all stakeholders. But there is a fatigue that comes from that kind of consultation and there's a way in which it's often a conscripted unpaid labor from people who are already exhausted from talking about the same things and having to explain their experience. So, I wonder how we mitigate that if the process is to be iterative. And I think it has to be iterative, and you get to a midpoint you realize that your assumptions were really wrong, or the policy is only as good as the people who are excluded from it. So, it needs to be revisited. So, perhaps we could discuss that.
Alia Kamlani: I mean so it depends. One very initial thought is compensation. So, in some cases, it's about you can't just be extracting from people their perspectives all the time. But how are you valuing. And it doesn't have to be monetary compensation. But how are you acknowledging that the value that they are bringing to the work and the value that they're contributing. And that kind of exchange of information, so it doesn't feel like you're just extracting all the time. And sometimes like you said, it can be very fatiguing and depending on the context, it can just be a bit retraumatization as well. So, I think there's got to be different ways of how we compensate for that exchange, sometimes it's monetary and sometimes it could be other things.
Dilip Soman: On this whole notion of evidence based, I think one of the key things is we need a lot more humility in governments. Because the current belief is that there are experts who know and as long as we have those experts making decisions or advising on, then we'll be OK. But like I said, I mean the number of times, I like to think of myself as an expert, but the number of times I've been wrong is astonishing. And I think we need more people who acknowledge that they don't know what's happening with people from other sub-groups and other ethnicities. And I think as long as we build that culture. So, let's even stop calling people experts. That might help.
Kate Bezanson: What do you recommend? Let's discuss. Another audience question that's right along these lines, that is, if I'm not in the position to gather new data, what is the best way to present outputs of, policy recommendations for example, that is based on inherently flawed data?
Dilip Soman: I was going to say that I actually don't agree that it is difficult to gather data. I think it's getting a lot easier. Maybe again like I said, maybe it's not the perfect data, maybe it's not you going to the communities and observing. But there are ways in which this could be crowd sourced. And, if you're interested, e-mail me. I'm happy to share some examples. But you could put up online portals, get people to contribute ideas, or to contribute observations. So it can be done. But again, I go back to the second best, I think. We shouldn't always strive for the best.
Alia Kamlani: I mean, what do you do in that situation I think is to acknowledge what assumptions you're making, where you think the flaws are. And perhaps this is not always easy depends on the kind of situation. But if you're using data to inform, like a decision or policy or program that's rolling out, can you test it first? So, part of it is you prototype something or can you test it in a smaller scale and use that as an opportunity to test some of the assumptions you think you know were incorrect, or the data you didn't have. Or if you feel like the data is flawed. can you start using that testing period as a bit of a way to gather the data you think you need? And then you use that to iterate on a pilot or a program before you're fully scaling it out. I think that's one strategy you can deploy.
Dilip Soman: Yeah, the buzzword, the phrase we love to use is TLA, Test, Learn, Adapt. So, start small, float a trial balloon, see if it sinks or swims, update it. And I think that's the way we should be thinking about all policy instead of coming up with this grand plan that's going to be rolled out over the next five years. And we've spent so much resource on that that we can't change anything. So, I think that there has to be a radical shift in how we make policy.
Kate Bezanson: OK, so this raises a question that is also an audience question, which I think is great, is really about the question of trust. And it's the relationship between trust and effective behavioral interventions. I'm aware that we're all sitting here talking about the importance of design and importance of thinking about the ways that we can tackle structural social inequalities in policy and in in our workplaces. And there is a very strong narrative in the popular discourse that rejects that premise entirely. And that is about a trust in institutions. And that's about a trust in intention. So, how can institutions develop and enhance trust to strengthen effective interventions?
Dilip Soman: if you start off working in areas where people want to change their behavior but can't and then you can successfully help them do that through your behavioral intervention, you earn their trust. It's all about figuring out what people want to do differently and not what you want them to do differently. You'll get trust.
Alia Kamlani: I think it's also about transparency. You have to be transparent what you're doing, what you're not doing, where the gaps are. And being transparent also builds trust.
Kate Bezanson: So, after all of the work you've done, do you think that there are any generalizable principles when it comes to people and policy, or do you really have to treat every context, person, group, problem, et cetera, as unique? What are the big take away reflections that you would like to leave with?
Dilip Soman: So, I'll say that while I firmly believe that every group is different, every situation is different, every context is different. We can develop a general approach to embrace those differences. Systematically studying heterogeneity or differences. We never do that in policy making. We focus on which is the one that's on average going to be the best. But I think we need to start developing the rigor of looking for differences as opposed to looking at averages. But I think we can create a process that embraces those differences.
Alia Kamlani: I think it's also about like it's not one and done right. You define the policy or the program and then now it's an implementation. It's just continuing to measure and evaluate how that's working and it comes back to how are you using the data, what data are you collecting, how are you measuring whether what you said or thought was the right thing to do, to balance the needs and the interests of the people that you're serving. Or how does that continue to be more of a living, breathing thing as opposed to something that you kind of reassess every 10 years? So, there's a measurement and data evaluation element to this. That's not unlinked to the outcomes that you set out at the beginning. So, you should be taking an outcomes based approach and thinking carefully about how you're defining outcomes and then measuring against those over time. I'm not in government, but I have the privilege sometimes working alongside government. Governments are in a tricky position, because they are in a position to be defining policies and programs for why, for all, for everybody. So, I think the big takeaway would be really think, rethink, how we're doing that in the first place. And then once we've done it and we’ve rolled something out, how do you keep that to be something that's iterative and you're learning from all the time?
Sarah Kaplan: That was another terrific panel discussion at our Gender Analytics: Possibilities. One of my biggest a ha’s was that getting interventions right means co-designing solutions with all stakeholders—and redefining what counts as impact in the end.
Thank you for listening to this special edition GATE Audio production podcast on “Designing for Everyone.” If you haven’t listened to them already, I hope you will check out the other 6 episodes in this limited series and other GATE Audio productions, including our signature podcast, BUSTED, where we bust common myths about gender and other forms of inequality. Just search for “Institute for Gender and the Economy” where you get your podcasts. Of course, you can help us get the word out by liking and following the podcast and telling your friends. We are nowhere without our community of listeners. If you want to keep learning, head to our website at GenderAnalytics.Org where you can discover our online course offerings and much more.
This podcast was produced by me, Sarah Kaplan, and edited by Ian Gormley. We are grateful for support from the Rotman School’s TD Management and Data Analytics Lab who co-hosted the Gender Analytics: Possibilities conference with GATE.
See you next time!