1 00:00:00,408 --> 00:00:03,889 Over the past two years, AI developers have been on a mission to achieve 2 00:00:04,009 --> 00:00:07,630 integration. Recently, that mission made its way to New Jersey's Labor Department, 3 00:00:08,050 --> 00:00:11,351 where unemployment insurance workers are using off-the-shelf models to 4 00:00:11,391 --> 00:00:14,652 make English-to-Spanish translations. And apparently, it's 5 00:00:14,692 --> 00:00:18,273 made them three times faster than they were before. Stick around, we're 6 00:00:18,293 --> 00:00:21,614 breaking it all down. But first, subscribe, hit the like button, 7 00:00:21,794 --> 00:00:33,449 drop a comment, you know, all that good stuff. The 8 00:00:33,489 --> 00:00:36,710 New Jersey Department of Labor, in collaboration with Google.org and 9 00:00:36,730 --> 00:00:40,872 the non-profit U.S. Digital Response, today released a free, open-source 10 00:00:40,912 --> 00:00:44,714 set of LLM training materials that are specifically designed to 11 00:00:44,834 --> 00:00:48,569 improve access to unemployment insurance for Spanish speakers. The 12 00:00:48,629 --> 00:00:51,811 idea here, which is already in action in New Jersey, is 13 00:00:51,851 --> 00:00:55,632 to fine tune existing off-the-shelf generative AI models, turning 14 00:00:55,652 --> 00:00:59,414 them into reliable translation assistants. The core of these materials 15 00:00:59,494 --> 00:01:02,776 is an English to Spanish unemployment insurance glossary. This 16 00:01:02,816 --> 00:01:06,377 was developed in collaboration with bilingual staff and policy experts, 17 00:01:06,817 --> 00:01:10,459 and it was tested and verified both for accuracy and efficacy. New 18 00:01:10,479 --> 00:01:14,081 Jersey's own implementation of a language translation assistant here has 19 00:01:17,534 --> 00:01:21,277 The idea here is that we have processes 20 00:01:21,377 --> 00:01:25,340 in place to try to make things 21 00:01:25,460 --> 00:01:28,963 more accessible. Unemployment, as an example of 22 00:01:29,023 --> 00:01:32,966 a benefit area in New Jersey, is very complicated. And professional translation 23 00:01:33,026 --> 00:01:36,889 is great, but it doesn't necessarily mean a professional translator 24 00:01:36,949 --> 00:01:40,352 who understands UI regulations and UI language and 25 00:01:40,552 --> 00:01:43,754 UI standards All of our work related to 26 00:01:43,894 --> 00:01:47,155 AI has been to center the human, and I mean the 27 00:01:47,215 --> 00:01:50,696 human expert. On the call center, those people are always getting humans. 28 00:01:50,756 --> 00:01:54,577 They're never getting a chatbot sort of answer. When they're talking, they're 29 00:01:54,637 --> 00:01:57,798 asking for someone to speak Spanish with them, to ask a question about a 30 00:01:57,858 --> 00:02:01,058 claim, that's always a human. We don't don't have it fully figured out yet 31 00:02:01,078 --> 00:02:05,060 but playing around with what are different ways that we can present bilingual information 32 00:02:05,080 --> 00:02:08,320 in the same email. Part of the reason that we spend so much 33 00:02:08,380 --> 00:02:11,621 time trying to figure out how do we test that this is as 34 00:02:11,701 --> 00:02:15,242 close caliber as we can to a human expert translator. How 35 00:02:15,322 --> 00:02:19,288 can we prove that this is efficient, right, 36 00:02:19,388 --> 00:02:23,070 is because we're trying to show governments that have different access 37 00:02:23,210 --> 00:02:26,653 to AI how they could build a similar thing 38 00:02:26,813 --> 00:02:30,215 and decrease their risk and increase their equity in this way. I 39 00:02:30,255 --> 00:02:33,878 believe we have over 80% of 40 00:02:33,958 --> 00:02:37,160 our staff have taken this training, and that's staff all the 41 00:02:37,220 --> 00:02:40,542 way, executive all the way to a call center agent to someone 42 00:02:43,719 --> 00:02:47,204 The materials on offer here are developer agnostic, meaning a given agency 43 00:02:47,264 --> 00:02:50,649 or state can use ChatGPT or Cloud or LLAMA or 44 00:02:50,689 --> 00:02:53,814 any other chatbot that they prefer. The idea behind this is 45 00:02:53,854 --> 00:02:57,019 that it would erase the cost of specialized long-term contracts with 46 00:02:57,059 --> 00:03:00,405 tech firms. The focus is also, importantly, on 47 00:03:01,926 --> 00:03:05,428 First, there's the problem of language barriers. Nearly 20 million 48 00:03:05,448 --> 00:03:09,792 Americans say that they speak English less than well. Second 49 00:03:09,892 --> 00:03:13,234 problem these trained materials solve is 50 00:03:13,274 --> 00:03:16,917 that it's a common problem across the United States. Any 51 00:03:16,977 --> 00:03:20,920 state can access them to create their own language assistant in 52 00:03:21,500 --> 00:03:24,683 the LLM of their choice. They can use these 53 00:03:24,723 --> 00:03:28,045 training materials and put them to work in Chet Chippy Tee 54 00:03:28,125 --> 00:03:31,187 or Claude or Gemini. This is one of those great examples of a 55 00:03:31,267 --> 00:03:34,469 public-private partnership that centered human beings at 56 00:03:34,509 --> 00:03:37,692 the center. This was not a collaboration about, how do we use 57 00:03:37,732 --> 00:03:41,334 this really cool technology, right? But rather, we took a human-first 58 00:03:41,374 --> 00:03:45,277 approach. And so what's key here is leveraging information 59 00:03:45,297 --> 00:03:48,819 that's publicly available, using that to develop 60 00:03:48,859 --> 00:03:52,602 the training materials, We don't think of this as being one of those situations 61 00:03:52,662 --> 00:03:56,625 where a job would be replaced. Rather, we see it as giving tools 62 00:03:56,746 --> 00:04:00,008 to government workers to better serve residents in 63 00:04:00,048 --> 00:04:03,631 their communities. We really wanted to ensure that there were no risks 64 00:04:04,172 --> 00:04:07,355 in terms of privacy or security that 65 00:04:07,375 --> 00:04:10,818 were a part of this approach, and I think we've struck that balance 66 00:04:11,839 --> 00:04:15,902 Now, the ethical issues of something like this are very clear. The 67 00:04:15,982 --> 00:04:19,325 obvious two that come to mind involve hallucinations and job loss. On 68 00:04:19,345 --> 00:04:22,807 that first point, hallucinations refer to the tendency of large language models 69 00:04:23,247 --> 00:04:26,810 to just produce false information. Since these models operate 70 00:04:26,830 --> 00:04:30,432 without a genuine understanding of any of the content that they produce. They 71 00:04:30,472 --> 00:04:33,573 tend to make these kinds of mistakes. The risk of hallucination in an 72 00:04:33,633 --> 00:04:36,835 environment like this seemed pretty severe. And there's also this fear of 73 00:04:36,895 --> 00:04:39,976 job loss that has spiraled alongside this rise we've seen in 74 00:04:40,016 --> 00:04:43,597 generative A.I. over the past couple of years. According to some early studies, 75 00:04:43,657 --> 00:04:47,278 that fear is not unfounded, though. So far, it's mainly targeting 76 00:04:47,318 --> 00:04:51,040 freelance industries at the moment. And there's evidence 77 00:04:51,180 --> 00:04:54,321 that it is starting to create other jobs right here. We 78 00:04:54,361 --> 00:04:57,702 talk about the idea of prompt engineering, right? 79 00:04:58,313 --> 00:05:02,354 But on top of that, there's also risks of data privacy and security breaches, again, 80 00:05:02,374 --> 00:05:06,315 in what seems to be a pretty sensitive environment. I pose these questions to 81 00:05:06,375 --> 00:05:09,676 the New Jersey Labor Department and the USDR. And according 82 00:05:09,696 --> 00:05:12,796 to both organizations, the reality is much more focused and 83 00:05:12,856 --> 00:05:16,937 much more secure than it might seem. Let's get into it. Jillian, 84 00:05:19,765 --> 00:05:23,548 You know, the, the place I want to start, right, we've got the training, uh, set 85 00:05:23,648 --> 00:05:26,991 coming out, um, and then going out 86 00:05:27,051 --> 00:05:31,294 being open sourced, uh, potentially across the country. Um, 87 00:05:32,135 --> 00:05:35,497 now I know New Jersey is the kind of first, 88 00:05:35,937 --> 00:05:39,260 uh, epicenter, I guess, that this is launching. And 89 00:05:39,280 --> 00:05:42,482 I just love to hear from you. how it's 90 00:05:42,582 --> 00:05:46,285 being used, how it's being used, what 91 00:05:46,325 --> 00:05:49,407 you're seeing on the ground so far in terms of how this is 92 00:05:49,487 --> 00:05:52,669 impacting the current processes, I 93 00:05:53,657 --> 00:05:56,819 I mean, I think to start, the 94 00:05:56,939 --> 00:06:00,641 idea here is that we have processes in 95 00:06:00,721 --> 00:06:04,524 place to try to make things 96 00:06:04,624 --> 00:06:08,486 more accessible. Unemployment as a example 97 00:06:08,526 --> 00:06:11,668 of a benefit area in New Jersey is very complicated. There's a 98 00:06:11,768 --> 00:06:15,490 lot of legal standards, federal requirements that 99 00:06:15,530 --> 00:06:18,772 people have to make or meet in order to 100 00:06:18,812 --> 00:06:22,255 be deemed eligible. And those things are really difficult to 101 00:06:22,295 --> 00:06:25,978 be understood by someone who speaks English fluently, nevermind 102 00:06:26,038 --> 00:06:29,281 someone who English isn't their first language. So when 103 00:06:29,321 --> 00:06:32,524 we try to think about how do we make equity and 104 00:06:32,704 --> 00:06:35,947 access to those benefits the priority, and whether it's unemployment or 105 00:06:36,087 --> 00:06:39,430 any other benefit that the government is trying to think about 106 00:06:39,490 --> 00:06:42,934 and make sure is reaching all the folks who are paying into 107 00:06:43,014 --> 00:06:47,137 it or who deserve it because it's a right in their workplace, One 108 00:06:47,177 --> 00:06:50,718 of the things we have to think about is language. And so for 109 00:06:50,938 --> 00:06:55,079 us, when we were rolling out our new unemployment application, an 110 00:06:55,139 --> 00:06:58,299 application that was built on modern technology, built in 111 00:06:58,339 --> 00:07:01,600 a way that it could be connected to our legacy mainframe, we 112 00:07:01,640 --> 00:07:05,060 had to think about what message do we want to send about how we 113 00:07:05,140 --> 00:07:08,361 as an administration feel about equity. And so for us, the 114 00:07:08,401 --> 00:07:11,882 most important thing was to deliver it at the same time 115 00:07:11,962 --> 00:07:15,172 in English and Spanish. And normally for us, 116 00:07:16,093 --> 00:07:19,375 that would require us, you know, to pull through all 117 00:07:19,395 --> 00:07:22,617 of the words that we were going to use and get 118 00:07:22,657 --> 00:07:26,260 those out to a professional translation. And professional 119 00:07:26,300 --> 00:07:29,442 translation is great, but it doesn't necessarily mean a 120 00:07:29,522 --> 00:07:33,005 professional translator who understands UI regulations and 121 00:07:33,185 --> 00:07:36,648 UI language and UI standards and 122 00:07:37,408 --> 00:07:40,550 what able and available means in UI speak and 123 00:07:40,711 --> 00:07:44,046 what being fired means versus being laid 124 00:07:44,206 --> 00:07:48,065 off means. in UI speak. So what 125 00:07:48,145 --> 00:07:52,048 this technology allowed us to do was pour 126 00:07:52,228 --> 00:07:55,491 in that knowledge of how do we, as 127 00:07:55,551 --> 00:07:59,174 bilingual call center agents, how do we explain that, those 128 00:07:59,254 --> 00:08:02,937 terms, those complicated ideas and requirements over 129 00:08:02,977 --> 00:08:06,139 the phone to someone who's calling in when they speak Spanish or they 130 00:08:06,179 --> 00:08:09,422 prefer to understand those terms in Spanish? How do we pour 131 00:08:09,462 --> 00:08:12,847 all that knowledge and experience that our agents have into 132 00:08:12,867 --> 00:08:16,110 the application itself. So that's when someone is 133 00:08:16,190 --> 00:08:19,333 reading those questions, they are getting the knowledge of 134 00:08:19,373 --> 00:08:22,656 that call center agent in how the translation is 135 00:08:22,696 --> 00:08:26,399 appearing to them. And so that 136 00:08:26,519 --> 00:08:29,738 sort of explosion of 137 00:08:29,798 --> 00:08:33,341 the use of all of that knowledge of our bilingual agents 138 00:08:33,421 --> 00:08:37,125 and that have that experience, being able to sort of magnify it 139 00:08:37,165 --> 00:08:40,628 across the different touch points that a human 140 00:08:40,668 --> 00:08:44,451 might have with her, with her unemployment application, 141 00:08:44,932 --> 00:08:48,335 right? The idea that you may, you're going to need it in the beginning when 142 00:08:48,355 --> 00:08:51,578 you're trying to understand, should I apply or not? You're going to need 143 00:08:51,618 --> 00:08:54,820 that knowledge again when you're doing the application. You're 144 00:08:54,840 --> 00:08:57,942 going to need that knowledge again when you get our 145 00:08:57,982 --> 00:09:01,225 communications of we need X and Y from you 146 00:09:01,285 --> 00:09:04,367 in order to figure this thing out. Or maybe you're going to 147 00:09:04,387 --> 00:09:07,649 file an appeal. How do you understand your appeal rights and 148 00:09:07,709 --> 00:09:11,151 the steps of an appeal? So 149 00:09:12,592 --> 00:09:15,835 that magnification of the impact of 150 00:09:16,676 --> 00:09:19,984 that specialized knowledge is one of the things that I'm 151 00:09:20,545 --> 00:09:24,113 so excited about in terms of increasing equity 152 00:09:26,642 --> 00:09:31,066 Right, and so what you're talking about, right, it's a very technical 153 00:09:31,186 --> 00:09:34,648 and important process. And the 154 00:09:34,728 --> 00:09:38,251 means, I guess, with which you're taking that specialized knowledge and 155 00:09:38,632 --> 00:09:43,635 turning it into Spanish, we 156 00:09:43,695 --> 00:09:47,078 know that these models have a tendency to confabulate or 157 00:09:47,118 --> 00:09:50,301 hallucinate. They kind of just output things that aren't 158 00:09:50,341 --> 00:09:54,164 necessarily grounded in truth. And so I'm wondering what procedures you 159 00:09:54,204 --> 00:09:57,878 have in place so that the person 160 00:09:57,918 --> 00:10:02,639 calling in looking for that information doesn't end up getting completely 161 00:10:02,719 --> 00:10:05,921 off-base information delivered and 162 00:10:07,621 --> 00:10:11,023 It's really important. All of our work 163 00:10:11,243 --> 00:10:14,944 related to AI has been to center the human, and 164 00:10:15,004 --> 00:10:18,369 I mean the human expert. And so on 165 00:10:18,409 --> 00:10:21,733 the call center, those people are always getting humans. They're never getting a 166 00:10:21,853 --> 00:10:25,156 chatbot sort of answer. When they're talking, they're asking for 167 00:10:25,216 --> 00:10:28,719 someone to speak Spanish with them, to ask a question about a claim, that's always 168 00:10:28,779 --> 00:10:32,003 a human. But what this is allowing us 169 00:10:32,123 --> 00:10:35,366 to do is speed up the 170 00:10:35,446 --> 00:10:39,049 process to that final human review. And 171 00:10:39,610 --> 00:10:43,233 I'll give you an example. So when we are updating 172 00:10:43,754 --> 00:10:47,037 the language in our UI application, like say we're adding a new use 173 00:10:47,097 --> 00:10:50,960 case that now people who served 174 00:10:51,000 --> 00:10:55,049 in the military can file through our online application. I 175 00:10:55,129 --> 00:10:58,811 am always looking to have parity with the English and the Spanish version, 176 00:10:59,171 --> 00:11:02,513 but how do I get that Spanish language that 177 00:11:02,593 --> 00:11:07,335 I can add in there so that both 178 00:11:07,415 --> 00:11:10,877 experiences are equal in English and Spanish? I'm 179 00:11:11,197 --> 00:11:15,239 using this system, the set of probs and this glossary, 180 00:11:15,619 --> 00:11:19,402 to give me the suggestion. We think We, 181 00:11:19,422 --> 00:11:22,785 the AI, think that this would be a good translation of 182 00:11:22,845 --> 00:11:26,328 the English version. And then the human, me, or 183 00:11:26,788 --> 00:11:30,592 one of the bilingual call center agents can go in and review and be 184 00:11:30,632 --> 00:11:34,535 like, yes, that's right, or no, I would tweak it in this way. But 185 00:11:34,575 --> 00:11:39,298 the time that it takes to get something close 186 00:11:39,338 --> 00:11:42,860 to perfect to a human and 187 00:11:42,880 --> 00:11:46,101 the amount of time that that human then has to review is 188 00:11:46,182 --> 00:11:51,164 much smaller because all of the complex terminology, 189 00:11:51,224 --> 00:11:54,565 all of the jargon that we have to work with, that's already been dealt with. 190 00:11:54,605 --> 00:11:57,967 That's already within the AI system. So the likelihood that 191 00:11:58,007 --> 00:12:01,758 I have to spend three 192 00:12:01,798 --> 00:12:05,079 hours approving the content, it's much more like now I have to 193 00:12:05,119 --> 00:12:08,281 spend 30 minutes, right? So the 194 00:12:08,341 --> 00:12:12,282 human is still there, both on the front end of determining 195 00:12:12,362 --> 00:12:17,445 what a applicable, what a realistic localized 196 00:12:17,505 --> 00:12:20,787 Spanish translation of that term is, and the human is still 197 00:12:20,827 --> 00:12:24,710 at the other end approving the version that 198 00:12:24,770 --> 00:12:28,153 spit back to us. But the time it takes to 199 00:12:28,213 --> 00:12:31,515 do that circle is going down. And 200 00:12:33,737 --> 00:12:37,480 the risk to the government that the translation is 201 00:12:37,600 --> 00:12:41,543 poor is lower because the expertise 202 00:12:41,583 --> 00:12:44,785 of the UI is being brought into the translation at the 203 00:12:44,825 --> 00:12:48,443 beginning. And so the reviewer knows 204 00:12:48,543 --> 00:12:52,568 what has gone in, what hasn't gone in, and can figure 205 00:12:52,668 --> 00:12:56,011 out, has the AI hallucinated? Is it not giving me back 206 00:12:56,111 --> 00:13:00,536 what I should have seen because I'm an expert? It 207 00:13:00,556 --> 00:13:05,035 just gives us a lot. a lot more confidence in 208 00:13:05,075 --> 00:13:08,177 the language that we're putting out there is the language that people are going to 209 00:13:08,237 --> 00:13:11,920 understand and reflects the reality of a very complicated 210 00:13:14,382 --> 00:13:17,604 Interesting. So at the end of the day, you still have not just 211 00:13:17,664 --> 00:13:21,427 human experts in UI, but bilingual human 212 00:13:21,507 --> 00:13:25,370 experts in UI and in Spanish that 213 00:13:25,390 --> 00:13:28,532 are then approving at, you know, every stage of the process. And 214 00:13:28,572 --> 00:13:31,835 so this has just enabled them to do a lot more, much 215 00:13:32,595 --> 00:13:35,838 Yeah, and I don't have to take them off the phone, right? Like, what I really want 216 00:13:35,878 --> 00:13:39,002 is for those humans to be on the phone, if that's their job, right? And 217 00:13:39,222 --> 00:13:42,605 using the example as the bilingual call center agents, they're there 218 00:13:42,645 --> 00:13:46,109 to help people on the phone who are calling because they can't figure something 219 00:13:46,289 --> 00:13:49,852 out, or they feel much more confident speaking to a human. The 220 00:13:49,893 --> 00:13:54,717 more time that they can spend on the phone and less time reviewing documents, 221 00:13:55,748 --> 00:13:59,030 The reviewing of documents has a value, but the impact to 222 00:13:59,070 --> 00:14:02,471 that human, if they're able to pick up the phone, is much more immediate and higher. 223 00:14:02,932 --> 00:14:06,533 So if I can take less time from them on the reviewing 224 00:14:06,593 --> 00:14:10,375 part because I'm giving them something much more accurate, then 225 00:14:10,415 --> 00:14:13,657 I feel like I'm getting 226 00:14:15,398 --> 00:14:18,759 And have you been seeing that play out, that kind of impact of enabling these 227 00:14:18,859 --> 00:14:22,601 experts to spend more time on the phone? Has 228 00:14:24,395 --> 00:14:27,739 Yeah, we see it in 229 00:14:27,759 --> 00:14:31,363 the review time that it takes someone to add 230 00:14:31,403 --> 00:14:35,548 this language into the intake application. 231 00:14:35,648 --> 00:14:39,272 I'm trying to think of another relevant example. 232 00:14:39,873 --> 00:14:43,949 The amount of time that it takes us to We've 233 00:14:43,969 --> 00:14:47,290 been working on some email communications and thinking about how do we, 234 00:14:47,890 --> 00:14:51,011 we don't have it fully figured out yet, but playing around with what are 235 00:14:51,051 --> 00:14:54,552 different ways that we can present bilingual information in the same email, 236 00:14:55,052 --> 00:14:58,453 knowing that about 95% of the people who call 237 00:14:58,533 --> 00:15:02,695 in asking for language support are asking for Spanish, prioritizing 238 00:15:02,755 --> 00:15:06,676 Spanish and putting those translations 239 00:15:06,776 --> 00:15:10,117 in some of the emails underneath the English and saying, you know, Español Abajo. 240 00:15:13,088 --> 00:15:16,631 the time now that it takes for us to do those translations and 241 00:15:16,651 --> 00:15:19,894 get them out in front of humans and see if they're working and see if they're not 242 00:15:19,954 --> 00:15:23,296 working is much quicker. Just 243 00:15:23,357 --> 00:15:26,739 because translations are typically an administrative process, right? 244 00:15:26,779 --> 00:15:30,002 You send them, you procure them, you go through a whole set of 245 00:15:30,783 --> 00:15:34,185 steps to do that, and that has its place. But for 246 00:15:35,966 --> 00:15:39,307 the type of nuance that a UI applicant needs 247 00:15:39,367 --> 00:15:42,888 in order to have true equity and access to their benefits 248 00:15:43,008 --> 00:15:46,789 and truly understand their rights and responsibilities. We 249 00:15:46,909 --> 00:15:50,190 feel like having the bilingual call center 250 00:15:50,250 --> 00:15:53,890 agents or the bilingual adjudication agents or the bilingual staff be 251 00:15:53,950 --> 00:15:57,311 part of that process is helping us 252 00:15:57,391 --> 00:16:00,712 both increase in equity, increase our trust that we're presenting what 253 00:16:01,972 --> 00:16:05,593 Right. And so I guess this approach, is 254 00:16:05,633 --> 00:16:09,634 this something that's being explored for areas beyond UI? And 255 00:16:12,935 --> 00:16:16,155 That's part of the reason that we spend so much time trying to 256 00:16:16,195 --> 00:16:20,750 figure out how do we test that this is of as 257 00:16:20,830 --> 00:16:24,371 close caliber as we can to a human expert translator. How 258 00:16:24,451 --> 00:16:28,411 can we prove that this is efficient, right, 259 00:16:28,531 --> 00:16:31,712 is because we're trying to show governments that 260 00:16:31,732 --> 00:16:35,073 have different access to AI how they could 261 00:16:35,153 --> 00:16:38,513 build a similar thing and decrease their risk and increase 262 00:16:38,533 --> 00:16:41,674 their equity in this way. So some of the steps that 263 00:16:41,694 --> 00:16:46,935 we've taken so far is to start to train other areas 264 00:16:46,975 --> 00:16:50,355 within Department of Labor on how they 265 00:16:50,395 --> 00:16:53,839 could take these steps and what sort of glossaries they 266 00:16:53,859 --> 00:16:57,402 would want to build with our Paid Family Medical Leave program, 267 00:16:57,422 --> 00:17:00,765 for example, or workers' rights programs. How 268 00:17:00,805 --> 00:17:04,489 do you build the right glossary in English? You build the right glossary 269 00:17:04,549 --> 00:17:08,072 in plain language. You build the right glossary in Spanish so 270 00:17:08,132 --> 00:17:13,023 that it could get you to a localized, 271 00:17:13,103 --> 00:17:16,185 acceptable, on like a 272 00:17:16,265 --> 00:17:20,126 risk and legal framework, translations faster. So 273 00:17:20,146 --> 00:17:23,347 we've been taking those steps and training some folks to see how they might 274 00:17:23,367 --> 00:17:26,548 be able to use it, what sort of work would be involved in 275 00:17:26,588 --> 00:17:30,290 developing a glossary that both the staff believe 276 00:17:30,310 --> 00:17:33,651 in, is legally sound, and then 277 00:17:36,663 --> 00:17:40,188 And so the last point I got for you, right, when we talk about the training 278 00:17:40,388 --> 00:17:43,792 and integration of these systems, I wonder to 279 00:17:43,832 --> 00:17:47,777 what degree that also includes AI 280 00:17:47,857 --> 00:17:51,302 education, AI literacy education on the workforce to 281 00:17:51,342 --> 00:17:54,586 make sure people understand what they're dealing with, how 282 00:17:55,738 --> 00:18:00,681 Yeah, it's been a big effort. I 283 00:18:00,701 --> 00:18:03,803 don't own it. It's owned by the governor's office and the 284 00:18:03,843 --> 00:18:07,365 New Jersey Office of Innovation to train our public sector 285 00:18:07,786 --> 00:18:12,589 on the safe use of AI. And there's 286 00:18:12,989 --> 00:18:16,591 extensive training, I think it's over two and a half hours, an extensive training 287 00:18:16,771 --> 00:18:20,634 on what is generative AI, how is it different than regular AI, what 288 00:18:20,674 --> 00:18:24,157 you should or shouldn't be putting into it, how 289 00:18:24,197 --> 00:18:30,014 to understand when and how it may hallucinate. And 290 00:18:30,534 --> 00:18:33,677 that training has been made available to all state staff. And 291 00:18:33,717 --> 00:18:37,560 in NJDOL, I believe we 292 00:18:37,620 --> 00:18:41,083 have over 80% of our staff have taken this training. 293 00:18:41,584 --> 00:18:44,867 And that's staff all the way, executive all the way to 294 00:18:44,907 --> 00:18:48,570 a call center agent to someone who works in the mailroom. That's across the 295 00:18:48,610 --> 00:18:52,233 board. So we've taken it very seriously. I don't 296 00:18:52,353 --> 00:18:57,466 know that necessarily everyone is using the the 297 00:18:57,726 --> 00:19:01,607 AI instance that we have for the public sector in New Jersey. So 298 00:19:01,707 --> 00:19:05,008 after you've taken this training, you can access this sandbox that's 299 00:19:05,068 --> 00:19:08,369 specific for state employees. But that's where 300 00:19:08,409 --> 00:19:11,750 we're doing our practice and our training with other 301 00:19:11,790 --> 00:19:14,931 staff agency is once you've taken that training, then we 302 00:19:14,971 --> 00:19:18,292 work with you to practice with 303 00:19:28,603 --> 00:19:31,747 Yeah. So, you know, we got this, I 304 00:19:31,767 --> 00:19:35,031 guess, pretty big launch today, right, with the new program, with the training 305 00:19:35,071 --> 00:19:39,115 materials. But where I want to start, right, tell me about the kind 306 00:19:39,135 --> 00:19:42,779 of motivation behind this. Why is this needed right 307 00:19:45,149 --> 00:19:48,352 You know, there's a couple problems happening concurrently, and it's 308 00:19:48,532 --> 00:19:52,696 part of why I'm so excited about the free and open source training 309 00:19:52,716 --> 00:19:56,179 materials we're putting out. First, there's the problem of language barriers. 310 00:19:57,080 --> 00:20:00,604 Nearly 20 million Americans describe their English skills 311 00:20:00,684 --> 00:20:03,887 as less than they speak English. Let me 312 00:20:03,907 --> 00:20:07,161 say that again. Nearly 20 million Americans say that 313 00:20:07,221 --> 00:20:11,127 they speak English less than well. And so for those 314 00:20:11,207 --> 00:20:14,472 20 million Americans, we need to make sure that, for 315 00:20:14,532 --> 00:20:18,393 example, when they experience job loss, They're 316 00:20:18,433 --> 00:20:21,795 not left in the cold in terms of getting the help they need to 317 00:20:21,815 --> 00:20:25,397 get back on their feet, get back in the labor force, and re-engage with 318 00:20:25,417 --> 00:20:28,699 the economy. So those language barriers are the first 319 00:20:28,739 --> 00:20:32,321 sort of key problem we want to solve. Second problem 320 00:20:32,961 --> 00:20:36,083 these training materials solve is that 321 00:20:36,123 --> 00:20:39,925 it's a common problem across the United States. So, 322 00:20:41,006 --> 00:20:44,889 you know, in this case, we're just so excited to release these 323 00:20:44,969 --> 00:20:48,332 materials with New Jersey. I'm a former New Jersey 324 00:20:48,352 --> 00:20:51,754 resident, go Jersey. But the fact of the matter is every 325 00:20:51,854 --> 00:20:55,397 state provides these services and these benefits. 326 00:20:56,398 --> 00:21:00,681 And we don't want each of them solving the problem themselves. 327 00:21:00,962 --> 00:21:04,366 So these training materials are replicable for all 328 00:21:04,446 --> 00:21:07,730 states. And so it means that there's sort 329 00:21:07,750 --> 00:21:11,595 of economies of scale with a free and open source solution. 330 00:21:12,692 --> 00:21:16,294 Last thing I'll mention, Ian, is that this solves the problem of 331 00:21:16,574 --> 00:21:19,795 lengthy and costly contracts with tech 332 00:21:19,835 --> 00:21:23,097 vendors. A lot of governments can get tied up when they 333 00:21:23,137 --> 00:21:27,038 have to purchase technology. There's really complicated contracts, 334 00:21:27,378 --> 00:21:31,020 it locks them in for the long term. And I think what's really important when 335 00:21:31,040 --> 00:21:35,022 we think about, you know, return on taxpayer investment, is 336 00:21:35,102 --> 00:21:38,563 making sure that governments can purchase the technology they 337 00:21:38,603 --> 00:21:42,045 need, but not be locked in. Because it's constantly evolving 338 00:21:42,085 --> 00:21:45,467 and changing. So we think for those sort of 339 00:21:45,867 --> 00:21:49,128 set of reasons, those are some of the problems we 340 00:21:52,210 --> 00:21:55,372 And so that last point, I guess, is why this is a launch of 341 00:21:55,452 --> 00:21:58,653 training materials, not a model itself. It's 342 00:21:58,713 --> 00:22:01,935 not just so no one is confused here. You're 343 00:22:01,955 --> 00:22:05,116 not launching a translation model. You're launching materials so that people can, 344 00:22:08,258 --> 00:22:11,539 That's right. These training materials can be used, 345 00:22:12,319 --> 00:22:16,500 any state can access them to create their own language assistant in 346 00:22:17,080 --> 00:22:20,261 the LLM of their choice. They can use these 347 00:22:20,301 --> 00:22:23,782 training materials and put them to work in CHAT GPT or 348 00:22:23,862 --> 00:22:27,363 CLAWD or Gemini. And so it's getting them 349 00:22:27,903 --> 00:22:31,284 most of the way there and then a state can access these 350 00:22:31,424 --> 00:22:34,725 training materials and create a language assistant of their own to 351 00:22:36,625 --> 00:22:39,748 Gotcha. Now, the materials themselves, how 352 00:22:39,788 --> 00:22:43,931 were they gathered, curated, verified, 353 00:22:44,051 --> 00:22:47,174 collected, right? We know that in AI, 354 00:22:47,214 --> 00:22:50,556 the data is king, and making sure the data is 355 00:22:50,997 --> 00:22:54,780 clean and accurate and legitimate, right, is a process. So 356 00:22:56,548 --> 00:23:00,129 Yeah, it involved a lot of collaboration. And I'm excited. This 357 00:23:00,189 --> 00:23:03,530 is one of those great examples of a public-private partnership that 358 00:23:03,590 --> 00:23:06,731 centered human beings at the center. This was not a 359 00:23:06,771 --> 00:23:10,532 collaboration about how do we use this really cool technology, right? But 360 00:23:10,572 --> 00:23:13,873 rather, we took a human-first approach. So in collaborating with 361 00:23:14,073 --> 00:23:18,115 the New Jersey Department of Labor and Google.org, we 362 00:23:18,175 --> 00:23:21,516 started by bringing together bilingual staff and 363 00:23:21,576 --> 00:23:25,317 policy experts to help make sense of the key information 364 00:23:26,017 --> 00:23:29,419 that individuals, residents in New Jersey 365 00:23:29,479 --> 00:23:33,260 would need to know to access unemployment insurance. So 366 00:23:33,360 --> 00:23:36,602 what's the key information? What do they have to know? We 367 00:23:36,622 --> 00:23:40,264 then, sort of as we developed the training materials, 368 00:23:40,604 --> 00:23:43,885 humans were involved at every step of the way. I think what's so 369 00:23:43,945 --> 00:23:47,227 exciting for me is that with the approach we've taken with these training 370 00:23:47,247 --> 00:23:51,349 materials, We're leveraging Gen AI 371 00:23:51,369 --> 00:23:54,871 to streamline processing the information, but we're not leveraging 372 00:23:54,911 --> 00:23:57,993 it to make decisions. We're still keeping that in 373 00:23:58,013 --> 00:24:01,615 the hands of the humans so that they can be doing the more complex work 374 00:24:02,015 --> 00:24:05,918 and putting the technology to work to help streamline information. 375 00:24:06,759 --> 00:24:09,962 I'm glad you brought up the decision-making side of things, right? Because I wanted to ask about 376 00:24:10,002 --> 00:24:13,666 that. We know issues of algorithmic 377 00:24:13,706 --> 00:24:17,990 discrimination are a documented problem, aside 378 00:24:18,030 --> 00:24:22,294 from, you know, when the algorithm just screws up, right? Hallucinations, confabulations, 379 00:24:22,354 --> 00:24:25,696 etc. These things are not necessarily completely reliable. So 380 00:24:25,756 --> 00:24:29,037 I wonder, you know, in terms of disseminating this open 381 00:24:29,077 --> 00:24:32,359 source kind of framework, right, to what degree does 382 00:24:32,419 --> 00:24:35,940 a sort of training go with that to make sure that whoever's using it knows, 383 00:24:36,441 --> 00:24:39,542 you know, verify on these points? Or is there some sort of 384 00:24:39,882 --> 00:24:43,023 trust and transparency explainability layer that kind of is 385 00:24:43,323 --> 00:24:46,705 designed to go hand in hand with this? 386 00:24:47,090 --> 00:24:50,314 That's a great question. And I think I can have Brian and Kate follow up 387 00:24:50,354 --> 00:24:53,858 with some more of the details and the technical information. But 388 00:24:53,878 --> 00:24:57,062 what I'll share is, I think that for us, what we've been wanting to do is 389 00:24:57,122 --> 00:25:01,027 to ensure that there is a way that we're responsibly 390 00:25:01,327 --> 00:25:04,671 leveraging the technology, but not at 391 00:25:04,732 --> 00:25:08,235 the risk of the determinations and decisions 392 00:25:08,295 --> 00:25:12,218 that are being made. And so what's key here is leveraging 393 00:25:12,278 --> 00:25:15,960 information that's publicly available, using that to 394 00:25:16,000 --> 00:25:19,123 develop the training materials, and then again, making it 395 00:25:19,143 --> 00:25:22,525 available for other states to access and customize on 396 00:25:22,565 --> 00:25:26,268 their end so that it works in their particular context. It 397 00:25:26,328 --> 00:25:29,810 works for them and the problem they're trying to solve. It's 398 00:25:30,051 --> 00:25:33,254 something I just appreciate. We're not trying to solve all the problems at once. 399 00:25:33,594 --> 00:25:37,979 But instead, we're doing some of the legwork to then empower folks 400 00:25:38,119 --> 00:25:41,362 in governments across the country to sort of take the 401 00:25:46,167 --> 00:25:49,549 risk factors, right? And one is 402 00:25:49,569 --> 00:25:52,690 a little messy. We haven't seen details on 403 00:25:52,730 --> 00:25:56,391 how it's playing out yet, but it's something that people kind of associate with similar 404 00:25:56,451 --> 00:26:00,013 pushes, right? Is will a chatbot 405 00:26:00,093 --> 00:26:03,835 take my job type of thing? And in this environment, right? 406 00:26:04,946 --> 00:26:08,148 Is that to a degree what's going on or is it quite simply there 407 00:26:10,850 --> 00:26:14,753 Yeah. You know, I think there is there is a talent 408 00:26:15,073 --> 00:26:18,395 issue at play here in terms of the talent we have in government having 409 00:26:18,435 --> 00:26:22,378 enough bilingual staff to be able to engage 410 00:26:22,518 --> 00:26:25,560 and translate the all of 411 00:26:25,600 --> 00:26:29,303 the government legalese that we have underpinning 412 00:26:29,683 --> 00:26:33,949 programs and services. Ultimately, what 413 00:26:33,989 --> 00:26:37,256 we're trying to do, though, is to think about 414 00:26:37,336 --> 00:26:41,003 how to, I guess, how to essentially give government workers 415 00:26:41,023 --> 00:26:44,437 a tool like a calculator. We don't 416 00:26:44,537 --> 00:26:48,239 see these training materials as replacing roles or getting 417 00:26:48,279 --> 00:26:51,580 rid of jobs. Rather, we think of these as 418 00:26:52,221 --> 00:26:55,663 being a key tool that government workers can use, 419 00:26:56,383 --> 00:26:59,785 process the information, get it in a way that is easily 420 00:26:59,885 --> 00:27:03,347 understandable for the residents you're trying to serve, so 421 00:27:03,387 --> 00:27:09,350 that then those human beings can focus on the more complex tasks 422 00:27:09,450 --> 00:27:13,172 that they have at hand, like making determination decisions, and 423 00:27:13,292 --> 00:27:16,594 figuring out the right next step, both on the government side 424 00:27:16,934 --> 00:27:20,116 and the resident side. So we don't think of this as being 425 00:27:20,156 --> 00:27:23,738 one of those situations where a job would be replaced. Rather, we 426 00:27:23,778 --> 00:27:27,080 see it as giving tools to government workers to better 427 00:27:36,005 --> 00:27:39,168 The other point, too, in talking about the risks, right, when we're 428 00:27:39,208 --> 00:27:42,491 talking about using off-the-shelf models, is 429 00:27:42,571 --> 00:27:46,494 there a risk of data privacy security 430 00:27:46,574 --> 00:27:49,737 issues by talking about 431 00:27:49,777 --> 00:27:53,360 potentially sensitive topics that get parsed through 432 00:27:56,955 --> 00:28:00,238 I love that you're asking about it because it's something that's really top of mind at 433 00:28:01,299 --> 00:28:05,181 U.S. Digital Response. We sort of put 434 00:28:05,282 --> 00:28:08,424 forth an approach in all of our work on generative AI that 435 00:28:08,464 --> 00:28:12,847 centers responsibility and ethical use of the technology. These 436 00:28:12,927 --> 00:28:16,111 training materials that we're releasing today do not 437 00:28:16,171 --> 00:28:19,314 touch PII, and rather what we're saying is we're 438 00:28:19,354 --> 00:28:22,858 taking the publicly available data and we're making 439 00:28:22,998 --> 00:28:26,361 better use of that and translating it, making 440 00:28:26,401 --> 00:28:29,763 it more accessible. But it's entirely staying 441 00:28:29,804 --> 00:28:33,024 on the government side in any state that chooses to use 442 00:28:33,064 --> 00:28:37,085 these materials, as New Jersey has done, to bring 443 00:28:37,185 --> 00:28:40,746 in any, you know, any private data. So 444 00:28:40,786 --> 00:28:43,887 we really wanted to ensure that there were no risks in 445 00:28:43,927 --> 00:28:47,188 terms of privacy or security that were 446 00:28:47,208 --> 00:28:50,709 a part of this approach. And I think we've struck that balance quite 447 00:28:51,129 --> 00:28:54,473 And the last point I want to leave off on, right, you mentioned that New Jersey is already 448 00:28:54,593 --> 00:28:57,876 using this. Yeah. I'm wondering what you can tell me 449 00:28:57,936 --> 00:29:01,180 about what that's looked like, responses so 450 00:29:01,260 --> 00:29:04,723 far as New Jersey. I am a 451 00:29:04,784 --> 00:29:08,027 native New Jerseyan myself, right? How that 452 00:29:08,047 --> 00:29:11,150 integration is going, what that kind of 453 00:29:13,299 --> 00:29:16,761 In New Jersey, what they found is that by leveraging these 454 00:29:17,521 --> 00:29:20,922 training materials, they have tripled their translation speed, 455 00:29:21,403 --> 00:29:25,424 but not at the cost of quality. So we did evaluation, 456 00:29:25,565 --> 00:29:29,666 and the translations were produced quicker, but 457 00:29:29,746 --> 00:29:33,008 are nearly on par with the level of quality that 458 00:29:33,168 --> 00:29:36,409 expert human translators would produce. I think that's 459 00:29:36,449 --> 00:29:40,111 really exciting. I think that that means that New Jersey residents 460 00:29:40,151 --> 00:29:43,552 who need these services so critically are going 461 00:29:43,572 --> 00:29:46,754 to get what they need, get it faster. And then 462 00:29:47,374 --> 00:29:50,455 on the government side, workers can be moving things along on 463 00:29:50,495 --> 00:29:54,057 their end, knowing that there's a high level of quality in 464 00:29:54,097 --> 00:29:57,299 terms of the information being given to residents. So, I 465 00:29:57,819 --> 00:30:01,182 think you're going to speak with our partner Jill 466 00:30:01,383 --> 00:30:04,566 at the State of New Jersey, and she'll be able to share more. I think 467 00:30:05,266 --> 00:30:08,669 that impact, that result in terms 468 00:30:08,830 --> 00:30:12,453 of the quicker translation speed 469 00:30:12,573 --> 00:30:16,016 is really exciting, and what we want to see, not at 470 00:30:17,298 --> 00:30:20,701 Of course. Well, thanks so much for joining me, Tita. Thank 471 00:30:22,197 --> 00:30:25,418 Thank you. And Ian, can I just say one thing 472 00:30:25,458 --> 00:30:28,979 that I'm so excited about 473 00:30:29,019 --> 00:30:32,740 this for sort of, if I can, just at a high level. The 474 00:30:32,780 --> 00:30:37,201 training materials we're putting out, they're fast, again. Translation 475 00:30:37,261 --> 00:30:41,342 speeds in New Jersey have tripled. They're free. Governments 476 00:30:41,402 --> 00:30:44,522 can use these resources without entering into lengthy or 477 00:30:44,562 --> 00:30:47,823 complex contracts with vendors. and 478 00:30:47,863 --> 00:30:51,404 doing so reduces risk and reduces the 479 00:30:51,464 --> 00:30:54,825 use of taxpayer dollars, and it benefits everyone. So 480 00:30:54,885 --> 00:30:58,486 constituents are getting better results, government workers 481 00:30:58,546 --> 00:31:02,106 are able to focus on complex tasks, and taxpayers know 482 00:31:02,126 --> 00:31:05,667 that their dollars aren't going to pay for some overly 483 00:31:05,707 --> 00:31:09,608 complex tool. I think it's just a really exciting