Subscribe
Share
Share
Embed
New Jersey's department of labor, in collaboration with USDR and Google.org, has assembled a set of training materials designed to turn off-the-shelf language models into bilingual unemployment insurance experts. We sat down with two of the people behind the launch to break it down.
Creators & Guests
What is The Deep View: Conversations?
Artificial intelligence is a complicated topic, bound by many complex threads — technical science, ethics, safety, regulation, neuroscience, psychology, philosophy, investment, and — above all — humanity. On The Deep View: Conversations, Ian Krietzberg, host and Editor in Chief at The Deep View, breaks it all down, cutting through the hype to make clear what's important and why you should care.
Over the past two years, AI developers have been on a mission to achieve
integration. Recently, that mission made its way to New Jersey's Labor Department,
where unemployment insurance workers are using off-the-shelf models to
make English-to-Spanish translations. And apparently, it's
made them three times faster than they were before. Stick around, we're
breaking it all down. But first, subscribe, hit the like button,
drop a comment, you know, all that good stuff. The
New Jersey Department of Labor, in collaboration with Google.org and
the non-profit U.S. Digital Response, today released a free, open-source
set of LLM training materials that are specifically designed to
improve access to unemployment insurance for Spanish speakers. The
idea here, which is already in action in New Jersey, is
to fine tune existing off-the-shelf generative AI models, turning
them into reliable translation assistants. The core of these materials
is an English to Spanish unemployment insurance glossary. This
was developed in collaboration with bilingual staff and policy experts,
and it was tested and verified both for accuracy and efficacy. New
Jersey's own implementation of a language translation assistant here has
The idea here is that we have processes
in place to try to make things
more accessible. Unemployment, as an example of
a benefit area in New Jersey, is very complicated. And professional translation
is great, but it doesn't necessarily mean a professional translator
who understands UI regulations and UI language and
UI standards All of our work related to
AI has been to center the human, and I mean the
human expert. On the call center, those people are always getting humans.
They're never getting a chatbot sort of answer. When they're talking, they're
asking for someone to speak Spanish with them, to ask a question about a
claim, that's always a human. We don't don't have it fully figured out yet
but playing around with what are different ways that we can present bilingual information
in the same email. Part of the reason that we spend so much
time trying to figure out how do we test that this is as
close caliber as we can to a human expert translator. How
can we prove that this is efficient, right,
is because we're trying to show governments that have different access
to AI how they could build a similar thing
and decrease their risk and increase their equity in this way. I
believe we have over 80% of
our staff have taken this training, and that's staff all the
way, executive all the way to a call center agent to someone
The materials on offer here are developer agnostic, meaning a given agency
or state can use ChatGPT or Cloud or LLAMA or
any other chatbot that they prefer. The idea behind this is
that it would erase the cost of specialized long-term contracts with
tech firms. The focus is also, importantly, on
First, there's the problem of language barriers. Nearly 20 million
Americans say that they speak English less than well. Second
problem these trained materials solve is
that it's a common problem across the United States. Any
state can access them to create their own language assistant in
the LLM of their choice. They can use these
training materials and put them to work in Chet Chippy Tee
or Claude or Gemini. This is one of those great examples of a
public-private partnership that centered human beings at
the center. This was not a collaboration about, how do we use
this really cool technology, right? But rather, we took a human-first
approach. And so what's key here is leveraging information
that's publicly available, using that to develop
the training materials, We don't think of this as being one of those situations
where a job would be replaced. Rather, we see it as giving tools
to government workers to better serve residents in
their communities. We really wanted to ensure that there were no risks
in terms of privacy or security that
were a part of this approach, and I think we've struck that balance
Now, the ethical issues of something like this are very clear. The
obvious two that come to mind involve hallucinations and job loss. On
that first point, hallucinations refer to the tendency of large language models
to just produce false information. Since these models operate
without a genuine understanding of any of the content that they produce. They
tend to make these kinds of mistakes. The risk of hallucination in an
environment like this seemed pretty severe. And there's also this fear of
job loss that has spiraled alongside this rise we've seen in
generative A.I. over the past couple of years. According to some early studies,
that fear is not unfounded, though. So far, it's mainly targeting
freelance industries at the moment. And there's evidence
that it is starting to create other jobs right here. We
talk about the idea of prompt engineering, right?
But on top of that, there's also risks of data privacy and security breaches, again,
in what seems to be a pretty sensitive environment. I pose these questions to
the New Jersey Labor Department and the USDR. And according
to both organizations, the reality is much more focused and
much more secure than it might seem. Let's get into it. Jillian,
You know, the, the place I want to start, right, we've got the training, uh, set
coming out, um, and then going out
being open sourced, uh, potentially across the country. Um,
now I know New Jersey is the kind of first,
uh, epicenter, I guess, that this is launching. And
I just love to hear from you. how it's
being used, how it's being used, what
you're seeing on the ground so far in terms of how this is
impacting the current processes, I
I mean, I think to start, the
idea here is that we have processes in
place to try to make things
more accessible. Unemployment as a example
of a benefit area in New Jersey is very complicated. There's a
lot of legal standards, federal requirements that
people have to make or meet in order to
be deemed eligible. And those things are really difficult to
be understood by someone who speaks English fluently, nevermind
someone who English isn't their first language. So when
we try to think about how do we make equity and
access to those benefits the priority, and whether it's unemployment or
any other benefit that the government is trying to think about
and make sure is reaching all the folks who are paying into
it or who deserve it because it's a right in their workplace, One
of the things we have to think about is language. And so for
us, when we were rolling out our new unemployment application, an
application that was built on modern technology, built in
a way that it could be connected to our legacy mainframe, we
had to think about what message do we want to send about how we
as an administration feel about equity. And so for us, the
most important thing was to deliver it at the same time
in English and Spanish. And normally for us,
that would require us, you know, to pull through all
of the words that we were going to use and get
those out to a professional translation. And professional
translation is great, but it doesn't necessarily mean a
professional translator who understands UI regulations and
UI language and UI standards and
what able and available means in UI speak and
what being fired means versus being laid
off means. in UI speak. So what
this technology allowed us to do was pour
in that knowledge of how do we, as
bilingual call center agents, how do we explain that, those
terms, those complicated ideas and requirements over
the phone to someone who's calling in when they speak Spanish or they
prefer to understand those terms in Spanish? How do we pour
all that knowledge and experience that our agents have into
the application itself. So that's when someone is
reading those questions, they are getting the knowledge of
that call center agent in how the translation is
appearing to them. And so that
sort of explosion of
the use of all of that knowledge of our bilingual agents
and that have that experience, being able to sort of magnify it
across the different touch points that a human
might have with her, with her unemployment application,
right? The idea that you may, you're going to need it in the beginning when
you're trying to understand, should I apply or not? You're going to need
that knowledge again when you're doing the application. You're
going to need that knowledge again when you get our
communications of we need X and Y from you
in order to figure this thing out. Or maybe you're going to
file an appeal. How do you understand your appeal rights and
the steps of an appeal? So
that magnification of the impact of
that specialized knowledge is one of the things that I'm
so excited about in terms of increasing equity
Right, and so what you're talking about, right, it's a very technical
and important process. And the
means, I guess, with which you're taking that specialized knowledge and
turning it into Spanish, we
know that these models have a tendency to confabulate or
hallucinate. They kind of just output things that aren't
necessarily grounded in truth. And so I'm wondering what procedures you
have in place so that the person
calling in looking for that information doesn't end up getting completely
off-base information delivered and
It's really important. All of our work
related to AI has been to center the human, and
I mean the human expert. And so on
the call center, those people are always getting humans. They're never getting a
chatbot sort of answer. When they're talking, they're asking for
someone to speak Spanish with them, to ask a question about a claim, that's always
a human. But what this is allowing us
to do is speed up the
process to that final human review. And
I'll give you an example. So when we are updating
the language in our UI application, like say we're adding a new use
case that now people who served
in the military can file through our online application. I
am always looking to have parity with the English and the Spanish version,
but how do I get that Spanish language that
I can add in there so that both
experiences are equal in English and Spanish? I'm
using this system, the set of probs and this glossary,
to give me the suggestion. We think We,
the AI, think that this would be a good translation of
the English version. And then the human, me, or
one of the bilingual call center agents can go in and review and be
like, yes, that's right, or no, I would tweak it in this way. But
the time that it takes to get something close
to perfect to a human and
the amount of time that that human then has to review is
much smaller because all of the complex terminology,
all of the jargon that we have to work with, that's already been dealt with.
That's already within the AI system. So the likelihood that
I have to spend three
hours approving the content, it's much more like now I have to
spend 30 minutes, right? So the
human is still there, both on the front end of determining
what a applicable, what a realistic localized
Spanish translation of that term is, and the human is still
at the other end approving the version that
spit back to us. But the time it takes to
do that circle is going down. And
the risk to the government that the translation is
poor is lower because the expertise
of the UI is being brought into the translation at the
beginning. And so the reviewer knows
what has gone in, what hasn't gone in, and can figure
out, has the AI hallucinated? Is it not giving me back
what I should have seen because I'm an expert? It
just gives us a lot. a lot more confidence in
the language that we're putting out there is the language that people are going to
understand and reflects the reality of a very complicated
Interesting. So at the end of the day, you still have not just
human experts in UI, but bilingual human
experts in UI and in Spanish that
are then approving at, you know, every stage of the process. And
so this has just enabled them to do a lot more, much
Yeah, and I don't have to take them off the phone, right? Like, what I really want
is for those humans to be on the phone, if that's their job, right? And
using the example as the bilingual call center agents, they're there
to help people on the phone who are calling because they can't figure something
out, or they feel much more confident speaking to a human. The
more time that they can spend on the phone and less time reviewing documents,
The reviewing of documents has a value, but the impact to
that human, if they're able to pick up the phone, is much more immediate and higher.
So if I can take less time from them on the reviewing
part because I'm giving them something much more accurate, then
I feel like I'm getting
And have you been seeing that play out, that kind of impact of enabling these
experts to spend more time on the phone? Has
Yeah, we see it in
the review time that it takes someone to add
this language into the intake application.
I'm trying to think of another relevant example.
The amount of time that it takes us to We've
been working on some email communications and thinking about how do we,
we don't have it fully figured out yet, but playing around with what are
different ways that we can present bilingual information in the same email,
knowing that about 95% of the people who call
in asking for language support are asking for Spanish, prioritizing
Spanish and putting those translations
in some of the emails underneath the English and saying, you know, Español Abajo.
the time now that it takes for us to do those translations and
get them out in front of humans and see if they're working and see if they're not
working is much quicker. Just
because translations are typically an administrative process, right?
You send them, you procure them, you go through a whole set of
steps to do that, and that has its place. But for
the type of nuance that a UI applicant needs
in order to have true equity and access to their benefits
and truly understand their rights and responsibilities. We
feel like having the bilingual call center
agents or the bilingual adjudication agents or the bilingual staff be
part of that process is helping us
both increase in equity, increase our trust that we're presenting what
Right. And so I guess this approach, is
this something that's being explored for areas beyond UI? And
That's part of the reason that we spend so much time trying to
figure out how do we test that this is of as
close caliber as we can to a human expert translator. How
can we prove that this is efficient, right,
is because we're trying to show governments that
have different access to AI how they could
build a similar thing and decrease their risk and increase
their equity in this way. So some of the steps that
we've taken so far is to start to train other areas
within Department of Labor on how they
could take these steps and what sort of glossaries they
would want to build with our Paid Family Medical Leave program,
for example, or workers' rights programs. How
do you build the right glossary in English? You build the right glossary
in plain language. You build the right glossary in Spanish so
that it could get you to a localized,
acceptable, on like a
risk and legal framework, translations faster. So
we've been taking those steps and training some folks to see how they might
be able to use it, what sort of work would be involved in
developing a glossary that both the staff believe
in, is legally sound, and then
And so the last point I got for you, right, when we talk about the training
and integration of these systems, I wonder to
what degree that also includes AI
education, AI literacy education on the workforce to
make sure people understand what they're dealing with, how
Yeah, it's been a big effort. I
don't own it. It's owned by the governor's office and the
New Jersey Office of Innovation to train our public sector
on the safe use of AI. And there's
extensive training, I think it's over two and a half hours, an extensive training
on what is generative AI, how is it different than regular AI, what
you should or shouldn't be putting into it, how
to understand when and how it may hallucinate. And
that training has been made available to all state staff. And
in NJDOL, I believe we
have over 80% of our staff have taken this training.
And that's staff all the way, executive all the way to
a call center agent to someone who works in the mailroom. That's across the
board. So we've taken it very seriously. I don't
know that necessarily everyone is using the the
AI instance that we have for the public sector in New Jersey. So
after you've taken this training, you can access this sandbox that's
specific for state employees. But that's where
we're doing our practice and our training with other
staff agency is once you've taken that training, then we
work with you to practice with
Yeah. So, you know, we got this, I
guess, pretty big launch today, right, with the new program, with the training
materials. But where I want to start, right, tell me about the kind
of motivation behind this. Why is this needed right
You know, there's a couple problems happening concurrently, and it's
part of why I'm so excited about the free and open source training
materials we're putting out. First, there's the problem of language barriers.
Nearly 20 million Americans describe their English skills
as less than they speak English. Let me
say that again. Nearly 20 million Americans say that
they speak English less than well. And so for those
20 million Americans, we need to make sure that, for
example, when they experience job loss, They're
not left in the cold in terms of getting the help they need to
get back on their feet, get back in the labor force, and re-engage with
the economy. So those language barriers are the first
sort of key problem we want to solve. Second problem
these training materials solve is that
it's a common problem across the United States. So,
you know, in this case, we're just so excited to release these
materials with New Jersey. I'm a former New Jersey
resident, go Jersey. But the fact of the matter is every
state provides these services and these benefits.
And we don't want each of them solving the problem themselves.
So these training materials are replicable for all
states. And so it means that there's sort
of economies of scale with a free and open source solution.
Last thing I'll mention, Ian, is that this solves the problem of
lengthy and costly contracts with tech
vendors. A lot of governments can get tied up when they
have to purchase technology. There's really complicated contracts,
it locks them in for the long term. And I think what's really important when
we think about, you know, return on taxpayer investment, is
making sure that governments can purchase the technology they
need, but not be locked in. Because it's constantly evolving
and changing. So we think for those sort of
set of reasons, those are some of the problems we
And so that last point, I guess, is why this is a launch of
training materials, not a model itself. It's
not just so no one is confused here. You're
not launching a translation model. You're launching materials so that people can,
That's right. These training materials can be used,
any state can access them to create their own language assistant in
the LLM of their choice. They can use these
training materials and put them to work in CHAT GPT or
CLAWD or Gemini. And so it's getting them
most of the way there and then a state can access these
training materials and create a language assistant of their own to
Gotcha. Now, the materials themselves, how
were they gathered, curated, verified,
collected, right? We know that in AI,
the data is king, and making sure the data is
clean and accurate and legitimate, right, is a process. So
Yeah, it involved a lot of collaboration. And I'm excited. This
is one of those great examples of a public-private partnership that
centered human beings at the center. This was not a
collaboration about how do we use this really cool technology, right? But
rather, we took a human-first approach. So in collaborating with
the New Jersey Department of Labor and Google.org, we
started by bringing together bilingual staff and
policy experts to help make sense of the key information
that individuals, residents in New Jersey
would need to know to access unemployment insurance. So
what's the key information? What do they have to know? We
then, sort of as we developed the training materials,
humans were involved at every step of the way. I think what's so
exciting for me is that with the approach we've taken with these training
materials, We're leveraging Gen AI
to streamline processing the information, but we're not leveraging
it to make decisions. We're still keeping that in
the hands of the humans so that they can be doing the more complex work
and putting the technology to work to help streamline information.
I'm glad you brought up the decision-making side of things, right? Because I wanted to ask about
that. We know issues of algorithmic
discrimination are a documented problem, aside
from, you know, when the algorithm just screws up, right? Hallucinations, confabulations,
etc. These things are not necessarily completely reliable. So
I wonder, you know, in terms of disseminating this open
source kind of framework, right, to what degree does
a sort of training go with that to make sure that whoever's using it knows,
you know, verify on these points? Or is there some sort of
trust and transparency explainability layer that kind of is
designed to go hand in hand with this?
That's a great question. And I think I can have Brian and Kate follow up
with some more of the details and the technical information. But
what I'll share is, I think that for us, what we've been wanting to do is
to ensure that there is a way that we're responsibly
leveraging the technology, but not at
the risk of the determinations and decisions
that are being made. And so what's key here is leveraging
information that's publicly available, using that to
develop the training materials, and then again, making it
available for other states to access and customize on
their end so that it works in their particular context. It
works for them and the problem they're trying to solve. It's
something I just appreciate. We're not trying to solve all the problems at once.
But instead, we're doing some of the legwork to then empower folks
in governments across the country to sort of take the
risk factors, right? And one is
a little messy. We haven't seen details on
how it's playing out yet, but it's something that people kind of associate with similar
pushes, right? Is will a chatbot
take my job type of thing? And in this environment, right?
Is that to a degree what's going on or is it quite simply there
Yeah. You know, I think there is there is a talent
issue at play here in terms of the talent we have in government having
enough bilingual staff to be able to engage
and translate the all of
the government legalese that we have underpinning
programs and services. Ultimately, what
we're trying to do, though, is to think about
how to, I guess, how to essentially give government workers
a tool like a calculator. We don't
see these training materials as replacing roles or getting
rid of jobs. Rather, we think of these as
being a key tool that government workers can use,
process the information, get it in a way that is easily
understandable for the residents you're trying to serve, so
that then those human beings can focus on the more complex tasks
that they have at hand, like making determination decisions, and
figuring out the right next step, both on the government side
and the resident side. So we don't think of this as being
one of those situations where a job would be replaced. Rather, we
see it as giving tools to government workers to better
The other point, too, in talking about the risks, right, when we're
talking about using off-the-shelf models, is
there a risk of data privacy security
issues by talking about
potentially sensitive topics that get parsed through
I love that you're asking about it because it's something that's really top of mind at
U.S. Digital Response. We sort of put
forth an approach in all of our work on generative AI that
centers responsibility and ethical use of the technology. These
training materials that we're releasing today do not
touch PII, and rather what we're saying is we're
taking the publicly available data and we're making
better use of that and translating it, making
it more accessible. But it's entirely staying
on the government side in any state that chooses to use
these materials, as New Jersey has done, to bring
in any, you know, any private data. So
we really wanted to ensure that there were no risks in
terms of privacy or security that were
a part of this approach. And I think we've struck that balance quite
And the last point I want to leave off on, right, you mentioned that New Jersey is already
using this. Yeah. I'm wondering what you can tell me
about what that's looked like, responses so
far as New Jersey. I am a
native New Jerseyan myself, right? How that
integration is going, what that kind of
In New Jersey, what they found is that by leveraging these
training materials, they have tripled their translation speed,
but not at the cost of quality. So we did evaluation,
and the translations were produced quicker, but
are nearly on par with the level of quality that
expert human translators would produce. I think that's
really exciting. I think that that means that New Jersey residents
who need these services so critically are going
to get what they need, get it faster. And then
on the government side, workers can be moving things along on
their end, knowing that there's a high level of quality in
terms of the information being given to residents. So, I
think you're going to speak with our partner Jill
at the State of New Jersey, and she'll be able to share more. I think
that impact, that result in terms
of the quicker translation speed
is really exciting, and what we want to see, not at
Of course. Well, thanks so much for joining me, Tita. Thank
Thank you. And Ian, can I just say one thing
that I'm so excited about
this for sort of, if I can, just at a high level. The
training materials we're putting out, they're fast, again. Translation
speeds in New Jersey have tripled. They're free. Governments
can use these resources without entering into lengthy or
complex contracts with vendors. and
doing so reduces risk and reduces the
use of taxpayer dollars, and it benefits everyone. So
constituents are getting better results, government workers
are able to focus on complex tasks, and taxpayers know
that their dollars aren't going to pay for some overly
complex tool. I think it's just a really exciting