Subscribe
Share
Share
Embed
The voice of the American Consortium for Equity in Education at ace-ed.org | Host Larry Jacobs facilitates rich discussions with innovative educators, thought leaders, authors and the leaders within the education industry to promote equity, access and opportunity for every student in every school.
I'm David Cicero, and this
is Education Talk Radio.
Most teachers have experienced
some version of this.
Someone comes into the classroom, watches
you teach, takes notes, and eventually
gives you feedback about what they saw.
Now, imagine if the observer
didn't have to be a person.
What if AI could watch and
listen to a classroom, recognize
patterns and instruction, and
give a teacher evidence about what
actually happened in that lesson?
That's what researchers at the University
of Virginia and the University at Albany
are exploring through the Artificial
Intelligence for Advancing Instruction
at Scale project, or AI2S, with the
support from the Gates Foundation and
technology developed with Edthena.
Peter Youngs is a professor at
the University of Virginia and the
project's principal investigator.
Peter, welcome to the show.
Thank you, David.
I'm looking forward to
talking with you today.
Me too.
Me too.
I, I, I kind of want to start
from the beginning here.
What was missing in the way
schools currently observe and
support classroom instruction
that made you want to build this?
When we think about a pre-service teacher
candidate or a first-year teacher, um,
who is working full-time, uh, responsible
for, um, their classroom for, for the
first, first time in their careers, or
even when we're thinking of someone with
five or 10 years of experience, we know
that one of the most important factors
in helping them continuing to advance and
strengthen their teaching is being able to
get regular feedback on their instruction.
And when we think about it from a
standpoint of a university-based teacher
education program, an alternate route
certification program, but also a
school or a school district, um, teacher
educators, principals, instructional
coaches are well aware of the value
of providing feedback to teachers.
At the same time, it can
be resource-intensive.
And we started our work several years
ago with the question: Could machine
learning models be a way of analyzing
instruction in videos and giving feedback
to pre-service teaching candidates,
first-year teachers, as well as
experienced teachers, um, that might be
available to them more frequently, might
be provided to them, uh, soon after they
have taught a given lesson, so kind of
rapid response, relatively quick feedback.
But also, could a system be set up that
involved AI machine learning models,
but that put teaching candidates,
first-year teachers, experienced
teachers in sort of an active role of
analyzing their teaching, um, looking
for ways to strengthen it, and engaging
with others, um, in that process.
Um, so we're really trying to address this
question of how can feedback be provided,
um, to teachers, um, that is happening
more frequently, that's happening rapidly,
but also really giving them an opportunity
to sort of direct the process of analyzing
and learning from their teaching.
Why did you think AI was the,
you know, an option or slash the
right way to tackle that problem?
I think we got the, you know,
immediate feedback, and obviously
it's not, you know, resource heavy,
but you mentioned the active role,
more active role for a teacher.
That kind of piqued my interest there.
Yeah.
I'll take a moment and just describe
the AI-based coaching process that our
research team has worked on with Edthena
over the last two or three years, and
that's now available to school districts
and to teacher education programs.
So the way this works is a
teacher would record a video of
their mathematics instruction,
might be 30 to 45 minutes long.
They upload the video to our system,
and within about 20 to 30 minutes,
they're able to sort of lead a
process of looking at information
related to one of four aspects or
dimensions of their math teaching.
So they might choose, for example, within
that lesson to focus on the dimension
of promoting mathematical language.
The first thing that happens is the
machine learning model generates some
descriptive information related to
promoting mathematical language across the
entire 30 or 40, 45 minutes of the lesson.
And we have a virtual agent that
shares some prompts, some questions
to begin to have the teacher analyze
and reflect across the entire lesson.
The heart of this process then comes next.
The teacher's actually able to see
the video that they've uploaded.
They're able to see metrics
associated with the aspect of math
teaching that they chose, and those
metrics are sort of arrayed across
the entire 30, 40, 45 minutes of
the lesson, so they would knowâ¦
Hi, Adam.
Hello, sir.
Sorry.
Welcome, Adam.
Welcome Um, really
quickly- Are we audio only?
We are audio only, yes.
Uh, Adam Geller is the founder and CEO
of Edthena and co-investigator on AI2S.
Welcome, Adam.
Thanks for joining.
Yes.
Sorry that I'm joining late.
No, no, no.
Uh, it's perfectly all right.
Adam, I'm just describing our, um,
AI-based coaching model, our prototype.
Um, and Adam, you should just feel
free to, to add as we go along.
So David, um, to describe sort of the
heart of the AI-based coaching process,
after the teacher has been able to
see descriptive information across the
entire lesson, they next are able to see
the video that they've uploaded to our
system, and they have access to metrics
associated with the dimen- dimension
of math instruction that they chose.
And that data, that information,
is arrayed across the 30 or
45 minutes of the lesson.
They have the opportunity
to do some timestamping.
They can make comments about strengths or
positive aspects of their teaching, but
they can also make comments about areas
where they have further questions, areas
where they may wanna come back and look at
refining or changing what they were doing.
And a key part of this is the virtual
agent sort of posing questions,
prompting them to continue to examine
and reflect on, um, the lesson in terms
of the math dimension they've chose.
So technically, they could come back and
go through additional coaching cycles
related to the same lesson by choosing,
um, some of the other, uh, dimensions of
math instruction, um, that we focus on.
And then after they have gone through
this process of sort of seeing the
video and seeing metrics associated
with one of the four math dimensions,
the process sort of concludes with an
opportunity to reflect back and, and
analyze the, the entire coaching cycle.
Um, and the virtual agent poses
some questions there as well.
Um, sometimes a teacher or a
pre-service teacher will go
through this process independently.
They have the option of going through this
with a peer, going through this with an
instructional coach or a teacher educator.
It is set up so that they are kind
of actively leading the process and
choosing whether to engage with others.
So you, you mentioned a
coaching cycle, right?
So it's not just I upload, um,
I get feedback, and next time I
upload there's, there's no context.
Just correct me if I'm, um, or, uh,
tell me if I'm onto something here.
So it sounds like the, the, the AI
sort of becomes an observer/assistant
throughout a specific cycle.
We've, we've chosen a dimension, and
we're going to look and see how we've
developed the lesson, um, towards
teaching well related to that dimension.
Does that sound right?
I think that's an accurate way
of, of describing it, absolutely.
Adam, do you wanna add anything to that?
Yeah, I mean, I think that this
question, we have to remember it's
being asked today, uh, in 2026.
Um, and the research experience,
uh, was built, um, you know,
about 18 months ago now, right?
So, uh, the state of the art
and, and what we expect out of
AI systems has certainly changed.
So this idea of the coach keeping
context over time from lesson to lesson
is not something that's, uh, possible
within the, the research experience.
However, uh, as you asked, David, it
is possible for the coach to kind of
keep the context within the lesson,
um, as the discussion is happening, as
the, the kind of analysis is going on,
so that, you know, if you, um, have
noticed something in the 20-minute
mark, but then you, you start kind of
analyzing something that happens at the
four-minute mark, all of that context and
awareness is present for the teacher as
they're working with the virtual coach.
Uh, but certainly, you know, I think, uh,
as we've worked at Athena, uh, to adapt,
um, some of what we've learned from the
AI2S project, we've been able to implement
more of that kind of across lesson
context that, that you're asking about.
And, and excited to tell you
more about that today as well.
Yeah.
And, and Peter just, uh, chatted in,
um, a- about what the four dimensions
actually are: using students'
contributions, promoting mathematical
language, sharing rich explanations, and
extending patterns into generalizations.
Why those, those four in particular?
So these four are associated with a
classroom observation instrument known as
the Mathematical Quality of Instruction.
Um, this was first developed by
Heather Hill and colleagues, and
this, um, observation tool is widely
used, um, for research purposes.
The, uh, individuals who developed it
were able to show that when teachers
are rated highly on this MQI tool
with their teaching, um, that there
are significant associations between
having high MQI scores and the
teachers being able to promote student
achievement gains in mathematics.
Um, so we, as a research team, felt
that the classroom observation tool
itself had a lot of value for the way
we wanted to sort of examine and promote
changes in mathematics instruction.
And David, one thing we've been
talking about is as the teacher goes
through a coaching cycle, they are
choosing to focus on one of these four
practices for a given coaching cycle.
We were concerned that in earlier
efforts by the EVA team to develop a
teacher dashboard, we were presenting,
uh, teachers with too much data
from the machine learning models.
We were sort of overlevel-
overwhelming them with a wall of data.
And one thing we've really tried
to do here is have them choose a
particular prac- practice, and then
as they go through the coaching
cycle, they're actually exposed to
a somewhat limited amount of data
related to that particular practice.
And it allows them and anyone they're
working with to sort of really focus, to
hone in on certain aspects of the lesson.
And then, as we mentioned earlier, they
have the option of coming back and going
through the coaching cycle a second
time and focusing on different practice.
So we felt that the MQI was a really
helpful frame of reference for the math
practices we wanna focus on, and then
we sort of made a decision for a given
coaching cycle, can we maintain sort of
a narrow focus and allow the teacher sort
of one at a time to look at, um, some
metrics associated with that practice
instead of seeing a wall of data or
being overwhelmed by too much data?
Hmm.
So, so I'm curious as to what
is actually being, uh, observed.
Adam, I'm not sure if you, if you, um,
would like to answer this question.
Um, what's that-- what is, what
is it actually looking for?
It's obviously listening to audio.
Is it also looking at, um, you know,
hands raising, things like that?
What is actually being observed?
Yeah, that's a, a really great question.
So one of the important things about this
study is that it was, uh, the first, as
far as we're aware, where we have, um,
essentially developed in partnership
with the computer science experts that
are part of the research team, what's
called a multimodal neural network.
Uh, and so that multimodal, uh,
aspect does in fact mean that the
audio and the video are analyzed,
and really importantly, they are
analyzed alongside each other.
Uh, so to kind of, um, push one
more detail deeper here, I just
wanna say that, uh, the system uses
transcripts alongside the video.
Uh, so that means that in this
current study, we're not able
to detect things like tone.
Um, but you know, we are able to see
w- you know, what tools the teachers
are using for the instruction, or
what is the type of instruction.
Is it small group?
Is it whole group?
Is it individual?
And align those data points alongside,
uh, an analysis of what's happening
within the speech of the lesson.
So that could be things like,
um, who is speaking, right?
The, the balance of student and
teacher talk time is, uh, something
that is considered valuable, uh,
to look at, um, among educators.
Uh, but it also digs into things like the
types of questions you're asking or, uh,
what type of thinking is being probed.
And importantly, um, sometimes when the
speech is not necessarily driving the
conversation forward or the lesson forward
and might be considered, um, task-oriented
or, um, related to student behaviors, um,
being able to separate that out so that,
um, again, in this context, when teachers
are looking at the content of the math
instruction itself, they can really focus
in and, uh, find the moments of interest.
So, you know, I think what happens is
the The computer system, the AI system
is, you know, coming up with hundreds,
potentially thousands of data points
for the teacher about their lesson.
And as Peter was explaining,
the teacher is not presented
with all of those data points.
It would be just very overwhelming.
You know, when I say thousands of
data points, uh, that'd be challenging
for anyone to analyze, let alone a
teacher who might be pressed for time.
So if a teacher, uh, has, you know,
used-- uh, picked using student
contributions, then we would present
a subset of the data, so let's say,
uh, the balance of student and teacher
talk time, visualized in a kind
of horizontal timeline with bars.
So in that way, it's really easy,
uh, to kind of see where in the video
was I speaking, or more importantly,
where was I speaking too long without
a break for students to speak?
And so that's the type of insight that
we're able to unlock for teachers when
we present them with data about their
video, so that then they can go and
watch the video and work on interpreting
it alongside the virtual coach.
So what would you, what would
you say that, I wanna say what
the goal is, but I, I think that
that's, that's oversimplification
of what my question really is.
But, you know, if I, if I
think about something like
Japanese lesson study, right?
We're observing, we are, um, collecting
information, we're continuing to refine
a single lesson or set of lessons.
Would you say that the, um, the goal
would be to say, "Hey, when we're teaching
this, um, standard," let's say, w-we've
got a record of what a good lesson or
we're hitting You know, three out of
four are all four of these dimensions.
Would you say that that's
where this eventually leads to?
Um, yeah, I, I understand the
question here, and, um, you know,
I think I want to maybe call out
for the listeners what this is not.
Um, I don't think it's exactly what
you're asking for, but I wanna say what
this technology and what at least our
intentions as a research team and as at
Dena as an organization, what it is not.
This is not a system that is
designed to evaluate teachers, right?
So, um, all the kind of data are presented
in a way that they're really helping
teachers understand what actually happened
within their teaching and learning,
uh, you know, within the classroom.
And I think this is really rooted
in kind of the, uh, more than 40
years now at this point, kind of, um,
emphasis within education that one
of the best ways to help a teacher
improve their practice is to have them
discuss their actual implementation
within their classroom, right?
And so that's different than, say, uh, a
pro- a professional development experience
that's focused mostly on, you know, trial
implementations within a workshop context.
So rooted in the kind of the academic
literature of what we know about coaching
and what we know leads to changes,
uh, both among teachers and, uh, among
adults generally, uh, you can think of
the process as really about helping the
teacher identify what is really happening
so that they can decide to make a change.
Because at the end of the day, um,
if making changes was as easy as
a computer telling us to, uh, you
know, go lift more weights at the
gym or take more steps, then we
would all be Olympic athletes, right?
So this is not a Fitbit for
teachers about their teaching.
This is really meant to be that kind of-
uh, mirror, if you will, that teachers
can look into, uh, and to see themselves.
And sometimes they might be seeing
themselves for the first time.
So why math?
Why start with math?
Um, is there something that makes math
instruction especially interesting
for this kind of technology?
Um, I can go first, and
Adam, feel free to jump in.
Um, one thing we know in terms
of the United States context is
that many, but not all elementary
teachers teach multiple subjects.
Uh, so they teach reading and language
arts, they teach math, and they
teach social studies and science.
And in the US, um, research has
shown that fairly high percentages
of elementary teaching candidates,
beginning elementary teachers, have less
teaching self-efficacy related to math.
Uh, they just feel less
confident about teaching it.
They may feel they have less math content
knowledge, um, compared in particular to
self-efficacy and knowledge associated
with teaching reading language arts.
Um, so it's a challenging area
for elementary, many elementary
pre-service candidates and many,
um, beginning elementary teachers.
Um, we think it's just a very important
part of, um, elementary school education.
We felt that this was an area, um,
where we could really look to, um,
make a difference in sort of the, uh,
instructional development of early career
teachers as well as experienced teachers.
Uh, yeah, I mean, I'll add, uh,
maybe another, uh, practical answer
to that, David, which is that, um,
if we look historically to the types
of research projects that have, uh,
happened, uh, there's been strong
data collection within math classrooms
and, uh, also literacy classrooms.
And so I think having that set of data
that is appropriately permissioned for
research, right, so the data is protected,
um, so that all these transcripts can
be analyzed, that's an important input
into the process of training a custom,
uh, model to be able to accurately and
reliably analyze classroom teaching.
Um, so, you know, to the extent there
are any, uh, funders out there who are
thinking, "Why not do a literacy model if
you have, uh, literacy data at the ready?"
Well, uh, w- we would welcome your,
your call or your email, uh, because,
uh, we do have the data and, and it's
really a question of the time and
resources to prepare that data, uh,
for the safe training of a new AI model
that then can be tested with teachers.
Um, I wanna add, you know, w- kind
of Peter was talking about, uh, you
know, kind of one of the reasons
that math is important is that, you
know, teachers perceive themselves,
uh, not to be good at teaching math.
And, um, you know, that can be for a
variety of different reasons, of course.
Uh, but what's interesting, a- and
I know we'll get more into detail
about what we found, uh, during our
field research, but that question of
perception of self-efficacy was, in
fact, one of the things that we wanted
to find out as we went and gave the
prototype experience to teachers, right?
Because it's one thing to say that
we can have an AI system that can
analyze your teaching, and you can
work privately with a virtual coach.
Um, you know, this is the Uh, it,
it, again, it feels maybe commonplace
based on what we read in the, the
news today, but this is still, you
know, on the edge science that,
that we are, are working with here.
Uh, but the really important question
is: What do the teachers think of it?
What happens when they use it?
Uh, do they like using it?
And, and how do their perceptions of
their- themselves change over time?
Um, and so I think that that was core
to our, uh, focus as we went into the
field and started working with teachers.
Yeah.
What are you finding?
What kind of feedback are you
getting and, and how is it, in
impacting practice, self-belief?
What, what are you, what are
you finding in that research?
Well, I'm going to let Peter answer
that- Okay ⦠because I sound
like the cheerleader when I report
what happened from the field.
So I wanna qualify everything
he's about to say, which is
that all of these- Sure, sure
data points from the field were
collected by the researchers at the
universities, not by Edthena team members.
Uh, so this is truly, uh,
research-grade data that we
are collecting from the field.
So David, thanks for the
question, and we'll answer it
in a couple different ways.
Um, the first thing we'll say is we
were able to collect data from about
20 teachers in two different states,
um, during this past school year.
Uh, one of the things we wanted to do as
each of them went through a total of four
coaching cycles, um, with our prototype,
was to interview them at multiple times
during the year and really learn some
things about their perceptions of the
AI coaching experience Um, we did found
pretty consistently across the 20 or so
teachers that they valued the sort of
rapid, um, feedback that they received.
They valued the fact that they could go
through multiple coaching cycles if they
chose to, related to the same lesson.
They felt that the fact that in
some ways the information they
were receiving was objective, um,
was, was really helpful to them.
One of the things they talked to us
about is that many of them also valued
having an instructional coach, a
principal, or an assistant principal who
knew their classroom context, who knew
their students, who kind of knew them.
And in interviews with them, they
would talk about potential ways as
our project moves forward, that we may
look at trying to combine or integrate
our AI-based coaching system with
sort of the efforts of human coaches.
Um, but a lot of them told us that in
terms of our prototype, the AI-based
coaching system, they trusted it.
They felt it was useful.
Um, they definitely found, like,
utility, um, in the experience.
And then we've started a process of
trying to understand whether going
through these multiple coaching
cycles might be associated with a
number of changes in sort of teacher
attributes or teacher outcomes.
And we'll preface this by saying that
our sample, uh, this past year was
small, so this is sort of initial work.
Uh, we would eventually want to look at
some of these same research questions
with closer to 100, 125, 150 teachers.
But across 20 teachers, we are finding
as they go through the multiple coaching
cycles, as Adam referenced, changes in
a positive way, improvements in their
mathematics teaching self-efficacy.
Uh, we found that very significant.
We're also examining the extent
to which their ability to sort of
analyze math instruction, uh, we refer
to that as their noticing skills,
um, whether we see changes there.
With this small sample, it's been
a bit indeterminate so far, but we
wanna keep looking at that question
with a larger number of teachers.
Um, third, we're interested in their
sort of beliefs or visions, their
perspectives on what constitutes
high-quality math instruction.
There are valid, reliable measures of
this construct, and we're interested in
whether going through coaching cycles is
associated with changes in some of their
beliefs about high-quality math teaching.
Um, the results there are
indeterminate at this point.
And then we're actually using the
classroom observation instrument that
we talked with you about earlier, the
mathematical quality of instruction tool,
or the MQI to actually analyze, um, videos
of their teaching before they started the
coaching cycles, and then, uh, two videos
of their teaching after they completed
the fourth and final coaching cycle.
That analysis is still in progress,
and again, it's a fairly small
sample of about 20 teachers.
Um, but as we continue the work,
we'll be interested to see, you know,
does participating in this AI-based
coaching lead you to actually teach
math better, um, according to the MQI?
Actually, I, I really love this,
and the timing is a bit uncanny.
I think last, my last publication on the
last show, I believe, um, I interviewed
the author of Compassionate Math, Dr.
Ali, and we talked about teachers'
perception of, uh, what good math
instruction looks like, and in some
cases, uh, not feeling super comfortable
sometimes with the math itself, and
also not, not feeling comfortable
expressing that, right, to their, uh,
to their coaches and to their, and
to their school leaders, um, as well.
And so I, I, I find it particularly
interesting one thing that, that
you found, if I, if I, if I wrote
this down right, was that, hey,
um, how can we use this information
in collaboration with other people
here, with other coaches here?
Um, and you know, in, in the back of
my mind I'm wondering, "Hmm, is this,
is this touching on something we, we
learned about in, in Compassionate Math?"
How do we share this in a safe way, right?
Um, I'm, I'm refining my practice.
Um, I have these gaps.
Let's, let's create and facilitate this
environment where we can all rise above,
create good math instruction, um, and
serve the needs of our, our students.
So, um, I, I just-- That, that,
that's kind of flagged for me
as you were talking, Peter.
Now, now Adam, we've-- You know, you
learned a lot in your field research,
so how is that affecting the, the AI
Coach, the solution moving forward?
Yeah, that's a, a great question.
And, um, you know, I think, uh, you
know, the AI Coach system, we launched
that originally in February of 2022,
uh, which I always like to say is
before ChatGPT, and, and certainly
have been evolving it as we go.
Um, but you know, the kind of field
research really enabled us to get that
close-up view to what happens when you
present teachers with the, the data about
a particular aspect of their teaching.
And as Peter mentioned, um, we continue to
find that, uh Uh, less is more, uh, so to
say, uh, when it comes to, uh, the data.
Not because teachers have any issues
interpreting it over time, but just
because, you know, how much you're going
to, to really engage with in one sitting,
um, is potentially limited in terms of
what you can, can make meaning out of.
So, uh, as of this fall, we actually
updated our AI coach experience,
and it now includes a visual
representation of those automated
detected aspects of the teaching.
So, um, you know, within the experience,
teachers are, uh, uploading a video.
First of all, they get a, a, now
an email five minutes later to say,
"Hey, here's, you know, a high-level
analysis from the AI system."
Uh, but when they, you know, go in for
that deep dive conversation with the
coach, uh, they can actually see the data
associated with one particular skill that
they might wanna be focusing on at a time.
So, um, what does that mean in practice?
If, for example, I were working on,
uh, eliciting student thinking, well,
one of the things I need to do for
that to occur is to ask questions.
Uh, I need to ask certain
types of questions.
I need to leave wait time after my
questions, and I need to then actually
analyze what students are saying.
Well, that's a, a complex,
multifaceted process.
And so, you know, certainly teachers
are at different starting points.
But, you know, let's say if you're,
uh, a novice teacher or a teacher who's
implementing these types of discussions in
a math classroom for the first time, you
just may not be asking enough questions.
So literally, the first data point might
be, how many questions am I asking?
Uh, and then we get into
what types of questions.
So each of those might be a
separate video and a separate,
uh, mini-analysis, if you will.
And the reason, uh, for that kind
of structure also, again, came out
of the feedback we were hearing
from teachers, which is they wanted
it to feel more flexible rather
than more, uh, heavily structured
cycles over and over and over again.
Thanks for joining us on Education Talk
Radio, a part of the B Podcast Network.
If there's a topic you'd like us to
tackle or have a guest idea, send me
a message at dciceroedutalk@gmail.com.
Thanks for listening, and
we'll see you next time.