1
01:00:00,033 --> 01:00:13,800
[MUSIC PLAYING]

2
01:00:14,400 --> 01:00:15,833
Hello, and welcome to

3
01:00:15,833 --> 01:00:17,699
another episode of Undercool,

4
01:00:17,699 --> 01:00:19,166
the materials education podcast.

5
01:00:20,133 --> 01:00:21,366
So today, Steve and I are

6
01:00:21,366 --> 01:00:22,633
hanging out in the office.

7
01:00:22,833 --> 01:00:24,066
And we're going to talk

8
01:00:24,066 --> 01:00:25,466
about the acronym that

9
01:00:25,466 --> 01:00:26,866
is no longer an acronym.

10
01:00:26,866 --> 01:00:28,533
It's just a name all by itself.

11
01:00:28,599 --> 01:00:29,733
That's right, ABET.

12
01:00:30,599 --> 01:00:31,633
Maybe once upon a time

13
01:00:31,633 --> 01:00:32,533
it stood for something.

14
01:00:32,533 --> 01:00:33,233
But we'll get into the

15
01:00:33,233 --> 01:00:34,233
history with Steve here.

16
01:00:34,833 --> 01:00:36,033
But today, we're going to talk about

17
01:00:36,033 --> 01:00:37,466
ABET, what it's good for,

18
01:00:37,466 --> 01:00:38,866
what it's useful for, what

19
01:00:38,866 --> 01:00:40,133
it might not be good for,

20
01:00:40,233 --> 01:00:42,033
and how it can inform what we

21
01:00:42,033 --> 01:00:43,433
do in our materials programs

22
01:00:43,433 --> 01:00:44,533
and help us teach better.

23
01:00:45,433 --> 01:00:46,733
So Steve, to kick things

24
01:00:46,733 --> 01:00:48,000
off, why don't you just tell us,

25
01:00:48,000 --> 01:00:49,733
what is ABET and why

26
01:00:49,733 --> 01:00:51,133
should materials departments

27
01:00:51,133 --> 01:00:51,900
care about it?

28
01:00:52,666 --> 01:00:55,099
So ABET used to stand

29
01:00:55,099 --> 01:00:55,966
for the Accreditation

30
01:00:56,433 --> 01:00:58,766
Board of Engineering and Technology.

31
01:00:59,466 --> 01:01:00,633
Now it's just ABET.

32
01:01:01,933 --> 01:01:04,133
And it's because they do

33
01:01:04,133 --> 01:01:05,300
more than just engineering

34
01:01:05,300 --> 01:01:06,066
and technology.

35
01:01:06,433 --> 01:01:08,033
They do applied stuff as well.

36
01:01:08,766 --> 01:01:10,466
But whatever, it's

37
01:01:10,466 --> 01:01:11,766
really the same organization.

38
01:01:12,766 --> 01:01:16,099
It's a federation of 30 to 40

39
01:01:16,099 --> 01:01:18,599
professional societies

40
01:01:19,233 --> 01:01:21,633
that get together and

41
01:01:21,633 --> 01:01:23,633
they make bylaws and set up

42
01:01:24,466 --> 01:01:25,433
all the rules and

43
01:01:25,433 --> 01:01:28,166
regulations for doing accreditation

44
01:01:28,633 --> 01:01:30,066
for engineering schools.

45
01:01:30,866 --> 01:01:33,233
And ultimately, it's about making sure

46
01:01:33,599 --> 01:01:34,533
that the people who are

47
01:01:34,533 --> 01:01:35,733
designing your bridges,

48
01:01:36,666 --> 01:01:40,166
your airplanes, your boats, actually

49
01:01:40,333 --> 01:01:41,466
know something about what

50
01:01:41,466 --> 01:01:42,466
they're supposed to know.

51
01:01:43,033 --> 01:01:45,266
So as a country, as a

52
01:01:45,266 --> 01:01:46,666
society, as the world,

53
01:01:48,099 --> 01:01:50,333
we can actually trust the engineers

54
01:01:50,533 --> 01:01:51,766
that our educational

55
01:01:51,766 --> 01:01:53,599
institutions are putting out.

56
01:01:53,966 --> 01:01:55,733
That's the big reason.

57
01:01:56,300 --> 01:01:56,733
And it's even

58
01:01:56,733 --> 01:01:58,833
sanctioned by the United States

59
01:01:58,833 --> 01:02:01,933
to do accreditation for all US programs,

60
01:02:02,199 --> 01:02:03,233
engineering programs.

61
01:02:04,133 --> 01:02:06,266
But ABET also performs many

62
01:02:06,266 --> 01:02:08,066
international accreditations

63
01:02:08,500 --> 01:02:09,666
at the request of

64
01:02:09,666 --> 01:02:11,133
schools in other countries

65
01:02:11,800 --> 01:02:13,733
because it's become

66
01:02:13,733 --> 01:02:15,133
kind of the gold standard

67
01:02:15,266 --> 01:02:17,066
for engineering accreditation.

68
01:02:18,066 --> 01:02:20,366
And other countries want to have--

69
01:02:21,199 --> 01:02:22,566
I mean, it's kind of funny.

70
01:02:23,033 --> 01:02:25,333
They want to say, we have the same

71
01:02:25,333 --> 01:02:27,833
accreditation as MIT does.

72
01:02:28,666 --> 01:02:29,533
So for whatever

73
01:02:29,533 --> 01:02:31,000
that's good for, but we'll

74
01:02:31,000 --> 01:02:33,566
get into that later on.

75
01:02:34,266 --> 01:02:37,266
So why do MSE programs care about this?

76
01:02:38,433 --> 01:02:43,500
Well, they should care because we usually

77
01:02:43,633 --> 01:02:46,099
have a few students who want

78
01:02:46,099 --> 01:02:48,633
to go into careers where they

79
01:02:48,633 --> 01:02:51,033
must become professional engineers.

80
01:02:51,833 --> 01:02:52,900
And as you know, Tim,

81
01:02:53,533 --> 01:02:56,666
the TMS has a committee

82
01:02:57,366 --> 01:02:58,666
that sets the

83
01:02:58,666 --> 01:03:00,233
professional engineering degree

84
01:03:00,366 --> 01:03:02,033
for materials.

85
01:03:02,266 --> 01:03:04,099
It's really metallurgy mostly.

86
01:03:05,099 --> 01:03:06,599
And although I don't think

87
01:03:06,599 --> 01:03:09,033
more than 20, 30 people a year

88
01:03:09,066 --> 01:03:10,866
take that test, it's

89
01:03:10,866 --> 01:03:13,199
there in case they need to be.

90
01:03:13,500 --> 01:03:14,966
Where would they need to do this?

91
01:03:14,966 --> 01:03:17,900
It would be companies like Exponent.

92
01:03:18,233 --> 01:03:22,366
These are consulting companies that often

93
01:03:22,366 --> 01:03:23,766
do a lot of litigation.

94
01:03:24,866 --> 01:03:26,366
When something goes wrong, in some

95
01:03:26,366 --> 01:03:27,800
company or some person,

96
01:03:28,000 --> 01:03:29,300
who's another company or

97
01:03:29,300 --> 01:03:31,166
another person, goes to court.

98
01:03:31,966 --> 01:03:33,833
And in court, there are all these rules

99
01:03:33,966 --> 01:03:35,966
about who's allowed to

100
01:03:35,966 --> 01:03:38,133
testify as an expert witness.

101
01:03:39,000 --> 01:03:40,566
Anything involving the government,

102
01:03:41,233 --> 01:03:44,666
only people with accredited--

103
01:03:45,966 --> 01:03:47,000
with professional

104
01:03:47,000 --> 01:03:49,166
engineering are going to count.

105
01:03:49,466 --> 01:03:51,300
But it's also to impress the jury.

106
01:03:51,900 --> 01:03:53,266
So the jury is going to want to know

107
01:03:53,366 --> 01:03:55,333
that it's a professional engineer.

108
01:03:55,966 --> 01:03:57,300
And guess what society

109
01:03:57,300 --> 01:03:59,066
is also a member of ABET,

110
01:03:59,666 --> 01:04:02,166
the Society of Professional Engineers.

111
01:04:02,699 --> 01:04:06,133
And a long time ago, they insisted that--

112
01:04:06,366 --> 01:04:07,933
well, hey, they make their own rules.

113
01:04:08,400 --> 01:04:09,366
If you want to be a

114
01:04:09,366 --> 01:04:10,566
professional engineer,

115
01:04:10,933 --> 01:04:13,033
you must have graduated from an

116
01:04:13,033 --> 01:04:15,300
accredited engineering program

117
01:04:15,966 --> 01:04:17,699
or have 11 years of

118
01:04:17,699 --> 01:04:18,900
equivalent experience.

119
01:04:19,766 --> 01:04:21,166
So do we really want our students

120
01:04:21,166 --> 01:04:22,366
to spend 11 years

121
01:04:22,366 --> 01:04:23,766
before they can start a job?

122
01:04:23,766 --> 01:04:24,733
They might want to start

123
01:04:24,733 --> 01:04:27,233
right after in our department.

124
01:04:28,533 --> 01:04:30,233
Maybe 1% or 2% of our

125
01:04:30,233 --> 01:04:31,866
students every year want to do it.

126
01:04:32,900 --> 01:04:33,833
So we do it for them.

127
01:04:34,733 --> 01:04:36,766
And it makes parents and

128
01:04:36,766 --> 01:04:38,933
alums feel really good.

129
01:04:39,666 --> 01:04:41,566
So those are the two main reasons

130
01:04:42,099 --> 01:04:44,333
why we partake in accreditation.

131
01:04:45,366 --> 01:04:47,199
OK, yeah, it definitely makes sense

132
01:04:47,266 --> 01:04:49,166
about wanting to give

133
01:04:49,166 --> 01:04:50,333
students the opportunity

134
01:04:50,599 --> 01:04:53,533
to earn certifications, to have access

135
01:04:53,666 --> 01:04:54,766
to different career

136
01:04:54,766 --> 01:04:56,066
paths, different opportunities

137
01:04:56,199 --> 01:04:57,599
down the road that

138
01:04:57,599 --> 01:05:00,533
ultimately rest on the program

139
01:05:00,533 --> 01:05:01,233
that they graduated

140
01:05:01,233 --> 01:05:03,133
from, having demonstrated

141
01:05:03,133 --> 01:05:04,233
a certain minimum

142
01:05:04,233 --> 01:05:06,866
quality of the education that

143
01:05:07,333 --> 01:05:08,266
is being provided there.

144
01:05:08,766 --> 01:05:10,033
So that seems reasonable.

145
01:05:10,800 --> 01:05:11,866
Now, my understanding

146
01:05:11,866 --> 01:05:13,400
is that to get certified

147
01:05:13,766 --> 01:05:15,233
as an ABET accredited

148
01:05:15,233 --> 01:05:17,466
institution, part of the process

149
01:05:17,466 --> 01:05:20,333
is having ABET, essentially,

150
01:05:20,333 --> 01:05:22,033
inspectors come to the program

151
01:05:22,533 --> 01:05:24,333
and do an on-site physical visit.

152
01:05:24,866 --> 01:05:26,566
What do those visits look like?

153
01:05:27,033 --> 01:05:29,766
And what's the hardest

154
01:05:29,766 --> 01:05:31,500
part of preparing for a visit?

155
01:05:31,500 --> 01:05:32,766
What do you have to do to get ready?

156
01:05:32,966 --> 01:05:34,066
That might not be obvious.

157
01:05:35,133 --> 01:05:36,033
So that's a great question.

158
01:05:36,533 --> 01:05:38,666
And first of all, it's

159
01:05:38,666 --> 01:05:40,533
very important to realize ABET

160
01:05:40,666 --> 01:05:41,633
doesn't accredit

161
01:05:41,633 --> 01:05:44,033
institutions or departments.

162
01:05:44,566 --> 01:05:46,933
They only accredit programs.

163
01:05:47,699 --> 01:05:48,966
So they look at the

164
01:05:48,966 --> 01:05:50,599
degrees that are being granted.

165
01:05:51,266 --> 01:05:53,133
And in our case, we have

166
01:05:53,133 --> 01:05:55,500
one material science degree

167
01:05:56,000 --> 01:05:58,133
that says a bachelor's in material

168
01:05:58,133 --> 01:05:59,166
science and engineering.

169
01:06:00,099 --> 01:06:00,933
That's a program.

170
01:06:01,433 --> 01:06:03,333
And that is what is accredited.

171
01:06:03,966 --> 01:06:05,566
And that's all ABET accredits.

172
01:06:06,433 --> 01:06:09,166
So what you need to do to be accredited,

173
01:06:09,733 --> 01:06:11,433
it's a little different the first time,

174
01:06:11,633 --> 01:06:12,066
but we've been

175
01:06:12,066 --> 01:06:13,733
accredited for so many years.

176
01:06:14,400 --> 01:06:15,566
I wasn't even here when

177
01:06:15,566 --> 01:06:16,599
we first had accredited.

178
01:06:17,199 --> 01:06:18,766
I've been involved in some, but I

179
01:06:18,766 --> 01:06:22,133
think you need to have graduated at least

180
01:06:22,133 --> 01:06:23,566
one student before you

181
01:06:23,566 --> 01:06:24,333
can be accredited.

182
01:06:24,866 --> 01:06:26,233
And when you go for your first

183
01:06:26,233 --> 01:06:27,333
accreditation visit,

184
01:06:27,866 --> 01:06:29,566
you're visited by two

185
01:06:29,566 --> 01:06:30,833
evaluators, not one.

186
01:06:31,599 --> 01:06:32,833
So that's a little different.

187
01:06:32,833 --> 01:06:35,133
But for most of us in most programs,

188
01:06:35,433 --> 01:06:36,633
it's about 125

189
01:06:36,633 --> 01:06:38,699
accredited materials programs

190
01:06:38,933 --> 01:06:39,599
in the country.

191
01:06:40,666 --> 01:06:42,733
And the vast majority of

192
01:06:42,733 --> 01:06:44,233
those have been accredited before.

193
01:06:46,133 --> 01:06:47,666
So what you need to do, your

194
01:06:47,666 --> 01:06:50,566
institution will call up ABET

195
01:06:50,566 --> 01:06:51,433
and say, hey, we're

196
01:06:51,433 --> 01:06:52,500
ready to be accredited.

197
01:06:52,500 --> 01:06:53,833
We're going to start the process.

198
01:06:53,833 --> 01:06:54,266
Here are all the

199
01:06:54,266 --> 01:06:55,866
programs we'd like to do,

200
01:06:56,166 --> 01:06:58,000
because ABET does a

201
01:06:58,000 --> 01:06:59,566
visit for all the programs

202
01:06:59,566 --> 01:07:01,500
at an institution at the same time.

203
01:07:03,199 --> 01:07:04,166
The first thing you need

204
01:07:04,166 --> 01:07:05,933
to do once all that-- that's

205
01:07:05,933 --> 01:07:06,866
all Dean stuff, so

206
01:07:06,866 --> 01:07:07,733
don't worry about that.

207
01:07:08,900 --> 01:07:11,566
First thing a program needs to do

208
01:07:11,900 --> 01:07:14,366
is start six years before that moment,

209
01:07:15,233 --> 01:07:16,766
because they really need to

210
01:07:16,766 --> 01:07:19,333
be preparing for this right

211
01:07:19,333 --> 01:07:21,599
from the day after their last visit.

212
01:07:21,599 --> 01:07:22,800
And I'll get back to that.

213
01:07:23,533 --> 01:07:26,900
So what you need to do is create what's

214
01:07:26,900 --> 01:07:27,866
called a self-study.

215
01:07:28,233 --> 01:07:30,466
And ABET gives you a template to fill in.

216
01:07:31,233 --> 01:07:33,566
ABET is all focused

217
01:07:33,566 --> 01:07:34,900
around their criteria.

218
01:07:35,633 --> 01:07:37,266
They have eight criteria.

219
01:07:38,266 --> 01:07:39,866
I'm not going to go through all of them.

220
01:07:39,866 --> 01:07:40,633
The most important

221
01:07:40,633 --> 01:07:41,966
ones are the first few.

222
01:07:42,766 --> 01:07:44,066
So criteria one is

223
01:07:44,066 --> 01:07:45,333
all about the students.

224
01:07:46,066 --> 01:07:47,099
Where the students come

225
01:07:47,099 --> 01:07:48,800
from, how are they advised,

226
01:07:49,800 --> 01:07:53,833
what kind of tracking do you do, how do

227
01:07:53,833 --> 01:07:55,133
you handle mental health

228
01:07:55,233 --> 01:07:56,400
issues, how do you do

229
01:07:56,400 --> 01:07:57,800
admissions, all that stuff.

230
01:07:58,566 --> 01:08:00,733
And the second

231
01:08:00,733 --> 01:08:03,766
criteria is all about program

232
01:08:04,000 --> 01:08:05,599
educational objectives.

233
01:08:06,233 --> 01:08:07,666
And these are those statements that

234
01:08:07,666 --> 01:08:10,900
talk about what a graduate

235
01:08:10,900 --> 01:08:12,566
should look like a few years

236
01:08:12,733 --> 01:08:13,866
after they graduate.

237
01:08:14,099 --> 01:08:15,733
What's our aspiration for that?

238
01:08:16,133 --> 01:08:19,100
And those don't even need to be measured,

239
01:08:19,633 --> 01:08:21,733
because some of our aspirations are we

240
01:08:21,733 --> 01:08:23,600
want them to be creative.

241
01:08:23,966 --> 01:08:25,533
We want them to solve

242
01:08:25,533 --> 01:08:26,666
the world's problems.

243
01:08:27,233 --> 01:08:27,733
And how do you

244
01:08:27,733 --> 01:08:29,166
actually measure those things?

245
01:08:29,633 --> 01:08:30,666
But it's important to

246
01:08:30,666 --> 01:08:32,033
have high aspirations,

247
01:08:32,366 --> 01:08:32,966
because that kind of

248
01:08:32,966 --> 01:08:34,233
drives the whole thing.

249
01:08:34,733 --> 01:08:37,500
I actually believe that the objectives

250
01:08:37,500 --> 01:08:40,366
are the most valuable part of the ABET

251
01:08:40,366 --> 01:08:41,766
process for a program.

252
01:08:42,433 --> 01:08:43,866
It's also one of the easiest

253
01:08:43,866 --> 01:08:46,433
parts to be compliant with,

254
01:08:46,633 --> 01:08:49,300
because all you need to do is consult

255
01:08:49,300 --> 01:08:50,733
with your constituencies

256
01:08:51,600 --> 01:08:57,033
at least once every six years, and ask

257
01:08:57,166 --> 01:09:00,233
them, are our objectives as written,

258
01:09:00,733 --> 01:09:02,266
meeting your needs as

259
01:09:02,266 --> 01:09:03,366
a constituent group?

260
01:09:04,600 --> 01:09:05,766
Our program has three

261
01:09:05,766 --> 01:09:07,500
constituencies, the students,

262
01:09:07,833 --> 01:09:09,333
the faculty, and our alums.

263
01:09:10,300 --> 01:09:12,733
Some programs get crazy, and they say,

264
01:09:13,300 --> 01:09:15,166
their constituents are the universe.

265
01:09:15,766 --> 01:09:16,366
Well, how are you

266
01:09:16,366 --> 01:09:17,633
going to ask the universe

267
01:09:17,633 --> 01:09:18,766
what their needs are?

268
01:09:19,333 --> 01:09:20,366
You know, we just can't

269
01:09:20,366 --> 01:09:21,633
get to some of those planets.

270
01:09:22,399 --> 01:09:24,133
So that would be a very

271
01:09:24,133 --> 01:09:26,133
unwise thing for a program to do.

272
01:09:26,533 --> 01:09:27,933
It's great to just have three.

273
01:09:28,333 --> 01:09:28,699
It works.

274
01:09:29,266 --> 01:09:30,533
But it's so important

275
01:09:30,533 --> 01:09:31,966
that you meet with them

276
01:09:32,166 --> 01:09:34,066
and document that thing.

277
01:09:34,066 --> 01:09:35,399
That's actually easy to do.

278
01:09:36,033 --> 01:09:37,933
The third criteria is the simplest,

279
01:09:38,333 --> 01:09:39,800
because ABET says, here

280
01:09:39,800 --> 01:09:41,366
are the student outcomes, one

281
01:09:41,366 --> 01:09:42,033
through seven.

282
01:09:43,199 --> 01:09:44,633
And you can add more,

283
01:09:45,033 --> 01:09:46,600
but what crazy programs

284
01:09:46,600 --> 01:09:48,266
can add more and add

285
01:09:48,266 --> 01:09:49,699
more work to their plate?

286
01:09:50,233 --> 01:09:52,766
So just do what ABET asks you to do.

287
01:09:52,766 --> 01:09:53,666
You just list it.

288
01:09:53,899 --> 01:09:54,766
Those are your outcomes.

289
01:09:55,233 --> 01:09:56,100
I think there's a table.

290
01:09:56,366 --> 01:09:57,300
You show how they're

291
01:09:57,300 --> 01:09:58,666
related to your objectives.

292
01:09:59,566 --> 01:10:00,433
That's easy to do.

293
01:10:00,766 --> 01:10:01,466
So that's easy.

294
01:10:02,166 --> 01:10:03,699
It's the fourth criteria

295
01:10:03,699 --> 01:10:05,800
that usually stumps everybody.

296
01:10:06,366 --> 01:10:07,633
And that's the criteria

297
01:10:07,633 --> 01:10:09,533
for continuous improvement.

298
01:10:10,966 --> 01:10:12,866
In that-- and it's very

299
01:10:12,866 --> 01:10:14,066
short if you read the words

300
01:10:14,066 --> 01:10:14,766
the criteria.

301
01:10:15,600 --> 01:10:17,433
It's not much language.

302
01:10:18,033 --> 01:10:19,433
But it says that you have

303
01:10:19,433 --> 01:10:24,566
to have a outcomes assessment

304
01:10:24,866 --> 01:10:29,166
process that is performed regularly.

305
01:10:30,066 --> 01:10:31,000
And in ABET, that

306
01:10:31,000 --> 01:10:32,733
means at least two cycles

307
01:10:33,033 --> 01:10:34,066
during the six years

308
01:10:34,066 --> 01:10:37,366
between when you started

309
01:10:37,766 --> 01:10:38,733
to when you get accredited.

310
01:10:41,033 --> 01:10:42,366
And it must be appropriate.

311
01:10:43,066 --> 01:10:44,600
And that's a big catch word.

312
01:10:44,899 --> 01:10:46,466
That could mean literally anything.

313
01:10:46,466 --> 01:10:47,066
Yes, it can.

314
01:10:47,666 --> 01:10:49,366
So those are the two words that usually

315
01:10:49,366 --> 01:10:51,066
catches most programs.

316
01:10:51,833 --> 01:10:53,966
And so it really all comes down to,

317
01:10:54,733 --> 01:10:56,500
how are you going to assess

318
01:10:56,500 --> 01:10:59,566
the outcomes for the students?

319
01:11:00,566 --> 01:11:01,733
How are you going to do it

320
01:11:01,733 --> 01:11:03,933
in a way that is regular?

321
01:11:04,933 --> 01:11:06,333
And how is it appropriate?

322
01:11:07,333 --> 01:11:09,800
So after being on many ABET visits

323
01:11:10,000 --> 01:11:12,333
and going to many ABET symposiums

324
01:11:12,333 --> 01:11:13,933
and even being on the board of directors

325
01:11:13,933 --> 01:11:15,566
for ABET for a while,

326
01:11:15,566 --> 01:11:17,033
I've gotten a good sense of what they

327
01:11:17,033 --> 01:11:18,399
actually mean by that,

328
01:11:18,666 --> 01:11:19,466
even though it's not

329
01:11:19,466 --> 01:11:21,866
explicitly written down.

330
01:11:23,066 --> 01:11:25,766
So my takeaways for a lot of this

331
01:11:25,800 --> 01:11:28,666
are that what ABET

332
01:11:28,666 --> 01:11:31,266
really cares deeply about--

333
01:11:31,266 --> 01:11:32,366
and you'll get this from

334
01:11:32,366 --> 01:11:34,033
anyone from ABET you talk to--

335
01:11:34,366 --> 01:11:36,100
they care deeply about

336
01:11:36,100 --> 01:11:37,633
continuous improvement.

337
01:11:38,333 --> 01:11:40,366
This all came from ISO 9000.

338
01:11:41,433 --> 01:11:46,766
And it turned into the EC2000, ABET,

339
01:11:46,766 --> 01:11:47,899
all of a sudden, with these words.

340
01:11:47,966 --> 01:11:48,966
But basically, they

341
01:11:48,966 --> 01:11:50,866
changed ABET dramatically

342
01:11:51,766 --> 01:11:54,833
around the year 2000, where they wanted

343
01:11:54,833 --> 01:11:57,500
to be more like what industry does

344
01:11:57,966 --> 01:11:59,566
for their continuous improvement.

345
01:12:00,399 --> 01:12:02,966
And there's a lot of good ideas in there.

346
01:12:02,966 --> 01:12:04,899
I have my own personal beliefs.

347
01:12:04,899 --> 01:12:07,266
I don't believe students in education.

348
01:12:07,600 --> 01:12:08,666
It's like a product

349
01:12:08,666 --> 01:12:10,133
that's on an assembly line.

350
01:12:10,766 --> 01:12:12,500
So it's a little harder to measure

351
01:12:12,500 --> 01:12:14,266
students' achievement

352
01:12:14,666 --> 01:12:16,066
of outcomes than it is

353
01:12:16,066 --> 01:12:18,033
to measure quality control

354
01:12:18,033 --> 01:12:19,366
on the dimensions of a

355
01:12:19,366 --> 01:12:21,766
part, the hardness of the metal,

356
01:12:22,533 --> 01:12:23,366
all those things.

357
01:12:23,366 --> 01:12:25,233
Those are very easy to measure.

358
01:12:26,366 --> 01:12:28,333
Measuring learning is really hard.

359
01:12:28,333 --> 01:12:30,300
And actually, none of us really know how

360
01:12:30,300 --> 01:12:32,633
to do it in a scalable way.

361
01:12:33,133 --> 01:12:35,366
So that's the hardest thing.

362
01:12:35,899 --> 01:12:36,899
But what ABET cares

363
01:12:36,899 --> 01:12:37,933
about is they at least

364
01:12:37,933 --> 01:12:40,466
have a process that makes a

365
01:12:40,466 --> 01:12:44,366
very serious attempt to measure.

366
01:12:44,399 --> 01:12:45,466
And what ABET cares about

367
01:12:45,466 --> 01:12:47,566
is you must, in the criteria,

368
01:12:48,133 --> 01:12:49,600
it says that you must

369
01:12:49,600 --> 01:12:54,533
measure the extent to which

370
01:12:55,333 --> 01:12:56,600
the graduates of your

371
01:12:56,600 --> 01:12:59,133
program have achieved the seven

372
01:12:59,266 --> 01:12:59,800
outcomes.

373
01:13:00,899 --> 01:13:02,366
Now, let's unpack that.

374
01:13:03,566 --> 01:13:06,333
The extent to which ABET doesn't say

375
01:13:06,333 --> 01:13:08,066
anything about that everyone

376
01:13:08,066 --> 01:13:10,033
has to have an A. They

377
01:13:10,033 --> 01:13:11,866
just say your program needs

378
01:13:12,066 --> 01:13:14,166
to know how well your

379
01:13:14,166 --> 01:13:16,399
graduates are doing so they can use

380
01:13:16,399 --> 01:13:19,166
that information if you need to to

381
01:13:19,166 --> 01:13:20,266
improve your program.

382
01:13:20,266 --> 01:13:21,033
It's a diagnostic.

383
01:13:22,466 --> 01:13:25,266
The other thing is the word "graduates."

384
01:13:26,300 --> 01:13:27,766
They're not looking for

385
01:13:27,766 --> 01:13:29,566
learning about what students

386
01:13:29,566 --> 01:13:31,533
at the first or second year are doing,

387
01:13:31,666 --> 01:13:33,699
because as you know, it's hierarchical.

388
01:13:34,066 --> 01:13:35,466
You build on the knowledge.

389
01:13:35,466 --> 01:13:36,433
And the kinds of things

390
01:13:36,433 --> 01:13:38,333
and the outcomes are--

391
01:13:38,666 --> 01:13:41,199
the outcome one is to solve science and

392
01:13:41,199 --> 01:13:42,266
engineering problems,

393
01:13:43,300 --> 01:13:44,266
complex engineering

394
01:13:44,266 --> 01:13:45,766
problems, which usually

395
01:13:45,766 --> 01:13:47,233
means open-ended problems.

396
01:13:47,833 --> 01:13:48,899
That's not something we

397
01:13:48,899 --> 01:13:50,533
expect our freshmen to do.

398
01:13:50,733 --> 01:13:51,933
It's something we expect

399
01:13:51,933 --> 01:13:53,633
our seniors to be able to do.

400
01:13:54,399 --> 01:13:56,333
So it really makes-- because of that,

401
01:13:56,333 --> 01:13:57,800
it makes little sense to do

402
01:13:57,800 --> 01:14:00,100
assessment in the first or second

403
01:14:00,233 --> 01:14:02,366
year if all you care about is

404
01:14:02,366 --> 01:14:03,966
meeting the objectives of ABET.

405
01:14:04,433 --> 01:14:06,166
Now, there's lots of people who might

406
01:14:06,266 --> 01:14:07,133
want to do assessment

407
01:14:07,133 --> 01:14:09,366
earlier for their own purposes,

408
01:14:09,366 --> 01:14:10,166
and that's fine.

409
01:14:10,866 --> 01:14:12,166
And in fact, the mantra--

410
01:14:12,633 --> 01:14:13,866
at least it used to be--

411
01:14:13,866 --> 01:14:17,133
I hope it still is-- at the TMS

412
01:14:17,133 --> 01:14:18,433
Accreditation Committee,

413
01:14:19,666 --> 01:14:22,166
used to be improve your

414
01:14:22,166 --> 01:14:24,000
program for yourself first

415
01:14:24,433 --> 01:14:26,199
and worry about ABET later.

416
01:14:27,000 --> 01:14:28,800
Because if you're doing a good job

417
01:14:28,800 --> 01:14:30,166
improving your program

418
01:14:30,166 --> 01:14:32,800
for yourself, you should have no problem

419
01:14:32,800 --> 01:14:34,533
documenting your processes

420
01:14:34,800 --> 01:14:36,533
and showing ABET that you did it.

421
01:14:36,933 --> 01:14:39,266
The converse is sort of

422
01:14:39,266 --> 01:14:40,899
a fool's errand, right?

423
01:14:41,366 --> 01:14:43,733
To just do it for compliance alone is

424
01:14:43,733 --> 01:14:45,000
kind of wasting your time.

425
01:14:45,433 --> 01:14:46,166
If you're going to do

426
01:14:46,166 --> 01:14:47,766
something, make it meaningful.

427
01:14:48,433 --> 01:14:50,699
So that's what you

428
01:14:50,699 --> 01:14:52,633
should really be doing.

429
01:14:53,300 --> 01:14:55,133
But at any rate, this is

430
01:14:55,133 --> 01:14:57,033
the number one criteria

431
01:14:57,800 --> 01:14:59,333
that shortcomings are

432
01:14:59,333 --> 01:15:03,500
delivered to that 

433
01:15:04,133 --> 01:15:07,266
there either is no good process or the

434
01:15:07,266 --> 01:15:09,166
program only did it once

435
01:15:09,399 --> 01:15:10,866
instead of at least twice,

436
01:15:10,866 --> 01:15:13,333
or that it's not appropriate.

437
01:15:13,566 --> 01:15:14,366
And what they mean by

438
01:15:14,366 --> 01:15:17,366
that is either doing it just

439
01:15:17,533 --> 01:15:19,633
running through the paces, but it's not

440
01:15:19,633 --> 01:15:21,433
giving them any information.

441
01:15:22,199 --> 01:15:24,433
The last part of the criteria says

442
01:15:25,000 --> 01:15:30,533
that you must use the results of the

443
01:15:30,533 --> 01:15:32,166
measurements, the assessment,

444
01:15:32,833 --> 01:15:34,366
as input to your

445
01:15:34,366 --> 01:15:36,500
continuous improvement process.

446
01:15:36,933 --> 01:15:38,333
And you must have a continuous

447
01:15:38,333 --> 01:15:40,466
improvement process as well.

448
01:15:42,666 --> 01:15:44,233
The last line of that

449
01:15:44,233 --> 01:15:46,033
criteria is the most valuable.

450
01:15:46,033 --> 01:15:49,433
It says you can also use as input

451
01:15:49,433 --> 01:15:52,166
anything else to improve your program.

452
01:15:53,399 --> 01:15:55,866
Now, this has been

453
01:15:55,866 --> 01:15:58,933
debated by many people at ABET,

454
01:15:59,333 --> 01:16:01,866
but ABET has made it exceedingly clear

455
01:16:01,866 --> 01:16:04,266
that the word input is

456
01:16:04,266 --> 01:16:05,399
there and not output.

457
01:16:06,500 --> 01:16:11,233
So if you do your assessment and you

458
01:16:11,233 --> 01:16:12,500
analyze your results

459
01:16:13,199 --> 01:16:15,800
and you show that everyone is doing great

460
01:16:15,800 --> 01:16:18,000
above whatever threshold

461
01:16:18,000 --> 01:16:19,433
you decide is important,

462
01:16:20,300 --> 01:16:23,133
then you may not be able to use the data

463
01:16:23,133 --> 01:16:24,966
you collected to

464
01:16:24,966 --> 01:16:26,399
actually improve your program.

465
01:16:27,466 --> 01:16:28,133
And that's OK.

466
01:16:28,666 --> 01:16:29,399
Because then you can

467
01:16:29,399 --> 01:16:30,866
bring in those other inputs,

468
01:16:31,366 --> 01:16:33,566
information from your advisory board or

469
01:16:33,566 --> 01:16:34,966
from industry partners

470
01:16:35,433 --> 01:16:36,899
to get that information instead.

471
01:16:37,666 --> 01:16:41,899
And so we have really good students.

472
01:16:42,366 --> 01:16:45,899
They come in, they all take our intro

473
01:16:45,899 --> 01:16:48,033
engineering course, Engin 100,

474
01:16:49,333 --> 01:16:51,366
which is really a fantastic course.

475
01:16:52,000 --> 01:16:53,766
It's really an English course, a

476
01:16:53,766 --> 01:16:54,766
communications course,

477
01:16:55,266 --> 01:16:58,833
but it's cast in a framework of a design,

478
01:16:58,833 --> 01:17:00,266
build, test environment,

479
01:17:01,066 --> 01:17:04,133
teaching freshmen how to do engineering

480
01:17:04,133 --> 01:17:06,433
design right from the get-go.

481
01:17:07,433 --> 01:17:10,100
So our students who take that, they learn

482
01:17:10,100 --> 01:17:11,399
how to work in teams,

483
01:17:11,899 --> 01:17:14,033
they learn about ethics, they learn about

484
01:17:14,033 --> 01:17:15,266
design, they learn

485
01:17:15,266 --> 01:17:16,633
about doing experiments.

486
01:17:17,766 --> 01:17:20,300
They do everything that our

487
01:17:20,300 --> 01:17:22,266
ABET outcomes ask us to do.

488
01:17:22,933 --> 01:17:24,566
So by the time they come in our

489
01:17:24,566 --> 01:17:25,666
department, in Michigan,

490
01:17:26,233 --> 01:17:27,966
they don't join our program

491
01:17:27,966 --> 01:17:29,566
until they've had at least one term.

492
01:17:31,233 --> 01:17:32,199
Wow, they're amazing.

493
01:17:32,566 --> 01:17:35,100
And I remember the days before Engin 100

494
01:17:35,100 --> 01:17:37,100
when it wasn't that way.

495
01:17:37,100 --> 01:17:38,433
I'm sure they were dark times.

496
01:17:38,766 --> 01:17:41,266
Yes. And so things have really gotten

497
01:17:41,266 --> 01:17:44,333
better in the sense of at

498
01:17:44,333 --> 01:17:46,333
least outcomes two through seven.

499
01:17:47,233 --> 01:17:49,533
Outcome one is the toughest outcome.

500
01:17:49,633 --> 01:17:53,833
Outcome one is using engineering, math,

501
01:17:53,833 --> 01:17:55,433
and science to solve

502
01:17:55,433 --> 01:17:57,233
complex engineering problems.

503
01:17:58,866 --> 01:17:59,633
That's difficult.

504
01:18:00,333 --> 01:18:01,466
That's the one we beat

505
01:18:01,466 --> 01:18:02,433
on the students hard.

506
01:18:02,933 --> 01:18:05,233
We have very high expectations for them.

507
01:18:05,766 --> 01:18:07,833
And so the scores for that outcome are

508
01:18:07,833 --> 01:18:09,833
always lower college-wide

509
01:18:09,833 --> 01:18:12,133
for all of these things.

510
01:18:12,766 --> 01:18:14,133
So all that's kind of cool.

511
01:18:14,766 --> 01:18:20,433
So at any rate, you know, that's what's

512
01:18:20,433 --> 01:18:21,633
the most difficult part.

513
01:18:21,899 --> 01:18:25,066
And so we've developed a whole new

514
01:18:25,066 --> 01:18:26,833
approach to doing this.

515
01:18:27,600 --> 01:18:29,366
We started this six years ago.

516
01:18:30,199 --> 01:18:31,933
We actually started in our department

517
01:18:31,933 --> 01:18:34,433
earlier, but we've rolled it out and are

518
01:18:34,433 --> 01:18:36,566
now doing it on a college-wide basis,

519
01:18:37,633 --> 01:18:41,566
automated, trying to follow some basic

520
01:18:41,566 --> 01:18:44,800
principles to make sure our process is

521
01:18:44,800 --> 01:18:46,533
completely sustainable.

522
01:18:47,066 --> 01:18:48,866
And so this is the process for measuring

523
01:18:48,866 --> 01:18:50,166
the student outcomes.

524
01:18:50,633 --> 01:18:50,833
Yes.

525
01:18:51,133 --> 01:18:55,033
OK, so from the instructor side, I can

526
01:18:55,033 --> 01:18:56,466
see my perspective on it.

527
01:18:56,466 --> 01:18:57,933
But I want to actually come back to

528
01:18:57,933 --> 01:19:00,699
something else you said earlier, which is

529
01:19:00,699 --> 01:19:04,166
the priority order of what to do for

530
01:19:04,166 --> 01:19:05,100
improving the program

531
01:19:05,100 --> 01:19:06,666
versus what to do for compliance.

532
01:19:07,366 --> 01:19:09,566
And when I'm thinking about making

533
01:19:09,566 --> 01:19:11,933
changes, hopefully improvements to my

534
01:19:11,933 --> 01:19:14,500
courses, I'm always saying, well, what do

535
01:19:14,500 --> 01:19:16,966
I really want to do with my class?

536
01:19:17,233 --> 01:19:18,833
And then looking at the ABET outcomes

537
01:19:18,833 --> 01:19:21,899
and saying, is there an outcome that this

538
01:19:21,899 --> 01:19:23,233
change I'd like to make

539
01:19:23,233 --> 01:19:26,133
happens to be well tied to?

540
01:19:26,433 --> 01:19:28,133
And if so, to me, that's a good indicator

541
01:19:28,133 --> 01:19:29,633
that it's something worth pursuing

542
01:19:29,633 --> 01:19:32,366
because most of the ABET outcomes are

543
01:19:32,366 --> 01:19:34,899
pretty transparently things that I think

544
01:19:34,899 --> 01:19:36,366
we should want our students to do.

545
01:19:36,600 --> 01:19:39,300
We should want them to consider societal,

546
01:19:39,300 --> 01:19:41,566
economic, environmental implications of

547
01:19:41,566 --> 01:19:42,533
their engineering work.

548
01:19:42,533 --> 01:19:45,266
We should want them to design experiments

549
01:19:45,266 --> 01:19:46,666
that have scientific validity.

550
01:19:46,966 --> 01:19:48,866
These are just good things to do anyway.

551
01:19:49,800 --> 01:19:51,933
So as I'm thinking about my courses,

552
01:19:52,233 --> 01:19:53,899
anytime I can point to an ABET outcome

553
01:19:53,899 --> 01:19:55,166
and say, by the way,

554
01:19:55,166 --> 01:19:56,533
I'm also achieving this.

555
01:19:56,966 --> 01:19:59,066
In addition to just doing what I believe

556
01:19:59,066 --> 01:20:01,266
is good teaching, that's always my anchor

557
01:20:01,266 --> 01:20:03,000
for how to make those changes.

558
01:20:03,000 --> 01:20:03,633
And that's great.

559
01:20:04,933 --> 01:20:06,800
There's kind of two viewpoints of this.

560
01:20:07,433 --> 01:20:10,166
There's the ABET viewpoint that they

561
01:20:10,166 --> 01:20:13,100
believe that they're driving all of our

562
01:20:13,100 --> 01:20:15,433
education by mandating these outcomes.

563
01:20:16,366 --> 01:20:17,966
But, you know, their

564
01:20:17,966 --> 01:20:19,833
outcomes are kind of motherhood.

565
01:20:20,466 --> 01:20:21,399
It's kind of obvious.

566
01:20:22,699 --> 01:20:24,733
And I'd like to think that our program

567
01:20:24,733 --> 01:20:28,466
actually has many more outcomes beyond

568
01:20:28,466 --> 01:20:30,366
just what ABET requires.

569
01:20:31,166 --> 01:20:33,166
And I think any good program will, of

570
01:20:33,166 --> 01:20:35,266
course, do all the things ABET wants.

571
01:20:36,500 --> 01:20:41,233
But the real improvement of the programs,

572
01:20:42,233 --> 01:20:44,666
at least my experience here in Michigan,

573
01:20:45,100 --> 01:20:47,166
has never really come from doing the

574
01:20:47,166 --> 01:20:48,833
assessment of the outcomes.

575
01:20:49,800 --> 01:20:52,866
The real improvement comes from people

576
01:20:52,866 --> 01:20:55,533
just like yourself having that good

577
01:20:55,533 --> 01:20:58,199
attitude of trying to do what's best for

578
01:20:58,199 --> 01:21:00,666
our students in a broad range of

579
01:21:01,500 --> 01:21:01,699
areas.

580
01:21:02,399 --> 01:21:05,633
So we have a very robust undergraduate

581
01:21:05,633 --> 01:21:08,933
committee that reviews

582
01:21:08,933 --> 01:21:10,600
constantly our curriculum.

583
01:21:10,833 --> 01:21:14,033
We do curricular reviews apart from ABET

584
01:21:14,033 --> 01:21:15,766
because we think it's important.

585
01:21:16,566 --> 01:21:19,366
We try to have meetings where we put

586
01:21:19,366 --> 01:21:21,633
together people who teach Thermo and

587
01:21:21,633 --> 01:21:25,466
Kinetics and ask, are you getting the

588
01:21:25,466 --> 01:21:26,800
right kind of

589
01:21:26,800 --> 01:21:28,333
background in your students?

590
01:21:28,600 --> 01:21:29,533
What's missing?

591
01:21:29,933 --> 01:21:31,633
And you know this better than anyone

592
01:21:31,633 --> 01:21:34,500
because you're teaching a math course

593
01:21:34,500 --> 01:21:38,000
because we're not getting students coming

594
01:21:38,000 --> 01:21:39,633
into our Thermo and our

595
01:21:39,633 --> 01:21:42,066
Kinetics courses with enough math.

596
01:21:42,633 --> 01:21:44,533
And so we're going to tailor the math

597
01:21:44,533 --> 01:21:46,866
they need to supplement from what the

598
01:21:46,866 --> 01:21:49,199
math department gives them so they can

599
01:21:49,199 --> 01:21:51,433
perform better in our Thermo

600
01:21:51,433 --> 01:21:53,966
and in our Kinetics courses.

601
01:21:54,933 --> 01:21:57,666
Another example of massive improvements

602
01:21:57,666 --> 01:22:01,300
in our system of our program didn't come

603
01:22:01,300 --> 01:22:04,933
out again from ABET but came by the work

604
01:22:04,933 --> 01:22:07,366
you did that we talked in another podcast

605
01:22:07,366 --> 01:22:09,033
about your alloy

606
01:22:09,033 --> 01:22:11,533
design module in the lab.

607
01:22:12,300 --> 01:22:13,766
You worked with the people who taught

608
01:22:13,766 --> 01:22:15,699
Thermo because while they're taking

609
01:22:15,699 --> 01:22:18,733
Thermo, our students are taking your lab,

610
01:22:18,733 --> 01:22:20,366
whether you're learning how to use tools

611
01:22:20,366 --> 01:22:23,133
like ThermoCalc and design their own

612
01:22:23,133 --> 01:22:24,699
materials, their own alloy.

613
01:22:25,533 --> 01:22:27,666
Using those thermodynamic principles

614
01:22:27,666 --> 01:22:30,566
they're learning in Thermo actually make it

615
01:22:30,566 --> 01:22:34,133
pour dog bone test specimens and test it

616
01:22:34,133 --> 01:22:35,966
and then try to understand why it didn't

617
01:22:35,966 --> 01:22:37,733
work because it rarely works.

618
01:22:38,433 --> 01:22:40,300
But you know it will once you get really

619
01:22:40,300 --> 01:22:41,399
good at it but you've

620
01:22:41,399 --> 01:22:42,399
got to start somewhere.

621
01:22:43,066 --> 01:22:44,866
So how is that captured

622
01:22:44,866 --> 01:22:47,266
in our assessment results?

623
01:22:47,866 --> 01:22:49,766
You need to go way beyond just

624
01:22:49,766 --> 01:22:52,733
assessment. You need to you know be

625
01:22:52,733 --> 01:22:55,766
creative and innovative and you need to

626
01:22:55,766 --> 01:22:58,866
inspire a culture in a program of the

627
01:22:58,866 --> 01:23:01,166
faculty caring about

628
01:23:01,166 --> 01:23:02,733
their undergraduate students.

629
01:23:03,333 --> 01:23:06,366
And I am so happy to say I think we've

630
01:23:06,366 --> 01:23:08,733
got an amazing culture here

631
01:23:08,733 --> 01:23:10,399
at Michigan doing just that.

632
01:23:11,000 --> 01:23:14,899
So the way I try to document that I know

633
01:23:14,899 --> 01:23:18,199
that every faculty member innovates in

634
01:23:18,199 --> 01:23:19,899
every single course they teach.

635
01:23:20,600 --> 01:23:22,266
And so when it comes time to write the

636
01:23:22,266 --> 01:23:24,766
self-study I put a note out to the

637
01:23:24,766 --> 01:23:27,166
faculty asking them to write me half a

638
01:23:27,166 --> 01:23:29,500
page to a page of what they're most proud

639
01:23:29,500 --> 01:23:31,733
of in developing in

640
01:23:31,733 --> 01:23:32,966
the previous six years.

641
01:23:33,966 --> 01:23:36,666
And that's the I think the best part of

642
01:23:36,666 --> 01:23:39,399
our self-study because it's honest it's

643
01:23:39,399 --> 01:23:41,933
from the heart. It's the real actual

644
01:23:41,933 --> 01:23:43,166
improvements that all

645
01:23:43,166 --> 01:23:44,899
these individuals do.

646
01:23:45,199 --> 01:23:48,033
Often with the help and support of the

647
01:23:48,033 --> 01:23:50,566
undergraduate committee or other things.

648
01:23:51,199 --> 01:23:54,366
But what a great way I think to document

649
01:23:54,366 --> 01:23:56,600
all of that and to really demonstrate

650
01:23:56,600 --> 01:24:00,233
that we're doing massive improvements to

651
01:24:00,233 --> 01:24:02,533
our program all the time.

652
01:24:02,533 --> 01:24:05,300
And it's coming from what the students

653
01:24:05,300 --> 01:24:08,333
tell us because we have we have town hall

654
01:24:08,333 --> 01:24:09,933
meetings every year with

655
01:24:09,933 --> 01:24:11,800
our students to hear comments.

656
01:24:12,333 --> 01:24:13,933
We heard a comment last term that the

657
01:24:13,933 --> 01:24:17,366
BioMed course is actually too biological

658
01:24:17,366 --> 01:24:19,033
and not enough material science.

659
01:24:19,866 --> 01:24:21,866
And guess what we're acting on that.

660
01:24:21,866 --> 01:24:24,866
And so Brian Love who also agrees with

661
01:24:24,866 --> 01:24:26,433
that assessment even though he's taught

662
01:24:26,433 --> 01:24:29,166
the course before is actively trying to

663
01:24:29,166 --> 01:24:30,500
change that course now.

664
01:24:31,899 --> 01:24:34,399
We also get comments from our external

665
01:24:34,399 --> 01:24:36,866
advisory board who are people in industry

666
01:24:36,866 --> 01:24:38,533
who are giving us heads

667
01:24:38,533 --> 01:24:40,266
up about things they need.

668
01:24:40,766 --> 01:24:43,699
We get information from our alums with

669
01:24:43,699 --> 01:24:45,233
whatever careers they followed.

670
01:24:46,000 --> 01:24:50,233
So it's you know it's a lot of good stuff

671
01:24:50,233 --> 01:24:52,433
that we use to improve our program.

672
01:24:53,233 --> 01:24:55,033
But I view this outcomes assessment

673
01:24:55,033 --> 01:24:58,733
process which we're mandated to do is

674
01:24:58,733 --> 01:25:01,233
actually a critical part of this process.

675
01:25:01,233 --> 01:25:03,866
And the way I think about it is these are

676
01:25:03,866 --> 01:25:06,833
the diagnostics that if we don't do we

677
01:25:06,833 --> 01:25:09,600
can get ourselves in serious trouble just

678
01:25:09,600 --> 01:25:12,500
like engineers design that little

679
01:25:12,500 --> 01:25:15,266
material inside your brake pads.

680
01:25:15,699 --> 01:25:17,100
To start squealing when

681
01:25:17,100 --> 01:25:18,966
the brake pads get small.

682
01:25:19,399 --> 01:25:22,100
You need that there even if you change

683
01:25:22,100 --> 01:25:22,866
your brake pads

684
01:25:22,866 --> 01:25:24,566
religiously and never hear it.

685
01:25:25,100 --> 01:25:27,333
You want to know before something bad

686
01:25:27,333 --> 01:25:29,699
happens that it's about to happen.

687
01:25:29,699 --> 01:25:32,300
And that's how I view our outcomes

688
01:25:32,300 --> 01:25:34,066
assessment at the very end.

689
01:25:34,433 --> 01:25:35,366
I talk about the

690
01:25:35,366 --> 01:25:37,366
future of what we can do.

691
01:25:37,366 --> 01:25:39,533
I think we can even learn more from our

692
01:25:39,533 --> 01:25:40,433
outcomes assessment.

693
01:25:41,866 --> 01:25:43,966
Well let's get to nuts and bolts for a

694
01:25:43,966 --> 01:25:48,166
minute here because we have this big

695
01:25:48,166 --> 01:25:50,366
picture aspirational vision of here's

696
01:25:50,366 --> 01:25:52,199
what all this assessment accomplishes.

697
01:25:52,666 --> 01:25:54,300
Right. Here's what it helps

698
01:25:54,300 --> 01:25:56,233
our program get better at.

699
01:25:56,500 --> 01:25:57,266
But there is still

700
01:25:57,266 --> 01:25:59,333
that implementation layer.

701
01:25:59,333 --> 01:26:02,866
The actually doing it which might not be

702
01:26:02,866 --> 01:26:04,566
obvious especially if a program is trying

703
01:26:04,566 --> 01:26:07,199
to step up their game in terms of having

704
01:26:07,199 --> 01:26:09,466
an easier more efficient time with

705
01:26:09,466 --> 01:26:11,199
checking the ABET boxes.

706
01:26:11,733 --> 01:26:12,933
While doing great teaching.

707
01:26:13,466 --> 01:26:14,333
So how do you actually

708
01:26:14,333 --> 01:26:16,000
do outcomes assessment.

709
01:26:16,366 --> 01:26:16,866
How is that

710
01:26:16,866 --> 01:26:18,833
implemented at a practical level.

711
01:26:19,133 --> 01:26:21,166
That is absolutely the critical question.

712
01:26:21,899 --> 01:26:23,866
We all know the biggest problem with

713
01:26:23,866 --> 01:26:25,866
doing outcomes assessment is

714
01:26:25,866 --> 01:26:27,666
getting our faculty to do it.

715
01:26:28,833 --> 01:26:32,399
The only people who can really probe the

716
01:26:32,399 --> 01:26:34,866
students in an efficient way where we

717
01:26:34,866 --> 01:26:37,433
distribute the load of the work is to

718
01:26:37,433 --> 01:26:38,366
have every single

719
01:26:38,366 --> 01:26:40,466
instructor of any course.

720
01:26:41,433 --> 01:26:44,666
That we're using to assess our students.

721
01:26:45,033 --> 01:26:46,366
They have to do the work.

722
01:26:47,133 --> 01:26:50,833
So it comes down to some fundamental

723
01:26:50,833 --> 01:26:56,100
concepts to make a build a process that

724
01:26:56,100 --> 01:26:58,733
everyone will participate in in the

725
01:26:58,733 --> 01:27:00,766
easiest possible way so

726
01:27:00,766 --> 01:27:02,066
that it becomes sustainable.

727
01:27:02,366 --> 01:27:03,933
And we do it all the time.

728
01:27:04,533 --> 01:27:06,833
So the first thing is although a bit you

729
01:27:06,833 --> 01:27:08,233
know they don't tell you how to do this.

730
01:27:08,500 --> 01:27:10,033
They just tell you have to do it.

731
01:27:11,366 --> 01:27:13,333
I've seen a lot of suggestions and I

732
01:27:13,333 --> 01:27:14,800
don't like most of them.

733
01:27:15,199 --> 01:27:16,533
Like ABET, symposia.

734
01:27:16,966 --> 01:27:18,266
When they say oh well you could do

735
01:27:18,266 --> 01:27:21,333
outcomes one and two this term and

736
01:27:21,333 --> 01:27:24,466
outcomes three and four the next term and

737
01:27:24,466 --> 01:27:27,133
then new outcomes five and six the term

738
01:27:27,133 --> 01:27:29,699
after that and do that and then cycle and

739
01:27:29,699 --> 01:27:30,666
by the time you're done six

740
01:27:30,666 --> 01:27:31,833
years you've done it two times.

741
01:27:32,733 --> 01:27:35,233
The problem with that is everyone forgets

742
01:27:35,233 --> 01:27:36,733
what they were supposed to be doing

743
01:27:36,733 --> 01:27:38,733
because we're creatures of habit.

744
01:27:40,000 --> 01:27:42,966
So we need to do all of the outcomes

745
01:27:42,966 --> 01:27:46,733
every single term in my opinion and find

746
01:27:46,733 --> 01:27:49,966
a way to make it as easy as possible for

747
01:27:49,966 --> 01:27:52,466
our instructors to actually do the work.

748
01:27:53,366 --> 01:27:58,533
So that's what I first make it easy for

749
01:27:58,533 --> 01:28:00,500
the faculty and instructors

750
01:28:00,500 --> 01:28:02,566
so that actually gets done.

751
01:28:03,366 --> 01:28:06,566
The next thing you know we need to do is

752
01:28:06,566 --> 01:28:08,600
how do we do this and

753
01:28:08,600 --> 01:28:10,366
also make it meaningful.

754
01:28:11,466 --> 01:28:14,366
So what we've done is we built and a lot

755
01:28:14,366 --> 01:28:16,300
of programs do this by the way I think

756
01:28:16,300 --> 01:28:18,033
people have learned over the years.

757
01:28:18,333 --> 01:28:19,766
So it's pretty standard now.

758
01:28:21,000 --> 01:28:23,566
We build a matrix of we look at all of

759
01:28:23,566 --> 01:28:25,899
our required courses and all of our

760
01:28:25,899 --> 01:28:31,133
elective courses and we assign we try to

761
01:28:31,133 --> 01:28:33,866
assign no more than one

762
01:28:33,866 --> 01:28:36,033
outcome to any given course.

763
01:28:36,833 --> 01:28:38,633
Now we can't do this for all courses

764
01:28:38,633 --> 01:28:41,266
because we lean very heavily into our

765
01:28:41,266 --> 01:28:44,600
design and our lab courses because those

766
01:28:44,600 --> 01:28:46,733
are the courses where it's more

767
01:28:46,733 --> 01:28:49,500
appropriate to measure things about

768
01:28:49,500 --> 01:28:53,033
teams, communications, design designing

769
01:28:53,033 --> 01:28:54,866
experiments, ethics.

770
01:28:55,766 --> 01:28:59,366
So those are outcomes two through six,

771
01:29:00,366 --> 01:29:02,199
two, three, four, five and

772
01:29:02,199 --> 01:29:03,333
six. That's five outcomes.

773
01:29:04,133 --> 01:29:07,600
But luckily we've got two lab courses and

774
01:29:07,600 --> 01:29:09,966
we have two design courses.

775
01:29:10,600 --> 01:29:12,633
So we're able to break it up so they only

776
01:29:12,633 --> 01:29:14,300
have three outcomes to do.

777
01:29:14,833 --> 01:29:18,899
Some programs wait until the last year

778
01:29:18,899 --> 01:29:22,866
and dump this all, all seven outcomes on

779
01:29:22,866 --> 01:29:25,699
the capstone design course, which makes

780
01:29:25,699 --> 01:29:27,533
sense because you're measuring with our

781
01:29:27,533 --> 01:29:28,933
graduates the extent to

782
01:29:28,933 --> 01:29:29,866
which they've learned it.

783
01:29:30,133 --> 01:29:30,866
But it makes it

784
01:29:30,866 --> 01:29:32,566
unsustainable in my opinion.

785
01:29:32,899 --> 01:29:34,933
Right. It's even more work for the person

786
01:29:34,933 --> 01:29:36,566
who's already doing the hardest class.

787
01:29:36,899 --> 01:29:39,066
Exactly. And that's just not fair.

788
01:29:39,066 --> 01:29:40,433
So we've broken it up.

789
01:29:41,066 --> 01:29:44,733
So our idea is let's collect a relatively

790
01:29:44,733 --> 01:29:48,433
sparse data set, but do it every single

791
01:29:48,433 --> 01:29:51,433
term so that we end up with a massive

792
01:29:51,433 --> 01:29:53,733
amount of data after six years.

793
01:29:53,733 --> 01:29:55,366
We do this 12 terms.

794
01:29:56,233 --> 01:29:59,933
And I have to say it kind of works well.

795
01:30:00,633 --> 01:30:01,966
So that's the first thing.

796
01:30:01,966 --> 01:30:04,633
And we also only do assessment in our

797
01:30:04,633 --> 01:30:06,266
junior and senior classes.

798
01:30:07,333 --> 01:30:11,466
We assess outcomes one

799
01:30:11,466 --> 01:30:14,766
through six in our required courses.

800
01:30:15,533 --> 01:30:18,600
So regardless of path, every student is

801
01:30:18,600 --> 01:30:20,833
assessed in our required courses because

802
01:30:20,833 --> 01:30:22,000
they all have to take it.

803
01:30:22,333 --> 01:30:24,866
Then our elective courses, because we

804
01:30:24,866 --> 01:30:27,266
want to distribute the load again, we

805
01:30:27,266 --> 01:30:30,333
have outcome seven is done in all of our

806
01:30:30,333 --> 01:30:34,100
which is lifelong learning, learning,

807
01:30:34,166 --> 01:30:36,466
different methods of learning, all that.

808
01:30:37,233 --> 01:30:39,033
And so we have all of our elective

809
01:30:39,033 --> 01:30:40,733
courses measure that outcome.

810
01:30:41,033 --> 01:30:43,899
So regardless of which elective courses

811
01:30:43,899 --> 01:30:44,933
our students take,

812
01:30:44,933 --> 01:30:45,633
they're being assessed.

813
01:30:46,333 --> 01:30:48,399
So we do assessment of all outcomes

814
01:30:48,399 --> 01:30:51,199
across regardless of path

815
01:30:51,199 --> 01:30:52,666
for all of our students.

816
01:30:53,633 --> 01:30:56,500
So once we've done that,

817
01:30:56,866 --> 01:30:58,566
we need one more thing.

818
01:30:58,866 --> 01:31:02,666
How can we make this even easier for our

819
01:31:02,666 --> 01:31:04,933
for our instructors?

820
01:31:05,666 --> 01:31:07,433
And that has to do with the actual

821
01:31:07,433 --> 01:31:09,766
mechanics of how they

822
01:31:09,766 --> 01:31:11,399
first they have to create.

823
01:31:11,399 --> 01:31:14,633
We expect our faculty to create the

824
01:31:14,633 --> 01:31:16,100
assessment measures.

825
01:31:16,533 --> 01:31:17,433
We have a choice.

826
01:31:18,166 --> 01:31:20,566
We could hire assessment professionals

827
01:31:20,566 --> 01:31:22,466
and have them tell us what

828
01:31:22,466 --> 01:31:23,766
the assessment should be.

829
01:31:24,433 --> 01:31:25,866
Or we can ask our faculty.

830
01:31:26,666 --> 01:31:29,066
I happen to think our faculty are better

831
01:31:29,066 --> 01:31:30,533
positioned because

832
01:31:30,533 --> 01:31:31,699
they're teaching the course.

833
01:31:32,266 --> 01:31:33,466
They know what they want

834
01:31:33,466 --> 01:31:34,766
their students to learn.

835
01:31:35,433 --> 01:31:37,600
And after all, don't we want all of our

836
01:31:37,600 --> 01:31:40,533
students to be able to solve complex,

837
01:31:40,533 --> 01:31:41,899
open ended problems?

838
01:31:42,433 --> 01:31:44,033
And all of our faculty do that.

839
01:31:44,333 --> 01:31:46,366
So they should find something within the

840
01:31:46,366 --> 01:31:47,666
context of their course.

841
01:31:47,966 --> 01:31:50,333
So it's meaningful for them for their

842
01:31:50,333 --> 01:31:53,366
course parameters, but use that to

843
01:31:53,366 --> 01:31:55,533
measure the particular outcomes.

844
01:31:57,000 --> 01:31:59,133
So what I've done is I

845
01:31:59,133 --> 01:32:00,766
made a series of videos.

846
01:32:01,533 --> 01:32:04,133
One short video for every single outcome,

847
01:32:04,800 --> 01:32:09,333
explaining to the faculty good, good

848
01:32:09,333 --> 01:32:11,033
approaches to build

849
01:32:11,033 --> 01:32:12,533
an outcome assessment.

850
01:32:13,066 --> 01:32:15,133
And that assessment measure, this measure

851
01:32:15,133 --> 01:32:16,833
for the assessment, it

852
01:32:16,833 --> 01:32:19,000
can be a homework problem.

853
01:32:19,000 --> 01:32:20,733
It can be an exam problem.

854
01:32:21,000 --> 01:32:23,333
It can be an activity that's not even

855
01:32:23,333 --> 01:32:25,766
graded for the course, whatever the

856
01:32:25,766 --> 01:32:27,366
faculty member wants to do.

857
01:32:28,433 --> 01:32:31,366
But if it's outcome one, for instance, it

858
01:32:31,366 --> 01:32:34,066
must involve a complex problem.

859
01:32:34,566 --> 01:32:36,666
And I show them the definition of what a

860
01:32:36,666 --> 01:32:38,266
complex problem is to ABET.

861
01:32:38,766 --> 01:32:40,566
And simply say, if you look at that, it's

862
01:32:40,566 --> 01:32:42,433
really just an open ended problem.

863
01:32:43,033 --> 01:32:44,399
And our faculty are great at

864
01:32:44,399 --> 01:32:45,899
writing open ended problems.

865
01:32:47,233 --> 01:32:49,566
So every faculty member, and they usually

866
01:32:49,566 --> 01:32:51,833
only have to do this once, because if

867
01:32:51,833 --> 01:32:53,666
they teach the same course over and over,

868
01:32:53,666 --> 01:32:54,766
they can keep using the

869
01:32:54,766 --> 01:32:56,833
assessment tool if it's good.

870
01:32:57,466 --> 01:33:00,133
And they can talk to other faculty if

871
01:33:00,133 --> 01:33:01,166
they're inheriting a

872
01:33:01,166 --> 01:33:03,266
course and borrow theirs.

873
01:33:03,266 --> 01:33:03,866
That's OK.

874
01:33:04,433 --> 01:33:06,266
But in each case, we have some faculty

875
01:33:06,266 --> 01:33:09,566
member to make a document where first

876
01:33:09,566 --> 01:33:11,166
they write down what is the

877
01:33:11,166 --> 01:33:12,533
outcome they're assessing.

878
01:33:13,066 --> 01:33:13,899
Because you want it first

879
01:33:13,899 --> 01:33:15,833
and foremost in their minds.

880
01:33:16,466 --> 01:33:18,466
Then what is the actual assessment?

881
01:33:18,466 --> 01:33:20,533
And they write down that problem in great

882
01:33:20,533 --> 01:33:22,666
detail, exactly what the student would

883
01:33:22,666 --> 01:33:23,666
see, what you're

884
01:33:23,666 --> 01:33:25,033
asking the student to do.

885
01:33:25,633 --> 01:33:27,966
And then finally, after that, they write

886
01:33:27,966 --> 01:33:31,266
a short little paragraph explaining why

887
01:33:31,266 --> 01:33:32,399
they believe this is

888
01:33:32,399 --> 01:33:33,733
an excellent outcome,

889
01:33:34,600 --> 01:33:36,033
a measure of the outcome

890
01:33:36,033 --> 01:33:36,966
they're trying to measure.

891
01:33:37,433 --> 01:33:39,433
And again, that's just to make sure that

892
01:33:39,433 --> 01:33:41,666
it's present in their minds and they've

893
01:33:41,666 --> 01:33:42,966
actually thought about it.

894
01:33:44,133 --> 01:33:45,666
Once they do that, they put that in a

895
01:33:45,666 --> 01:33:46,866
document and make a PDF.

896
01:33:47,533 --> 01:33:48,933
And at the end of the term,

897
01:33:49,300 --> 01:33:50,899
they upload that document.

898
01:33:50,899 --> 01:33:52,166
So they'll get a link.

899
01:33:53,166 --> 01:33:55,333
This comes from our college because we've

900
01:33:55,333 --> 01:33:56,366
developed this system

901
01:33:56,366 --> 01:33:58,566
across all of our 12 programs.

902
01:33:59,366 --> 01:34:03,500
And the college has our matrix and the

903
01:34:03,500 --> 01:34:06,966
every program has an ABED coordinator on

904
01:34:06,966 --> 01:34:08,966
the ABED coordinator for our program.

905
01:34:09,766 --> 01:34:12,933
So every term I review the list of who's

906
01:34:12,933 --> 01:34:14,166
teaching what courses that

907
01:34:14,166 --> 01:34:15,633
are being used for assessment.

908
01:34:16,333 --> 01:34:17,966
I confirm that with the college.

909
01:34:18,666 --> 01:34:20,666
They send a special link to each

910
01:34:20,666 --> 01:34:23,766
instructor with everything pre-filled.

911
01:34:24,333 --> 01:34:26,300
So what term it is, what course they're

912
01:34:26,300 --> 01:34:27,833
teaching, their name, all that stuff

913
01:34:27,833 --> 01:34:28,933
that's all in there.

914
01:34:28,933 --> 01:34:30,533
It's done for the faculty member.

915
01:34:31,699 --> 01:34:33,800
All the faculty member has to do is

916
01:34:33,800 --> 01:34:36,333
upload the PDF of what was the metric.

917
01:34:37,733 --> 01:34:39,033
And they have to upload

918
01:34:39,033 --> 01:34:40,833
the scores of the students.

919
01:34:41,166 --> 01:34:42,233
This is where it gets tricky.

920
01:34:43,133 --> 01:34:47,133
So we want to know the actual unique

921
01:34:47,133 --> 01:34:48,633
name, the student

922
01:34:48,633 --> 01:34:50,633
identifier for every score.

923
01:34:51,333 --> 01:34:53,833
This is going to be critical for another

924
01:34:53,833 --> 01:34:56,433
onerous thing that ABED has made us do.

925
01:34:57,333 --> 01:34:59,699
ABED demands that we

926
01:34:59,699 --> 01:35:01,399
disaggregate the data.

927
01:35:01,966 --> 01:35:02,833
How does that mean?

928
01:35:03,500 --> 01:35:06,933
That means that we can only consider

929
01:35:06,933 --> 01:35:10,866
students who are in our program when we

930
01:35:10,866 --> 01:35:13,100
do the analysis of our results.

931
01:35:13,966 --> 01:35:16,766
They don't want to contaminate it by a

932
01:35:16,766 --> 01:35:19,333
graduate student is in the class, by a

933
01:35:19,333 --> 01:35:20,566
student from another department.

934
01:35:21,733 --> 01:35:22,766
I still don't understand

935
01:35:22,766 --> 01:35:24,366
why, but we have to do it.

936
01:35:24,833 --> 01:35:26,866
If we know the unique names of every

937
01:35:26,866 --> 01:35:29,033
student, we know that information.

938
01:35:29,366 --> 01:35:30,466
We have a big data

939
01:35:30,466 --> 01:35:32,533
warehouse that's got all that stuff.

940
01:35:33,000 --> 01:35:35,666
And in the back end, we can easily filter

941
01:35:35,666 --> 01:35:37,833
the students who are in our program and

942
01:35:37,833 --> 01:35:40,433
not making a faculty

943
01:35:40,433 --> 01:35:42,399
member do that is insane.

944
01:35:43,033 --> 01:35:44,866
Because if you have 60 people in your

945
01:35:44,866 --> 01:35:46,766
class, you don't know who they you don't

946
01:35:46,766 --> 01:35:48,133
know what program they're in.

947
01:35:48,666 --> 01:35:50,133
And you don't even want to

948
01:35:50,133 --> 01:35:51,433
know what program they're in.

949
01:35:51,433 --> 01:35:53,333
You want to treat them all equally.

950
01:35:53,966 --> 01:35:55,533
You don't want to have a bias.

951
01:35:56,066 --> 01:35:57,466
Oh, they're graduate students.

952
01:35:57,466 --> 01:36:00,533
They should do more work or oh, these are

953
01:36:00,533 --> 01:36:03,399
students from, you know, from.

954
01:36:04,000 --> 01:36:05,833
Well, I won't say the name, but that

955
01:36:05,833 --> 01:36:08,100
other major that we like to pick on, you

956
01:36:08,100 --> 01:36:09,533
know, that we know we're going to you

957
01:36:09,533 --> 01:36:11,833
don't want to know that you want to know

958
01:36:11,833 --> 01:36:13,266
that there are students in your class.

959
01:36:13,966 --> 01:36:16,866
So and plus, it's really hard.

960
01:36:16,866 --> 01:36:18,033
A faculty member is going to have to

961
01:36:18,033 --> 01:36:20,033
download, you know, look up

962
01:36:20,033 --> 01:36:21,666
what program they're all in.

963
01:36:22,033 --> 01:36:22,699
What a pain.

964
01:36:23,233 --> 01:36:24,366
It's easier to just do your

965
01:36:24,366 --> 01:36:26,133
whole class and dump it in.

966
01:36:26,433 --> 01:36:27,533
And now we're

967
01:36:27,533 --> 01:36:29,333
launching a new way to do this.

968
01:36:29,666 --> 01:36:31,500
So if you have a spreadsheet, you just

969
01:36:31,500 --> 01:36:33,666
highlight the two columns, the unique

970
01:36:33,666 --> 01:36:35,233
names and the score.

971
01:36:35,233 --> 01:36:37,866
And we ask all faculty to put in a score

972
01:36:37,866 --> 01:36:41,399
normalized zero to 100 so we can combine

973
01:36:41,399 --> 01:36:44,666
the scores with other courses to see how

974
01:36:44,666 --> 01:36:46,133
they're doing across an outcome.

975
01:36:47,333 --> 01:36:48,233
They just highlight

976
01:36:48,233 --> 01:36:49,600
those things and copy.

977
01:36:50,366 --> 01:36:52,133
Go to a line. I never knew you could do

978
01:36:52,133 --> 01:36:54,433
this in a Google form right in the line.

979
01:36:54,433 --> 01:36:57,000
You just paste it and it's kind of like

980
01:36:57,000 --> 01:36:58,666
comma separated values.

981
01:36:59,366 --> 01:37:01,666
But in the back end, they use regular

982
01:37:01,666 --> 01:37:05,533
expressions to parse the data to put it

983
01:37:05,533 --> 01:37:06,699
back into a spreadsheet.

984
01:37:07,333 --> 01:37:08,466
We're doing this because these

985
01:37:08,466 --> 01:37:10,333
spreadsheets we were getting from faculty

986
01:37:10,333 --> 01:37:13,366
in the past were all over the map and it

987
01:37:13,366 --> 01:37:16,500
became really ugly for the back end

988
01:37:16,500 --> 01:37:18,533
people to deal with this data.

989
01:37:18,666 --> 01:37:19,199
So this should be

990
01:37:19,199 --> 01:37:21,333
much better at any rate.

991
01:37:22,666 --> 01:37:24,933
The way the reason it works, because the

992
01:37:24,933 --> 01:37:27,066
first thing you might think of is, oh, my

993
01:37:27,066 --> 01:37:28,766
God, that's a FERPA violation.

994
01:37:29,533 --> 01:37:32,033
That's student records. You can't make

995
01:37:32,033 --> 01:37:34,066
that public. And we don't.

996
01:37:34,733 --> 01:37:36,933
But the only way it works is because our

997
01:37:36,933 --> 01:37:40,933
university signed a legal agreement with

998
01:37:40,933 --> 01:37:45,000
Google to make sure that the instance of

999
01:37:45,000 --> 01:37:47,933
Google that we have is FERPA compliant.

1000
01:37:48,866 --> 01:37:51,733
So we're allowed to use Google mail from

1001
01:37:51,733 --> 01:37:54,500
the university to talk about student

1002
01:37:54,500 --> 01:37:55,833
issues and all of that.

1003
01:37:56,433 --> 01:37:58,699
If your university doesn't have that

1004
01:37:58,699 --> 01:38:00,300
agreement, you'll have

1005
01:38:00,300 --> 01:38:01,800
to find another solution.

1006
01:38:01,800 --> 01:38:04,533
But it works really well for us. And that

1007
01:38:04,533 --> 01:38:06,899
student identifier is incredible, as

1008
01:38:06,899 --> 01:38:08,166
you'll see in a few minutes.

1009
01:38:08,733 --> 01:38:10,666
So that's all. But from a faculty's point

1010
01:38:10,666 --> 01:38:13,066
of view, they just give that assessment,

1011
01:38:13,533 --> 01:38:16,466
collect that data, upload a PDF of their

1012
01:38:16,466 --> 01:38:18,533
metric and upload the student data.

1013
01:38:18,766 --> 01:38:20,833
And they're done. It takes very little

1014
01:38:20,833 --> 01:38:24,100
time. And believe me, I remind everybody

1015
01:38:24,100 --> 01:38:25,933
to make sure they do that assessment so

1016
01:38:25,933 --> 01:38:26,833
they have that data.

1017
01:38:27,533 --> 01:38:30,066
And then it works. And after we've done

1018
01:38:30,066 --> 01:38:33,066
this now for six and a half years now, we

1019
01:38:33,066 --> 01:38:36,000
have 100 percent compliance, except the

1020
01:38:36,000 --> 01:38:39,166
very first year, one of our 80 year old

1021
01:38:39,166 --> 01:38:41,433
faculty members who didn't understand

1022
01:38:41,433 --> 01:38:43,366
Google Forms didn't do it.

1023
01:38:44,333 --> 01:38:46,433
But that's pretty good. Yeah, I have to

1024
01:38:46,433 --> 01:38:47,666
say on the user side.

1025
01:38:48,300 --> 01:38:50,266
Being tasked with three outcomes per

1026
01:38:50,266 --> 01:38:53,466
semester due to the lab class, it takes

1027
01:38:53,466 --> 01:38:56,633
me an hour, maybe an hour and a half tops

1028
01:38:56,633 --> 01:38:58,399
to do this once a semester.

1029
01:38:59,000 --> 01:39:00,699
And as you said, I just I

1030
01:39:00,699 --> 01:39:03,533
have my assessment items.

1031
01:39:03,533 --> 01:39:04,333
They're already in our

1032
01:39:04,333 --> 01:39:05,399
learning management system.

1033
01:39:05,399 --> 01:39:07,899
So I just grab the student scores,

1034
01:39:08,600 --> 01:39:11,366
normalized to 100, grab the student IDs,

1035
01:39:12,233 --> 01:39:13,233
paste columns in a sheet,

1036
01:39:13,933 --> 01:39:16,566
upload to our collection form.

1037
01:39:16,766 --> 01:39:19,666
And I'm done. It's really quite painless.

1038
01:39:19,899 --> 01:39:21,566
Yeah. And from the ABET coordinator

1039
01:39:21,566 --> 01:39:23,566
point of view, I just go to the Google

1040
01:39:23,566 --> 01:39:25,633
spreadsheet that they build and I can

1041
01:39:25,633 --> 01:39:26,833
instantly see who's

1042
01:39:26,833 --> 01:39:28,333
done it and who hasn't.

1043
01:39:28,333 --> 01:39:29,800
I can click on the documents

1044
01:39:29,800 --> 01:39:31,333
and check them very quickly.

1045
01:39:31,899 --> 01:39:33,466
And if there's a problem with a faculty

1046
01:39:33,466 --> 01:39:35,066
member, I just go visit the faculty

1047
01:39:35,066 --> 01:39:37,733
member and talk to them

1048
01:39:37,733 --> 01:39:39,466
and help them make it better.

1049
01:39:40,133 --> 01:39:43,300
So it's great. So the really cool thing

1050
01:39:43,300 --> 01:39:45,933
is when it's all done, there's this

1051
01:39:45,933 --> 01:39:46,699
program called the A BET.

1052
01:39:46,733 --> 01:39:47,800
It's called Tableau that

1053
01:39:47,800 --> 01:39:50,100
I'm sure many places use.

1054
01:39:50,699 --> 01:39:52,500
It's, you know, a database program that

1055
01:39:52,500 --> 01:39:54,933
ingest spreadsheets and you can have you

1056
01:39:54,933 --> 01:39:57,466
can build a user UI that does different

1057
01:39:57,466 --> 01:39:58,933
filtering on what you see.

1058
01:39:59,600 --> 01:40:03,066
But the data that we care about, what we

1059
01:40:03,066 --> 01:40:06,233
want to know is we want to be able to see

1060
01:40:06,233 --> 01:40:10,066
a histogram of the actual values of the

1061
01:40:10,066 --> 01:40:11,566
scores that our students got,

1062
01:40:11,566 --> 01:40:12,899
where every single piece of

1063
01:40:12,899 --> 01:40:14,333
data is on that histogram.

1064
01:40:15,100 --> 01:40:17,466
In the old days, we used to report an

1065
01:40:17,466 --> 01:40:20,033
average and we would say if the average

1066
01:40:20,033 --> 01:40:21,766
is above this number, we're good.

1067
01:40:22,966 --> 01:40:26,000
And of course, A-BET evaluators through

1068
01:40:26,000 --> 01:40:28,766
no fault of ABET, but because evaluators

1069
01:40:28,766 --> 01:40:31,366
like to come up with new stuff, started

1070
01:40:31,366 --> 01:40:33,100
saying, yeah, well, what

1071
01:40:33,100 --> 01:40:34,500
about the standard deviation?

1072
01:40:35,133 --> 01:40:37,500
And what about the modality? And it's

1073
01:40:37,500 --> 01:40:38,766
like, yeah, well, what about it?

1074
01:40:39,199 --> 01:40:42,433
And, you know, eventually it's going to

1075
01:40:42,433 --> 01:40:46,633
become part of ABET evaluator lore that

1076
01:40:46,633 --> 01:40:48,233
if this isn't done, you're going to get a

1077
01:40:48,233 --> 01:40:49,300
shortcoming and there's

1078
01:40:49,300 --> 01:40:50,733
nothing you can do about it.

1079
01:40:51,199 --> 01:40:53,966
So how do you protect yourself against

1080
01:40:53,966 --> 01:40:58,266
what I call the ABET virus that rapidly,

1081
01:40:58,266 --> 01:41:02,833
you know, because, you know, 13 people,

1082
01:41:03,133 --> 01:41:05,199
visitors on a team hear

1083
01:41:05,199 --> 01:41:06,800
about some great idea,

1084
01:41:07,033 --> 01:41:07,966
which has nothing to

1085
01:41:07,966 --> 01:41:09,866
do with the real thing.

1086
01:41:10,833 --> 01:41:13,933
And then the next year they go to 13

1087
01:41:13,933 --> 01:41:15,666
different programs and they

1088
01:41:15,666 --> 01:41:17,666
infect all those new teams.

1089
01:41:17,866 --> 01:41:20,366
Well, when I did this, it was and all of

1090
01:41:20,366 --> 01:41:21,433
a sudden it's growing

1091
01:41:21,433 --> 01:41:23,933
exponentially like a virus.

1092
01:41:24,699 --> 01:41:25,500
So how do you protect

1093
01:41:25,500 --> 01:41:26,733
yourself against that?

1094
01:41:26,733 --> 01:41:28,366
Well, you know, we're scientists.

1095
01:41:29,066 --> 01:41:31,233
We know that the best thing to do always

1096
01:41:31,233 --> 01:41:32,766
is just to look at all the data.

1097
01:41:33,466 --> 01:41:36,533
And so what we plot is the histogram of

1098
01:41:36,533 --> 01:41:40,600
all the data so we instantly can see what

1099
01:41:40,600 --> 01:41:42,233
the standard deviation is, what the

1100
01:41:42,233 --> 01:41:44,466
modality is, if there's skewness, we can

1101
01:41:44,466 --> 01:41:46,566
look at the tails, you know, all of that.

1102
01:41:46,899 --> 01:41:48,733
We just see in an instant.

1103
01:41:49,666 --> 01:41:51,666
And in a way, it makes the analysis of

1104
01:41:51,666 --> 01:41:53,466
the data very straightforward because you

1105
01:41:53,466 --> 01:41:55,100
just look at it and you can tell.

1106
01:41:55,866 --> 01:41:58,433
So what we do is for each outcome, some

1107
01:41:58,433 --> 01:42:00,033
way of a spreadsheet and the

1108
01:42:00,033 --> 01:42:01,600
spreadsheet has little boxes.

1109
01:42:02,233 --> 01:42:05,066
And you first you choose what department.

1110
01:42:06,300 --> 01:42:08,466
And when you choose the department, you

1111
01:42:08,466 --> 01:42:09,199
can choose which

1112
01:42:09,199 --> 01:42:10,633
outcome you want to look at.

1113
01:42:11,033 --> 01:42:13,333
Then you choose which terms and you can

1114
01:42:13,333 --> 01:42:16,966
check boxes and choose, you know, like

1115
01:42:16,966 --> 01:42:19,966
one term or one academic year or two

1116
01:42:19,966 --> 01:42:21,166
academic years or

1117
01:42:21,166 --> 01:42:22,366
three, whatever you want.

1118
01:42:22,633 --> 01:42:24,100
And you'll pull in all the data.

1119
01:42:24,633 --> 01:42:26,199
Then you'll see for that particular

1120
01:42:26,199 --> 01:42:28,333
outcome, here's this

1121
01:42:28,333 --> 01:42:30,933
curve, which is a histogram.

1122
01:42:30,966 --> 01:42:32,533
And there's no curve drawn.

1123
01:42:32,533 --> 01:42:34,066
It's just the actual data.

1124
01:42:35,066 --> 01:42:36,666
And it gives you an average.

1125
01:42:37,266 --> 01:42:39,433
We have to filter out zeros sometimes

1126
01:42:39,433 --> 01:42:42,633
because when students drop a class and it

1127
01:42:42,633 --> 01:42:43,633
still shows up in the

1128
01:42:43,633 --> 01:42:45,199
instructors grade books.

1129
01:42:45,833 --> 01:42:47,633
So the zeros usually are not meaningful.

1130
01:42:48,000 --> 01:42:49,433
Sometimes they are, but whatever.

1131
01:42:51,266 --> 01:42:54,433
And then you do the magic because, like I

1132
01:42:54,433 --> 01:42:57,899
said, ABIT doesn't accredit departments,

1133
01:42:58,166 --> 01:42:59,600
they accredit programs.

1134
01:43:00,566 --> 01:43:02,966
And your department might have a course

1135
01:43:02,966 --> 01:43:05,699
like we have our structures course MSE

1136
01:43:05,699 --> 01:43:09,466
350 that has a huge number of students

1137
01:43:09,466 --> 01:43:11,566
who weren't in our program in that course

1138
01:43:11,566 --> 01:43:13,466
because we offer a minor.

1139
01:43:14,666 --> 01:43:16,833
And one of the requirements for the minor

1140
01:43:16,833 --> 01:43:18,233
is our structures course.

1141
01:43:18,933 --> 01:43:20,833
So we have students from biomedical

1142
01:43:20,833 --> 01:43:22,833
engineering, from aerospace, chemical

1143
01:43:22,833 --> 01:43:25,366
engineering, from mechanical engineering

1144
01:43:25,366 --> 01:43:27,233
all over the place that

1145
01:43:27,233 --> 01:43:28,166
are not in our program.

1146
01:43:29,066 --> 01:43:31,733
So when we have another checkbox that

1147
01:43:31,733 --> 01:43:33,066
says which program do

1148
01:43:33,066 --> 01:43:34,100
you want to filter by?

1149
01:43:34,100 --> 01:43:35,866
And we can choose our material science

1150
01:43:35,866 --> 01:43:38,399
and engineering undergraduate program.

1151
01:43:38,866 --> 01:43:41,066
And you'll want you click that button and

1152
01:43:41,066 --> 01:43:43,633
you watch the numbers drop because now

1153
01:43:43,633 --> 01:43:44,533
you're only getting the

1154
01:43:44,533 --> 01:43:46,199
students from your program.

1155
01:43:46,633 --> 01:43:50,066
And it allows us to then prepare these

1156
01:43:50,066 --> 01:43:52,733
histograms so that I can go to a faculty

1157
01:43:52,733 --> 01:43:54,633
meeting and I do this once a year.

1158
01:43:55,366 --> 01:43:57,766
And I show the entire faculty how are our

1159
01:43:57,766 --> 01:44:00,366
students doing on outcomes one through

1160
01:44:00,366 --> 01:44:03,533
seven for the last academic year of just

1161
01:44:03,533 --> 01:44:04,766
students in our program.

1162
01:44:05,533 --> 01:44:06,933
And we look at the data,

1163
01:44:07,199 --> 01:44:10,199
we discuss it and off we go.

1164
01:44:10,333 --> 01:44:11,666
And that's continuous improvement.

1165
01:44:12,066 --> 01:44:13,633
We've used it as input if

1166
01:44:13,633 --> 01:44:15,166
it's not going to help us.

1167
01:44:15,166 --> 01:44:17,533
That's OK. At least we've considered it

1168
01:44:17,533 --> 01:44:20,399
as input to our program at

1169
01:44:20,399 --> 01:44:21,733
the level that's appropriate.

1170
01:44:23,033 --> 01:44:24,033
So that's what we do.

1171
01:44:24,066 --> 01:44:27,466
Yeah. Well, the process sounds great.

1172
01:44:27,733 --> 01:44:31,233
I love. I actually enjoy that annual

1173
01:44:31,233 --> 01:44:33,100
faculty meeting where we look at the data

1174
01:44:33,100 --> 01:44:35,733
is we're visually interrogating this

1175
01:44:35,733 --> 01:44:39,633
database of student performance in a

1176
01:44:39,633 --> 01:44:40,833
variety of different ways.

1177
01:44:40,833 --> 01:44:43,233
And it's quite interesting to see where

1178
01:44:43,233 --> 01:44:46,433
the program is doing particularly well

1179
01:44:46,433 --> 01:44:47,466
and where there are

1180
01:44:47,466 --> 01:44:48,866
improvements to be made.

1181
01:44:48,933 --> 01:44:51,866
So I'm a fan, but that's just my opinion.

1182
01:44:53,066 --> 01:44:55,566
Would you say that this method is working

1183
01:44:55,566 --> 01:44:57,266
from the ABET point of view?

1184
01:44:57,266 --> 01:44:59,833
Are they satisfied with the process that

1185
01:44:59,833 --> 01:45:00,666
you've described here?

1186
01:45:01,800 --> 01:45:03,699
Yeah. Well, we just had our ABET review

1187
01:45:03,699 --> 01:45:07,866
last fall and we have 12 programs that

1188
01:45:07,866 --> 01:45:09,933
are accredited and 10

1189
01:45:09,933 --> 01:45:12,133
of those 12 got NGR.

1190
01:45:13,233 --> 01:45:14,899
No general. I forgot.

1191
01:45:15,433 --> 01:45:18,366
NGR is perfect. No shortcomings.

1192
01:45:19,566 --> 01:45:21,233
And so, yeah, I think it

1193
01:45:21,233 --> 01:45:22,600
worked really, really well.

1194
01:45:23,533 --> 01:45:25,233
Were there problems? Yeah.

1195
01:45:25,500 --> 01:45:27,166
That's why the two programs that got

1196
01:45:27,166 --> 01:45:29,199
dinged for shortcomings got dinged.

1197
01:45:29,833 --> 01:45:31,833
And guess what? They deserve to be dinged

1198
01:45:31,833 --> 01:45:36,233
because they just adopted our process.

1199
01:45:36,800 --> 01:45:39,133
But forgot a really important part.

1200
01:45:39,533 --> 01:45:42,866
They forgot to document how they did

1201
01:45:42,866 --> 01:45:44,966
continuous improvement of their program.

1202
01:45:45,333 --> 01:45:46,699
They just said, we did this

1203
01:45:46,699 --> 01:45:48,133
and it's not going to help.

1204
01:45:48,933 --> 01:45:51,500
That's not OK. You must show continuous

1205
01:45:51,500 --> 01:45:52,800
improvement of your program.

1206
01:45:53,300 --> 01:45:54,300
And they forgot to do

1207
01:45:54,300 --> 01:45:55,833
that. Now, they had done it.

1208
01:45:55,833 --> 01:45:57,300
They just didn't document it.

1209
01:45:57,300 --> 01:45:58,333
So they had to go back.

1210
01:45:59,066 --> 01:46:00,699
And of course, both of these programs had

1211
01:46:00,699 --> 01:46:01,366
pretty major

1212
01:46:01,366 --> 01:46:03,366
curricular reviews and changes.

1213
01:46:03,733 --> 01:46:04,633
And they just didn't

1214
01:46:04,633 --> 01:46:06,133
document it or write about it.

1215
01:46:06,533 --> 01:46:08,733
So it made them complacent because it

1216
01:46:08,733 --> 01:46:10,833
worked too well, I would say.

1217
01:46:11,166 --> 01:46:12,666
So don't get complacent.

1218
01:46:12,699 --> 01:46:15,866
You still have to make sure that you

1219
01:46:15,866 --> 01:46:16,899
document and

1220
01:46:16,899 --> 01:46:19,366
continuously improve your program.

1221
01:46:19,899 --> 01:46:20,833
And this is not going to

1222
01:46:20,833 --> 01:46:22,566
get rid of that part at all.

1223
01:46:25,366 --> 01:46:28,666
Well, it sounds like the process works as

1224
01:46:28,666 --> 01:46:29,566
long as you follow the

1225
01:46:29,566 --> 01:46:33,633
process, which sounds almost

1226
01:46:33,733 --> 01:46:35,333
tautological. I'll have to

1227
01:46:35,333 --> 01:46:36,866
logic that one out for a minute.

1228
01:46:38,500 --> 01:46:39,833
Well, how I think about

1229
01:46:39,833 --> 01:46:41,666
that, let me ask this.

1230
01:46:41,666 --> 01:46:44,033
You've mentioned that we sort of started

1231
01:46:44,033 --> 01:46:46,233
this process six years

1232
01:46:46,233 --> 01:46:47,466
ago, seven years ago now.

1233
01:46:48,566 --> 01:46:51,800
What do next steps look like for our

1234
01:46:51,800 --> 01:46:54,533
continuous improvement of our process for

1235
01:46:54,533 --> 01:46:55,633
continuous improvement?

1236
01:46:55,833 --> 01:46:58,066
How are we going to get better at

1237
01:46:58,066 --> 01:47:00,633
inoculating ourselves against ABET?

1238
01:47:02,033 --> 01:47:06,733
So when I look at these histograms, I

1239
01:47:06,733 --> 01:47:07,166
think when anyone

1240
01:47:07,166 --> 01:47:08,533
looks at these histograms,

1241
01:47:09,266 --> 01:47:16,866
although we have something like 90% of

1242
01:47:16,866 --> 01:47:18,733
our students are above

1243
01:47:18,733 --> 01:47:20,466
our threshold for minimal

1244
01:47:20,533 --> 01:47:23,366
acceptance, you can graduate from the

1245
01:47:23,366 --> 01:47:26,066
University of Michigan with a 2.0 GPA.

1246
01:47:27,633 --> 01:47:29,666
That's like a 60% for the

1247
01:47:29,666 --> 01:47:31,733
way most of us grade exams.

1248
01:47:32,966 --> 01:47:35,366
And so that's our minimum level because

1249
01:47:35,366 --> 01:47:37,033
that's just being honest,

1250
01:47:37,600 --> 01:47:40,166
because a 2.0 can graduate you.

1251
01:47:40,899 --> 01:47:43,866
But when you look at the data, there's

1252
01:47:43,866 --> 01:47:45,833
always a few stragglers in

1253
01:47:45,833 --> 01:47:47,433
the tails that are below that.

1254
01:47:48,566 --> 01:47:49,466
And you have to ask

1255
01:47:49,466 --> 01:47:51,066
yourself, what about them?

1256
01:47:52,033 --> 01:47:55,133
And so this has actually been a good

1257
01:47:55,133 --> 01:47:56,933
process by looking at

1258
01:47:56,933 --> 01:47:59,300
these histograms over and over.

1259
01:47:59,966 --> 01:48:02,466
It's made me think about what about that

1260
01:48:02,466 --> 01:48:04,800
group? Even at our

1261
01:48:04,800 --> 01:48:06,433
university level, we are

1262
01:48:07,399 --> 01:48:12,133
very, very proud of our graduation rate

1263
01:48:12,133 --> 01:48:13,366
at University of Michigan.

1264
01:48:14,366 --> 01:48:17,266
Our five-year graduation rate is like

1265
01:48:17,266 --> 01:48:21,666
93%. That's the envy of the whole world.

1266
01:48:22,733 --> 01:48:26,166
Only a very few schools can say that. I

1267
01:48:26,166 --> 01:48:27,600
remember going on college

1268
01:48:27,600 --> 01:48:29,833
visits with my daughter and the

1269
01:48:29,833 --> 01:48:31,733
people standing up and saying, "We are

1270
01:48:31,733 --> 01:48:33,266
very proud of our five-year

1271
01:48:33,266 --> 01:48:35,899
graduation rate. It's 65%."

1272
01:48:36,533 --> 01:48:39,733
And I'm like, "What? How can you be proud

1273
01:48:39,733 --> 01:48:40,699
of that?" And I came

1274
01:48:40,699 --> 01:48:42,533
back, and when I talked to

1275
01:48:42,966 --> 01:48:45,066
some of the deans about that, they said,

1276
01:48:45,066 --> 01:48:46,566
"Yeah, we're unusual."

1277
01:48:48,033 --> 01:48:54,233
Even with our amazing 93%, and by the

1278
01:48:54,233 --> 01:48:57,333
way, some of the 7% are

1279
01:48:57,333 --> 01:48:59,166
accounted for by people who dropped

1280
01:48:59,166 --> 01:49:01,466
out of engineering but still ended up

1281
01:49:01,466 --> 01:49:03,166
graduating from other

1282
01:49:03,166 --> 01:49:04,666
colleges at our university,

1283
01:49:06,466 --> 01:49:08,733
there's still the stragglers of the 3% to

1284
01:49:08,733 --> 01:49:11,800
5% that do leave our university.

1285
01:49:12,766 --> 01:49:15,800
I just went to a second provost seminar

1286
01:49:15,800 --> 01:49:18,366
on teaching where the

1287
01:49:18,366 --> 01:49:22,199
provost stood up and

1288
01:49:22,433 --> 01:49:25,600
talked about this and said, "What can we

1289
01:49:25,600 --> 01:49:28,366
do about that 3%? How can we

1290
01:49:28,366 --> 01:49:30,066
make sure that they graduate?

1291
01:49:30,633 --> 01:49:33,433
That should be our focus." And why

1292
01:49:33,433 --> 01:49:35,133
shouldn't that be our focus in our

1293
01:49:35,133 --> 01:49:37,933
program? And maybe we can use

1294
01:49:37,966 --> 01:49:40,633
the ABET data we're collecting to help us

1295
01:49:40,633 --> 01:49:41,899
understand that

1296
01:49:41,899 --> 01:49:43,666
because guess what? We know

1297
01:49:43,666 --> 01:49:46,366
who those people are. We haven't accessed

1298
01:49:46,366 --> 01:49:47,966
that data yet because

1299
01:49:47,966 --> 01:49:49,133
it's a little complicated.

1300
01:49:49,466 --> 01:49:52,366
You still, I believe, we would need IRB

1301
01:49:52,366 --> 01:49:54,366
approval to actually do a

1302
01:49:54,366 --> 01:49:55,433
study. I'm not sure. I'm going

1303
01:49:55,433 --> 01:49:58,233
to have to find out the rules. But a new

1304
01:49:58,233 --> 01:50:00,966
tool just came available

1305
01:50:00,966 --> 01:50:04,666
that might let us easily do this

1306
01:50:05,133 --> 01:50:08,066
without hiring education researchers or

1307
01:50:08,066 --> 01:50:12,766
data specialists. And that tool is MAIZY.

1308
01:50:13,699 --> 01:50:18,166
MAIZY is U of M's generative AI bot.

1309
01:50:18,533 --> 01:50:19,566
That's a private

1310
01:50:19,566 --> 01:50:22,333
generative AI bot system.

1311
01:50:22,966 --> 01:50:26,433
So it means we can look at that data

1312
01:50:26,433 --> 01:50:29,100
without exposing student

1313
01:50:29,100 --> 01:50:31,199
data to the world. We can stay

1314
01:50:31,433 --> 01:50:35,933
FERPA compliant. MAIZY has a way to link

1315
01:50:35,933 --> 01:50:39,600
to a SQL database, which is what our

1316
01:50:39,600 --> 01:50:40,633
whole data warehouse

1317
01:50:40,666 --> 01:50:44,133
is built on. And so I've been talking to

1318
01:50:44,133 --> 01:50:47,000
those folks and we're

1319
01:50:47,000 --> 01:50:49,966
going to see, can we ask the

1320
01:50:49,966 --> 01:50:56,066
generative AI bot questions like, the

1321
01:50:56,066 --> 01:50:59,300
students in the tails for outcome one,

1322
01:50:59,333 --> 01:51:02,766
you know, can we do a longitudinal study?

1323
01:51:02,766 --> 01:51:03,733
Because we have several

1324
01:51:03,733 --> 01:51:05,600
years of data from when there are

1325
01:51:05,600 --> 01:51:09,133
sophomores until they graduate. Do we see

1326
01:51:09,133 --> 01:51:11,033
them improving? Do they

1327
01:51:11,033 --> 01:51:13,033
graduate? What happens to

1328
01:51:13,033 --> 01:51:16,133
those students? Can we talk about race

1329
01:51:16,133 --> 01:51:18,633
and gender? Can we talk about first

1330
01:51:18,633 --> 01:51:19,766
generation students,

1331
01:51:20,466 --> 01:51:23,266
Pell Grant students? What can we learn

1332
01:51:23,266 --> 01:51:24,633
from the data of the

1333
01:51:24,633 --> 01:51:25,866
students in our tails?

1334
01:51:26,833 --> 01:51:30,966
That could significantly help us improve

1335
01:51:30,966 --> 01:51:33,333
our program. And I'll

1336
01:51:33,333 --> 01:51:34,766
tell you what, if we did it,

1337
01:51:35,033 --> 01:51:37,000
I'm not putting this in our self-study

1338
01:51:37,000 --> 01:51:38,733
because I'm not doing this

1339
01:51:38,733 --> 01:51:40,933
for ABET. I'm doing this for

1340
01:51:40,933 --> 01:51:43,033
ourselves because of what I said way

1341
01:51:43,033 --> 01:51:45,399
back. First, we want to improve our

1342
01:51:45,399 --> 01:51:47,066
program for ourselves and

1343
01:51:47,066 --> 01:51:49,266
worry about ABET later. But I think it's

1344
01:51:49,266 --> 01:51:50,899
a wonderful exercise, a

1345
01:51:50,899 --> 01:51:52,466
wonderful way to use our data.

1346
01:51:52,533 --> 01:51:55,633
I should also mention that our Center for

1347
01:51:55,633 --> 01:51:57,733
Research on Learning and Teaching,

1348
01:51:58,133 --> 01:52:01,100
CRLT, which is solely focused on

1349
01:52:01,100 --> 01:52:03,633
improving education in Michigan, when

1350
01:52:03,633 --> 01:52:04,966
they saw our system,

1351
01:52:04,966 --> 01:52:07,133
they came and asked me, can we use your

1352
01:52:07,133 --> 01:52:08,666
system where we put in,

1353
01:52:08,666 --> 01:52:10,466
you know, our own questions?

1354
01:52:10,466 --> 01:52:12,933
Because they're doing curricular revision

1355
01:52:12,933 --> 01:52:14,133
pilots with a few

1356
01:52:14,133 --> 01:52:15,533
different programs or departments

1357
01:52:16,066 --> 01:52:18,233
because they do care about departments.

1358
01:52:19,233 --> 01:52:21,566
And so they're going

1359
01:52:21,566 --> 01:52:22,933
to be using our system

1360
01:52:23,733 --> 01:52:25,666
for their purposes, even though it has

1361
01:52:25,666 --> 01:52:27,966
nothing to do with ABET. So I think it's

1362
01:52:27,966 --> 01:52:29,133
really, really cool.

1363
01:52:29,133 --> 01:52:30,666
We have a lot of data. We've got data

1364
01:52:30,666 --> 01:52:33,533
from, you know, 13,000

1365
01:52:33,533 --> 01:52:35,166
students in this last pass.

1366
01:52:36,000 --> 01:52:38,199
In any given term, we're looking at 3,000

1367
01:52:38,199 --> 01:52:39,766
students. That's a lot

1368
01:52:39,766 --> 01:52:41,266
of data. And it's only

1369
01:52:41,333 --> 01:52:43,066
growing because I think other programs

1370
01:52:43,066 --> 01:52:44,733
have learned how valuable

1371
01:52:44,733 --> 01:52:48,266
this is. So that's what we do

1372
01:52:48,966 --> 01:52:51,933
for our ABET outcomes assessment. And

1373
01:52:51,933 --> 01:52:53,266
it's made getting

1374
01:52:53,266 --> 01:52:56,199
through criterion four a breeze.

1375
01:52:57,233 --> 01:53:00,266
I really love how this program

1376
01:53:00,266 --> 01:53:02,166
essentially started as, let's make sure

1377
01:53:02,166 --> 01:53:03,633
bridges don't fall down.

1378
01:53:03,899 --> 01:53:06,766
And now we're at a place with it where we

1379
01:53:06,766 --> 01:53:08,566
can interrogate this

1380
01:53:08,566 --> 01:53:10,166
really rich, robust data set

1381
01:53:10,233 --> 01:53:12,366
to ask questions about equity in our

1382
01:53:12,366 --> 01:53:14,133
programs and to say, how can

1383
01:53:14,133 --> 01:53:16,366
we better serve the students

1384
01:53:16,366 --> 01:53:18,066
who we are not currently serving well

1385
01:53:18,066 --> 01:53:19,233
enough? I think that's

1386
01:53:19,233 --> 01:53:22,966
fantastic. Well, that is all the

1387
01:53:22,966 --> 01:53:24,833
time that we have for today, but that was

1388
01:53:24,833 --> 01:53:27,866
a pretty good day one deep dive into

1389
01:53:27,866 --> 01:53:30,666
ABET. I think we might

1390
01:53:30,666 --> 01:53:32,433
have to revisit this topic in the future

1391
01:53:32,433 --> 01:53:33,899
now that we're asking interesting

1392
01:53:33,899 --> 01:53:35,866
questions about what we

1393
01:53:35,866 --> 01:53:37,866
can do with everything that we've learned

1394
01:53:37,866 --> 01:53:39,433
from these self-studies and

1395
01:53:39,433 --> 01:53:40,633
how we can use it exactly,

1396
01:53:40,866 --> 01:53:42,566
as you said, to make our own programs

1397
01:53:42,566 --> 01:53:43,566
better for ourselves

1398
01:53:43,566 --> 01:53:44,533
and for our students.

1399
01:53:45,166 --> 01:53:47,033
And I should mention, so, you know, I

1400
01:53:47,033 --> 01:53:49,166
agree, Tim. I'd like to,

1401
01:53:49,166 --> 01:53:50,566
you know, maybe bring on,

1402
01:53:50,766 --> 01:53:53,399
you know, some people from ABET. I'd like

1403
01:53:53,399 --> 01:53:54,533
to see if Jeff Fergus

1404
01:53:54,533 --> 01:53:56,000
can talk to us. Jeff Fergus

1405
01:53:56,533 --> 01:53:59,000
has probably been one of the most

1406
01:53:59,000 --> 01:54:00,666
influential people from the

1407
01:54:00,666 --> 01:54:03,066
materials community at ABET.

1408
01:54:03,699 --> 01:54:08,633
And I really think his principles and

1409
01:54:08,633 --> 01:54:10,133
values align closely with

1410
01:54:10,133 --> 01:54:13,633
mine. You know, Jeff was the

1411
01:54:14,133 --> 01:54:16,699
head of the EAC, the Engineering

1412
01:54:16,699 --> 01:54:19,466
Accreditation Commission. He's been in

1413
01:54:19,466 --> 01:54:20,766
charge of training for

1414
01:54:20,766 --> 01:54:23,433
ABET. So he's very, very well known at

1415
01:54:23,433 --> 01:54:25,899
ABET. And so I think it'd be

1416
01:54:25,899 --> 01:54:28,566
great to get his ideas because,

1417
01:54:28,766 --> 01:54:30,366
you know, one of the reasons we're doing

1418
01:54:30,366 --> 01:54:31,933
this podcast is to try to

1419
01:54:31,933 --> 01:54:33,566
share best practices with

1420
01:54:33,566 --> 01:54:36,933
our materials community. And to that end,

1421
01:54:37,166 --> 01:54:39,100
I hope everybody comes to

1422
01:54:39,100 --> 01:54:41,033
the North American Materials

1423
01:54:41,466 --> 01:54:43,133
Education Symposium. There's the plug.

1424
01:54:43,666 --> 01:54:45,699
It's a plug because on

1425
01:54:45,699 --> 01:54:47,733
Friday, I'm going to be doing a

1426
01:54:47,866 --> 01:54:51,066
free workshop on, I will show everybody

1427
01:54:51,066 --> 01:54:52,566
our outcomes assessment

1428
01:54:52,566 --> 01:54:55,766
stuff in detail. You get

1429
01:54:55,833 --> 01:54:58,833
to play with our Tableau system and I'm

1430
01:54:58,833 --> 01:54:59,933
available to give, you

1431
01:54:59,933 --> 01:55:01,233
know, to answer any questions,

1432
01:55:01,699 --> 01:55:04,600
not from a official ABET standpoint,

1433
01:55:04,600 --> 01:55:05,433
because I don't have any

1434
01:55:05,433 --> 01:55:07,399
official standing in ABET,

1435
01:55:07,733 --> 01:55:10,866
but just from a colleague in materials

1436
01:55:10,866 --> 01:55:13,133
who would love to help all of our

1437
01:55:13,133 --> 01:55:15,733
programs get through ABET

1438
01:55:15,866 --> 01:55:18,666
with a minimal amount of work and do a

1439
01:55:18,666 --> 01:55:20,666
maximal job on their ABET.

1440
01:55:21,066 --> 01:55:22,733
So come to our symposium,

1441
01:55:23,033 --> 01:55:26,233
get your chair to send you and we'll help

1442
01:55:26,233 --> 01:55:27,533
you with ABET for free.

1443
01:55:28,399 --> 01:55:30,633
Excellent. Well, thanks for being in the

1444
01:55:30,633 --> 01:55:32,766
hot seat today, Steve, and

1445
01:55:32,766 --> 01:55:33,933
to everyone else out there in

1446
01:55:33,933 --> 01:55:35,766
the world. We'll see you next time on

1447
01:55:35,766 --> 01:55:37,300
another episode of

1448
01:55:37,300 --> 01:55:38,866
Undercooled. See you later.