1 01:00:00,000 --> 01:00:12,111 [MUSIC] 2 01:00:13,666 --> 01:00:14,111 Well, hello. 3 01:00:14,777 --> 01:00:15,500 Welcome to another 4 01:00:15,500 --> 01:00:18,000 episode of Undercooled. 5 01:00:18,277 --> 01:00:19,166 Today, Tim and I are 6 01:00:19,166 --> 01:00:20,444 gonna talk about grading. 7 01:00:21,000 --> 01:00:21,333 Why? 8 01:00:21,666 --> 01:00:23,277 Because we just graded our 9 01:00:23,277 --> 01:00:25,722 class from the end of the term, 10 01:00:25,722 --> 01:00:28,555 and it's foremost on our minds. 11 01:00:29,222 --> 01:00:32,500 So why don't we start by talking about 12 01:00:32,500 --> 01:00:36,388 what grading is, why it exists, 13 01:00:37,000 --> 01:00:38,777 and maybe some of the 14 01:00:38,777 --> 01:00:40,333 problems we've identified. 15 01:00:40,888 --> 01:00:43,611 So we'll start with what's wrong with 16 01:00:43,611 --> 01:00:44,833 traditional grading, 17 01:00:45,388 --> 01:00:47,611 and let's start with curving. 18 01:00:48,555 --> 01:00:50,388 So Tim, what do you think about curves? 19 01:00:51,944 --> 01:00:52,833 I think they're very 20 01:00:52,833 --> 01:00:53,833 statistically interesting, 21 01:00:54,444 --> 01:00:57,833 and they reflect a few assumptions and 22 01:00:57,833 --> 01:00:59,333 biases in our grading system. 23 01:01:00,000 --> 01:01:02,111 First is this idea that there should even 24 01:01:02,111 --> 01:01:03,944 be a Gaussian distribution 25 01:01:04,500 --> 01:01:05,611 in our populations 26 01:01:05,611 --> 01:01:06,777 that we're working with. 27 01:01:07,444 --> 01:01:09,888 And like a lot of these assumptions about 28 01:01:09,888 --> 01:01:11,611 grading, there's a nugget of truth to it. 29 01:01:11,833 --> 01:01:13,388 If you looked at the entire nation, 30 01:01:13,777 --> 01:01:15,166 there'd probably be something pretty 31 01:01:15,166 --> 01:01:16,055 close to a normal 32 01:01:16,055 --> 01:01:18,388 distribution of proficiency in 33 01:01:18,388 --> 01:01:19,277 lots of different areas. 34 01:01:19,944 --> 01:01:23,111 But at a university, we're performing 35 01:01:23,111 --> 01:01:24,666 this biased selection 36 01:01:24,666 --> 01:01:26,222 from a certain subset of 37 01:01:26,222 --> 01:01:28,388 the population to try to get the kind of 38 01:01:28,388 --> 01:01:29,333 people that we want to get. 39 01:01:30,000 --> 01:01:32,500 So believing that there should be an 40 01:01:32,500 --> 01:01:34,055 underlying normal distribution 41 01:01:34,055 --> 01:01:36,222 is already quite a flawed model. 42 01:01:36,666 --> 01:01:39,166 We tend to have a much more skewed 43 01:01:39,166 --> 01:01:40,611 distribution with the 44 01:01:40,611 --> 01:01:41,444 students that we have, 45 01:01:41,444 --> 01:01:43,333 because we're selecting who we admit, 46 01:01:43,777 --> 01:01:44,944 some of those selection criteria. 47 01:01:45,555 --> 01:01:46,277 Actually, you know, that's 48 01:01:46,277 --> 01:01:47,722 another episode entirely. 49 01:01:48,000 --> 01:01:49,888 We can talk about admissions another day, 50 01:01:50,666 --> 01:01:51,944 but we don't have a Gaussian 51 01:01:51,944 --> 01:01:53,166 distribution to begin with. 52 01:01:53,166 --> 01:01:54,777 So why would you try to enforce one? 53 01:01:55,222 --> 01:01:56,722 I think curving is 54 01:01:56,722 --> 01:01:58,388 pretty silly in that sense. 55 01:02:00,166 --> 01:02:03,222 Yet we're always told that you should fit 56 01:02:03,222 --> 01:02:05,000 your distribution to a bell curve, 57 01:02:05,000 --> 01:02:06,000 which is a Gaussian 58 01:02:06,000 --> 01:02:07,277 distribution, of course. 59 01:02:07,500 --> 01:02:10,388 And that's the standard way that most 60 01:02:10,388 --> 01:02:11,833 faculty just approach it. 61 01:02:12,000 --> 01:02:15,277 And it's a shame because it's really 62 01:02:15,277 --> 01:02:16,333 flawed thinking like 63 01:02:16,333 --> 01:02:17,277 you just pointed out. 64 01:02:18,777 --> 01:02:19,000 Yeah. 65 01:02:19,277 --> 01:02:21,222 And there's another problem there, which 66 01:02:21,222 --> 01:02:22,722 is to a lot of students, 67 01:02:22,722 --> 01:02:25,277 this conveys a very competitive, 68 01:02:25,722 --> 01:02:27,666 anti-collaborative mindset, right? 69 01:02:28,055 --> 01:02:28,777 If they're going to 70 01:02:28,777 --> 01:02:29,944 be stuck onto a curve, 71 01:02:30,333 --> 01:02:31,833 then they have to out-compete their 72 01:02:31,833 --> 01:02:33,833 classmates in order to get a grade, 73 01:02:34,222 --> 01:02:35,555 which maybe they should or should not 74 01:02:35,555 --> 01:02:36,333 actually care about, 75 01:02:36,333 --> 01:02:37,944 but intrinsically a lot of them do. 76 01:02:38,722 --> 01:02:41,555 And as soon as you throw away the curve, 77 01:02:41,555 --> 01:02:42,444 you can get to a space 78 01:02:42,444 --> 01:02:44,055 where everyone can succeed 79 01:02:44,055 --> 01:02:45,055 and everyone can help each 80 01:02:45,055 --> 01:02:46,277 other achieve at a higher level 81 01:02:46,444 --> 01:02:49,166 to actually meet and hopefully exceed 82 01:02:49,166 --> 01:02:50,555 your expectations for the course. 83 01:02:51,888 --> 01:02:53,277 Yeah. That's so true. 84 01:02:54,222 --> 01:02:56,277 Whoever said that only 10% 85 01:02:56,611 --> 01:02:57,944 of the class should get A's. 86 01:02:58,333 --> 01:02:58,944 What does that mean? 87 01:02:59,722 --> 01:03:02,222 You know, if you take teaching 88 01:03:02,222 --> 01:03:04,500 seriously, like I think we do, 89 01:03:05,111 --> 01:03:06,222 you want everybody to 90 01:03:06,222 --> 01:03:07,222 be able to get an A. 91 01:03:07,722 --> 01:03:10,388 But you have to have the bar pretty high 92 01:03:10,388 --> 01:03:13,333 so that everybody actually earns an A. 93 01:03:13,666 --> 01:03:14,666 But our goal should be 94 01:03:14,666 --> 01:03:15,666 to get everybody there. 95 01:03:16,111 --> 01:03:18,611 Why have an arbitrary number that makes 96 01:03:18,611 --> 01:03:19,944 absolutely no sense. 97 01:03:21,000 --> 01:03:25,055 And another thing that makes absolutely 98 01:03:25,055 --> 01:03:26,833 no sense is why we're 99 01:03:26,833 --> 01:03:28,444 so focused on teaching 100 01:03:28,888 --> 01:03:32,055 grading with a zero to 100 scale, 101 01:03:33,000 --> 01:03:35,388 as if we have the precision to say 102 01:03:35,388 --> 01:03:36,666 anything about that. 103 01:03:37,277 --> 01:03:39,555 So if you think about it, when we grade 104 01:03:39,555 --> 01:03:41,555 on a zero to 100 scale, 105 01:03:42,222 --> 01:03:45,055 we normally look at the standard grading, 106 01:03:45,333 --> 01:03:48,777 which says you need a 93% to get an A, a 107 01:03:48,777 --> 01:03:50,777 90% to get an A minus, 108 01:03:51,388 --> 01:03:54,722 an 87% to get a B plus, et cetera. 109 01:03:55,722 --> 01:03:57,555 Why do we have those numbers? 110 01:03:58,388 --> 01:04:00,888 And when we do it like that, what happens 111 01:04:00,888 --> 01:04:03,944 is the entire class gets compressed 112 01:04:04,388 --> 01:04:07,944 between usually 75 and 100. 113 01:04:08,833 --> 01:04:10,777 And then we look at all the scores, 114 01:04:10,777 --> 01:04:12,444 especially in a large class, 115 01:04:12,888 --> 01:04:16,111 and we look for natural breaks because 116 01:04:16,111 --> 01:04:18,777 we're so omniscient 117 01:04:18,777 --> 01:04:21,500 about what these things mean. 118 01:04:22,000 --> 01:04:23,277 And we know that this 119 01:04:23,277 --> 01:04:26,111 moves people to the right area. 120 01:04:26,833 --> 01:04:29,666 So we sit there and we look at grades and 121 01:04:29,666 --> 01:04:31,055 we look at the boundaries 122 01:04:31,055 --> 01:04:33,055 when we do our final grades, 123 01:04:33,222 --> 01:04:35,388 because we usually never publish these 124 01:04:35,388 --> 01:04:36,055 because then the 125 01:04:36,055 --> 01:04:37,555 students will scream at us. 126 01:04:38,333 --> 01:04:40,833 And we look and see, 127 01:04:40,833 --> 01:04:46,333 well, this person had a 89.95. 128 01:04:47,277 --> 01:04:49,222 And okay, you go, well, that's pretty 129 01:04:49,222 --> 01:04:51,333 close to a 90. Let's give them a 90. 130 01:04:51,833 --> 01:04:54,555 Except you realize in your class of 150 131 01:04:54,555 --> 01:04:58,000 students that you've got 10 students, 132 01:04:59,111 --> 01:05:03,611 four of which have 89.95. 133 01:05:04,388 --> 01:05:09,000 And then just below that, you have 89.89. 134 01:05:09,944 --> 01:05:12,333 And you go, ah, that's a big jump. 135 01:05:13,777 --> 01:05:14,777 But think about that. 136 01:05:15,388 --> 01:05:24,166 That is 89.95 versus 89.99 even. 137 01:05:24,388 --> 01:05:29,055 That is 0.04%. 138 01:05:33,111 --> 01:05:37,555 So the fraction is 0.0004. 139 01:05:40,277 --> 01:05:43,777 How in our wildest dreams do we feel that 140 01:05:43,777 --> 01:05:44,833 that is significant? 141 01:05:45,555 --> 01:05:47,722 If we were to put error bars on this and 142 01:05:47,722 --> 01:05:48,777 think thoughtfully of 143 01:05:48,777 --> 01:05:50,055 what those error bars mean, 144 01:05:51,055 --> 01:05:54,000 probably we can only measure learning 145 01:05:54,000 --> 01:05:55,666 through exams and 146 01:05:55,666 --> 01:05:57,166 grades and stuff like that 147 01:05:57,666 --> 01:06:02,166 to at best plus or minus 10%, which is a 148 01:06:02,166 --> 01:06:05,000 lot bigger than 0.04%. 149 01:06:06,777 --> 01:06:08,611 Yeah, there's a great irony there that 150 01:06:08,611 --> 01:06:10,944 especially in a lab context, 151 01:06:10,944 --> 01:06:13,500 I'm always mentioning to students the 152 01:06:13,500 --> 01:06:14,777 importance of significant figures 153 01:06:14,777 --> 01:06:15,611 and thinking about the 154 01:06:15,611 --> 01:06:16,611 precision of your measurement 155 01:06:16,888 --> 01:06:19,111 and how the number of digits you use 156 01:06:19,111 --> 01:06:21,777 conveys the intrinsic precision of the 157 01:06:21,777 --> 01:06:22,722 tools that you're using 158 01:06:23,277 --> 01:06:26,500 and for significant figures on a grade. 159 01:06:28,388 --> 01:06:29,722 That's borderline meaningless. 160 01:06:30,333 --> 01:06:31,833 I could be convinced we have two sig 161 01:06:31,833 --> 01:06:35,166 figs, as you said, to order of 10% error. 162 01:06:35,333 --> 01:06:36,555 Yeah, that's actually pretty achievable. 163 01:06:37,444 --> 01:06:39,555 And hard to do but possible. 164 01:06:41,444 --> 01:06:43,055 But if we were going to grade ourselves 165 01:06:43,055 --> 01:06:44,277 on that kind of 166 01:06:44,277 --> 01:06:45,944 effort, we'd probably fail. 167 01:06:47,166 --> 01:06:51,444 And yet this is done in just about every 168 01:06:51,444 --> 01:06:53,611 class all over the world. 169 01:06:54,388 --> 01:06:55,277 And it's absurd. 170 01:06:55,777 --> 01:06:56,833 And it's not right. 171 01:06:57,277 --> 01:07:00,000 And so unless we can find a way to 172 01:07:00,000 --> 01:07:04,611 magically improve the precision of our 173 01:07:04,611 --> 01:07:05,722 measurement of learning, 174 01:07:06,222 --> 01:07:08,777 which maybe can be done, I mean, you can 175 01:07:08,777 --> 01:07:11,555 have precise measurements that are good 176 01:07:11,555 --> 01:07:13,333 down to 10 to the minus 7. 177 01:07:13,777 --> 01:07:16,388 Delta E over E or 178 01:07:16,388 --> 01:07:17,555 however you want to measure it. 179 01:07:18,111 --> 01:07:19,888 Look at the LIGO project, measuring 180 01:07:19,888 --> 01:07:21,055 gravitational waves. 181 01:07:21,500 --> 01:07:23,055 That was the hero 182 01:07:23,055 --> 01:07:25,833 project of improving precision. 183 01:07:26,111 --> 01:07:27,555 Unbelievable what they did. 184 01:07:28,388 --> 01:07:30,777 But that took billions of dollars and it 185 01:07:30,777 --> 01:07:33,000 was in an area where you actually could 186 01:07:33,000 --> 01:07:34,833 make those measurements and 187 01:07:34,833 --> 01:07:36,888 could improve the precision. 188 01:07:37,611 --> 01:07:40,055 I'd argue it's a lot harder to measure 189 01:07:40,055 --> 01:07:41,944 learning than it is to measure 190 01:07:41,944 --> 01:07:42,555 gravitational fields. 191 01:07:42,722 --> 01:07:47,888 And so why are we even doing it? 192 01:07:48,388 --> 01:07:50,555 So the simplest thing, of course, to do 193 01:07:50,555 --> 01:07:53,111 is not grade on a zero to 100 scale, 194 01:07:53,722 --> 01:07:56,500 but something like zero to 10 or even 195 01:07:56,500 --> 01:08:01,722 zero, one and two or a five point scale 196 01:08:01,722 --> 01:08:02,944 or something like that, 197 01:08:02,944 --> 01:08:06,666 where the data is a more meaningful 198 01:08:06,666 --> 01:08:09,722 reflection of the actual precision that we're trying to measure. 199 01:08:10,388 --> 01:08:16,333 And so, but it still comes back to how do 200 01:08:16,333 --> 01:08:17,888 we measure learning? 201 01:08:18,833 --> 01:08:21,222 And that's a deep subject. 202 01:08:21,777 --> 01:08:24,333 And I'd argue that is the hardest problem 203 01:08:24,333 --> 01:08:26,888 in education is how do 204 01:08:26,888 --> 01:08:27,777 you measure learning? 205 01:08:27,777 --> 01:08:28,944 And I wish more people 206 01:08:28,944 --> 01:08:32,166 studied it, but they really don't. 207 01:08:32,388 --> 01:08:34,111 They just keep giving tests. 208 01:08:34,944 --> 01:08:35,833 So, and tests are a problem. 209 01:08:35,944 --> 01:08:38,444 Maybe you can talk about 210 01:08:38,444 --> 01:08:39,888 the problem with exams. 211 01:08:41,333 --> 01:08:41,888 Oh, goodness. 212 01:08:43,611 --> 01:08:45,500 I, you know, I first have to come from 213 01:08:45,500 --> 01:08:48,222 the research perspective, having done 214 01:08:48,222 --> 01:08:50,055 education research for enough years, 215 01:08:50,055 --> 01:08:53,444 that there is this gap between measuring 216 01:08:53,444 --> 01:08:56,444 learning in a rigorous 217 01:08:56,444 --> 01:08:57,833 research context versus 218 01:08:57,833 --> 01:09:00,166 what you have to do to survive as a 219 01:09:00,166 --> 01:09:01,833 teacher with hundreds of 220 01:09:01,833 --> 01:09:02,222 students in one semester, 221 01:09:02,277 --> 01:09:08,611 developing a tool to precisely and 222 01:09:08,611 --> 01:09:10,777 reliably and validly measure student 223 01:09:10,777 --> 01:09:12,611 learning in one small topic 224 01:09:12,611 --> 01:09:14,500 area is a years long effort. 225 01:09:14,777 --> 01:09:16,833 And that's not something that any of us 226 01:09:16,833 --> 01:09:21,944 have the luxury of time to do when we're 227 01:09:21,944 --> 01:09:23,611 just trying to get through our day 228 01:09:23,611 --> 01:09:25,722 and make sure, hopefully try to make sure 229 01:09:25,722 --> 01:09:27,222 that our students are learning something 230 01:09:27,222 --> 01:09:29,777 rather than spending all of our time 231 01:09:29,777 --> 01:09:30,555 developing measurement tools. 232 01:09:31,222 --> 01:09:35,444 You know, these measurements can be done, 233 01:09:35,944 --> 01:09:38,944 but I don't think it's practical to 234 01:09:38,944 --> 01:09:42,666 expect most practicing teachers to get 235 01:09:42,666 --> 01:09:45,722 into so much of the minutia of the 236 01:09:45,722 --> 01:09:48,222 psychometrics and the statistics and the 237 01:09:48,222 --> 01:09:50,222 validation procedures to say, 238 01:09:50,222 --> 01:09:51,333 "Yes, I'm actually 239 01:09:51,333 --> 01:09:52,833 measuring learning with this test." 240 01:09:52,833 --> 01:09:53,333 That's just not 241 01:09:53,333 --> 01:09:54,777 something we can practically do. 242 01:09:55,666 --> 01:09:58,055 So instead, you have to ask, "How should 243 01:09:58,055 --> 01:09:58,388 we assess our students? 244 01:09:59,111 --> 01:10:03,500 What can we do as regular practitioners 245 01:10:03,500 --> 01:10:05,944 of this craft of teaching to say, "I 246 01:10:05,944 --> 01:10:08,388 think I have a gauge, you know, a certain 247 01:10:08,388 --> 01:10:11,166 level of understanding of which of my 248 01:10:11,166 --> 01:10:12,555 students have learned how much." 249 01:10:13,833 --> 01:10:16,611 So the other problem is that almost all 250 01:10:16,611 --> 01:10:20,555 exams are focused on having a student 251 01:10:20,555 --> 01:10:22,666 demonstrate a certain level of knowledge 252 01:10:22,666 --> 01:10:25,000 on the day and time of that exam. 253 01:10:25,777 --> 01:10:28,777 And very little thought usually goes into 254 01:10:28,777 --> 01:10:31,277 the idea of retention. 255 01:10:32,444 --> 01:10:35,222 And this is famously born out with Father 256 01:10:35,222 --> 01:10:38,722 Guido Sarducci's little comedy skit, The 257 01:10:38,722 --> 01:10:40,166 Five-Minute University. 258 01:10:40,166 --> 01:10:41,444 I have none of you out 259 01:10:41,444 --> 01:10:42,888 there have ever seen that. 260 01:10:42,888 --> 01:10:44,944 Just type into Google, "Five-Minute 261 01:10:44,944 --> 01:10:47,666 University," or in YouTube, and watch it. 262 01:10:48,555 --> 01:10:51,000 And he makes the point that, you know, in 263 01:10:51,000 --> 01:10:54,500 five minutes, he can teach students what 264 01:10:54,500 --> 01:10:57,111 the average college graduate remembers 265 01:10:57,111 --> 01:10:59,777 five years after they leave college. 266 01:11:00,555 --> 01:11:03,222 And sadly, it's kind of true. 267 01:11:03,222 --> 01:11:05,222 And there's a large body of educational 268 01:11:05,222 --> 01:11:09,000 research that bears this out, that by 269 01:11:09,000 --> 01:11:11,888 giving students exams that are largely, 270 01:11:11,888 --> 01:11:15,500 largely factual recall and retrieval, 271 01:11:16,333 --> 01:11:18,500 has caused generations of 272 01:11:18,500 --> 01:11:20,277 students, myself included, 273 01:11:21,666 --> 01:11:23,944 headed to the library two days before the 274 01:11:23,944 --> 01:11:27,555 exam to cram like crazy to remember all 275 01:11:27,555 --> 01:11:29,611 the things that I might be tested on. 276 01:11:29,611 --> 01:11:32,277 Then I go and do a data dump on the test. 277 01:11:32,500 --> 01:11:35,666 I do really well, but they show that even 278 01:11:35,666 --> 01:11:39,388 two hours later, the students couldn't 279 01:11:39,388 --> 01:11:42,055 even do 50% retrieval 280 01:11:42,055 --> 01:11:43,000 of what they just did. 281 01:11:43,777 --> 01:11:46,777 And two months later, it's down to like, 282 01:11:46,777 --> 01:11:49,111 you know, 20% or some crazy low number. 283 01:11:50,000 --> 01:11:53,055 And so that's the whole point of Father 284 01:11:53,055 --> 01:11:55,333 Guido Sarducci's little skit. 285 01:11:56,111 --> 01:11:59,666 And why don't we think about retention? 286 01:11:59,666 --> 01:12:01,333 Retention is what really matters. 287 01:12:02,333 --> 01:12:04,000 Now, in some fields where it's 288 01:12:04,000 --> 01:12:06,166 conceptually based, we 289 01:12:06,166 --> 01:12:07,500 can think about concepts. 290 01:12:07,944 --> 01:12:10,111 And if you can understand a concept, the 291 01:12:10,111 --> 01:12:11,888 chances are you'll retain that 292 01:12:11,888 --> 01:12:13,333 understanding much 293 01:12:13,333 --> 01:12:15,277 longer than just memorization. 294 01:12:16,222 --> 01:12:18,444 And so the physics community has come up 295 01:12:18,444 --> 01:12:20,611 with concept tests, which is really an 296 01:12:20,611 --> 01:12:22,055 excellent way to do that. 297 01:12:22,833 --> 01:12:25,444 But it's not so easy to translate that to 298 01:12:25,444 --> 01:12:28,222 other fields because most of the concepts 299 01:12:28,222 --> 01:12:30,111 in material science, for instance, 300 01:12:30,611 --> 01:12:31,611 are really chemistry 301 01:12:31,611 --> 01:12:33,333 concepts or physics concepts. 302 01:12:34,500 --> 01:12:36,277 And we don't test those. 303 01:12:36,722 --> 01:12:40,055 We try to test the ability to use those 304 01:12:40,055 --> 01:12:42,388 concepts in the context of material 305 01:12:42,388 --> 01:12:45,666 science to be able to answer a question. 306 01:12:46,944 --> 01:12:48,833 Like earlier, we were just playing around 307 01:12:48,833 --> 01:12:51,722 with the new voice model for chat GPT. 308 01:12:52,555 --> 01:12:54,944 And Tim, you asked it about Martin site. 309 01:12:55,277 --> 01:12:55,777 What is it? 310 01:12:56,555 --> 01:12:59,000 And chat GPT was confused. 311 01:12:59,333 --> 01:13:02,333 It just said it's an interesting phase, 312 01:13:02,333 --> 01:13:03,722 but I don't really see it 313 01:13:03,722 --> 01:13:05,000 on any of the phase diagrams. 314 01:13:05,222 --> 01:13:07,055 So I'm not quite sure what it is. 315 01:13:08,055 --> 01:13:10,722 And so chat GPT just scrubbing the 316 01:13:10,722 --> 01:13:13,000 internet didn't get that 317 01:13:13,000 --> 01:13:14,277 it's a metastable phase. 318 01:13:14,500 --> 01:13:16,666 And you don't see metastable phases on 319 01:13:16,666 --> 01:13:18,111 equilibrium phase diagrams. 320 01:13:18,333 --> 01:13:19,500 And that's the whole point. 321 01:13:19,500 --> 01:13:21,277 So there's an underlying concept there. 322 01:13:22,055 --> 01:13:24,944 And so that's why chat GTP didn't get it 323 01:13:24,944 --> 01:13:25,388 because it doesn't 324 01:13:25,388 --> 01:13:27,000 really understand anything. 325 01:13:27,000 --> 01:13:28,444 It just repeats things. 326 01:13:29,555 --> 01:13:35,388 So giving exams is a dangerous way, in my 327 01:13:35,388 --> 01:13:38,722 opinion, to assess learning. 328 01:13:39,611 --> 01:13:42,277 And, you know, there is one type of exam 329 01:13:42,277 --> 01:13:43,944 that's fantastic for actually 330 01:13:43,944 --> 01:13:44,611 understanding whether 331 01:13:44,611 --> 01:13:45,722 your students know something. 332 01:13:46,333 --> 01:13:46,833 That's right. 333 01:13:46,833 --> 01:13:48,833 But that's not a written exam, is it? 334 01:13:49,500 --> 01:13:49,833 No. 335 01:13:50,055 --> 01:13:50,222 Yeah. 336 01:13:50,222 --> 01:13:50,666 I think you know 337 01:13:50,666 --> 01:13:51,722 where I'm going with this. 338 01:13:52,833 --> 01:13:53,222 Go ahead. 339 01:13:55,333 --> 01:13:56,833 Oral exams can be so powerful. 340 01:13:57,500 --> 01:13:58,611 Both on the student side 341 01:13:58,611 --> 01:13:59,833 and on the teacher side. 342 01:14:00,888 --> 01:14:02,666 You know, this ability to interact with 343 01:14:02,666 --> 01:14:06,055 someone in real time to ask them 344 01:14:06,055 --> 01:14:07,444 questions, to give them hints, 345 01:14:07,444 --> 01:14:10,277 to see exactly where in the process of 346 01:14:10,277 --> 01:14:11,444 working out a problem in 347 01:14:11,444 --> 01:14:13,444 real time, where they get stuck, 348 01:14:13,444 --> 01:14:14,777 where they have moments of insight. 349 01:14:15,222 --> 01:14:18,611 That can be very powerful for revealing 350 01:14:18,611 --> 01:14:20,222 learning in our students. 351 01:14:20,944 --> 01:14:23,333 And it's the original 352 01:14:23,333 --> 01:14:24,722 adaptive testing, right? 353 01:14:25,222 --> 01:14:25,500 Yeah. 354 01:14:25,500 --> 01:14:26,388 The student is crushing it. 355 01:14:26,388 --> 01:14:27,611 You make the question harder. 356 01:14:27,777 --> 01:14:29,111 You see how far they can get. 357 01:14:29,111 --> 01:14:30,055 The student is struggling. 358 01:14:30,222 --> 01:14:31,833 You back off and you give them support 359 01:14:31,833 --> 01:14:33,666 and you see what you can 360 01:14:33,666 --> 01:14:35,277 help them with collaboratively. 361 01:14:36,444 --> 01:14:36,833 That's right. 362 01:14:37,777 --> 01:14:39,166 We're trying to develop all this 363 01:14:39,166 --> 01:14:41,388 technology to replicate that experience. 364 01:14:43,277 --> 01:14:47,222 But either way, the oral exam, as good as 365 01:14:47,222 --> 01:14:49,666 it is, is not very scalable. 366 01:14:50,166 --> 01:14:51,777 If you've got a class of 100 students, 367 01:14:52,388 --> 01:14:53,722 how are you going to have 100 368 01:14:53,722 --> 01:14:54,833 half-hour-long conversations 369 01:14:55,277 --> 01:14:56,444 with all of the 370 01:14:56,444 --> 01:14:57,666 individual people in your class? 371 01:14:58,444 --> 01:14:59,833 And that's the key point. 372 01:15:00,500 --> 01:15:02,777 Some people say, "Oh, you only have to 373 01:15:02,777 --> 01:15:05,944 give a 10-minute oral exam 374 01:15:05,944 --> 01:15:07,222 and you can learn everything." 375 01:15:07,777 --> 01:15:09,222 I'm not sure I agree with that. 376 01:15:09,833 --> 01:15:11,388 And then they say, "Well, to grade 377 01:15:11,388 --> 01:15:13,388 somebody's paper takes about 10 minutes, 378 01:15:13,388 --> 01:15:14,444 so it should be scalable." 379 01:15:15,333 --> 01:15:16,888 But I think you said the key thing. 380 01:15:16,888 --> 01:15:18,055 It kind of takes more like a 381 01:15:18,055 --> 01:15:20,000 half an hour for every student. 382 01:15:20,666 --> 01:15:25,833 Because all of us who went and got PhDs, 383 01:15:26,166 --> 01:15:27,444 most of us had a PhD 384 01:15:27,444 --> 01:15:29,833 defense, which is an oral exam. 385 01:15:30,277 --> 01:15:32,166 And that takes at least an hour, 386 01:15:32,166 --> 01:15:33,666 sometimes two hours. 387 01:15:33,944 --> 01:15:35,388 But that's a big body of work. 388 01:15:35,944 --> 01:15:37,888 And I'll never forget, I was a math major 389 01:15:37,888 --> 01:15:39,611 as an undergrad, and I 390 01:15:39,611 --> 01:15:41,166 had a really interesting 391 01:15:41,222 --> 01:15:44,000 professor who he was in the National 392 01:15:44,000 --> 01:15:45,055 Academy of Sciences. 393 01:15:45,944 --> 01:15:48,000 And he used to tell us 394 01:15:48,000 --> 01:15:49,555 about thought process. 395 01:15:49,611 --> 01:15:50,722 He brought up this thing, 396 01:15:50,722 --> 01:15:52,277 the Epicurean model of thought. 397 01:15:53,111 --> 01:15:54,722 And so he thought that when you were 398 01:15:54,722 --> 01:15:56,944 trying to solve, prove a 399 01:15:56,944 --> 01:15:59,277 theorem, very abstract, right? 400 01:15:59,500 --> 01:16:02,777 And you would sit there and have this 401 01:16:02,777 --> 01:16:04,277 experience of pounding your 402 01:16:04,277 --> 01:16:05,500 head on the table and getting 403 01:16:05,500 --> 01:16:08,444 very, very frustrated while you tried to 404 01:16:08,444 --> 01:16:09,555 figure out the problem. 405 01:16:09,944 --> 01:16:12,222 And often the best thing to do would be 406 01:16:12,222 --> 01:16:14,055 to work for a few hours 407 01:16:14,055 --> 01:16:15,111 and then go take a break, 408 01:16:15,111 --> 01:16:16,111 do something else, go 409 01:16:16,111 --> 01:16:16,888 to the bar, whatever. 410 01:16:17,666 --> 01:16:20,555 And maybe in the middle of the earth, of 411 01:16:20,555 --> 01:16:22,222 sleeping that night, you'll 412 01:16:22,222 --> 01:16:24,277 wake up with this aha moment. 413 01:16:25,055 --> 01:16:27,111 And he viewed that as the, he called it 414 01:16:27,111 --> 01:16:29,055 the Epicurean model of thought. 415 01:16:29,055 --> 01:16:30,000 I have no idea if this is 416 01:16:30,000 --> 01:16:31,666 accurate, but it's a good model. 417 01:16:32,333 --> 01:16:34,055 While you're pounding your head on the 418 01:16:34,055 --> 01:16:36,611 table, you are taking 419 01:16:36,611 --> 01:16:39,611 relevant ideas off that pegboard in 420 01:16:39,611 --> 01:16:41,888 your brain and throwing them into a bingo 421 01:16:41,888 --> 01:16:43,944 pot where like the 422 01:16:43,944 --> 01:16:45,111 kinetic theory of gases, 423 01:16:45,222 --> 01:16:47,666 you get random connections of 424 01:16:47,666 --> 01:16:49,333 these ideas coming together. 425 01:16:49,944 --> 01:16:52,222 And when the right idea comes together by 426 01:16:52,222 --> 01:16:55,055 random, the power of the 427 01:16:55,055 --> 01:16:56,944 aesthetics of that beautiful 428 01:16:57,222 --> 01:17:00,222 solution are so powerful, they crash 429 01:17:00,222 --> 01:17:01,888 through to your conscious mind. 430 01:17:02,944 --> 01:17:06,000 So he gave us math exams. 431 01:17:06,277 --> 01:17:08,333 We only had 14 kids in the class because 432 01:17:08,333 --> 01:17:09,666 it was, you know, who was a math major. 433 01:17:10,333 --> 01:17:14,277 And he would give us a piece of chalk and 434 01:17:14,277 --> 01:17:15,777 the chalkboard and he'd 435 01:17:15,777 --> 01:17:17,277 tell us something to prove 436 01:17:17,277 --> 01:17:19,388 that was pretty easy. And we'd start 437 01:17:19,388 --> 01:17:21,000 writing on the board and doing it. 438 01:17:21,000 --> 01:17:22,944 As soon as he realized that we knew what 439 01:17:22,944 --> 01:17:24,111 we were doing, he 440 01:17:24,111 --> 01:17:25,055 said, oh, enough of that. 441 01:17:25,444 --> 01:17:28,000 Let's do this. And you knew you did well. 442 01:17:28,333 --> 01:17:30,277 If you could never finish any of the 443 01:17:30,277 --> 01:17:31,166 things he asked you 444 01:17:31,166 --> 01:17:32,500 because he could tell that you 445 01:17:32,500 --> 01:17:34,277 knew how to do it. So I waste the time 446 01:17:34,277 --> 01:17:35,611 and he wanted to push you 447 01:17:35,611 --> 01:17:37,166 and he'd eventually get you 448 01:17:37,222 --> 01:17:39,277 to a place where you had no idea what to 449 01:17:39,277 --> 01:17:41,000 do. And then he'd start to 450 01:17:41,000 --> 01:17:43,500 drop hints and see how we did 451 01:17:43,500 --> 01:17:45,944 with the hints. And when it was all done, 452 01:17:45,944 --> 01:17:47,222 because this was a very 453 01:17:47,222 --> 01:17:50,555 subjective exam, he gave us two 454 01:17:50,555 --> 01:17:53,611 grades and those two grades were averaged 455 01:17:53,611 --> 01:17:54,833 for our final grade. And 456 01:17:54,833 --> 01:17:59,111 those two grades were how good 457 01:17:59,444 --> 01:18:03,722 was our conscious thinking and how good 458 01:18:03,722 --> 01:18:04,888 the other grade was, how 459 01:18:04,888 --> 01:18:06,055 good was our subconscious 460 01:18:06,555 --> 01:18:09,166 thinking going back to this Epicurean 461 01:18:09,166 --> 01:18:10,666 model that he told us about. 462 01:18:11,611 --> 01:18:14,500 And that was where he gave us 463 01:18:14,500 --> 01:18:17,888 hints. How could we synthesize those to 464 01:18:17,888 --> 01:18:18,666 crash through to our 465 01:18:18,666 --> 01:18:20,222 conscious mind to do the technical 466 01:18:20,333 --> 01:18:23,444 aspects of actually doing the proof. And 467 01:18:23,444 --> 01:18:25,277 so it was pretty amazing. 468 01:18:26,500 --> 01:18:27,888 But that took a lot of time. 469 01:18:27,888 --> 01:18:29,555 I mean, I was in there with him for over 470 01:18:29,555 --> 01:18:32,777 a half an hour and I got a 471 01:18:32,777 --> 01:18:34,222 C for my conscious thinking, 472 01:18:34,500 --> 01:18:36,222 but I got an A for my subconscious 473 01:18:36,222 --> 01:18:38,000 thinking. So I got a B for 474 01:18:38,000 --> 01:18:40,388 the exam and I was pretty happy, 475 01:18:40,777 --> 01:18:42,944 but I learned a lot. It was an amazing 476 01:18:42,944 --> 01:18:45,555 experience, but it's just 477 01:18:45,555 --> 01:18:47,611 not scalable to 150 people. 478 01:18:49,000 --> 01:18:51,500 That's just too much time. 479 01:18:53,166 --> 01:18:55,833 You know, that reminds me of one question 480 01:18:55,833 --> 01:19:00,555 from my PhD qualifying exam. And 481 01:19:01,500 --> 01:19:04,944 the question was very simple. You have a 482 01:19:04,944 --> 01:19:06,777 gas of charged particles, 483 01:19:07,611 --> 01:19:09,000 the box that the gases in 484 01:19:09,000 --> 01:19:11,500 expands, does the temperature go up or 485 01:19:11,500 --> 01:19:15,000 down? And so of course, having done 486 01:19:15,000 --> 01:19:16,277 however many years of 487 01:19:16,277 --> 01:19:17,777 physics at that point, I went straight to 488 01:19:17,777 --> 01:19:18,722 the nuclear option. I'm 489 01:19:18,722 --> 01:19:19,555 like, I'm going to write the 490 01:19:19,555 --> 01:19:21,055 partition function for this thing. 491 01:19:21,333 --> 01:19:23,277 Because if I know that I can calculate 492 01:19:23,277 --> 01:19:24,777 anything. So, you know, 493 01:19:24,777 --> 01:19:27,722 I go through the jumping through the 494 01:19:27,722 --> 01:19:29,666 hoops of writing down the 495 01:19:29,666 --> 01:19:31,500 states of the system. And I 496 01:19:31,500 --> 01:19:33,833 start to write integral of, and then I 497 01:19:33,833 --> 01:19:35,055 have this moment of I'm 498 01:19:35,055 --> 01:19:36,388 never going to be able to do the 499 01:19:36,388 --> 01:19:38,277 math for this. This is completely 500 01:19:38,277 --> 01:19:40,000 intractable what I've set 501 01:19:40,000 --> 01:19:42,777 myself up for. And one of the 502 01:19:42,888 --> 01:19:44,833 people on the committee saw me have that 503 01:19:44,833 --> 01:19:46,222 moment where I had 504 01:19:46,222 --> 01:19:48,388 finally realized that, you know, 505 01:19:48,666 --> 01:19:50,166 integrating this by hand was not the 506 01:19:50,166 --> 01:19:51,500 correct approach. And he 507 01:19:51,500 --> 01:19:53,166 says, Okay, good. So now just 508 01:19:53,222 --> 01:19:55,277 think about it physically. Where is the 509 01:19:55,277 --> 01:19:56,722 energy in the system? And 510 01:19:56,722 --> 01:19:59,277 I'm like, Oh, yeah, huh. And 511 01:19:59,777 --> 01:20:01,388 five minutes later, I was at the answer. 512 01:20:01,388 --> 01:20:03,944 But, you know, it was just 513 01:20:03,944 --> 01:20:05,555 one of those experiences that 514 01:20:05,555 --> 01:20:08,111 sticks with you. Yeah, I wish we could 515 01:20:08,111 --> 01:20:09,222 have more of our students to 516 01:20:09,222 --> 01:20:10,333 have more of those epiphanies 517 01:20:10,500 --> 01:20:13,111 on their exams instead of just grinding 518 01:20:13,111 --> 01:20:14,944 out written calculations. 519 01:20:15,777 --> 01:20:17,055 And speaking of written, 520 01:20:17,444 --> 01:20:20,000 another option is to think more 521 01:20:20,000 --> 01:20:23,166 physically and have them actually write 522 01:20:23,166 --> 01:20:25,555 essays and write papers 523 01:20:26,055 --> 01:20:29,000 about the concepts that underlie it. And 524 01:20:29,000 --> 01:20:31,111 this used to be a pretty good way. Now 525 01:20:31,111 --> 01:20:32,333 that also takes a lot 526 01:20:32,333 --> 01:20:36,055 of time to grade. That's not so easy. But 527 01:20:36,055 --> 01:20:37,277 the whole world has changed 528 01:20:37,277 --> 01:20:39,277 in the last year. First of all, 529 01:20:39,277 --> 01:20:41,888 now chat GPT will help the students write 530 01:20:41,888 --> 01:20:44,388 those without necessarily 531 01:20:44,388 --> 01:20:46,444 showing that they understood 532 01:20:46,722 --> 01:20:49,944 it, but it will also let the instructor 533 01:20:49,944 --> 01:20:51,388 grade them more 534 01:20:51,388 --> 01:20:54,333 accurately. And maybe there's a way 535 01:20:54,333 --> 01:20:57,388 with the oral exam, if you record it, 536 01:20:57,388 --> 01:20:58,888 that we could transcribe 537 01:20:58,888 --> 01:21:03,611 it and, you know, grade it 538 01:21:03,611 --> 01:21:05,388 automatically. But unfortunately, you 539 01:21:05,388 --> 01:21:06,222 still have to sit there 540 01:21:06,222 --> 01:21:07,777 through the whole oral exam and 541 01:21:07,777 --> 01:21:11,055 participate in the interaction. So until 542 01:21:11,055 --> 01:21:13,444 chat GTP voice models can 543 01:21:13,444 --> 01:21:15,000 actually do that interrogation 544 01:21:15,500 --> 01:21:17,722 on a curated set of data, so it's 545 01:21:17,722 --> 01:21:20,833 accurate, and be able to do assessment, 546 01:21:21,555 --> 01:21:22,833 this hasn't happened yet. 547 01:21:23,500 --> 01:21:26,722 Maybe it can. Maybe chat GPT will finally 548 01:21:26,722 --> 01:21:30,222 give us a way to evaluate 549 01:21:30,222 --> 01:21:31,666 the quality of learning. 550 01:21:32,777 --> 01:21:36,111 But I think that's a long way off. But 551 01:21:36,111 --> 01:21:36,777 it's a nice thought. 552 01:21:36,777 --> 01:21:44,222 At any rate, so we've talked a lot about 553 01:21:44,222 --> 01:21:45,111 what's wrong with 554 01:21:45,111 --> 01:21:47,055 grading, what's wrong with exams, 555 01:21:48,000 --> 01:21:50,555 why, you know, oral exams, which are 556 01:21:50,555 --> 01:21:53,722 great, really aren't scalable. But what 557 01:21:53,722 --> 01:21:55,388 do we do? Are there new 558 01:21:55,388 --> 01:21:58,833 methods out there? And when I say new 559 01:21:58,833 --> 01:22:00,222 methods, I've got to put that 560 01:22:00,222 --> 01:22:02,611 in quotes because there are, 561 01:22:02,833 --> 01:22:04,944 but they're really not that new. They've 562 01:22:04,944 --> 01:22:06,166 been around for a while. 563 01:22:06,611 --> 01:22:09,333 And so, for example, you know, 564 01:22:09,333 --> 01:22:11,500 people talk a lot about mastery grading 565 01:22:11,500 --> 01:22:13,444 as one of these new things 566 01:22:13,444 --> 01:22:15,055 that we're going to do. But if 567 01:22:15,055 --> 01:22:17,666 you go back in the literature, the person 568 01:22:17,666 --> 01:22:20,000 who coined the phrase 569 01:22:20,000 --> 01:22:21,722 mastery learning and first wrote 570 01:22:21,777 --> 01:22:24,000 about it was none other than Benjamin 571 01:22:24,000 --> 01:22:26,555 Bloom, who's very famous for his Bloom's 572 01:22:26,555 --> 01:22:27,888 taxonomy back in the 573 01:22:27,944 --> 01:22:32,555 60s. And so, but mastery grading is 574 01:22:32,555 --> 01:22:35,666 starting to gain a lot of traction. So 575 01:22:35,666 --> 01:22:37,722 why don't you tell us 576 01:22:37,722 --> 01:22:40,277 what you think mastery grading means? 577 01:22:40,777 --> 01:22:44,111 Interesting. So this is not 578 01:22:44,111 --> 01:22:45,944 something I use in my classes. 579 01:22:47,277 --> 01:22:49,722 We can get to that. But my understanding 580 01:22:49,722 --> 01:22:51,444 of mastery grading is that 581 01:22:51,444 --> 01:22:53,666 you have your course broken into 582 01:22:54,333 --> 01:22:58,388 many small components. And for each of 583 01:22:58,388 --> 01:23:00,333 those components, you're 584 01:23:00,333 --> 01:23:01,888 assessing the students almost 585 01:23:01,888 --> 01:23:05,388 on a binary scale of have you achieved a 586 01:23:05,388 --> 01:23:07,388 sufficient level of proficiency on this 587 01:23:07,388 --> 01:23:09,166 or have you not? And 588 01:23:09,166 --> 01:23:11,500 then by totaling how many course 589 01:23:11,500 --> 01:23:13,777 components students achieve mastery of 590 01:23:13,777 --> 01:23:14,833 over the course of the 591 01:23:15,166 --> 01:23:16,944 term, the quarter semester year, whatever 592 01:23:16,944 --> 01:23:19,333 it is, then you can assign 593 01:23:19,333 --> 01:23:20,777 a final grade based on how 594 01:23:20,777 --> 01:23:22,277 much of the course content they have 595 01:23:22,277 --> 01:23:24,166 mastered. Is that how you 596 01:23:24,166 --> 01:23:25,666 implement it in your classes? 597 01:23:27,166 --> 01:23:30,555 Sort of, but I'd say that the K through 598 01:23:30,555 --> 01:23:31,666 12 community has been 599 01:23:31,666 --> 01:23:33,111 using this for a long time, 600 01:23:33,444 --> 01:23:36,000 especially with the younger students. And 601 01:23:36,000 --> 01:23:37,277 so I know when my kids 602 01:23:37,277 --> 01:23:38,166 were going to school, 603 01:23:38,666 --> 01:23:40,333 they weren't getting grades until like 604 01:23:40,333 --> 01:23:41,555 after fourth grade or 605 01:23:41,555 --> 01:23:43,444 fifth grade until then they 606 01:23:43,444 --> 01:23:46,444 were just getting one of three things 607 01:23:46,444 --> 01:23:50,444 exceeds expectation meets 608 01:23:50,444 --> 01:23:53,166 expectations or is approaching 609 01:23:53,333 --> 01:23:56,555 expectations because the idea wasn't to 610 01:23:56,555 --> 01:23:58,500 stigmatize the student with something 611 01:23:58,500 --> 01:24:00,277 horrible, but rather 612 01:24:00,333 --> 01:24:03,166 talk about where they are relative to 613 01:24:03,166 --> 01:24:04,611 what the teachers wanted 614 01:24:04,611 --> 01:24:07,166 them to do to give them flags so 615 01:24:07,166 --> 01:24:08,777 that they knew who to give more attention 616 01:24:08,777 --> 01:24:10,166 to. Cause you know, our 617 01:24:10,166 --> 01:24:11,555 public school system doesn't have 618 01:24:12,388 --> 01:24:14,166 that much money. Teachers aren't paid 619 01:24:14,166 --> 01:24:16,166 that well. And most teachers find 620 01:24:16,166 --> 01:24:17,277 themselves in classes 621 01:24:17,777 --> 01:24:19,833 that are way too big for them to 622 01:24:19,833 --> 01:24:22,444 adequately teach. So they do 623 01:24:22,444 --> 01:24:24,388 a triage and that's exactly 624 01:24:24,388 --> 01:24:27,555 what that is. You know, except unlike a 625 01:24:27,555 --> 01:24:29,888 triage, the worst group, 626 01:24:29,888 --> 01:24:31,222 they don't just leave to die. 627 01:24:31,500 --> 01:24:33,166 They actually spend more time on that 628 01:24:33,166 --> 01:24:34,666 worst group, a little more 629 01:24:34,666 --> 01:24:36,000 time with the people who are 630 01:24:36,666 --> 01:24:38,666 just starting to meet those expectations. 631 01:24:39,277 --> 01:24:40,500 And then maybe even enlist 632 01:24:40,500 --> 01:24:42,277 the help of those students who 633 01:24:42,277 --> 01:24:44,833 exceed to help the younger students do a 634 01:24:44,833 --> 01:24:46,555 little bit better. That's kind of the 635 01:24:46,555 --> 01:24:47,777 idea of the one room 636 01:24:48,388 --> 01:24:50,666 schoolroom, the schoolhouse, right? Where 637 01:24:50,666 --> 01:24:52,222 all grades used to be mixed 638 01:24:52,222 --> 01:24:53,888 together. The older students 639 01:24:53,888 --> 01:24:57,555 would help the younger students. And, you 640 01:24:57,555 --> 01:24:58,444 know, it's a pretty 641 01:24:58,444 --> 01:25:00,277 powerful technique and I think it 642 01:25:00,444 --> 01:25:03,222 works really well. So why don't we use 643 01:25:03,222 --> 01:25:04,944 that in higher education? 644 01:25:05,888 --> 01:25:08,555 That's kind of my thought on 645 01:25:08,555 --> 01:25:11,333 the whole thing. Well, since you gave me 646 01:25:11,333 --> 01:25:12,222 the perfect launching 647 01:25:12,222 --> 01:25:13,666 point, I'll have to say that's 648 01:25:13,666 --> 01:25:16,722 very close to what I do in my classes. I 649 01:25:16,722 --> 01:25:18,277 am bringing together a few of 650 01:25:18,277 --> 01:25:19,444 the topics we covered today. 651 01:25:20,000 --> 01:25:21,944 Most of my assignments are scored on a 652 01:25:21,944 --> 01:25:23,777 five point scale because 653 01:25:23,777 --> 01:25:25,277 that's course enough that I can 654 01:25:25,277 --> 01:25:27,833 actually be reliable about it. I can 655 01:25:27,833 --> 01:25:28,888 distinguish between a four 656 01:25:28,888 --> 01:25:30,277 and a five, a four and a three. 657 01:25:31,222 --> 01:25:35,666 And my criteria are exactly what you've 658 01:25:35,666 --> 01:25:36,611 just stated. I have 659 01:25:36,611 --> 01:25:38,333 meets expectations, exceeds, 660 01:25:38,611 --> 01:25:41,277 does not meet. And then descriptive 661 01:25:41,277 --> 01:25:43,222 rubrics that say, here's what I'm 662 01:25:43,222 --> 01:25:44,333 expecting. If you achieve 663 01:25:44,333 --> 01:25:46,333 this, that's a four. If you exceed that, 664 01:25:46,333 --> 01:25:48,944 that's a five. And there is a lot of 665 01:25:48,944 --> 01:25:50,388 student angst around 666 01:25:50,500 --> 01:25:52,111 that, especially at the start of the 667 01:25:52,111 --> 01:25:53,611 semester. Oh my God, I got a 668 01:25:53,611 --> 01:25:54,944 three out of five. I'm failing 669 01:25:54,944 --> 01:25:57,222 out of school. No, it means you're almost 670 01:25:57,222 --> 01:25:58,222 there. You're not quite 671 01:25:58,222 --> 01:26:01,277 there yet. And it's something 672 01:26:01,277 --> 01:26:05,055 that I find makes the grading go a lot 673 01:26:05,055 --> 01:26:07,277 smoother because I can focus on the 674 01:26:07,277 --> 01:26:08,722 qualitative feedback 675 01:26:08,722 --> 01:26:11,111 that I give to the students about where 676 01:26:11,111 --> 01:26:12,222 they succeeded and where 677 01:26:12,222 --> 01:26:13,222 they still need to develop 678 01:26:13,222 --> 01:26:15,222 more as opposed to spending my effort 679 01:26:15,222 --> 01:26:16,222 trying to turn it into a 680 01:26:16,222 --> 01:26:18,888 number. Right. And of course, 681 01:26:18,944 --> 01:26:22,166 by using rubrics, you're moving towards 682 01:26:22,166 --> 01:26:23,777 another form of mastery 683 01:26:23,777 --> 01:26:26,166 grading. You're probably doing 684 01:26:26,166 --> 01:26:28,055 something closer to specifications 685 01:26:28,055 --> 01:26:30,666 grading. Yes, that's 686 01:26:30,666 --> 01:26:32,222 right. So I'm, I'm giving the 687 01:26:32,222 --> 01:26:34,444 specifications for what the different 688 01:26:34,444 --> 01:26:35,888 levels of achievement are for each 689 01:26:35,888 --> 01:26:37,444 assignment. And then 690 01:26:38,222 --> 01:26:40,388 just comparing what the students produce 691 01:26:40,388 --> 01:26:41,722 to those specifications. 692 01:26:42,555 --> 01:26:46,000 So you've put in a lot of transparency to 693 01:26:46,000 --> 01:26:47,500 your grading, even though 694 01:26:47,500 --> 01:26:48,777 the students may be fearful 695 01:26:48,888 --> 01:26:51,444 because it's new, it's actually a lot 696 01:26:51,444 --> 01:26:55,000 less opaque than just, you 697 01:26:55,000 --> 01:26:57,111 know, getting a percentage on an 698 01:26:57,111 --> 01:27:00,000 exam where then the professor arbitrarily 699 01:27:00,000 --> 01:27:03,555 draws the lines later on. There's no 700 01:27:03,555 --> 01:27:04,833 transparency to that. 701 01:27:06,777 --> 01:27:10,000 So that's pretty cool. And then there's 702 01:27:10,000 --> 01:27:11,333 another movement that 703 01:27:11,333 --> 01:27:13,777 has grown up around mastery 704 01:27:14,000 --> 01:27:16,555 grading specifications, grading, and 705 01:27:16,555 --> 01:27:19,333 there was a book published that talks 706 01:27:19,333 --> 01:27:20,722 about this quite a bit, 707 01:27:21,055 --> 01:27:24,055 and it's called ungrading. And the 708 01:27:24,055 --> 01:27:26,055 ungrading movement is also 709 01:27:26,055 --> 01:27:29,444 not a new idea. For example, 710 01:27:29,444 --> 01:27:33,055 the president of University of Chicago 711 01:27:33,055 --> 01:27:35,333 decided that grades were 712 01:27:35,333 --> 01:27:38,222 evil back in the 1930s, and he 713 01:27:38,277 --> 01:27:41,444 banned grades, but that only lasted for 714 01:27:41,444 --> 01:27:42,555 about a year before they 715 01:27:42,555 --> 01:27:45,055 threw the guy out because none of 716 01:27:45,055 --> 01:27:47,222 the faculty could deal with it. But the 717 01:27:47,222 --> 01:27:48,500 truth is that if you go 718 01:27:48,500 --> 01:27:51,611 way back, grades didn't exist 719 01:27:52,500 --> 01:27:56,222 until the late 1800s. And they were 720 01:27:56,222 --> 01:27:58,277 started in England at 721 01:27:58,277 --> 01:27:59,333 Cambridge and Oxford, 722 01:28:00,000 --> 01:28:03,333 not grades, but levels, right? You 723 01:28:03,333 --> 01:28:06,000 achieve some level. And do you 724 01:28:06,000 --> 01:28:07,555 know where grades were? A, B, 725 01:28:07,611 --> 01:28:10,111 C, D were first instituted in the United 726 01:28:10,111 --> 01:28:11,555 States in the whole world. 727 01:28:13,611 --> 01:28:16,333 I don't. I would guess University of 728 01:28:16,333 --> 01:28:18,333 Chicago just for pure irony. 729 01:28:18,944 --> 01:28:21,277 No, for real irony, it started at the 730 01:28:21,277 --> 01:28:22,444 University of Michigan. 731 01:28:23,000 --> 01:28:25,555 Oh, no. Yes. So we were 732 01:28:25,555 --> 01:28:26,944 the leaders there as well. 733 01:28:27,277 --> 01:28:30,500 Yes, we were. And it caught on like 734 01:28:30,500 --> 01:28:32,500 wildfire. And at least that's what the 735 01:28:32,500 --> 01:28:34,555 book ungrading says. I 736 01:28:35,500 --> 01:28:38,500 will trust them. It's a fascinating book. 737 01:28:38,888 --> 01:28:39,500 And so this whole 738 01:28:39,500 --> 01:28:41,666 ungrading movement is to get rid of 739 01:28:41,666 --> 01:28:45,000 all of those concepts. And if you take it 740 01:28:45,000 --> 01:28:46,500 to its logical conclusion, 741 01:28:47,388 --> 01:28:48,944 the logical conclusion of 742 01:28:48,944 --> 01:28:51,166 ungrading means you don't give grades. 743 01:28:52,111 --> 01:28:54,388 And instead, you just ask the students, 744 01:28:55,222 --> 01:28:56,000 what do you think you 745 01:28:56,000 --> 01:28:58,666 deserve? And it's kind of interesting 746 01:28:58,666 --> 01:29:01,277 because, you know, that 747 01:29:01,277 --> 01:29:02,833 will actually work sometimes, 748 01:29:03,833 --> 01:29:05,666 but not at the University of Michigan. 749 01:29:06,666 --> 01:29:09,777 And I watched a podcast that 750 01:29:09,777 --> 01:29:13,222 Eric Mazur did on ungrading 751 01:29:13,777 --> 01:29:16,944 with two faculty members from the Eastern 752 01:29:16,944 --> 01:29:19,444 Kentucky University, which has a very 753 01:29:19,444 --> 01:29:20,777 different demographic 754 01:29:20,888 --> 01:29:24,444 than Michigan or Harvard. And they 755 01:29:24,444 --> 01:29:26,777 actually talked about their students. 756 01:29:27,444 --> 01:29:29,000 When they let their students 757 01:29:29,500 --> 01:29:32,444 give themselves grades, they almost 758 01:29:32,444 --> 01:29:34,611 always give themselves a 759 01:29:34,611 --> 01:29:36,111 lower grade than they actually 760 01:29:36,111 --> 01:29:39,277 deserved. But that's because they have 761 01:29:39,277 --> 01:29:40,555 mostly non-traditional 762 01:29:40,555 --> 01:29:43,833 students who, you know, are mostly 763 01:29:43,833 --> 01:29:47,388 first-generation college students. They 764 01:29:47,388 --> 01:29:48,722 know why they're in school 765 01:29:48,722 --> 01:29:50,666 and they're pretty tough on 766 01:29:50,666 --> 01:29:55,111 themselves. And a great comment, one of 767 01:29:55,111 --> 01:29:57,333 the faculty members said, but this must 768 01:29:57,333 --> 01:29:58,111 be great at Harvard. 769 01:29:58,277 --> 01:30:00,555 Your students are like so highly evolved. 770 01:30:00,555 --> 01:30:01,611 And Eric just shook his head 771 01:30:01,611 --> 01:30:03,833 and said, no, think about how 772 01:30:03,833 --> 01:30:07,166 they got into Harvard. They all got in by 773 01:30:07,166 --> 01:30:09,333 focusing entirely on 774 01:30:09,333 --> 01:30:11,944 grades. And so there's a lot of 775 01:30:11,944 --> 01:30:14,666 dishonesty in that whole process. And 776 01:30:14,666 --> 01:30:15,777 sadly, University of 777 01:30:15,777 --> 01:30:19,222 Michigan is close to that. So it 778 01:30:19,222 --> 01:30:21,333 just won't work to let students give 779 01:30:21,333 --> 01:30:23,555 themselves grades because they'll just 780 01:30:23,555 --> 01:30:24,611 give themselves A's. 781 01:30:24,944 --> 01:30:26,833 I don't want to be that, you know, 782 01:30:26,833 --> 01:30:30,166 cynical, but I'd say for 90% 783 01:30:30,555 --> 01:30:33,055 of the time, it will be cynical. 784 01:30:33,277 --> 01:30:36,111 So we can't quite go that route. It might 785 01:30:36,111 --> 01:30:37,944 work some places, but it's 786 01:30:37,944 --> 01:30:41,722 that to me is a bridge too far. 787 01:30:42,555 --> 01:30:45,055 Yeah. As we're talking about students 788 01:30:45,055 --> 01:30:47,944 grading themselves and the 789 01:30:47,944 --> 01:30:48,833 sort of student obsession 790 01:30:48,944 --> 01:30:53,388 with the A, it really makes me think 791 01:30:53,388 --> 01:30:55,777 about the fact that there's 792 01:30:55,777 --> 01:30:57,444 a very wide gap between how 793 01:30:58,000 --> 01:31:00,722 important our students believe grades are 794 01:31:00,722 --> 01:31:02,333 and how important grades 795 01:31:02,333 --> 01:31:04,944 actually are not. You know, 796 01:31:04,944 --> 01:31:07,166 that as we're sending students to their 797 01:31:07,166 --> 01:31:10,333 employers or to graduate schools, the 798 01:31:10,333 --> 01:31:11,722 large majority of what 799 01:31:11,944 --> 01:31:15,277 we're writing in our recommendations is 800 01:31:15,277 --> 01:31:16,444 not about the grades they 801 01:31:16,444 --> 01:31:17,944 got. It's about what they did 802 01:31:18,277 --> 01:31:20,777 and the interactions that they had. And 803 01:31:20,777 --> 01:31:22,277 certainly when I'm writing 804 01:31:22,277 --> 01:31:23,222 a letter of recommendation, 805 01:31:23,611 --> 01:31:25,500 I'll say about two sentences of this 806 01:31:25,500 --> 01:31:27,222 student got an A and then two 807 01:31:27,222 --> 01:31:29,055 pages about the actual human 808 01:31:29,111 --> 01:31:33,222 being, not the letter. That's right. And 809 01:31:33,222 --> 01:31:35,555 what's sweeping the country 810 01:31:35,555 --> 01:31:38,777 is this whole idea of holistic 811 01:31:40,000 --> 01:31:44,777 evaluation of admissions. And for the 812 01:31:44,777 --> 01:31:46,166 graduate programs, you 813 01:31:46,166 --> 01:31:48,277 know, it's well known that 814 01:31:49,555 --> 01:31:52,722 all grades are as a predictor is whether 815 01:31:52,722 --> 01:31:54,222 or not students will score 816 01:31:54,222 --> 01:31:57,444 well in exams in graduate 817 01:31:57,444 --> 01:32:00,777 school. And it says nothing about how 818 01:32:00,777 --> 01:32:02,277 well they will do with 819 01:32:02,277 --> 01:32:04,444 research, which is what PhDs are all 820 01:32:04,555 --> 01:32:07,666 about. And for research, we know that 821 01:32:07,666 --> 01:32:09,777 we'd rather have a student who's 822 01:32:09,777 --> 01:32:13,388 perseveres, who's fearless, 823 01:32:14,222 --> 01:32:19,944 who embraces failure, learns from it, has 824 01:32:19,944 --> 01:32:22,333 all these other things. 825 01:32:22,611 --> 01:32:24,277 We would much rather have a 826 01:32:24,277 --> 01:32:26,833 student who's overcome adversity in their 827 01:32:26,833 --> 01:32:28,111 undergraduate career 828 01:32:28,111 --> 01:32:30,166 to go from a pretty low 829 01:32:30,222 --> 01:32:32,833 level to a high level that shows an 830 01:32:32,833 --> 01:32:35,166 ability to overcome the kind 831 01:32:35,166 --> 01:32:37,722 of barriers that are actually 832 01:32:37,777 --> 01:32:40,444 important for doing research. Because 833 01:32:40,444 --> 01:32:41,555 that's what research is all 834 01:32:41,555 --> 01:32:43,388 about. Better get used to failing 835 01:32:43,388 --> 01:32:47,000 a lot and embracing it because if we knew 836 01:32:47,000 --> 01:32:47,944 what the answer was, it 837 01:32:47,944 --> 01:32:49,222 wouldn't be called research. 838 01:32:50,333 --> 01:32:55,222 So, you know, with all of that, I, again, 839 01:32:56,111 --> 01:32:57,277 students have this, you 840 01:32:57,277 --> 01:32:58,500 know, how do we educate students 841 01:32:58,666 --> 01:33:01,388 that grades aren't important when all of 842 01:33:01,388 --> 01:33:02,666 society is telling them 843 01:33:02,666 --> 01:33:04,944 that they are. And it's really, 844 01:33:04,944 --> 01:33:12,777 really sad. So anyway, we've talked a lot 845 01:33:12,777 --> 01:33:14,333 about some background 846 01:33:14,333 --> 01:33:15,833 about what you and I believe 847 01:33:16,333 --> 01:33:19,666 is important in grading or not important 848 01:33:19,666 --> 01:33:22,000 in grading. Why don't you 849 01:33:22,000 --> 01:33:22,944 tell me what you actually 850 01:33:23,111 --> 01:33:25,055 do for grading? You started by telling me 851 01:33:25,055 --> 01:33:26,666 you have a multi-point 852 01:33:26,666 --> 01:33:28,500 scale. Can you be a little 853 01:33:28,500 --> 01:33:31,277 more specific about how you do grading in 854 01:33:31,277 --> 01:33:33,500 your class right now? Sure. 855 01:33:34,444 --> 01:33:35,444 Yeah. On the implementation 856 01:33:35,666 --> 01:33:37,777 side for each of these assignments, 857 01:33:37,777 --> 01:33:40,888 whether it's a report or a 858 01:33:40,888 --> 01:33:42,777 lab notebook or a homework or, 859 01:33:44,222 --> 01:33:45,611 no, actually, I guess that's about it. 860 01:33:46,833 --> 01:33:50,944 The rubric will have a description of 861 01:33:51,944 --> 01:33:54,611 what students should be accomplishing in 862 01:33:54,611 --> 01:33:56,333 the assignment. So as a 863 01:33:56,333 --> 01:33:57,611 concrete example, in this 864 01:33:57,611 --> 01:34:00,444 last lab report that my students have 865 01:34:00,444 --> 01:34:02,388 just turned in, I'll have a 866 01:34:02,388 --> 01:34:04,388 line that says, for example, 867 01:34:05,333 --> 01:34:07,888 must compare predictions of a theoretical 868 01:34:07,888 --> 01:34:09,944 model to experimental data 869 01:34:09,944 --> 01:34:11,555 and evaluate the accuracy of 870 01:34:11,611 --> 01:34:16,777 the model. And then at that point, the 871 01:34:16,777 --> 01:34:18,111 different levels of grading 872 01:34:18,111 --> 01:34:22,000 are a five does an excellent 873 01:34:22,000 --> 01:34:24,000 job of providing a convincing explanation 874 01:34:24,000 --> 01:34:25,055 that combines multiple 875 01:34:25,055 --> 01:34:27,166 sources of data. And a four 876 01:34:28,333 --> 01:34:31,722 accurately compares the theoretical model 877 01:34:31,722 --> 01:34:33,055 to the experimental data, 878 01:34:33,055 --> 01:34:35,000 but does not bring in other 879 01:34:35,000 --> 01:34:37,055 sources of information. And there's this 880 01:34:37,055 --> 01:34:38,833 kind of tiered structure of 881 01:34:38,833 --> 01:34:40,833 what will really convince me 882 01:34:40,888 --> 01:34:42,722 that you know what you're doing and then 883 01:34:42,722 --> 01:34:45,000 sort of taking away some of 884 01:34:45,000 --> 01:34:46,500 those pieces of the argument 885 01:34:46,500 --> 01:34:49,944 as you get to lower score levels. And an 886 01:34:49,944 --> 01:34:51,388 important distinction that I 887 01:34:51,388 --> 01:34:52,555 think is worth pointing out 888 01:34:52,555 --> 01:34:56,055 here is that I'm telling students what 889 01:34:56,055 --> 01:34:56,833 they need to have 890 01:34:56,833 --> 01:34:59,055 accomplished at the end of the day. I'm 891 01:34:59,055 --> 01:35:00,555 telling them that here's what will 892 01:35:00,555 --> 01:35:03,777 convince me, but I'm not actually giving 893 01:35:03,777 --> 01:35:06,222 them a list of boxes 894 01:35:06,333 --> 01:35:09,166 to check, right? I'm not saying, okay, 895 01:35:09,166 --> 01:35:10,111 you have to get some error 896 01:35:10,111 --> 01:35:11,444 bars on this thing. And I'm not 897 01:35:11,444 --> 01:35:15,000 telling them, oh, yes, you should run the 898 01:35:15,000 --> 01:35:16,000 model with different model 899 01:35:16,000 --> 01:35:17,388 parameters to see whether 900 01:35:17,388 --> 01:35:21,166 there are numerical issues because that 901 01:35:21,166 --> 01:35:23,277 that's part of what's being 902 01:35:23,277 --> 01:35:25,444 assessed is do they have the 903 01:35:25,944 --> 01:35:28,222 understanding of this process of science 904 01:35:28,222 --> 01:35:30,222 to be able to do a 905 01:35:30,222 --> 01:35:32,944 meaningful comparison. And sometimes 906 01:35:33,111 --> 01:35:34,388 students look at this rubric and they're 907 01:35:34,388 --> 01:35:35,888 like, so you want a meaningful 908 01:35:35,888 --> 01:35:37,388 comparison? What does that 909 01:35:37,388 --> 01:35:39,888 mean? But then that's a great question. 910 01:35:39,888 --> 01:35:40,722 That's something we can have a 911 01:35:40,722 --> 01:35:42,444 conversation about. And 912 01:35:42,444 --> 01:35:44,055 when they're ready to ask that question, 913 01:35:44,055 --> 01:35:45,444 and we can talk about it, 914 01:35:45,444 --> 01:35:47,222 then that's a place where I think 915 01:35:47,222 --> 01:35:49,944 true learning can really happen. So I 916 01:35:49,944 --> 01:35:52,555 would argue that there's value in an 917 01:35:52,555 --> 01:35:54,388 interactive course that 918 01:35:54,555 --> 01:36:01,277 leaving some things not fully written out 919 01:36:01,277 --> 01:36:03,111 explicitly in these 920 01:36:03,111 --> 01:36:04,777 specifications is good 921 01:36:04,777 --> 01:36:06,500 because it gets students to question the 922 01:36:06,500 --> 01:36:08,833 meaning of the words in the 923 01:36:08,833 --> 01:36:10,611 specification to help them 924 01:36:10,611 --> 01:36:12,166 unpack any gaps in their own 925 01:36:12,166 --> 01:36:13,277 understanding of it. 926 01:36:15,111 --> 01:36:16,888 Right. So I think it's important to 927 01:36:16,888 --> 01:36:18,444 mention that you teach a 928 01:36:18,444 --> 01:36:20,944 laboratory class, and that you 929 01:36:21,944 --> 01:36:25,666 typically have, you spend a lot of time 930 01:36:25,666 --> 01:36:27,000 with the students. So 931 01:36:27,000 --> 01:36:28,666 your teaching schedule is 932 01:36:29,666 --> 01:36:32,000 four days a week, every afternoon, you're 933 01:36:32,000 --> 01:36:34,111 in the lab with groups of 934 01:36:34,111 --> 01:36:35,833 students and you typically have 935 01:36:36,166 --> 01:36:38,444 groups of four in each team. Is that 936 01:36:38,444 --> 01:36:41,500 right? Yeah, usually three to four. And 937 01:36:41,500 --> 01:36:43,333 how many teams do you 938 01:36:43,333 --> 01:36:44,888 have because you have them all rotating 939 01:36:44,888 --> 01:36:47,222 on stations around to 940 01:36:47,222 --> 01:36:48,166 utilize the equipment? 941 01:36:48,500 --> 01:36:52,111 Yeah, typically a section will be four 942 01:36:52,111 --> 01:36:54,055 teams and then I'll have however many 943 01:36:54,055 --> 01:36:55,055 sections I have based 944 01:36:55,055 --> 01:36:57,222 on enrollment, usually three. Right. So 945 01:36:57,222 --> 01:36:58,944 you have like up to 16 946 01:36:58,944 --> 01:37:01,111 students at any given time. 947 01:37:01,833 --> 01:37:03,777 So you actually have time to interact 948 01:37:03,777 --> 01:37:05,777 with each student. Because 949 01:37:05,777 --> 01:37:07,333 how long are your labs are like 950 01:37:07,722 --> 01:37:09,166 two hours? They're four hours, four 951 01:37:09,166 --> 01:37:11,277 hours. Yeah. So you have, you 952 01:37:11,277 --> 01:37:13,222 have a lot of time to interact 953 01:37:13,555 --> 01:37:18,500 with each student. And that just you just 954 01:37:18,500 --> 01:37:20,944 have to spend a lot of time. So in a way, 955 01:37:21,611 --> 01:37:24,388 you're able to do oral exams with all 956 01:37:24,388 --> 01:37:26,444 your students almost every day. 957 01:37:27,611 --> 01:37:29,555 Yeah, I just don't call them that. And 958 01:37:29,555 --> 01:37:30,388 that's right. That takes 959 01:37:30,388 --> 01:37:33,055 away a lot of the stress around, 960 01:37:33,055 --> 01:37:35,444 Oh my God, I'm being examined. No, we're 961 01:37:35,444 --> 01:37:37,388 just having a conversation so I can see 962 01:37:37,666 --> 01:37:39,166 what you get and what you don't get. So I 963 01:37:39,166 --> 01:37:41,777 can help you get what you don't yet get. 964 01:37:42,111 --> 01:37:44,000 Right. But you know, 965 01:37:44,000 --> 01:37:46,277 this approach. Go on. 966 01:37:47,055 --> 01:37:48,888 I was just going to say that this 967 01:37:48,888 --> 01:37:51,222 approach is going to hit a 968 01:37:51,222 --> 01:37:54,222 scaling limit where in a 100, 969 01:37:54,222 --> 01:37:57,222 120 person course, there's just no way to 970 01:37:57,222 --> 01:37:59,444 do this unless I had an army of grad 971 01:37:59,444 --> 01:38:00,444 students who were all 972 01:38:00,722 --> 01:38:03,444 trained in pedagogy and able to tease out 973 01:38:03,444 --> 01:38:04,833 students ideas in the same 974 01:38:04,833 --> 01:38:06,777 way. So that kind of leads to 975 01:38:06,833 --> 01:38:09,388 the question of in a bigger class, 976 01:38:09,833 --> 01:38:11,166 because like Steve, your classes are 977 01:38:11,166 --> 01:38:12,333 significantly larger than 978 01:38:12,333 --> 01:38:14,833 mine. What are other approaches that you 979 01:38:14,833 --> 01:38:16,333 can employ for grading 980 01:38:16,333 --> 01:38:18,333 that would work well with 981 01:38:18,333 --> 01:38:21,055 a larger number of students? That's 982 01:38:21,055 --> 01:38:23,444 right. So I have 140 983 01:38:23,444 --> 01:38:25,500 students in my class this term. 984 01:38:25,944 --> 01:38:31,444 We had a 24 teams of six people per team, 985 01:38:31,444 --> 01:38:33,222 roughly. Some people, 986 01:38:33,722 --> 01:38:35,000 some teams were five people. 987 01:38:35,833 --> 01:38:40,666 It was in a very large room. And so, you 988 01:38:40,666 --> 01:38:41,777 know, I've tried to 989 01:38:41,777 --> 01:38:43,833 structure my course really around 990 01:38:44,055 --> 01:38:46,055 the way that I'm going to do grading. And 991 01:38:46,055 --> 01:38:48,722 so I think one thing that 992 01:38:48,722 --> 01:38:50,388 was really important in what 993 01:38:50,388 --> 01:38:54,166 you do, Tim, is a best practice that lots 994 01:38:54,166 --> 01:38:55,333 of people talk about 995 01:38:55,333 --> 01:38:57,444 that if you're going to give 996 01:38:57,611 --> 01:39:01,444 assignments for a grade, it's better to 997 01:39:01,444 --> 01:39:04,055 give lots of low stakes 998 01:39:04,055 --> 01:39:07,277 assignments. So instead of giving 999 01:39:07,277 --> 01:39:11,222 three exams, it's better if you gave 14 1000 01:39:11,222 --> 01:39:13,111 exams that were all equally 1001 01:39:13,111 --> 01:39:15,388 weighted, because that way, 1002 01:39:15,833 --> 01:39:18,166 if you do poorly on one of them, it 1003 01:39:18,166 --> 01:39:19,611 doesn't crash your whole 1004 01:39:19,611 --> 01:39:22,777 grade. And so by working with them 1005 01:39:22,944 --> 01:39:27,166 every single week and interacting with 1006 01:39:27,166 --> 01:39:28,500 all of them, you know, 1007 01:39:28,500 --> 01:39:30,944 you've kind of created a 1008 01:39:33,055 --> 01:39:36,944 large number of evaluations with low 1009 01:39:36,944 --> 01:39:38,777 stakes for each one that 1010 01:39:38,777 --> 01:39:41,722 conclude with your judgment, 1011 01:39:42,444 --> 01:39:44,055 where you can use your rubric to grade 1012 01:39:44,055 --> 01:39:45,944 them. And so I do try 1013 01:39:45,944 --> 01:39:47,666 to do the same thing. So 1014 01:39:48,166 --> 01:39:52,277 with as many students as I have, it's 1015 01:39:52,277 --> 01:39:56,888 kind of hard to just redo everything. I 1016 01:39:56,888 --> 01:39:57,888 have to do it in steps. 1017 01:39:58,666 --> 01:40:01,000 So I looked at what I had been doing. So 1018 01:40:01,000 --> 01:40:01,777 I give reading 1019 01:40:01,777 --> 01:40:04,277 assignments for every class session. 1020 01:40:05,444 --> 01:40:09,666 And I give, we usually have, I meet two 1021 01:40:09,666 --> 01:40:12,944 hours, twice a week. And 1022 01:40:12,944 --> 01:40:14,611 so each week has two class 1023 01:40:14,666 --> 01:40:18,333 sessions. And so I usually give two 1024 01:40:18,333 --> 01:40:20,055 reading assignments for 1025 01:40:20,055 --> 01:40:23,277 each class session. Each week, 1026 01:40:23,888 --> 01:40:27,722 I have a homework assignment. Each week, 1027 01:40:28,111 --> 01:40:29,500 not the same week, but I 1028 01:40:29,500 --> 01:40:31,277 have a what I call a readiness 1029 01:40:31,277 --> 01:40:34,611 assurance activity, which is a more 1030 01:40:34,611 --> 01:40:38,722 formative based test, but it's weighted heavily towards the team 1031 01:40:38,722 --> 01:40:45,833 participation. I also have outcomes 1032 01:40:45,833 --> 01:40:50,166 assessment reflections, where I, at the 1033 01:40:50,166 --> 01:40:51,222 end of every module, 1034 01:40:51,500 --> 01:40:53,277 I have students reflect on what they 1035 01:40:53,277 --> 01:40:54,944 actually learned. So 1036 01:40:54,944 --> 01:40:56,944 again, my whole course is set up 1037 01:40:56,944 --> 01:40:59,944 to focus on retention, because I think 1038 01:40:59,944 --> 01:41:00,888 that's really important. 1039 01:41:01,722 --> 01:41:02,722 And one way to get retention 1040 01:41:03,333 --> 01:41:09,555 is to have students revisit a concept 1041 01:41:09,555 --> 01:41:12,611 multiple times over a 1042 01:41:12,611 --> 01:41:14,555 relatively long period of time. 1043 01:41:15,333 --> 01:41:19,055 So all of my modules, so a module starts 1044 01:41:19,055 --> 01:41:20,555 with a one week effort. 1045 01:41:21,222 --> 01:41:23,500 And that module then extends 1046 01:41:23,500 --> 01:41:25,722 for three weeks before they actually 1047 01:41:25,722 --> 01:41:27,277 finish all the things. So while they're working on other 1048 01:41:28,000 --> 01:41:33,055 aspects of that module, a new module is 1049 01:41:33,055 --> 01:41:35,166 being initiated the next 1050 01:41:35,166 --> 01:41:37,777 week. At any rate, I have 1051 01:41:37,777 --> 01:41:42,722 roughly 24 reading assignments in the 1052 01:41:42,722 --> 01:41:45,833 term. We have about 12 weeks, 14 weeks, 1053 01:41:45,833 --> 01:41:46,611 something like that. 1054 01:41:47,444 --> 01:41:49,888 So I have, you know, 24 reading 1055 01:41:49,888 --> 01:41:52,944 assignments. I have 14 homework 1056 01:41:52,944 --> 01:41:53,111 assignments. I have 14 readiness 1057 01:41:53,500 --> 01:42:00,833 assurance assignments. I have 14 outcomes 1058 01:42:00,833 --> 01:42:02,500 reflection assignments. 1059 01:42:02,944 --> 01:42:04,333 And then I also do projects. 1060 01:42:05,111 --> 01:42:09,333 And I've been doing three projects, but 1061 01:42:09,333 --> 01:42:11,222 the first one's just for practice. I'll 1062 01:42:11,222 --> 01:42:12,222 be changing that next 1063 01:42:12,222 --> 01:42:15,833 year, mostly because drop bad deadlines 1064 01:42:15,833 --> 01:42:16,833 destroy teams in my class for the first three weeks. So 1065 01:42:16,888 --> 01:42:22,166 my first project has pretty much always 1066 01:42:22,166 --> 01:42:23,666 been a bust because the 1067 01:42:23,666 --> 01:42:25,666 teams got disrupted so much. 1068 01:42:26,055 --> 01:42:28,833 It wasn't working, but I finally figured 1069 01:42:28,833 --> 01:42:31,555 out how to fix that. I can 1070 01:42:31,555 --> 01:42:34,222 now go in the day after class 1071 01:42:34,222 --> 01:42:38,000 starts and drop the enrollment number so 1072 01:42:38,000 --> 01:42:38,944 nobody can join the 1073 01:42:38,944 --> 01:42:40,555 class. People can still drop, 1074 01:42:40,555 --> 01:42:42,666 but nobody can add my class after the 1075 01:42:42,666 --> 01:42:42,777 deadline. And that's going to let me have three projects 1076 01:42:43,333 --> 01:42:50,222 which will be worth two units each. When 1077 01:42:50,222 --> 01:42:51,555 it's all said and done, I 1078 01:42:51,555 --> 01:42:54,666 add this all up and I have 72 1079 01:42:54,666 --> 01:42:58,722 units make up my course. And what I tell 1080 01:42:58,722 --> 01:43:00,777 the students is they have 1081 01:43:00,777 --> 01:43:02,611 to get an A on every single 1082 01:43:02,611 --> 01:43:04,944 unit. Now I'm a little flexible there and A minus is just as good as an A because I can't really 1083 01:43:05,111 --> 01:43:12,222 distinguish. So they pretty much have to 1084 01:43:12,222 --> 01:43:14,777 score like a 90%, but it's 1085 01:43:14,777 --> 01:43:17,222 not a strict 90%. I just do it 1086 01:43:17,222 --> 01:43:24,055 as a zero one or two and a two is an A. A 1087 01:43:24,055 --> 01:43:25,555 one is approaching expectations and a zero means 1088 01:43:25,611 --> 01:43:32,722 they just didn't do it. And I try to give 1089 01:43:32,722 --> 01:43:35,888 each of those units where 1090 01:43:35,888 --> 01:43:37,611 I can, I can't always do it 1091 01:43:37,611 --> 01:43:40,555 where I can. I give them the ability to 1092 01:43:40,555 --> 01:43:42,444 retake them multiple times 1093 01:43:42,444 --> 01:43:44,277 until they achieve a two. 1094 01:43:46,000 --> 01:43:48,166 All the reading assignments, which are, 1095 01:43:48,166 --> 01:43:49,444 you know, 24 out of 72, a big chunk, all of that 1096 01:43:49,944 --> 01:43:56,222 are reading assignments on perusal and 1097 01:43:56,222 --> 01:43:58,166 perusal gives me the ability 1098 01:43:58,166 --> 01:44:01,500 to allow them to continually 1099 01:44:01,500 --> 01:44:07,555 add annotations that score highly or to 1100 01:44:07,555 --> 01:44:12,277 do high quality interactions. 1101 01:44:12,277 --> 01:44:14,500 So they get credit for being 1102 01:44:14,500 --> 01:44:16,777 social because I believe social learning 1103 01:44:16,777 --> 01:44:17,444 is a powerful tool. But I think that's a really 1104 01:44:17,444 --> 01:44:20,888 powerful tool. Social learning is hard 1105 01:44:20,888 --> 01:44:22,277 baked into our DNA. 1106 01:44:22,944 --> 01:44:24,944 Writing's only been around for 5,000 1107 01:44:24,944 --> 01:44:27,166 years. Social learning has been around 1108 01:44:27,166 --> 01:44:29,333 for 3 million years. That's 1109 01:44:29,333 --> 01:44:31,333 our primal instinct. That's 1110 01:44:31,333 --> 01:44:34,666 how we learn. And so by being social, by 1111 01:44:34,666 --> 01:44:36,500 interacting with other students on 1112 01:44:36,500 --> 01:44:38,555 perusal, I really highly 1113 01:44:38,722 --> 01:44:42,777 value that. So they can do that up to a 1114 01:44:42,777 --> 01:44:45,333 deadline. They, you know, once the 1115 01:44:45,333 --> 01:44:47,555 deadline goes too late. 1116 01:44:50,000 --> 01:44:52,500 So that's how, that's how I'm trying to 1117 01:44:52,500 --> 01:44:55,277 do mastery grading. I also 1118 01:44:55,277 --> 01:44:57,166 have these reflections. So 1119 01:44:57,444 --> 01:44:59,555 what I grade is the homework reflection. 1120 01:44:59,555 --> 01:45:01,000 I don't grade the 1121 01:45:01,000 --> 01:45:02,222 initial homework effort. 1122 01:45:02,777 --> 01:45:05,500 I only grade that on effort, not on 1123 01:45:05,500 --> 01:45:08,111 accuracy. They have to scan 1124 01:45:08,111 --> 01:45:09,333 their work and put it into 1125 01:45:09,333 --> 01:45:11,444 canvas. They have to do it, but I pretty 1126 01:45:11,444 --> 01:45:12,722 much just, you know, 1127 01:45:12,722 --> 01:45:15,166 virtually weigh the packet of 1128 01:45:15,166 --> 01:45:18,222 information to see that they tried. And 1129 01:45:18,222 --> 01:45:19,222 that's just a check mark. 1130 01:45:19,555 --> 01:45:21,500 They have to do that if they 1131 01:45:21,500 --> 01:45:24,666 want to get an A on the assignment or 1132 01:45:24,666 --> 01:45:26,555 pass that unit. If they 1133 01:45:26,555 --> 01:45:27,888 don't do that, it doesn't matter 1134 01:45:27,944 --> 01:45:30,111 what they do later on. They're not going 1135 01:45:30,111 --> 01:45:32,277 to pass it. Then they have 1136 01:45:32,277 --> 01:45:34,166 a homework activity in class 1137 01:45:34,166 --> 01:45:36,222 where they make a better homework 1138 01:45:36,222 --> 01:45:40,388 solution with their team. And I hire all 1139 01:45:40,388 --> 01:45:41,555 of these instructional 1140 01:45:41,666 --> 01:45:46,277 aides, our senior level material students 1141 01:45:46,277 --> 01:45:48,555 who were just in that class two years 1142 01:45:48,555 --> 01:45:50,555 earlier. And so they, 1143 01:45:51,500 --> 01:45:54,277 I have one undergrad for every two 1144 01:45:54,277 --> 01:45:57,333 tables, maybe every three tables, and 1145 01:45:57,333 --> 01:45:59,333 they walk around and 1146 01:45:59,333 --> 01:46:01,500 provide help. But it turns out the 1147 01:46:01,500 --> 01:46:02,500 students don't really need 1148 01:46:02,500 --> 01:46:06,444 help. The help they need is those 1149 01:46:06,444 --> 01:46:09,222 students that are introverts and don't 1150 01:46:09,222 --> 01:46:12,500 talk need to talk. So I tell the whole 1151 01:46:12,500 --> 01:46:13,333 class, we're going to 1152 01:46:13,333 --> 01:46:15,777 do this. And I train the instructional 1153 01:46:15,777 --> 01:46:17,222 aides to pull those folks 1154 01:46:17,222 --> 01:46:19,722 aside and say, Hey, you know, 1155 01:46:20,000 --> 01:46:22,722 Johnny, you're not talking enough. How is 1156 01:46:22,722 --> 01:46:24,111 anyone in your unit team? 1157 01:46:24,111 --> 01:46:26,055 How is anyone in your unit team 1158 01:46:26,166 --> 01:46:29,111 realize that you're adding value? And 1159 01:46:29,111 --> 01:46:30,222 unless they realize you're 1160 01:46:30,222 --> 01:46:31,666 adding value, they won't respect 1161 01:46:31,666 --> 01:46:34,555 you. And good teams are teams that 1162 01:46:34,555 --> 01:46:37,277 respect each other. So we help the 1163 01:46:37,277 --> 01:46:38,722 introverts become less 1164 01:46:38,722 --> 01:46:40,722 introverted. We also talk to the 1165 01:46:40,722 --> 01:46:43,166 extroverts and explain how important 1166 01:46:43,166 --> 01:46:44,722 learning how to listen is. 1167 01:46:45,444 --> 01:46:50,111 And it's sort of like an audio, we do 1168 01:46:50,111 --> 01:46:51,500 compression on the high 1169 01:46:51,500 --> 01:46:53,500 end. And we do, you know, 1170 01:46:54,055 --> 01:46:56,833 we boost the levels on the low end, to 1171 01:46:56,833 --> 01:46:58,222 try to make teams that 1172 01:46:58,222 --> 01:47:00,000 perform better, because ultimately, 1173 01:47:00,722 --> 01:47:02,555 they need to teach each other these 1174 01:47:02,555 --> 01:47:04,277 concepts, if they want to 1175 01:47:04,277 --> 01:47:07,277 have retention. So that's what I 1176 01:47:07,333 --> 01:47:11,333 focus on. And does it work? Well, I'm not 1177 01:47:11,333 --> 01:47:12,555 there yet. I still have a 1178 01:47:12,555 --> 01:47:13,555 lot of problems with it. 1179 01:47:13,611 --> 01:47:16,888 But I'm getting there and I'm learning 1180 01:47:16,888 --> 01:47:17,888 right now. I have way 1181 01:47:17,888 --> 01:47:20,444 too many A's. So although I 1182 01:47:20,944 --> 01:47:24,222 want to have everyone get an A, I know in 1183 01:47:24,222 --> 01:47:25,166 my heart that the 1184 01:47:25,166 --> 01:47:26,444 students who a lot of students 1185 01:47:26,444 --> 01:47:28,333 are getting A's who really don't know 1186 01:47:28,333 --> 01:47:29,833 anything. And how do I know 1187 01:47:29,833 --> 01:47:33,000 that? Because when I walk around 1188 01:47:33,000 --> 01:47:35,500 the classroom, I will pick out a few 1189 01:47:35,500 --> 01:47:37,944 activities that I give them that I know 1190 01:47:37,944 --> 01:47:38,722 are very conceptual. 1191 01:47:39,555 --> 01:47:42,222 And I go and make them write stuff on the 1192 01:47:42,222 --> 01:47:43,666 whiteboard. And I talk 1193 01:47:43,666 --> 01:47:44,833 to every single team, 1194 01:47:45,277 --> 01:47:46,777 takes me about a half an hour to get 1195 01:47:46,777 --> 01:47:48,500 around the whole room. But 1196 01:47:48,500 --> 01:47:50,333 I can get a really good feel 1197 01:47:50,722 --> 01:47:52,555 for whether they learned anything from 1198 01:47:52,555 --> 01:47:53,833 their reading, whether they 1199 01:47:53,833 --> 01:47:54,833 learned anything from what 1200 01:47:54,833 --> 01:47:59,000 I talk about. And sadly, sometimes I walk 1201 01:47:59,000 --> 01:48:00,333 around the room like I tried 1202 01:48:00,333 --> 01:48:02,277 to tell them with extrinsic 1203 01:48:02,333 --> 01:48:05,555 semi-conductivity, where the acceptor 1204 01:48:05,555 --> 01:48:07,388 level is, where the donor 1205 01:48:07,388 --> 01:48:09,833 level is, where the Fermi level 1206 01:48:09,944 --> 01:48:12,000 is for an intrinsic semiconductor, and 1207 01:48:12,000 --> 01:48:13,388 where it moves when you have 1208 01:48:13,388 --> 01:48:15,833 an extrinsic or acceptor or 1209 01:48:15,833 --> 01:48:18,000 donor. And I try to get them to draw 1210 01:48:18,000 --> 01:48:19,000 these pictures on the 1211 01:48:19,000 --> 01:48:21,166 board. And then two days later, 1212 01:48:21,166 --> 01:48:24,055 I give them that exact problem on the 1213 01:48:24,055 --> 01:48:25,666 readiness assurance activity. 1214 01:48:25,666 --> 01:48:27,277 And on the individual round, 1215 01:48:28,222 --> 01:48:32,166 30% get it right. So I failed. I thought 1216 01:48:32,166 --> 01:48:35,055 I was giving them one-on-one instruction. 1217 01:48:35,666 --> 01:48:37,444 I thought they understood what I was 1218 01:48:37,444 --> 01:48:40,555 saying, but other factors 1219 01:48:40,555 --> 01:48:43,555 always come in. This was on the 1220 01:48:43,555 --> 01:48:46,833 second to last day of class. And I'm 1221 01:48:46,833 --> 01:48:50,555 sorry, students, by the time they hit the 1222 01:48:50,555 --> 01:48:52,388 very last week of the term 1223 01:48:53,055 --> 01:48:55,666 are pretty much brain dead because their 1224 01:48:55,666 --> 01:48:56,666 other classes are 1225 01:48:56,666 --> 01:48:58,333 weighing so heavily on them. 1226 01:48:58,944 --> 01:49:03,666 They're cramming like crazy. And it's a 1227 01:49:03,666 --> 01:49:05,055 really tough environment 1228 01:49:05,055 --> 01:49:07,222 to learn something new that 1229 01:49:07,222 --> 01:49:10,833 last week of class. So, you know, we all 1230 01:49:10,833 --> 01:49:12,666 suffer from this. I'm sure you've 1231 01:49:12,666 --> 01:49:13,944 observed that phenomena 1232 01:49:14,222 --> 01:49:18,166 as well. Oh, sure. And so those are my 1233 01:49:18,166 --> 01:49:20,166 problems. But what I'm 1234 01:49:20,166 --> 01:49:22,166 going to try to do next time, 1235 01:49:22,166 --> 01:49:24,333 I'm going to go back to something that's 1236 01:49:24,333 --> 01:49:27,777 really like an oral exam. You know, 1237 01:49:27,777 --> 01:49:29,388 it's called the Feynman technique. It's 1238 01:49:29,388 --> 01:49:31,444 what Richard Feynman told all of us. 1239 01:49:31,944 --> 01:49:35,111 If we want to know that we actually 1240 01:49:35,111 --> 01:49:37,500 understand something, we 1241 01:49:37,500 --> 01:49:39,833 should use the litmus test of, 1242 01:49:40,500 --> 01:49:43,166 can you teach this to a five-year-old? 1243 01:49:43,833 --> 01:49:45,055 And if you can teach 1244 01:49:45,055 --> 01:49:46,388 it to a five-year-old, 1245 01:49:46,666 --> 01:49:50,444 you probably really deeply understand it. 1246 01:49:50,444 --> 01:49:52,222 And if you deeply understand anything, 1247 01:49:52,888 --> 01:49:54,388 chances are you're going to retain that 1248 01:49:54,388 --> 01:49:56,888 for a long time. So I'm 1249 01:49:56,888 --> 01:49:58,222 going to play around with 1250 01:49:58,555 --> 01:50:00,833 things like I do for my projects, which 1251 01:50:00,833 --> 01:50:03,277 is whiteboard video. And I'm 1252 01:50:03,277 --> 01:50:04,722 going to try to get students to 1253 01:50:04,722 --> 01:50:07,333 make short whiteboard videos. My 1254 01:50:07,333 --> 01:50:09,222 whiteboard videos for my projects are 1255 01:50:09,222 --> 01:50:10,388 like two minutes long. 1256 01:50:11,277 --> 01:50:14,055 I think for certain concepts that I'm 1257 01:50:14,055 --> 01:50:16,444 going to ask them to do, I'm going to 1258 01:50:16,444 --> 01:50:17,944 embody the idea of a 1259 01:50:17,944 --> 01:50:21,944 YouTube short or a TikTok video. They 1260 01:50:21,944 --> 01:50:25,777 have 30 to 45 seconds to teach the 1261 01:50:25,777 --> 01:50:27,833 concept. And we'll see how 1262 01:50:27,833 --> 01:50:30,055 that goes. I haven't quite figured out 1263 01:50:30,055 --> 01:50:32,500 how to do it. But the truth 1264 01:50:32,500 --> 01:50:34,000 is they're all pretty skilled 1265 01:50:34,000 --> 01:50:36,444 in editing video. They come to college 1266 01:50:36,444 --> 01:50:38,611 knowing how to do that. So 1267 01:50:38,611 --> 01:50:40,000 let's exploit those skills. 1268 01:50:40,555 --> 01:50:44,666 Let's get stuff up. And maybe they can 1269 01:50:44,666 --> 01:50:45,944 help each other enough. And 1270 01:50:45,944 --> 01:50:47,555 if I build a library of the 1271 01:50:47,555 --> 01:50:50,055 good ones, those can even help students 1272 01:50:50,055 --> 01:50:54,444 learn small little chunks. But we'll see. 1273 01:50:54,722 --> 01:50:57,222 Yeah, I love that idea of producing short 1274 01:50:57,222 --> 01:50:59,555 form content. One trouble 1275 01:50:59,555 --> 01:51:02,166 that I have in my course with 1276 01:51:02,333 --> 01:51:04,277 the specifications that I'm using is 1277 01:51:04,277 --> 01:51:06,555 students come in believing 1278 01:51:06,555 --> 01:51:08,555 that more is better. And, 1279 01:51:09,055 --> 01:51:11,055 maybe I'll just write 30 pages instead of 1280 01:51:11,055 --> 01:51:11,833 20. And then all the 1281 01:51:11,833 --> 01:51:12,888 right answers will be in there 1282 01:51:12,888 --> 01:51:15,611 somewhere. And they will, but that's not 1283 01:51:15,611 --> 01:51:16,388 really helping the 1284 01:51:16,388 --> 01:51:19,000 underlying problem. So this is 1285 01:51:19,000 --> 01:51:21,500 something that I'm always trying to find 1286 01:51:21,500 --> 01:51:22,444 ways to do better in my 1287 01:51:22,444 --> 01:51:25,222 classes to give assignments that 1288 01:51:25,277 --> 01:51:27,222 really have students distill something 1289 01:51:27,222 --> 01:51:29,222 down to just the minimum 1290 01:51:29,222 --> 01:51:32,000 essence of the concept or the 1291 01:51:32,000 --> 01:51:34,055 problem of the calculation. And do it in 1292 01:51:34,055 --> 01:51:34,944 a paragraph. Don't even 1293 01:51:34,944 --> 01:51:36,555 do it in a page. If I could 1294 01:51:37,277 --> 01:51:40,777 really get them to be more concise and to 1295 01:51:40,777 --> 01:51:42,000 think about how short can 1296 01:51:42,000 --> 01:51:43,888 I make my product instead? 1297 01:51:43,888 --> 01:51:46,222 How long can I make it that would serve 1298 01:51:46,222 --> 01:51:48,000 all of us very well, but 1299 01:51:48,000 --> 01:51:49,388 we'll get there someday. 1300 01:51:49,388 --> 01:51:51,111 We all know it's much harder to write 1301 01:51:51,111 --> 01:51:52,055 something that's short 1302 01:51:52,055 --> 01:51:53,444 than something that's long. 1303 01:51:54,555 --> 01:51:56,944 And the same is true for video or 1304 01:51:56,944 --> 01:51:59,166 anything else. The thing I like about 1305 01:51:59,166 --> 01:52:01,111 video is that it's way 1306 01:52:01,222 --> 01:52:04,500 ahead of where chat GGP can quite get to 1307 01:52:04,500 --> 01:52:06,500 right now. I can start to 1308 01:52:06,500 --> 01:52:07,444 make videos, but they're not 1309 01:52:07,444 --> 01:52:11,166 going to be very good. And so I don't 1310 01:52:11,166 --> 01:52:12,444 have a problem with having 1311 01:52:12,444 --> 01:52:15,000 the students use chat GPT to 1312 01:52:15,000 --> 01:52:18,777 make their script, but to make the script 1313 01:52:18,777 --> 01:52:19,944 and make it concise 1314 01:52:19,944 --> 01:52:22,166 enough for a 30 second spot. 1315 01:52:22,777 --> 01:52:25,111 That's a pretty challenging thing that 1316 01:52:25,111 --> 01:52:27,111 requires you actually think 1317 01:52:27,111 --> 01:52:29,055 about what chat GPT is telling 1318 01:52:29,222 --> 01:52:31,888 you. And as long as you think about it, 1319 01:52:32,222 --> 01:52:34,055 that's a really good 1320 01:52:34,055 --> 01:52:36,555 activity in my book. Because once 1321 01:52:36,555 --> 01:52:38,888 you've thought about it, you've gone 1322 01:52:38,888 --> 01:52:41,333 beyond just memorizing and 1323 01:52:41,333 --> 01:52:44,666 you've edited what chat GPT says. 1324 01:52:45,333 --> 01:52:48,666 And so then chat GPT becomes a tool and 1325 01:52:48,666 --> 01:52:50,722 it's a useful tool, but you 1326 01:52:50,722 --> 01:52:52,222 must edit it. You must think 1327 01:52:52,222 --> 01:52:54,944 about what it's saying in order to make 1328 01:52:54,944 --> 01:52:56,388 something very concise. So I 1329 01:52:56,388 --> 01:52:57,500 think that's a good strategy. 1330 01:52:59,222 --> 01:53:01,277 Evaluate whether it's accurate, decide 1331 01:53:01,277 --> 01:53:02,944 how it can be done better, and then 1332 01:53:02,944 --> 01:53:04,000 actually execute the 1333 01:53:04,000 --> 01:53:06,166 doing it better. And that's a really good 1334 01:53:06,166 --> 01:53:07,333 use of time for learning. 1335 01:53:07,333 --> 01:53:08,555 And I'm even thinking maybe I 1336 01:53:08,555 --> 01:53:11,222 can come up with an activity where they 1337 01:53:11,222 --> 01:53:12,833 use their short form video 1338 01:53:12,833 --> 01:53:15,055 to teach other people on their 1339 01:53:15,055 --> 01:53:16,944 team and have the other people on the 1340 01:53:16,944 --> 01:53:19,666 team evaluate the quality of those 1341 01:53:19,666 --> 01:53:23,722 videos. So we'll say, 1342 01:53:24,555 --> 01:53:26,888 not sure how I'm going to pull this all 1343 01:53:26,888 --> 01:53:28,888 off. But that's what I'm 1344 01:53:28,888 --> 01:53:30,111 thinking about for next year. 1345 01:53:30,888 --> 01:53:31,500 What are you thinking 1346 01:53:31,500 --> 01:53:32,833 about for next year? 1347 01:53:35,222 --> 01:53:35,611 Ah, 1348 01:53:38,222 --> 01:53:41,833 I'm thinking about the biggest change 1349 01:53:41,833 --> 01:53:43,555 that I want to make on 1350 01:53:43,555 --> 01:53:45,722 this topic of grading is 1351 01:53:45,777 --> 01:53:53,222 doing a better job of mapping this five 1352 01:53:53,222 --> 01:53:57,222 point scale onto a 1353 01:53:57,222 --> 01:53:59,111 numerical scale that will make sense 1354 01:53:59,111 --> 01:54:01,666 to students who have grown up in zero to 1355 01:54:01,666 --> 01:54:05,055 100 land without getting 1356 01:54:05,055 --> 01:54:06,555 to this compression problem 1357 01:54:06,555 --> 01:54:07,833 that you were bringing up at the 1358 01:54:07,833 --> 01:54:12,388 beginning. If I can actually spread out 1359 01:54:12,388 --> 01:54:14,166 the individual numbers 1360 01:54:14,500 --> 01:54:18,555 over a wider numerical range so that the 1361 01:54:18,555 --> 01:54:20,500 values are distinct and have 1362 01:54:20,500 --> 01:54:21,388 separate meanings from each 1363 01:54:21,388 --> 01:54:24,055 other without getting students into the 1364 01:54:24,055 --> 01:54:26,222 panic of why do I have a C minus 1365 01:54:26,222 --> 01:54:28,277 territory? That's really 1366 01:54:28,666 --> 01:54:32,111 what I'm focused on right now is to help 1367 01:54:32,111 --> 01:54:33,833 students distinguish the 1368 01:54:33,833 --> 01:54:35,222 score on the assignment 1369 01:54:36,333 --> 01:54:38,277 from the grade you're going to get in the 1370 01:54:38,277 --> 01:54:39,722 course. Because as 1371 01:54:39,722 --> 01:54:40,666 much as I would like to 1372 01:54:41,166 --> 01:54:42,500 just tell them that grades don't matter 1373 01:54:42,500 --> 01:54:44,277 and have them actually believe it, 1374 01:54:44,666 --> 01:54:48,333 that's an impossible challenge. So have 1375 01:54:48,333 --> 01:54:50,777 to work around, work with 1376 01:54:50,777 --> 01:54:52,722 what our students believe 1377 01:54:52,722 --> 01:54:54,500 instead of what we wish they believed. 1378 01:54:55,333 --> 01:54:57,444 Well, that's a good point 1379 01:54:57,444 --> 01:54:58,833 to end with. I think we've 1380 01:54:58,833 --> 01:55:00,666 been talking for a long time now longer 1381 01:55:00,666 --> 01:55:03,500 than we intended. It 1382 01:55:03,500 --> 01:55:04,722 sure is hard to be concise. 1383 01:55:05,222 --> 01:55:09,833 Yes, it is. Especially when it comes to 1384 01:55:09,833 --> 01:55:11,611 something as intractable as grading. 1385 01:55:13,000 --> 01:55:15,666 So I guess with that, we'll say goodbye 1386 01:55:15,666 --> 01:55:18,555 and looking forward to our 1387 01:55:18,555 --> 01:55:20,500 next chat. See you later. 1388 01:55:21,555 --> 01:55:22,722 Okay. See you next time.