Subscribe
Share
Share
Embed
The Transform your Teaching podcast is a service of the Center for Teaching and Learning at Cedarville University in Cedarville, Ohio. Join Dr. Rob McDole and Dr. Jared Pyles as they seek to inspire higher education faculty to adopt innovative teaching and learning practices.
This is the Transform Your Teaching Podcast. The Transform Your Teaching Podcast is a service of the Center for Teaching and Learning at Cedarville University in Cedarville, Ohio.
Speaker 2:Hello, and welcome to this episode of Transform Your Teaching. In today's episode, Doctor. Rob McDole and Doctor. Jared Pyles continue our series on AI resilient course design by talking about assessments. Thanks for joining us.
Speaker 3:Rob, we're back, and now we're gonna talk about assessments, essentially part two from our last episode about objectives. So now let's talk assessments. We're going to revisit the objectives that we fixed, made them more measurable in our objectives episode. And now we're going to talk about the assessments that can measure those measurable objectives. And to
Speaker 4:be clear, we're not going to talk about the work that we did with doctor Rogers.
Speaker 3:That is correct.
Speaker 4:This is the episode... We're referring back to the episode that we did before we talked to doctor Rogers about objectives. So That is correct. We had four of those, business, health sciences, humanities, and social sciences, and we did not discuss assessments. But we're gonna go through, remind our listeners of what those objectives are that we came up with.
Speaker 4:We'll probably just quickly read the weak one and then the one we believed provided us with a better objective. And then we're gonna talk assessment. Yep. We're gonna get into the specifics and and have some conversations about that.
Speaker 2:Yep.
Speaker 4:So let's do it. So Okay. Example one was business.
Speaker 3:Yes.
Speaker 4:And the intentionally weak objective was students will understand how to make good business decisions. And what did we replace that with?
Speaker 3:Well, I wanna add in the other stuff that you didn't mention. I wanna mention why it's bad.
Speaker 4:Understand is not measurable. Good is undefined. So we replaced it with evaluate a business decision using stated ethical criteria. So this brings us into the assessment portion of where we'd start looking at how would we design an assessment that's going to actually measure, evaluate a business decision using stated ethical criteria. So what evidence would show that a student can evaluate?
Speaker 4:What would you say?
Speaker 3:Well, I think giving them a case study would be a great way of doing that because it allows them to look at something that happened, and then if you ask them to give their opinion or evaluate it based on ethical criteria, like in your ethics course, for example, if you throw a scenario at them... I remember that you do this with some usage. I don't if you do this anymore. You used to bring in guests from the actual industry.
Speaker 4:Yes.
Speaker 3:And the industry person would give them actual scenarios of what goes on, and then you'd have the students respond to it. We still do that. Oh, do you? Yeah. This is a great example of that because it allows them to see real world, real life scenarios and then have them evaluate it based on the stated criteria that you have in class.
Speaker 4:Yeah. And that could be... Like like I had mentioned or in the early episode, that could be gap for accounting Mhmm. FTC for finance. You could have others, OSHA, if you're talking management or anything that has to do with manufacturing, OSHA is always gonna be in there Mhmm.
Speaker 4:Telling you, you know, my favorite, I think, of all the OSHA requirements is you have to have three connections to a ladder at any one time.
Speaker 3:Really?
Speaker 4:Yes. So in other words, two feet, one hand. Oh, no way. Two hands, one foot. You're supposed to always have three points of connection with a ladder at all times.
Speaker 3:That blew my mind.
Speaker 4:That is an actual OSHA standard.
Speaker 3:You know, that every time I get on a ladder down, I'm gonna think about this. And I wanna follow-up
Speaker 4:with myself. The subjective. No. But, you know... So we've defined those things.
Speaker 4:I think you're right. Like, a case based evaluation is something that would work really well here in terms of evaluating one. Something that could also work along those lines where it becomes more AI resilient.
Speaker 3:Yeah. And we should mention we're doing AI resilience. So
Speaker 4:we have this. And if it's an old... If... How would you make it AI resilient is the next question. Sure.
Speaker 4:I mean, the case based solution is good. The problem is if it's old and it's been around a long time, more than likely AI is gonna already have access to that data. Yep. So one way that you can do this is to create your own Mhmm. Scenarios that aren't historical, or you literally pull something from the news headlines that day.
Speaker 4:Yeah. And you tell students you tell students, you have to actually find a case the day that you are going to do this assignment and you have to show me the date and then you'd have to work through it yourself. You're not allowed to use AI to evaluate this. You need to do it yourself. I think that's gonna make it a lot harder.
Speaker 4:The AI generated recommendation, especially when you start talking about things that are new. Now, can AI go out, read the news story? Yes, it can. But I think you can also provide some things that would make it difficult for students. At least, hopefully, that would be my my hope.
Speaker 3:Well, I think if you go back to the stated ethical criteria and the objective, if that's something that you create in your class
Speaker 4:Yeah.
Speaker 3:Then that's something you'd want to see in their response. And if it's an AI generated response, especially if it's a home brewed, ethical criteria that you're using, you'd wanna see that in an AI generated text or response wouldn't necessarily have that unless it's universal.
Speaker 4:Right. Or students feed
Speaker 3:Oh, they could feed the ethical criteria criteria
Speaker 4:to it. Yeah. So that that can be, you know, obviously, that can be difficult. Or something that I found is helpful is to actually preload AI with these scenarios and require students to show how they would evaluate it in a conversation with AI. Yep.
Speaker 4:And then they have to copy and paste or they have to do a screenshot PDF and give it to me.
Speaker 3:I want you to create a scenario in which an OSHA person comes in and sees someone going up a ladder with one leg and one hand. And I want you to like give them an assessment and make them, sir, clearly this person is in violation. Well, sir, this person only has one arm. So what do you expect me to do?
Speaker 4:Well... They're not allowed to climb a ladder.
Speaker 3:They can't... They need to be off of ladders.
Speaker 4:They need to use a a power lift.
Speaker 3:Do you have a scissor lift, sir, that you can have the person do? No?
Speaker 4:Yeah. I mean, you... We kinda laugh and giggle, but at some point, like, having students using AI and making them actually use AIs in this particular situation will will help us be able to see how they evaluate a business decision
Speaker 3:Yeah.
Speaker 4:Based on, certain criteria that we can feed to the bot and that it's basically saying, I'm... You have to answer within this. So I think that's really a good one. Let's move on to, another one, the health sciences. This was our next one.
Speaker 3:Yeah. Here's the intentionally weak one. Students will identify and respond appropriately to patient safety concerns. This is the one where I said, probably not a great idea because if you could be like, you know, Oh, you're coughing? Let me amputate your leg, kind of a thing.
Speaker 3:So probably not the best idea. Our measurable version of it was prioritize patient care actions in a simulated scenario using stated safety criteria. The assessment idea would be like something, you know, Rob, I'll throw it back to you like you threw it back to me with the first one. What evidence would show that a student can prioritize in this situation?
Speaker 4:Well, our objective really made that explicit early on by saying a simulation. So we already had an idea of how we can meet this idea of prioritizing patient care. Where do you see it? Well, you see it in the actual work
Speaker 3:Mhmm.
Speaker 4:Of caring for patients. So scenarios are a very good way of doing this. So in a simulated response, we can see how the student orders their actions, how they identify safety evidence, and that supports their actions. So are they actually following whatever the criteria? It could be a hospital.
Speaker 4:Mhmm. So there are certain... Even though they learn everything they need to in terms of nursing to pass their nursing exams to be able to license nurses, hospitals will also have their own specific regulations on how care should be handled
Speaker 3:Right.
Speaker 4:In lots of different scenarios. So taking them through these kinds of scenarios is really providing them with what they're going to face in the real world. And again, I think you could do... To make this AI resilient, you could take these scenarios, put them into AI, and have students have to work through them, respond to the AI bot. Again, you're gonna see this and you're see timestamps as to when they did these things.
Speaker 4:So, you know, I could hear somebody also saying, well, couldn't I just have AI talk to AI? Yeah. You probably could. And if you have a student at that point that's doing that, I think that's gonna come out at some point.
Speaker 3:Well, in the health sciences, they've got simulations that they do. Right. And they watch them
Speaker 4:Yes.
Speaker 3:And they evaluate them. So if someone keeps using AI to do that, eventually they're going to pull out their phone and use ChatGPT there beside the dummy as it's hemorrhaging or something like that and be like, please help me Chat. What do I do? You know?
Speaker 4:My patient's dying. Gonna my only hope.
Speaker 3:It's gonna come out in the wash eventually. You know what I mean?
Speaker 4:Yeah. And you definitely don't want it there. This is where I would say an AEI's resilient assessment as well would be what you just said. Like, have them watch Yep. In class these scenarios and they have to write on a piece of paper what they would do during the scenario and hand that into you before they leave.
Speaker 4:And somebody would say to me, well, Rob, that's gonna take me more work. Well, here's where you can use AI to help yourself out. Most of our our faculty here at Cedarville have multifunction scanners and printers somewhere in their offices.
Speaker 3:Hopefully, it's but by it's there.
Speaker 4:It's there. And they have document feeders and, you know, you can have student you can have student papers that they've written. AI is really good at recognizing handwriting. It has blown me away.
Speaker 3:Really? I've never actually tried that.
Speaker 4:Yeah. Some of the... Like, I've I've intentionally scrawled things to see if I... To see how much it could understand, how much it could pull out. That's interesting....
Speaker 4:Shocked at, its ability to use other programs that also do what's called OCR, which is Yep. Optical character recognition. So it's not just recognizing a picture of a printed letter, but also one that has been scripted by hand. But not only does it recognize it, it can also pretty much figure out even if you misspell a word. Like, oh, this is the word you were looking for within the context.
Speaker 4:So I would say doing that, one, keeps things grounded in terms of your students and and, you know, they are the ones that are having to write on the piece of paper. But when you go to actually evaluate it, you can go ahead and have ChatGPT. You scan those into scanners into PDFs, then it can read those... It can turn those into legible text for you if you wanna do it that way, or you can have it summarize their responses to kinda give you a first pass before you go through and review it all. Yeah.
Speaker 4:I think those are really helpful ways to provide for AI resilient assessment.
Speaker 3:That's good. This next one on the humanities, students will engage meaningfully with great works of the past. This was our... That was our intentionally weak one. We wanted to do something measurable, so instead we did compare how two authors frame a shared moral question.
Speaker 3:And this one I feel is is pretty self explanatory when it comes to the evidence of assessment for that. We... I... I've talked about this over and over again with my students this semester. In order to fully, you know, do something like a comparison and contrast or some sort of literary analysis, you have to use specific evidence from the text.
Speaker 3:And that's really how you show examples of, compare and contrast or, like I said, analyze or anything like that when it comes to literature. So a textual comparison using specific evidence from both works to distinguish the author's point of view and framing. And again, as far as AI resilience goes, this is something where you can have a shared language that you have in the course. Like I have, developed an art framework, which is affirm, reject, transform when it comes to how we, as Christians approach literature from biblical worldview. And so I don't ex...
Speaker 3:Like, again, they could feed it into AI. Yeah. But odds are good that when they... I'm looking for affirm, reject, and transform in their writings, it's gonna be something that they have created in, unless they feed the framework into the the bot. But...
Speaker 4:But they're also gonna still have to evaluate the output of the bot. Right. Because you and I both know when you put things in, it's not a 100%. You have to go back and do editing. You have to refine what you're expecting out of it.
Speaker 4:And that requires knowledge.
Speaker 3:Mhmm.
Speaker 4:It requires actually more knowledge than you would think. Yep. You can't depend on AI and that will come out in the result. Like, when I'm reading results, I can always tell when somebody's just taken something from AI and used it, especially in an area where I have knowledge and they didn't have that knowledge. They just took it and they put it in there as if somehow that was the right thing.
Speaker 4:Mhmm. I mean, I've I've seen this in many different occasions. So, you know, that puts you down that whole road of having to have difficult conversations, which is another episode for another day.
Speaker 3:Yeah. Definitely.
Speaker 4:But I I would say, again, this might be a place where you could use AI as a, you know, as a tutor or in this compare and contrast, use it as a Socratic. Yes. Literally, you put these issues into ChatGPT or a skill of some sort, and you have students do a formative assessment of sorts where it asks leading questions that they have to then answer.
Speaker 3:Yeah. That's good.
Speaker 4:Again, could you use a bot to to answer that going back and forth? Yes. But like we've said before, on earlier, you know, episodes, it's not just one piece of evidence here in terms of an assessment that you need to use for a student. It's the whole... A whole arc of different points of data Mhmm.
Speaker 4:That need to be evaluated more now than ever.
Speaker 3:Yeah. For sure.
Speaker 4:It can't just be one. It's gotta be multiple.
Speaker 3:Formative assessments.
Speaker 4:Yeah. Fill them up. And, yeah, and summative. You need those summative ones there
Speaker 3:Mhmm.
Speaker 4:To just check.
Speaker 3:Mhmm.
Speaker 4:You know? And, yes, they need to be protected, and, yes, they need to be, valid. So I just want our listeners to to know we're giving these kinds of, assessment strategies. They're not necessarily going to save you every single time and you need to make sure that you're you're, again, making this resilient within the ecosystem of a of a larger assessment strategy for your student. One more.
Speaker 4:One more. Social sciences. Our week one is students will read, discuss, and learn about foundational theories of human behavior. The measurable objective one was apply one foundational theory of human behavior to interpret a specified social scenario. So what evidence would show that a student can apply?
Speaker 4:You wanna take that?
Speaker 3:Yeah. So case interpretation that uses the theory's concepts to account for the scenario, rather than simply recalling definitions or repeating class discussion. So you wanna see, the... How that theory comes into practice when it comes to, a scenario or a case study.
Speaker 4:Well, real life scenarios though are very helpful.
Speaker 3:Oh, yeah.
Speaker 4:You know, especially if you've got a story and you provide that story to students and nobody's ever heard it before.
Speaker 3:Mhmm.
Speaker 4:But you know they're gonna have to think about it. And AI hasn't heard it. Those can be valid assessments.
Speaker 3:Anything's better than read, discuss, and learn. Yeah.
Speaker 4:I mean,
Speaker 3:let's be real here.
Speaker 4:Yeah. Again, how would we make this one how would we make this one AI resilient? I mean, we've we've already talked about case studies. We've talked about some of the things we can do there. No need to rehash that.
Speaker 4:But what would you what would you recommend?
Speaker 3:I mean, you could use... I mean, again, we could use AI to generate more scenarios or have students respond. I mean, students could use AI to respond to it as well. I think there's... Going back to the same idea of a shared language idea, would help with that.
Speaker 3:But we really wanna see the evaluation or the application of the theory. Because apply is rather lower in Bloom's, there's more of a chance that a bot could just go ahead and take that over and, do it for them because it's rather low. And, you know, if you give the theory and then have the... Give the case study, the bot could probably do a pretty good job of, coming up with that answer. But a way of doing it entirely is just to do it in class and have a student come up and be like, alright.
Speaker 3:Just like you did today with your Yeah. Your stools in class, had students come up, he did a q and a, like a mock roundtable, and you could do something like that where you throw a scenario at them and say, apply this foundational theory to
Speaker 4:it. I was inspired I was inspired by your fishbowl and and we use... We've used fishbowl...
Speaker 3:It's not my fishbowl.
Speaker 4:I know. But you were the one who kinda introduced it to me personally. It's not, you know, in a formal sense. I had... I've done things like it before, but in terms of what you brought to the table with it, it was it was very helpful.
Speaker 4:It gave me some insights in terms of how can I actually view what students are kind of thinking?
Speaker 3:You should explain what fishbowl is.
Speaker 4:So the fishbowl is you have... I think the way you set it up was you have a specified number of students that sit in the middle kind of in a circle or in a group, and then you have a specified number, equal number or so that are outside also in the group, and they're kind of like the people looking through the fishbowl. And the fish are the ones in the middle having the conversation, but the others are supposed to be paying attention.
Speaker 3:Mhmm.
Speaker 4:And they may, from time to time, ask questions at some point, but they're supposed to evaluate. Listen. They're supposed to listen and potentially evaluate what's going on the inside because they're gonna flip at some point, and you're gonna have different questions, but it's still gonna be on the same topic, and it might be, deeper. So I took that today, and I gave that in a sense to my students. I couldn't get people around them, but I just kept putting six new people up there.
Speaker 4:Ask a question everybody had to answer. And if it... If the question had already been answered, then I'd I'd go a little deeper and make them think. So we went on this whole journey of value assessment and how do you evaluate someone's values like Elon Musk. They they got on the whole story.
Speaker 4:Somebody mentioned Elon Musk, and so I just used that. It became a thing where I could use that and take them down a path to discuss not only being able to see what people value in their own lives, but also to reflect upon their own lives to figure out how do I value things and is there something I'm valuing highly that I shouldn't. All that to say, you could do something very similar in the class where you're using some sort of ad hoc oral presentation. And I know I've got some students I could tell they were just, like, frozen Mhmm. In fear.
Speaker 4:But it... As seniors in the business program who've all had, lord willing, at least if they've been here, speech, right, as part of our gen ed, they should be able to do this. So Right. I think there was a part of me that didn't feel so bad about that. I felt confident that my students would be able to get up there and do it, and they did a great job both both hours.
Speaker 3:So yeah, like, to give the other half, I gave a sequence idea, at the end of the objectives episode, so I can just add onto it now with this assessment piece. Identify the immeasurable or overloaded objective, use one objective that has one measurable verb, and then design the assessment that elicits the evidence of that measurable verb. One of my favorite things to do, and I've mentioned this in I think every objective episode that we've done or any time objectives have come up, A really great way of linking your objective to your assessment is using the measurable verb in your assessment details. So if you're having them, apply one foundational theory of human behavior, you would put that somewhere in your assessment description. Students will apply one or have them put it in student facing language.
Speaker 3:Apply one foundational theory of human behavior to this social scenario and spell it out. That's a great way of making sure that the objective and the assessment are at least linked in some capacity. And once you do that, only then decide whether AI supports the practice, belongs in the assessment portion, there you have it do the Socratic method like we talked about, or if it even belongs at all. Or would you rather do this in class instead?
Speaker 4:So that kind of wraps up the assessment portion of our AI resilient backwards design.
Speaker 3:Mhmm.
Speaker 4:And I'm looking forward to talking to doctor Rogers as we go through assessments that he's gonna bring to the table where we can have these conversations.
Speaker 3:I'm looking forward to it as well.
Speaker 2:Thanks for listening to this episode of Transform Your Teaching. If you have any questions or comments about our conversation about assessments today, feel free to email us at CTLPodcastcedarville dot edu. You can also connect with us on LinkedIn. And finally, don't forget to check out our blog at cedarville.edu/focusblog. Thanks for listening.