Subscribe
Share
Share
Embed
The Transform your Teaching podcast is a service of the Center for Teaching and Learning at Cedarville University in Cedarville, Ohio. Join Dr. Rob McDole and Dr. Jared Pyles as they seek to inspire higher education faculty to adopt innovative teaching and learning practices.
This is the Transform Your Teaching Podcast. The Transform Your Teaching Podcast is a service of the Center for Teaching and Learning at Cedarville University in Cedarville, Ohio.
Ryan:Hello, and welcome
Ryan:to this episode of Transform Your Teaching. In today's episode, Doctor. Rob McDole and Doctor. Jared Pyles continue our series on AI resilient course design by chatting again with Doctor. Trent Rogers in our lab.
Ryan:Thanks for joining us. Well, Rob. Yeah.
Jared:We're back. Another lab episode in our AI Resilience series. Trent Rogers decided to stick around.
Rob:He's back.
Jared:He didn't ditch us after the first one. So apparently, we're having a good conversation, which I I thoroughly enjoyed our last lab episode with him.
Rob:Yeah. Because where we left it off last time, doctor Rogers
Jared:Cliffhanger.
Rob:Again. Thank you for joining us.
Trent Rogers:Yeah. Happy to be here again.
Jared:But we are on to assessments. We we established your objectives, and we thought they were thumbs up. Good. Now we're moving on to the assessment portion that is then going to assess these objectives. And we're really gonna focus on the AI resilient part of this, because we mentioned in the last lab that, you know, you could make your objectives AI resilient.
Jared:Really the payoff is during the assessment portion. That's when you make it really AI resilient.
Rob:And I and I think we identified the second objective. Just let's state that again, if you would, Trent, for our audience as the one that needed more AI assist resilience help.
Trent Rogers:Develop teaching outlines and messages to express the author's intended meaning.
Jared:That was the one we were like, we gotta really make sure that we make this one AI resilient, the assessment portion of it. So maybe we should start there.
Rob:Right. So obviously, AI can make those outlines and messages pretty easily.
Trent Rogers:You know, chat can generate you something. Right. But what I really want is the student to be able to do that, not for them to derive that from something else. And I and, know, I I do think that that is a challenge because students sometimes confuse the product of what I'm trying to achieve. The product is actually them.
Trent Rogers:Like, I want them to be shaped. I want them to be formed.
Rob:Right. So it it... I think one of those ways that it comes out would be in their judgment of whether or not that outline is correct or not. So I think there's a way that you can get to that using AI actually. And that is you could create a skill because that's where things are right now.
Rob:Instead of a custom GPT, you could actually create a skill that would intentionally create outlines of passages, let's say, first John, that have enough good stuff in them that you could say, oh, yeah. Okay. That's right. But then it it slightly puts in error, and the students actually have to interact with this and correct it. So that's a possibility.
Trent Rogers:When I think about how do I create this resiliency, I have a... I teach... So just to recall from the first episode of this lab, this is primarily designed in the online format.
Rob:Right.
Trent Rogers:But I teach similar skills in a residential format. So students can have the same temptation Oh, yeah. To run to chat or in any other Mhmm. Sure. AI platform to create a a written document.
Rob:Mhmm.
Trent Rogers:One of the ways that I've created some resiliency in person has been through the series of worksheets that I actually make them handwrite. So thing about a worksheet, well, I'll just describe, I'll describe the worksheet. So we think about a series of exegetical steps that's needed to arrive at the ability to construct a, I'll call it a sermon or a teaching outline. So in a residential format, we're starting them with text criticism. Well, I actually make them, you know, list out things in handwritten form and they have to, you know, list things in certain boxes of these worksheets.
Trent Rogers:And then they have to give a written evaluation and assessment. And then we go to a next phase, we go to like word study and I actually make them write it out in these worksheets so that if I don't think chat in its current form could do that. And even if it did, they would have to then transcribe it all with handwritten form, which would again seem... It goes back to to use AI is is longer than actually developing the skill that I want them to develop.
Jared:Yeah.
Rob:That wouldn't... But it still would not surprise me if you had had a student actually do that, which would... That would even give me even greater concern given the program that you're
Trent Rogers:you're taking
Rob:them through.
Trent Rogers:I mean, I I read a report of a student who trained AI... Trained in AI to use his script of handwriting.
Rob:Yes.
Trent Rogers:And to be able to... So... Yeah. I mean, that is... It is possible, but I think that's...
Trent Rogers:Maybe that's not resilient. Maybe that's AI resistance that I'm creating in there. But at least I'm making them demonstrate incremental steps Mhmm. Towards the product rather than just give me the product. Yeah.
Trent Rogers:Because if I if if I just say give me the product, it's so easy just to say to an AI, give me the product.
Jared:Yeah. Yeah. I wonder though, you've also mentioned that this this specific objective, this isn't new to them. Already established practice. So I wonder at this point, since they've done it so much, hopefully on their own, that they're less likely just by nature to use AI because they've done it for so long and they're like, oh, I've done this.
Jared:So it's, I have a practice, a routine, I know what I'm doing, and maybe I'm less likely to reach out to ChatGPT to ask you to do that for me.
Rob:Yeah. What the reason that's probably primarily the reason I went the way I went in terms of how you could use it. Mhmm. Because what you're really getting at is judgment at a high level and being able to spot error. And spotting error, honestly, is probably one of the more difficult things to do because if you don't know it Yep.
Rob:If you don't understand how to make it in the first place, you're not gonna be able to spot the errors. So it almost becomes like... You can almost get into one of those situations where I know it's not an oral examination, but it is an examination, one that's bound by time. And here's why. Because if you say, you have got x amount of time to do this with the bot, then you require them to give you the transcript.
Rob:Yeah. You then have something that you can evaluate because you've got time. And if they weren't able to get through it in a matter of time, then you know they're not where they need to be. Right? Because when they sit down to do some of these things like what you're talking about, they should be able to do that, I would think, fairly quickly.
Trent Rogers:One of the challenges with our point two of develop these teaching outlines and messages is there is something hesitant to use the term subjective to the way that I would express the biblical text. So, you know, the three of us, if we were to sit down and we were to carefully exegete, since we're using first John in this course, some aspect of first John and then construct it in a teaching outline, even though our outlines would, I think, demonstrate the same mastery of understanding the biblical text, the way that we choose to express it for an audience will have some personality
Jared:Right.
Trent Rogers:Of Right. The teacher involved. So you could be dealing with expressions of the text that might not be categorically wrong, but just wouldn't represent my own work.
Jared:Now important question. Of the three of us, who do you think would do the better job of
Trent Rogers:A fourth person. Just
Jared:curious.
Rob:I appreciate that. And and I think that illustrates to our listeners, you need to make sure that you know what you're trying to get to. Right. So I would say, in this case, you've you've identified those things that are most important for the students to be actually do.
Trent Rogers:Mhmm.
Rob:And that's really the key in AI resilience is, one, identifying what that is. So you're saying, I need to see the personality. So however you use the bot or use that, you need to be able to see it. You need to be able to assess it, it sounds like.
Jared:Yeah. Even in the word choice, you would see if you, especially if it was something developed by a bot, would see them struggle over or use words that... I mean, you see that in people's writing too when they use a bot or use ChatGPT to develop something that's not in their voice, which call it voice and composition. But it's that idea where it doesn't sound like them or it doesn't feel like them. You would usually see that if you were to, if someone were to use a bot in that manner.
Trent Rogers:That would be an interesting experiment to have a student teach their outline and then maybe have a bot evaluate their written and oral expression and compare the word choice between those two.
Rob:Also a very good practice for AI resilience. Yeah. Yeah. No. Because then you force it, and you could also bring it in to compare to, like...
Rob:Since you've got this program, you probably have some residual artifacts. Yeah. Portfolio almost. Some portfolio where you can say, no. And it would be helpful for the students too to reflect on where they've been and where they are now.
Rob:But I think what you're actually trying to capture is the relational human perspective, which... That's what this is. Mhmm. It's more than just teaching content. That's right.
Rob:It's the content that's changed you that you're offering to others to have it also be offered up because it's something else when you have somebody just tell you content versus, oh, I know this guy. Yeah. I know this guy, Trent. Mhmm. He's relating something personally.
Rob:And people know when you're being personal and when you're not. And so that's... Well, that's that's difficult to
Jared:assess.
Trent Rogers:Yeah. This is a high level skill. The ability to exposit a text to people in your own personality. I mean, that that that that that is rightfully the capstone
Rob:Oh, yeah.
Trent Rogers:Of a ministry degree. Sure.
Rob:Yeah.
Jared:So yes, we did park on the second one because that was the, you know, I'm curious about the first one though, interpret the biblical texts using proper exegetical methods. I feel like that's one you could easily spoof with AI as well.
Trent Rogers:Yeah. So there are elements. So I I mentioned one of the ways that we we create AI resiliency in a residential format is making them demonstrate incremental skills Mhmm. On the way to the final product and making making the incremental skill development accessible to me and making them turn that in. Mhmm.
Trent Rogers:And I make them do it handwritten just because, again, that's just one more element that that makes him think, it's gonna be a whole lot harder to to do this with chat than to just learn
Rob:the skill.
Jared:Gotcha.
Trent Rogers:So... But I think we've built some things into the course Mhmm. That actually get to the assessing of their ability to interpret biblical texts. So let's just use... I'm gonna use one example, and this is something that we we talked through.
Trent Rogers:So historically, we have exam prep. This this course has a midterm and a final exam. So historically, what we've done is we've said, okay, I'm gonna give you 10, 20 possible essay questions. Questions.
Rob:Mhmm.
Trent Rogers:And the idea behind that was then the student goes back and says, this is an essay question that I need to be able to understand and express so that the student formation actually happened in the preparation for the exam.
Jared:Correct.
Trent Rogers:Right? So that, like, we mentioned, like, let's say, the Christological emphases of first John. Okay. So the student has to go back and to to develop that that essay as a prepared essay question, and really the formation was not in them coming to the exam and and, you know, verbatim saying what they had thought through in person. Mhmm.
Trent Rogers:The formation was actually having to to do that themselves. The problem now... Or the... I shouldn't say the... At least the challenge now is input into chat.
Trent Rogers:Give me the crystalized parentheses, and then and then all they're doing is memorizing that artifact that chat produced
Jared:Right.
Trent Rogers:Rather than actually being formed by it. So one of the things we built into the course is to say, actually, in the exam, you're gonna have to demonstrate more mastery than merely regurgitating a prepared outline.
Jared:Mhmm.
Trent Rogers:So let's say I have, well, there's a section on the gospel of John. So maybe we say, maybe one of the prepared essay questions is describe the Christological emphases of the gospel of John. And then we have another prepared essay question, describe the Christological emphases of first John.
Rob:Mhmm.
Trent Rogers:And then the... And we tell them ahead of time that these exam questions... These essay questions will not appear in their exact format. They actually might ask you to use two of these responses to answer the exam question. Mhmm.
Trent Rogers:So that when they come to the exam, the question is not just give me the Christological emphases of one of these books, but it's actually compare and contrast the Christological emphases of the gospel of John and first John, which requires a greater mastery. Right? So Right. Even if they even if they took a shortcut and used an AI to create those initial things. They have to own those Sure.
Trent Rogers:And think through those in a way in the exam that actually makes the exam a little, I mean, a little more formative Mhmm. Than historically it might have been. That's just their moment to, you know, jot down the outline that I've memorized and fill it in.
Jared:That's great.
Trent Rogers:Do you wanna go to objective three? Sure. Should we remind them what that is?
Jared:Well, I'll give you the one that I updated it to be.
Trent Rogers:Modification. It's not it's not a rewrite.
Jared:It's small. Express the author's intended textual meaning through written and oral communication in a contemporary setting.
Trent Rogers:Maybe it's best to focus here on on the oral communication because we've talked about written communication. Mhmm. Mhmm. But oral communication, at least at the current... There was current moment in AI, it's a little harder to manufacture.
Trent Rogers:Right? To to... For... Maybe I'm wrong on that, but
Rob:Depend on the tools, and it depends on what you're having them do.
Trent Rogers:It it probably won't always be hard for a student to, you know, take some video of themselves and have it, you know, produce a delivered sermon, if you will. Yep. So we'll probably have to think about ways that we adjust it as AI technology develops. Right now, we just have them record a video of which again is gonna draw on all of those skills. And we're actually evaluating not just the content of what they say, but how they say it, how they express it.
Trent Rogers:So, you know, we'll have instructions for... On some of these, oh, if you're doing a devotional, maybe I just need to see your headshot. But there's gonna be some of these that I'm gonna need to see your whole body in the presentation. Mhmm. Because we want students to to be able to to stand and teach God's word.
Trent Rogers:Mhmm. Not every setting requires that. Correct. But some do. And so a student that graduates from our program, we want them to have that ability.
Rob:Yeah. I don't I don't know if there's anything more that you could make it any more AI resilient than that. I mean, the only thing I could think of would be... I mean, if you wanted to use AI as a critic.
Jared:I do like... Yeah. I was gonna say that that idea earlier that you brought up of having it look at the written outline and then compare it with a recording or a transcript of what they actually do and percent would be interesting to look at. Because that's using the tool in a way that enhances or reinforces, refines what the student's doing.
Rob:It's kinda similar to what I do with my students. It's like, have them actually put their written work in there, and then I have three questions that they have to have AI. They have to ask AI, and it critiques their work. And then I don't leave it there. I'm like, you have to decide then whether you're going to actually use what it told you if you think it's helpful or not.
Rob:And then, you know, you're gonna need to tell me, okay, why?
Trent Rogers:So I I wanna ask the question for you. Are we... Am I allowed to to ask questions? Okay. So let's envision a setting in which maybe three years from now, four years, I don't know how long it'll take, in which it's very easy for a student to produce a video of themself teaching a text even in a ministry context.
Trent Rogers:Right? They they... They've got a congregation who, you know, is queued in to laugh at their jokes. Right? You know?
Trent Rogers:That's scary in one sense, but it's actually not that hard for us to envision that. Sure. The the technology would get there. Mhmm. How would you build in AI resilience in that in that technological environment?
Trent Rogers:So in an ideal scenario, we might be able to do it live. Right? Mhmm. The the challenge with that is most online students are an online student because synchronous learning, what... Well, I should say synchronous residential learning was not an option for them for any number of reasons.
Jared:Right.
Trent Rogers:And it's not just the residential component. It's also the synchronous component. So, you know, in an ideal setting, they would be in front of me, and I would be evaluating. Mhmm. And next to ideal setting, I would be, you know, synchronous live watching them teach Given that maybe we can't do either of those Mhmm.
Trent Rogers:How do I build in AI resiliency in oral delivery
Rob:For three years from now?
Trent Rogers:Let's say ten years from now. Ten because let's envision. I think ten years from now, let's we probably agree that that technology is going to be more easily accessible. So ten years from now, how do I build in AI resiliency watching to evaluate students competency in teaching the biblical text?
Rob:I already have an answer, but I wanna hear what You do? I do. Yes.
Jared:I was gonna say holograms, like Star Wars.
Rob:You put holograms in there. I I think you can go back with old technology. I would actually refer back to our previous conversation. Same way you use a human, the same way you use a human proctor, you could use a human proctor in the sense of this oral communication where you have you have them verify that they are actually in the room listening to or watching watching this go on, and you have them show up on the video and make a statement.
Trent Rogers:Well, but I can... If I can if I can create the... I can spoof that too.
Rob:Yeah. Right. You can spoof it. But here's the thing though, like, if you verify through your proctoring that this person that they're working with in a church setting is going to actually
Trent Rogers:Yeah. I think I agree.
Rob:Be there, then then
Trent Rogers:I think a proctor can help. I don't... I think having the proctor appear on the video is superfluous since we're we're thinking about the idea of them creating... Being able to create themselves in a video.
Rob:Understood. I was just I was just saying it's like, sometimes it's nice to see people. I mean, genuinely, like, having worked with this from a distance ed perspective and, again, when I was younger, I appreciated actually meeting people.
Trent Rogers:Yeah.
Rob:Even proctors, believe it or not, like, because it's really cool when you actually put a face to a name and you have a conversation. And so, yeah, it is superfluous. I won't... I'll give you that. If if you've got a verified proctor and they said, yeah, I sat there and I watched this, then you're good at a minimum.
Jared:It's gonna require a whole new way of assessing that we don't... That maybe ten years down the road will exist. But right now, I don't... That's a great idea, but I... That's a really hard question.
Jared:I would hope, though, that a student who's trying to finish a degree and then go and lead a congregation would not stoop to that level to do that.
Trent Rogers:But if that's the case, then we don't need... If that were always the case Yeah. Then we wouldn't need to have this conversation about Yeah. About my programs. Right?
Trent Rogers:We wouldn't have to talk about AI resiliency
Jared:and Right.
Trent Rogers:And AI detection. Right. But the reality is there is some sense in which we have to form their character as we're forming their skill set.
Rob:Well, and having boundaries is an appropriate thing to do that. And I think that's what we're really talking about here. I would I would say this to anyone who's who's listening. I say this to you as well. There's no way with any of this that we're gonna get a 100%.
Rob:Right. And that's why I call it AI resilience. Like, within reason, should this keep my students, your students, from thinking twice. I mean, at least cause them to say, is this really the way I wanna go? Like, make that easy go to chat, do this for me that much more difficult.
Rob:And I think that's ultimately, you know, you said you want them to be formed in that moment of working with that material, and I think we're providing boundaries that at least they have to hit up against those and and ask themselves those questions.
Trent Rogers:I I think part of what you're saying is they have to be aware that they're they're consciously making a a moral choice for bad. Correct. Right. And I think part of what we're having to do is to train their conscience because
Jared:Yeah.
Trent Rogers:You know, there are a lot of students that are just assuming like, oh, yeah, I can produce the product and I can do that with chat. That's fine.
Rob:Can means
Trent Rogers:should. That's right. That's right. So there... The...
Trent Rogers:Maybe the term you're saying is is hit up against a boundary. You're you're... And and hitting against something, you're you're training them on legitimate and illegitimate uses of AI. Okay. My next question is this.
Trent Rogers:How do we... What's the role of timed assessments in creating AI resiliency?
Rob:Oh, those are... Timed assessments, I think, are really good. I use them a lot.
Jared:Not even just for AI resiliency. Yeah. Just in general. Yeah. I don't know if they necessarily fit the box of you do this as a result of AI resiliency.
Jared:You do them because they're a way of, keeping test integrity. Yeah. In the same way as like shuffling answers or, pulling from a test bank That that... But for AI resiliency specifically, I'm not.
Rob:I'm... And I use that in mastery quizzing. So timing, you can't cheat time. Mastery quizzing, basically, I take a... Create a test bank on certain topics or whatever it is I'm quizzing on, and I usually use it in a formative way.
Rob:Mhmm. So it's like if it's over reading, then I'll have them... I'll have... I'll create formative quizzes, and it'll be a bank of questions that we'll swap out, and they'll be able to use, they'll be able to take that within a given time limit. So it's usually, like, 10 questions.
Rob:And, you know, when they see the question and it's... It is synthetic in the sense that requires synthesis for the student to not only know the... At least mine, to know what the fact is, but they have to be able to recognize that fact and then synthesize it in some way to answer the question. So I usually give anywhere from thirty to forty five seconds per question, and I have 10 questions. And they can take that.
Rob:And then if they don't do well on it, they can take it again, but it'll give them a different set of questions on the same material and... But it will space that next thing out by an hour, or you could do it even longer than that depending on what kind of time you had. But that's essentially when I say master quizzing what I'm talking about.
Trent Rogers:Yeah. So full full disclosures, we go back to the lab of this particular course. There's a mastery quiz every week Yep. That we've embedded within the course. Mhmm.
Trent Rogers:And now sometimes when you do the mastery quizzes, you let them... So they get 10 questions out of a question. So you've got 200 questions in your bank. They get a random set of 10. Let's say they get a five out of 10.
Trent Rogers:Sometimes you design it so, hey. They can go back, do the reading again Mhmm. Mhmm. And come back. And sometimes you use a time thing.
Trent Rogers:Right? You say, they've gotta wait ten minutes before they take it again. So make sure they don't sit there and just click, click, click until Mhmm. But that's a way that timing
Rob:Yep.
Trent Rogers:Creates a little AI resilience. Right? Yeah. It's pushing them back to make... To form them.
Rob:Because if they know it, they should be able to come out with it pretty quickly.
Trent Rogers:That's right. Yeah. Yeah. I think this is an important discussion, though. It's a discussion that, in some ways, I started with COVID because when COVID hit, I'd been teaching full time for two years.
Trent Rogers:Wow. So I was just starting to think about, am I assessing the things that most... That matter most?
Rob:Yeah. Mhmm.
Trent Rogers:And then COVID basically shaped everything and then, like, we... When we we went online fully for a little bit of time, like, changed.
Rob:Yep. Yep.
Trent Rogers:And we had to really reevaluate, are we... Are assessments getting to the heart of the matter that we most care about? And in some ways, AI makes us do that again. So when we think about faculty, I think if we have an exhortation to faculty, this is a pedagogical challenge of our current environment. And that's what we're do...
Trent Rogers:We're educators. So we're not just masters of our subject matter area. We're all... We also need to master how we deliver and assess that as educators.
Rob:I think that's a great way to end.
Ryan:Thanks for listening to this episode of Transform Your Teaching. If you have any questions or comments about our conversation with Doctor. Rogers, feel free to send us an email at CTLPodcast cedarville dot edu. You can also connect with us on LinkedIn. And finally, don't forget to check out our blog at cedarville.edu/focusblog.
Ryan:Thanks for listening.