Solutions Economic and Market Watch

Join Economic & Market Watch producer Kathy Ryan for a bonus, behind-the-scenes look at how AI has changed the economic research team and the show in the past year. 

Links:
Contact the Economic & Market Watch team at economicresearch@nrucfc.coop.
Visit us, download the dashboard and explore other Solutions media on our website, nrucfc.coop/Solutions.

Creators and Guests

Host
Antony Davies
Antony Davies is director of economic research at CFC. Visit his full bio at https://www.nrucfc.coop/content/solutions/en/author/antony-davies.html for more.
Host
Sam Kem
Sam Kem is a senior economic research analyst at CFC, where she is responsible for the management and creation of economic research, writing and presentation materials for internal and external audiences. See Sam's full bio at https://www.nrucfc.coop/content/solutions/en/author/sam-kem.html.
Producer
Bonus Episode
Highlighting features of the show, CFC's economic publications and financial tools and the lighter side of the research team.

What is Solutions Economic and Market Watch?

Listen to the latest economic insights from CFC experts John Suter, Sam Kem, and Antony Davies.

Kathy Ryan:

Hello, and welcome to this bonus episode of Economic and Market Watch. I'm the producer, Kathy Ryan.

Kathy Ryan:

Our usual content is on hiatus as we observe Memorial Day, but I'm here this week with a behind the scenes look at the show. It's about AI, which well, we do cover that a lot. This time, though, instead of covering economics of AI or AI's effect on the economy, we're looking at AI's effect on the economists.

Kathy Ryan:

CFC launched AI access company wide last summer, which seemed like quite a historic milestone to me. Being a financial history buff with a microphone, I wanted to document how it would change how John, Sam, and Antony do what they do. So I started leaving the recorder running after the last take some weeks and asking what was new in their AI lives each time.

Kathy Ryan:

Before we get started, a few caveats. First, the economists aren't AI experts, just AI users. In Sam and Antony's case, very frequent and enthusiastic users, but still.

Kathy Ryan:

Second, these were informal chats and they took place over eight months. We don't always specify which AI we're talking about. It's usually ChatGPT or Copilot, but each of these have changed so much and so often that August 2025 is practically ancient history. So some of these observations just may not apply anymore.

Kathy Ryan:

Third, Sam and Antony were using CFC enterprise accounts for their research and following CFC's AI governance policies. I can't stress enough how important it is to know your co op's policies and follow them. Don't take my word for it though. Brian Sloboda of our industry research team did a great video about exactly this, and I'll put a link to it in the show notes for you.

Kathy Ryan:

Here's Sam explaining one of the drawbacks she was finding with her AI last August.

Kathy Ryan:

So Sam, you told me between takes that the AI you're using is bad at numbers. Could you tell me more about that?

Sam Kem:

It kind of makes sense because if you think about it, AI is actually you know, it's a language model. So it's actually pretty good with words. So, you know, if you ask it to do tasks like summarize a text, it would do a fabulous job at it. Or even even if you ask it to draft something with a certain word limit, they typically do a a good job with that. But when it comes to numbers, especially if you ask for precise numbers, AI will most likely hallucinate.

Sam Kem:

So that's why I do not rely on AI for things that require precision. Because if you have to, you know, you have to keep in mind that some a lot of time, the answer AI gives you is not the exact answers, especially when it comes to numbers. It's an illustration.

Kathy Ryan:

And this is Antony, back in September, with a few reservations of his own.

Antony Davie:

At your prompting, I was thinking about doing an episode with AI, and I thought the interesting twist would be to actually have the AI with a verbal interface, so I can talk to it in real time and it responds on the podcast just like a guest would. And I was experimenting with sorts of things that that I would ask it. And again, at your prompting, was asking, "Okay, give me examples of ways in which I've changed your mind or ways in which you've changed your mind." And the particular AI I'm using has a memory, so it knows, you know, going back twelve months, all of our interactions.

Antony Davie:

And it starts saying to me, oh, well, ways I've changed your mind are the following things. And it gives these examples of things where there was some error and the error was fixed.

Antony Davie:

And the darn AI, it's taking credit for my work. In these examples, it was wrong. I was correcting it. But somehow in his mind now, it thinks, "Oh, no. That was my idea all along."

Kathy Ryan:

But Ant was also discovering that his AI was useful for working on scripts.

Antony Davie:

But nonetheless, this is one of the the great things I have found great uses for AI in my work, is when I write scripts for the podcast here and when I write the intelligence brief and pretty much any other document. Before I send it anywhere, I upload it to AI and I say, fact check this.

Antony Davie:

It's been incredibly useful. AI will come back and, you know, if there's some factual numeric error, it flags it. Often when it finds a numeric error, it's because it's looking at data that's more outdated than what I have. But nonetheless, it's an interesting check.

Antony Davie:

Most of the more useful checks are when AI tells me I've stated something too strongly. I need to back off. And you know, I get a little bit excited, carried away with some of the things I say, right? So I think, yeah, AI's right. I need to soften this a little bit.

Antony Davie:

I found it really good as a check in that manner. I like to think of AI in general as a very talented graduate assistant. It can do wonderful stuff, but always be looking over its shoulder. Don't ever let it do something and just send it out into the wild.

Kathy Ryan:

By October, we were experimenting even more with script development. Here's Sam and I during a recording session where she had tried out something new.

Kathy Ryan:

Looking at the script for today, which our listeners can't see, but which includes not only pauses for you to breathe and shorter sentences so it's easier to get through any given sentence in one take. It also includes script notation. How did you ask AI to do that for you?

Sam Kem:

Other than explaining it, that this is a podcast related to economics and that's specifically for cooperative members, I also tell it that to ensure that it's the script is smooth, that I have room to breathe, and also that it sounds natural. And then it produces the text and then it asked me if I wanted it to add things like pause, slow down. It even tells me to smile in my voice. I don't know what that's supposed to mean, but apparently it's a thing.

Kathy Ryan:

Oh, I can hear it. I can hear it.

Sam Kem:

Okay. Yeah. So, yes, exactly. And then I -- I said, yes, do that. And that's the result.

Kathy Ryan:

Okay. To contrast this, John is also working on a script and as an experiment, you and John fed that into AI and asked it to clean things up a bit.

Kathy Ryan:

That one, I was disappointed in the result because it was a little cleaner on the surface and certainly, it touched up the grammar and it gave pauses for breath and all those kinds of things that are helpful. But when it was done, it didn't sound like John anymore.

Kathy Ryan:

A large part of producing the podcast, I like to think, is that you sound like you, Antony sounds like Antony, and most of all, John sounds like John because everybody knows John. He's very distinctive, and the result from this pass through AI sounded a lot more like a a BuzzFeed listicle article and not like John's take on economics.

Sam Kem:

Well, I would say that the biggest difference between when I did it for my script versus when we help him do it for his scripts is that I use AI more often. So I use you know, it's it's my account. Mhmm. So my AI actually knows me. After a little while, AI will will pick up your tone, especially if you feed it your own writing many times.

Sam Kem:

And this, you know, have to remember it was John's first time. So his AI doesn't know him. His AI, it hasn't had the time to or, you know, the examples of his writing, his tone to work with just yet. The thing with AI is that, you know, the more it knows you, the better it works for you.

Sam Kem:

There's another thing that I would use it outside of, you know, like text and scripts and things like that. Have your AI critique your work. And it's not just with writing. You can actually do it with other things too.

Sam Kem:

For example, if you have any sort of statistical model, forecasting model, you know, it doesn't have to be full forecast. What you can do is either type in the model or upload the file or, you know, whatever.

Sam Kem:

Normally, I prefer is to have it in a Word document or any sorts of document and upload it. AI reads it better that way. So, yes, always upload the file and, you know, I would do that.

Sam Kem:

Like, "Here's my model. Look at it. Here are the reasons and the assumptions behind the model."

Sam Kem:

You know, I list all of them. Like, I'm I'm doing this because of this and these are my assumptions, blah blah blah. And then I would tell AI, you know, whether that's Copilot or ChatGPT, "If there's any mistake in this, tell me what it is." Now, the reason why you have to say that because, again, constraint.

Sam Kem:

Because if you just ask it for mistakes, AI will come up with a mistake. So that's why you have to make sure that you give that constraint that, "Only if there's this mistake, do tell me and why. And if there's a way to improve that, tell me how." It's the same thing that you would normally do that everybody tells you to do with AI except adding more constraints. Adding more constraints would reduce hallucinations, which is basically the biggest problem with AI.

Kathy Ryan:

Here they are in November, and they're starting to get a real handle on the AI's strengths and weaknesses.

Antony Davie:

I use AI on a multi daily basis. I find two things, one in which it's not that great and one in which it's pretty cool. And the not that great thing is pretty much any time I ask it some question, a factual question, it seems to just go through its memory and spit out stuff. And there's a, I'd say, 40% chance that the stuff it spits out is just wrong. And you don't know this unless you challenge it.

Antony Davie:

Oftentimes, it will come back. In fact, it did just this morning. I was asking it about treasury auctions, and it told me a bunch of stuff, and I started digging deeper. I said, tell me more about this. It ended up contradicting itself.

Antony Davie:

I said, "Hang on. Earlier on, you said X and now you're saying not X." It comes back as if it had discovered this wonderful thing. Said, "Oh, yes. Quite right. Good catch. Here's what's going on here," completely ignoring the fact that it contradicted itself. It's just going to say more things.

Antony Davie:

So the first my first thing is a word of warning.

Antony Davie:

Be really, really careful. Do not take anything it says as as fact. Take it as a starting point. That's where AI, I find fails. Where AI does really well is when I give it something I've written and I say, "Fact check this."

Antony Davie:

And AI is really good at coming back at me and saying, "No. The these various things you've said are incorrect and here's why." Now, again, AI will err. I'd say 30% of the time, things it says to me and it says, "Look, you've said this thing, it's incorrect." AI is actually incorrect. I'm right.

Antony Davie:

But that's okay. I'd rather err on the side of it flagging too many things than too little. But often it will find things that I realize, yeah, I need to I need to rephrase this or or find a different source or whatever it is. So in terms of using AI to replace the human, I find AI consistently fails.

Antony Davie:

Using it to augment the human, I find AI is absolutely superb.

Sam Kem:

I find that my AI is an overachiever because we were working on trying to, like, have several items of electric utility equipment or components that we could track prices of. I went to AI and just kinda like, "Hey. Give me some equipment components that, you know, electric co-ops use and break them down by categories." AI gave me, like, a 100 of those. Yeah.

Sam Kem:

And I'm like, yeah. Sure. I I mean, I'm not gonna do all those. It's an overachiever sometimes. It's good because it gives me all more options.

Sam Kem:

It's bad because it gives me more options. I was writing a monthly snapshot, basically just kinda try to describe the economy for the month. And, you know, it's a bunch of things, you know, weak labor market, blah blah blah. And then I realized that, okay, it probably sounds a little too gloomy, so I uploaded what I wrote to AI and told it to make it less gloomy. It generated something that's pretty much the same as what I wrote, except it added a sentence that say, well, on the brighter side, the UK inflation dipped to blah blah blah.

Sam Kem:

And I was like, okay, I guess we don't have anything nice to say about the US.

Sam Kem:

There's a follow-up story to that too.

Kathy Ryan:

Okay. Great.

Sam Kem:

The next day, you know, I asked it to provide a summary of something, blah blah blah. And then I noticed in the results that it gave me, my AI turned British.

Sam Kem:

What?

Sam Kem:

Yeah. It's spelled emphasized with an 's' instead of a 'z,' labor with a 'u', and all of that.

Sam Kem:

And then I asked it, "Why are you British?" And it responded that "I'm trying to bring culture to this conversation." I got no response to that.

Kathy Ryan:

I checked in again with the team in March. John and AI were still just getting acquainted. Sam wasn't asking for script notation anymore, and ChatGPT wasn't telling her when to smile. And Antony had promoted his AI from graduate student.

Kathy Ryan:

Things had really changed in the way Sam and Antony were doing research... but I'll let them tell it.

Antony Davie:

I've come to appreciate what it is and what it isn't. And in so doing, it's become much more distinctly a tool for me. When I first started messing with it, as most people do, the thing looks like a sentient being. And I came to the conclusion after dealing with it intensely for a long time that AI is really a mirror. It's a distorted mirror, so it's not obvious that you're looking at yourself. But that's really what's happening.

Antony Davie:

And so as you talk to it and it comes back at you and it seems like it's a sentient person, of course, seems like a sentient person. It's you, Talking to yourself. You just don't realize it. So in any event, that that was an interesting insight that got me solidly on the track of using it as a tool.

Antony Davie:

And when you use it as tool, you have to be very careful because it's not a tool like a computer despite the fact that you're dealing with a computer. The nice thing about a computer is you tell it to do something and it does it. And if it messes up, you know it's not because the computer made an error, it's because you made an error in telling it what it is you wanted it to do. Not so AI. AI has kind of a mind of its own.

Antony Davie:

But here's what I have come to use it for here at CFC. I started off developing what I call an economics expert that basically, this is an AI set up with with instructions and background to look at the world kind of the way I do as as an economist.

Antony Davie:

"Look at it. There are certain things I'm interested in, other things I'm not. We're gonna look with skepticism. Everything we're going to say is is gonna be backed by data, that sort of thing."

Antony Davie:

And I've come to to rely on this tool. Anything that I write from from articles that are going out to even podcast scripts here with Economic and Market Watch. Before I release it, I'll give it to the economics expert and I'll say, "Verify this. Fact check it."

Antony Davie:

And it will go out to the Internet and it will find sources to refute or support what it is I've said and it will come back and it will say, you know, the following things are okay. This thing is is ambiguous. This thing over here is completely wrong. And it's caused me to catch a number of of errors before they've gone out the door. It's really useful.

Sam Kem:

So the way I work with AI now is a bit different than before, and I think it does provide better results. The way I do it now, I don't go in and ask what I want right away. I warm up AI first. I would first start having a conversation with my AI about the topic that I'm getting into.

Sam Kem:

So I would go, what's going on the war in Iran right now? How much of global exports in oil that will be impacted and things like that? And then I was like, how about natural gas? And then it start giving me answers, and then I start picking on that. And then what are some other hidden things that we don't always talk about?

Sam Kem:

And, again, I was trying to get at something, and then it eventually gave me that answer.

Sam Kem:

You know, the byproduct of oil and gas refining is sulfur. And sulfur is the quiet but essential input for industrial activity. Then I asked AI. I was like, well, do tell me what are some of the things, what essential in electric utilities that sulfur would impact, and it named copper. And it's correct because sulfuric acid is essential in copper mining, copper processing.

Sam Kem:

And after that, I was like, okay. Well, I did not know this about copper. So I did my own research outside of outside of AI, and then I went back to the conversation.

Kathy Ryan:

I'm gonna have to cut Sam short here, but if you're interested in the article she was researching about the Strait of Hormuz and risk to rural America, There's a link to it in the show notes.

Sam Kem:

I went into AI knowing different things already, and I asked AI a bunch of questions. And then when they give you answers, then you kind of pick up a few things in those answers too. So having that conversation was very helpful. And then in the end, that's when I have my final instruction.

Sam Kem:

So my final instruction would be, "Look, I am going to present this to the public audience. These are the points I would like to make. I want to connect all the points, all the answers that we just discussed above."

Sam Kem:

It's not like it provided me something new, but it create that chain better than I would have myself because I would I think if I had to explain myself, I would probably go all over the place.

Sam Kem:

By asking AI all these questions ahead of time, it's not answering, and now it knew exactly what I wanted. And I and I think that's the way that I'm going to be working with AI moving forward is not asking it right away, asking, like, warm up my AI first.

Kathy Ryan:

There's one more thing we've been doing with AI in the past year on the production side. It's given us the chance to add a feature to the show, transcripts. AI generates them, I correct them, and you can access them through your podcast app or on the Economic and Market Watch page by clicking the "More info" link inside the podcast player. Lastly, I'd like to thank all of our listeners and subscribers. If you're not already a subscriber, it's easy and it's free.

Kathy Ryan:

Look at the top of the show's solutions page for links to get you started. Thanks again.