Accelerate Data Engineering using MCP Tools

Emil Krause & Jonny Daenen explore how to accelerate dbt development by integrating MCP (Model Context Protocol) with Postgres and Cursor. Emil demonstrates how to solve a database bug by allowing AI agents to interact directly with databases. 

They discuss the setup of a database MCP server, demonstrate its capabilities in troubleshooting data inconsistencies, and highlight the importance of understanding data even when using advanced tools. The conversation also touches on the potential pitfalls of using such tools and the need for technical expertise in leveraging them effectively.

Resources:

• Demo code: https://github.com/datamindedbe/demo-technology-exploration/tree/main/demos/postgres_mcp
• MCP 101: https://www.youtube.com/watch?v=fIr55-koOJQ
• Click here to watch a video of this episode.
  
• Full playlist: https://www.youtube.com/playlist?list=PLJ_da7qdfL80rA7byzC_CmyrfJWjcCTnb

Chapters:

• (00:00) - Introduction: MCP + Postgres
• (02:20) - Demo: debugging salary percentiles
• (06:29) - Creating and testing dbt models
• (07:11) - Benefits and dangers of AI assistance
• (09:42) - Setting up Postgres MCP in Cursor
• (12:57) - Challenges & pitfalls
• (14:54) - MCP vs semantic models
• (17:16) - Other dev tasks
• (18:39) - Claude Desktop vs Cursor
• (19:59) - Summary

Data & AI: Technology Explorations is a biweekly show from Dataminded. Each episode a Dataminded engineer demos a tool or technique worth knowing about -- working code, honest takes, no hype.

Music by Aleksandr Karabanov from Pixabay

Creators and Guests

Host

Jonny Daenen

Head of Knowledge at Dataminded

Guest

Emil Krause

Data Engineer at Dataminded

What is Technology Explorations in Data & AI?

Deep dives and practical demos on the technologies shaping modern data and AI development. Join the Dataminded team as we explore, unbox, and critically review the latest tools, from building AI agents and RAG systems to optimizing cloud costs and accelerating data pipelines. We cut through the hype to show you what actually works in real data engineering practice, complete with demo code!

Working with databases during development often slows you down. You're constantly switching between code, your database, trying to understand schemas, setting up test data. All of this breaks your flow. AI agents can help, but by default, they cannot reach your database. That's where MCP comes in. MCP can act as a bridge, extending your AI agent with the ability to query and interact with your database directly.

In this session, we'll show you how to do this with Postgres using an MCP server. This way, you can supercharge your developer workflow without compromises. Hi everyone, welcome to Technology Explorations at Dataminded. In this series, we aim to give you an initial look at new or interesting technologies. My name is Jonny, knowledge lead here at Dataminded. And for today, I've invited Emil to dive into the MCP and Postgres world.

Hi Emil. Hi, how are you doing? I'm good here. How are you? Yeah, all good as well. Okay, I propose we will dive right into what you're going to show us.

But maybe a first question. What made you decide to explore a database connection with an AI agent? I don't like writing SQL very, very much. And I find that it takes me quite a long time to write good SQL, even with AI help. And I thought maybe I could just talk to my database using natural language and not use SQL.

Or at least not. But I think I was able to do that in the initial stages of my exploration. And this led me to MCP servers connecting directly to, in my case, Postgres database. Okay, so the goal here is avoiding SQL. Yes, or without hard feelings towards SQL, streamlining your processing by not spending that much time writing queries.

Yeah. Okay, I propose we have a look at what it can do. Absolutely. Let me share my screen. So here I'm going to jump right in and write a couple of questions I have about my data in the prompt.

And we can see the MCP server in action. So I'm using here an employee database with fake HR data that we can use for illustration purposes. So let's start with something simple. How many employees do we have? And let's see what it does.

It knows it has to call the tool right away and lists tables, gets some information and calls a query that you can see by the way yourself in case you'd like to double check other information. And this is the correct result. Let's ask something a little bit more involved if all of them are assigned to a department at the moment.

So in this case, the AI agent knows what to call. We'll dive into that later. It knows which tools are available. And here I see it executing. Right. So it's using real queries to answer your question.

Exactly. Exactly. It knows on its own what tool is best fit for a job. And all of them are assigned to department. Let's have a third and last question about the data. If all of them are in a current role or maybe someone is maybe in between roles or not currently assigned to any role.

And the describe table, could you open that one up while it's thinking? So it fetches. So we see always. So we see the parameters it passes and then the result it gets back. Right. And we see that 15% are currently not in an active assignment.

Yeah, it did a good job so far. But now let's tackle something more that corresponds more to my day to day life. People come with certain complaints or data inconsistencies. And my job is to troubleshoot and debug them. So one of those inconsistencies is just to give you a little bit of the background. Here we have a table where we see the percentile of your salary on the department basis.

So we've been this department who is at which percentile. And we had a report from a person who is the head of customer service and earning 80k. And he's at the 70th percentile in his department, which is a bit strange, him supposedly being the highest earner. So let's use MCP to diagnose this and dive into this. Okay.

So my prompt is I've been hearing that the salary percentile on the department level aren't correct. Investigate. Okay. So in this case, Cursor's model makes a plan first, and then it goes to the steps. And in the steps, it will use your MCP agent to call the database.

Correct. So let's go through what it's doing. So this is a dbt project. Right. So this is a project that uses source tables and builds some other data marts and other business intelligence tables here.

So here, after getting some table samples from the database, it already jumps into the dbt code that generated the table. So here we see the intersection between this being used as a query tool or a fast query tool. And actually something that can repair. What you see in the database, which is, I think the unique, the unique mix of the two.

So so let's see what else. This is really nice. So it looks at the code, which is in the repository of this project, which is SQL wrapped in dbt, of course, or using dbt as a framework. And it also can look at the data at the same time. So it can bring those two together to solve your problem in this case.

Exactly. And here it points me to the database. So I can see that I have a dbt model where I was missing a partition by department and to fix it, I need to add this line in my model. So first it verified using the MCP that the data indeed is not correct in the database. And then it already see how this data is generated and it fixes this.

So would you like me to create a fix for this issue in a dbt model? So let's say yes. So if you say yes, I expect it will do an addition of one line in your model. Aha. Okay. And you're asking to write a unit test.

I see. Good practice. You will avoid future issues. Absolutely. So we see it added the correct partition. So the bug was about, it was giving you the global percentile, not per department.

That was the whole bug. That's why we saw what we saw. And then we... We get a new test. And we add a new test where it takes actual data from seller analytics. And maybe it's not the best test.

Maybe we could give it a bit more prompts. But it's a good direction. Yeah. Yeah. I think this is really cool because it shows how your development flow is now. You have a bug.

You can ask the agent to actually connect your code to the real scenario, the real data in the database. It proposes a fix. And you can ask it to write at least a starting point for a unit test. And it's like, you need some mental capacity to verify that what it does is actually good. So you still need to go a bit deeper to make sure that what it actually does is valid.

So you still need your expert knowledge. Absolutely. And that was one of the main points I wanted to make is that this is not a substitute for understanding your data because you need to be the judge whether the solutions are correct. And especially the unit test. If you don't understand the data, you will not be able to write good unit tests.

So this is a very, very great tool. Yeah. It's very useful to have quick hypothesis about what's happening, both on the output and on the input side in your SQL and in your tables. You still need to understand the data and take your time doing this. But once you do it, you're unstoppable.

You are so much faster using this. Yeah, indeed. And so during development, I can imagine this is a big difference or a big time saver if you can use such a tool. Yes. I had situations where I could diagnose a bug.

I could do this in a minute or two versus if I did it manually, I would have a chain of SQL statements. I would track in different tabs in my pasting it into Google Docs not to lose it. And it would be such a long chain of queries that will lead me to this hypothesis already midway. You forget what you ran a couple of times before.

So this is very good, especially for very chained queries that are not just one query or one. It's a simple bug. And yes, it really reduces your mental capacity needed to actually arrive somewhere. Yeah, it also seems like a dangerous tool. If you don't double check, things can happen that you're out of control of or the tests might not be the right ones.

So you still need to be ready to review whatever comes out of it. Yes. And yeah, this is also running on my local database. Yeah. So I can see where I upload the latest snapshot from a database. We're dealing with very, very small amounts of data.

Of course, as we saw, it just did dbt run. So we need to make sure where you're connected to and. Yeah. Okay, cool. Now I'm curious. This is something you set up, right?

That's the MCP server bridging the gap between the model, Claude 4 Sonnet I see here, and then the Postgres database. Yes. So the very starting point where you have the config of the server. Yeah. So you have the server itself.

You just go to the cursor settings and here you have MCP integrations. This is the server we've been using. And behind this, it's just a JSON file where we have the Python executable and the main entry point to my server. And that's all it is to integrate this to cursor. And then the actual MCP server I also have here.

I created it. It's a fast MCP framework. And it's really simple. It's just actually one file server.py where I have various tools. And the most important one is the one that queries execute query. Here it's important to also note about safety.

So we generally don't want it to drop tables or insert records or update records. So I have this manual safety tool built in that. Here I'm only allowing for select queries. And this will stop any attempt to drop and recreate tables or to do anything else. At the same time, I built it kind of from scratch using the fast MCP framework. I had a look at the landscape today.

There are many ready solutions where no code is required. And you can have a Postgres MCP server just as a Docker image where all you need to do is to pass it the connection string and the mode. Okay. So you can have the only mode or the read and write mode. So it has evolved.

This is probably not necessary anymore unless you want some custom behaviors and then have more of a control. If you like more control, maybe this is nicer. I'm also a bit amazed by the ease that you can integrate this into Cursor. So you had that settings. You can also control, I guess, the different tools you expose.

Like your server exposes 16 tools I see. Yes. Sometimes it's integrated easily. Sometimes it doesn't know it should call the MCP server. When you ask a data question, it's not sure if you want to know about dbt models or if you want to know about the real data.

So sometimes you need to guide it. I want you to use the MCP tool for this. I also have two databases and switching database wasn't there out of the box. So I just created my own switch database. And then it intelligently knows. It can use it to switch the database.

Yeah, neat. You can add some customized options for your context. But given this file, anybody could actually just run their local MCP server and do what you just did, right? Absolutely. It's that easy.

Anyone with basic Python skills can do it. That's why I think the uplift of productivity versus effort spent, it's enormous here. Everyone should do it who has similar use cases at work. Yeah. Besides the knowledge you still need in this case, you still need SQL knowledge. You still need knowledge about your data.

Are there any other pitfalls that you see in hosting something like this? This is an easy scenario because this is hosted locally. It's the Postgres running installed via Homebrew. So this is a local server. I'm sure it gets more complex when it's a remote server or maybe you want to port forward and connect to a remote database.

I'm sure there are some connectivity issues. There are some connectivity or authentication questions that could make it more difficult. I think this is for this barebone example, I didn't come across any pitfalls or anything that I found unintuitive or difficult. I do feel like this is a very good fit for a development use case because you're working with development data, dummy data.

You're working locally, the MCP servers running locally, but you are still using a remote model in this case. Claude 4 Sonnet, which is used by Cursor in this case, as we see in the bottom, that will receive your data, right? It will get the data that is used in your prompt, so it will see everything that's going on there.

So I would be thinking this should not be used with production data, obviously. What's your thought on that, on sending data around? So the data I pull in here, generally it's from test environment or dev environment, and I work with the latest backup. But you do have a point. It is something that your organization should agree on and clarify if it's okay to have production data being sent for this.

Sometimes some bugs surface only in production or are best reproducible in production, or your other environments don't use production data as often is the case. So I would be curious to know how it could still be used in these situations if it's not reproducible. But I just wanted to maybe talk about alternatives. I think that's a good question.

I think there are a lot of options to using it this way, and I think MCP servers, postgres MCP servers, can be built into other tools as well. For example, Claude Desktop, where you can talk to data and extract exactly the same insights we did, but it's disconnected from, in this case, dbt, how data was created. So I think this is the unique selling point here is the intersection of your code base and the actual output of your data.

So that is what drove me towards this. to this. I still think it's a tool for technology experts. I would be very cautious giving this tool to non-technical people or to managers just because they won't be able to verify if this data is correct or don't understand underlying tables and relationships. I think it's still something that should stay and be verified by an actual data expert. And so I think we still have, we'll have our jobs for a while before this is a decision-making tool.

So you're not getting rid of SQL yet, but writing it yourself will be more limited. I do write SQL still because sometimes it's just the beginning of an exploration or I don't trust it. So it's still, but it's much, much less. I was thinking about, we did an experiment with Snowflake where we have the semantic model capabilities that they have offered in previews. And there you add some metadata to your analytical warehouse. You say this column is connected to that column. That's the kind of relationship. And essentially what you're saying here is because we add dbt also into the mix, the model has access to these kinds of things because of dbt. So somehow you supply that semantic model, what is in Snowflake directly into the model using dbt in this case. And I think it's very powerful because it shows it.

It didn't do the wrong queries. It did the right queries, it seems. Yes. And yeah, this is where the strength lies for me. It finds the, it confirms the problem and it fixes it at the very source. And yeah, there's, it's just the full, full, full flow. Yeah. And the flow that you showed was mainly a bug fix. Somebody notifies you, I'm seeing some weird data. You enter it in the LLM and it combines, it's a code, it combines the data. It gives you a fix and it gives you a test. And how does this help you in other development tasks, like non-bug related, but adding new models or something like that. Is that also where you think it could be valuable?

Absolutely. You can also use it, mainly rely on the LLM to, to create models. And then well, that's what I did for, for this project as well. I, I instructed the model to create this dbt project on top of the source code that was already existing in my Postgres that was accessible via the MCP. And then it queried the database for existing staging tables. It knew what I had.

And then it built the whole dbt project thinking, how can I enrich this data? And that's how I came up with analytical data like salary percentiles. So it can also build things from scratch based on what you have already. So it goes both ways. This is interesting. So you could use it to generate sources in dbt and also generate derived tables, enrich that information that you have as metadata in dbt with better or correct information.

So it's going to cross validate. Okay. This is very interesting. You also mentioned, Emil, that if you would use Claude Desktop, for example, it would not be as powerful. Did you do a test with that?

Yes, I did. And we're here. We have Claude Desktop and I have this MCP server connected here. And we can ask the same kind of questions. So trying to use Claude Desktop and present it with the same issue as we did with Cursor, it still runs the queries. But all it does, it confirms that the problem exists.

So it says, yes, you're right. These statistics are not correct. But there is nothing that actually points me to the right problem. Because it doesn't work. It doesn't have the code. It cannot see how the data was created.

So it also cannot provide a good fix or at least a fix that would make sense. It has to guess. Yes. So my starting point now would be to still ask it to write a couple of SQL queries and just go about it the manual way versus I got everything ready out of the box in the previous demo. So this is a big benefit of actually doing this in Cursor as you did. Yeah.

Between the code and the data. That's the thing that's crucial to getting good results. Yes. All right. We can wrap it up. I can summarize.

All right. You can speed up your analytical queries and get insights from data very easily using MCP servers with Postgres. It can accompany you and support you throughout the entire flow from the development and generation of the data and understanding of the data that currently lives. Yes. So that's a huge productivity boost for me that was easy to implement in just a couple of hours. So if you have a similar use case, give it a shot.

Cool. And my takeaway is it's very accelerating. You showed me that in a few minutes you can go from a bug to a solution because you mix in the code with the data. And I'm also I'm going to try out these MCP servers in Cursor. I didn't know it was that easy to set up. You triggered me in using that.

The code you wrote is something we will be put in public. We have a public repository. Yes. Okay. I'll add the link to the notes as well so everybody can test out your server. All right.

Thanks a lot, Emil, for this explanation. I learned quite a few things. And thanks, everybody, for watching. This was our MCP Postgres integration by Emil. So I hope to see you next time. Thanks for watching.

Bye-bye. Bye-bye. Bye-bye.