Technology Explorations in Data & AI

Code reviews are often considered a pain, resulting in quick approvals and bugs reaching production. CodeRabbit and Sourcery aim to solve this by letting an AI agent review your changes early on.
In this episode, data & platform engineer Hannes De Smet shows Jonny what he learned after exploring several AI Code Reviewers. Hannes demos both tools on a real code change, allowing a critical look at the quality of the suggestions, as well as the user experience. It turns out that, depending on the context, both could use some improvements.
Resources:
  • CodeRabbit: https://www.coderabbit.ai
  • Sourcery: https://www.sourcery.ai/
  • Multi-workspace AI video: https://www.youtube.com/watch?v=E_kOAvmeTJ0
Note: This video is not sponsored or affiliated with CodeRabbit or Sourcery.


Resources:
Chapters:
  • (00:00) - Intro: why AI code reviews?
  • (02:53) - AI Reviewer 1: CodeRabbit
  • (10:32) - What CodeRabbit catches (and misses)
  • (12:18) - When AI comments become noise (80% disregard)
  • (13:27) - Catching a PII issue
  • (15:15) - AI Reviewer 2: Sourcery
  • (19:14) - Cost & comparison
  • (19:59) - What's the future of AI code reviews?
  • (20:41) - Summary & takeaways


Data & AI: Technology Explorations is a biweekly show from Dataminded. Each episode a Dataminded engineer demos a tool or technique worth knowing about -- working code, honest takes, no hype.

Music by Aleksandr Karabanov from Pixabay

Creators and Guests

Host
Jonny Daenen
Head of Knowledge at Dataminded
Guest
Hannes De Smet
Data Engineer at Dataminded

What is Technology Explorations in Data & AI?

Deep dives and practical demos on the technologies shaping modern data and AI development. Join the Dataminded team as we explore, unbox, and critically review the latest tools, from building AI agents and RAG systems to optimizing cloud costs and accelerating data pipelines. We cut through the hype to show you what actually works in real data engineering practice, complete with demo code!

Jonny Daenen (00:00)
code reviews can be a pain, right? A lot of people don't like them, and we see many cases where code reviews are not that great. And again, surprise, AI can help. We see a rise in AI code reviewers that can help you assess your code or improve it before you reach production. Let's have a look.

Jonny Daenen (00:25)
Hi everyone, welcome to technology explorations at Dataminded. In this series, we give you an initial look in new or interesting technologies. My name is Jonny, knowledge lead here at Dataminded. And today we're going to look into AI assistance for code reviews. And for that, I brought with me Hannes. Welcome Hannes.

Hannes (00:42)
Hi, thanks for having me.

Jonny Daenen (00:43)
How are you doing today? All good?

Hannes (00:45)
All good, all good. Ready to show you some things.

Jonny Daenen (00:47)
So what brought you to this topic? AI assisted code reviews. What is the reason this triggered you?

Hannes (00:54)
Yes, so I think we were all quite amazed when we first used whatever model it was. Was it a GPT model or Gemini model or Claude or whatever. or a lot of people are...

utilizing that or at least to some extent should, I think, that's my opinion. But you've also noticed, think, Jonny, that it sometimes doesn't know really what it's doing. It doesn't have enough context, which is most of our time our fault, but sometimes it also just makes mistakes and you've been there. It's not always a walk in the park. if you accept that and

want to go to production, you go through PRs. But after a while, the PRs start to become so big because people are producing so much code because it's so easy to produce code. And I was really interested, if you're automating software engineering itself, writing the code, can we also automate

or leverage AI to do the PRs itself.

Jonny Daenen (01:42)
And so if I understand correctly, we generate a lot more code due to AI systems. And essentially you were looking for a system that would reduce the burden of the code review mechanism.

Hannes (01:53)
Yes, and I think it's twofold. Indeed, it removes the burden but on the other hand, you will always miss things as a human,

it's easy to overlook something, especially if you don't know the repo, and you only have a few people to get things done. The chance that you miss a mistake, even a small one, it's very big. So I think that's another benefit next to the burden itself of doing it.

Jonny Daenen (02:17)
So also increasing the quality of the reviews.

Hannes (02:20)
Yes, yes,

Jonny Daenen (02:21)
If I look at my own experience, what I personally like in pull request reviews is if they go beyond a code review. to me this is beyond coding. This is also about checking the logic. checking whether...

everything works, doing a local test with the code and giving feedback on that also on the design. quite often I do see at companies that there's lots of rubber stamping, like people just approving stuff

so I do think there is a huge chance to optimize this workflow to have better reviews So I'm very looking forward to what you're going to show me. So I would say let's dive in.

Hannes De Smet (02:53)
I've looked at a few tools that I can test out. the space is quite big already, so it's really growing, And here are a few of the big players out there. There is Code Rabbit, which we will take a look at. There is Qodo,

We have also Greptile, Sourcery, Beto and quite a few others. we'll try to do just two different ones, but I, in general, validating AI seems always as a hard task, at least to me.

Jonny Daenen (03:18)
Yeah, indeed. There's a few benchmarks you could use, but I think it would be good if we could already get a feeling on how these things work.

Hannes De Smet (03:19)
⁓

Yes, indeed. So let's just dive into the first. So what I've been doing, Jonny, is this project It's called the Sandi Insights Project. It's a little internal project for the people listening or watching. And Jonny asked to review something, to do a few changes

⁓ Me, not knowing the repo, I started vibe coding, So I asked some feedback of what it was, I checked, okay, what do need to change, where, made a few changes. I have cursor.

it's VS Code with an extra layer with AI integration basically. And one of the extensions I installed is CodeRabbit. So in here you see I'm logged in as myself on GitHub.

and I installed the CodeRabbit extension. And in here I can see I'm on a separate branch and I can compare to main. And in here I can review all my changes. So I will already start that because it takes a while. It's setting up, reviewing the changes, et cetera. And...

Jonny Daenen (04:20)
Okay, so you

made a branch and then you, you vibe coded some things or you programmed yourself and now you have the chance to let it review the pull request already that you could create. This is not a pull request yet, this is just the two branches that we'll compare.

Hannes De Smet (04:23)
Exactly. Yes.

Yes.

Yes, very good point. So indeed it's not yet a pull request. I will make a pull request right now. I don't think I have any local changes.

Everything is up to date. So let me quickly go to GitHub while we're waiting for the review anyway. So this is the code base.

So let's contribute, let's open a request. I will just create a pull request for now. I know Jonny in the future, I would like to add some extra descriptions, But for now I'll just create a pull request. ⁓ For once this is okay, I'm allowed.

Jonny Daenen (05:00)
Yes, please. Okay, for once this is okay.

Hannes De Smet (05:06)
So let's go back to CodeRabbit. So CodeRabbit has been very helpful. The rabbit has been working and you see here quite a few things popping up. So again, we're looking at a certain branch and we're comparing it to main. it's checked 12 files and it's detected nine issues. So what you can do is either I go here through the comments or you click here on the files. So

Jonny Daenen (05:23)
Okay.

Hannes De Smet (05:28)
This is the biggest file I've changed to the file with the most changes of myself. So there's a potential issue. So I can say, click on it. And that brings me to the actual line question. So here it tells me there's a type mismatch for origin column prefill. So the multiselect column says it's a streamlit column type. So it's a streamlit application and the multiselect column allows you to select multiple aspects. So the prefill here.

is currently set on an empty string, but that does not make sense since the default value should be a list because it can be multiple values. that's CodeRabbit is complaining and it seems to me that this is actually a valid I could do is indeed look this up and validate with actual documentation of Streamlit that this is actually the case. So let's assume

that the AI is correct in this regard. Then I can either apply the suggested change, fix with AI, or I can also ignore when I think it's not relevant in this case. So let's fix it with AI. And something very cool is happening. I don't know if you saw what happened, but.

Jonny Daenen (06:34)
It

popped up, it opened up a new chat.

Hannes De Smet (06:36)
Yes, exactly. So in Cursor, it directly start a new chat with in this file, around lines, blah, blah, blah. And then it says actually what it found. So it gives a bit of context where it should find it and then it gives the error. So this allows me to fix it with AI.

Jonny Daenen (06:53)
Okay, so.

So you basically had these different options. The first one was actually the diff that it proposed to directly apply it, which is already suggested by CodeRabbit, but you chose the option to fix with AI, which then transfers this request to fix to a chat window for Cursor, which can then use different models and maybe more powerful things to fix it.

Hannes De Smet (07:07)
Yes.

Yes.

Exactly. So I'm not sure how this indicates, for example, to co-pilot or if you're using another integration within VS code. But for example, for Cursor, it works out of the So here.

Jonny Daenen (07:27)
Hannes, when would

you opt for this AI system to kick in like here with the supplied context versus just pressing the apply now button?

Hannes De Smet (07:35)
Yes, good question. So I think applying now, you should always and only do when It is actually correct and you agree with it and you say, yes, this is indeed much better. A simple rename here, indeed a simple default value. Either this is my research and I ask, is this the case?

And I basically ask for a second opinion here

So I would use this mainly for either you want to ask more context, more feedback, or if you're unsure about the implementation, sometimes

Jonny Daenen (08:04)
it does say it can confirm the issue. It knows about the issue from the context that was given to it. And it will, in this case, confirm that the fix is indeed what Code Rabbit proposed.

Hannes De Smet (08:15)
Yes, and it even suggested a second edit because somewhere else I seem to have made the same mistake. So it even went one step above. Maybe I'll open a second.

which here is nice, it seems like a critical logic error. So.

Here it starts complaining again about regarding the same column that returns a list of selected values and the conversion expecting it by comma produces incorrect results. It gives an example, client work training and I expected to split on comma which wouldn't work because then string, etc. So here it proposes yet another change.

And it also asks to update the error raising because here I use the same logic. So let's say, okay, I agree with this. I've checked this. I validated it. I tested it myself in my Python terminal. I can now apply the suggested change. Whoop, you see something going over. Looks very fancy.

And then it's done. I can also unresolve the comment and go in again. But now it changed a few lines. So you can here again, see the changes it made from the test strip, And that's fixed it. And now can go all through the rest. I can either ignore. If I say, well, this issue, I don't think this is an issue, I press ignore. And then you go through the list.

Jonny Daenen (09:11)
Okay.

Hannes De Smet (09:30)
I do it actually one by one on my own and I go through this.

What CodeRabbit can also do is if you take a look at the website,

Jonny Daenen (09:34)
Yeah, indeed.

Hannes De Smet (09:38)
next to CLI work, which is actually what I did now, you can also comment on PRs itself. So what I did now is actually, I call it a pre-flight check.

So basically before I even create the PR, I check myself locally, well, did I do anything wrong or whatever? But if I make the PR, if you have it enabled and pay for it,

Jonny Daenen (09:48)
Yeah.

Hannes De Smet (09:59)
then it can actually do it in the UI itself. So here you can see a few screenshots and commenting on PR itself, which is really cool. So you have two places in your whole development process where you can run this app. So before you're creating a PR and after creating the PR,

Jonny Daenen (10:16)
And for the PRs, then it needs a GitHub integration, suppose, or Bitbucket or whatever you use.

Hannes De Smet (10:18)
Yes, then it needs a GitHub integration.

So you need to allow access to your GitHub repo to make changes.

Jonny Daenen (10:27)
It's essentially

a bot, assume that just visits your PR and then gives comments. And what I wonder is what you showed locally, Does it only consider code that was committed on that branch or does it also consider uncommitted code?

Hannes De Smet (10:31)
Exactly,

Good question. So it basically, there is your answer. It can review uncommitted changes, committed changes or all changes, which is quite useful indeed. So you can even before you commit the change. So even if you do test-driven development and you're very meticulous in the way you never commit non-working code well, this is a nice feature that it even reviews uncommitted changes before you.

Jonny Daenen (10:44)
Yep.

And so the comments that are generated, Are these all applicable only to the changes that you've made or does it go beyond that? Cause there could be a bug in your main code already. will simply ignore it.

Hannes De Smet (11:11)
So it only...

Yeah, so it will ignore code that's already there in the sense of it will not review it, but it will look for it as but it will not assume say, hey, last year or last month you made a change to this or looks like the method is not really correct.

Jonny Daenen (11:22)
Yeah.

Hannes De Smet (11:32)
it will not comment on it. So it's really only on your changes.

Jonny Daenen (11:35)
Yeah, okay. I also see there in the pro version, you can use coding guidelines. That sounds very interesting to me. Like if you have specific standards, if you can ship them and the review also checks that.

Hannes De Smet (11:46)
Yeah, I think this is very useful. also, there is a lot of options as well in AI assets and coding, both with GitHub Copilot, with Cursor, with Joony, whatever you use. I think all products now have an option to give you guidelines, in combination with pre-commit hooks that allow you to automatically do other checks as well. I think...

It also limits the need of this, but it is useful nonetheless.

So this is something I use myself actually. So I highly recommend testing it out for yourself to see if it works. For a repo I know pretty well it can be that 50 % or more or 80 % of the comments can disregard. Basically because I have enough context about what's going on I can say I can ignore this and this and this because I'm quite sure

It's an overkill what they suggest, I think it's not needed or too defensive programming in that case and then I just ignore the comments.

Jonny Daenen (12:39)
But that's interesting. that means like 80 % of the comments are not really useful for you personally.

Hannes De Smet (12:45)
So it depends, in this case, I think it was very useful because I vibe-coded quite a lot the first two comments seemed very plausible and I don't understand the repo well. If I have a repo where I handwritten, so to say the code base 95 % or 90%, then often I feel the tendency that indeed 80 % of the comments is,

not really that useful, but I have got, for example, few variables that I used the wrong variable name for example, which got through a unit test, And if he didn't catch it, it would have been a bug later on. So I actually caught a few bugs which would have caused at least some trouble in production with this.

Jonny Daenen (13:27)
I do want to dive in maybe on one of these issues, like there's this PII exposure, always interesting. So I'm actually curious what it suggests there.

Hannes De Smet (13:27)
it.

Yes.

Jonny Daenen (13:36)
Remove or mask the email address from the UI.

Hannes De Smet (13:38)
Yes, so this is a point which doesn't make that much sense, right? So I added this as a debug information that I am logged in as hannes.desmet[at]dataminded.com

is that personal information? Yes, but it's displayed only for me. So this doesn't make that much sense because it's in the session state of the current user shows you the user email. So does it make sense that it's actually a PII issue? I don't think so. So in this case, I would ignore

Jonny Daenen (14:08)
indeed. And the solution gives us like, it says remove this from the UI, which I can see it does there. It only shows the number of ⁓ user answers and it also adds something extra in the if test. It also adds like, and user answers. So when I saw this, had to actually extend my mental context because I was looking for ah It will just remove the email, but it actually added a few things to the test as well.

Hannes De Smet (14:15)
Mm-hmm

Jonny Daenen (14:33)
which I had to think about. I would prefer if it gave me a bit more explanation on what it did there, but it seems to be correct. And then it gives you a second option. So it says you can mask it and it gives you a way of masking the email address, which I also like that it does that. But now we don't get an in place button, like apply this, you get different options I see.

Hannes De Smet (14:39)
Yeah.

Yes.

Yeah, it's...

Yes, so it's not always there, the button of automatically accepting. So sometimes you have to do it yourself, either copy pasting it and changing it, or just manually typing it. sometimes it makes you think also about other aspects, which is quite nice. If you, maybe you wouldn't just thought of the PII data or you wouldn't think of security logging

Jonny Daenen (15:10)
Yeah.

Exactly.

Hannes De Smet (15:16)
tool I tested out as well, which I can't unfortunately anymore, which is quite interesting and I would like to mention as well is ⁓ Sourcery So Sourcery is also a code quality tool. And it also states it does code review for the AI era. So it does also security scanning and team.

analytics, not sure what that means. But the code review, very interesting, is free for open source projects. So it also comments on the PR itself, like in in GitHub, but it's also a tool you can use in your IDE as code review, So it also works on

both places in your development flow, And I also got quite nice results. And the nice thing about Sourcery, at least with the comments here in the IDE was that my Cursor could actually see them all. It had the context of all the comments it posted.

I could just start typing and it could see all the comments, and it integrated really well with Cursor, which was quite nice. if you want to use it for private repos, at least in the IDE, you have to take a pro license. it is another nice option, especially for open source projects.

Jonny Daenen (16:19)
Is this possible to demo it on one of our public repositories?

Hannes De Smet (16:24)
Yes, I think so. Live, we can try that.

Jonny Daenen (16:26)
Life, I mean.

have the technology explorations repository, which is public.

Yeah, that one.

Hannes De Smet (16:32)
Demo

technology explorations.

Opening new window.

I don't know if the files are already... Plugin. Yes, the extension is installed.

Jonny Daenen (16:36)
This is already the Sourcery ⁓ plugin that you have, or extension.

Hannes De Smet (16:42)
I am currently... I don't know... Yeah, I think so.

Jonny Daenen (16:44)
You're on the main branch, so you might want to

Hannes De Smet (16:47)
compare with main

here it gave an overview. There's one general issue found, the issue division by zero in percentage calculation, which is one of the most default risks you can have, think, in programming. But at least it is indeed a possible, if your amount of employees is zero, you have an issue,

Jonny Daenen (16:57)
Ha.

Hannes De Smet (17:06)
Here is the comment, so if I hover over the blue squiggly line is actually the problem. So I don't know how it integrates with it. But here it says on both lines, because it's two lines, it takes it together again I can fix it in chat. Enter.

Jonny Daenen (17:10)
one of the comments.

And

in terms of comparison to CodeRabbit, CodeRabbit gave already a proposal quite often. It says like, this is what you can do to change. Here we don't get a proposal and we need to use the chat to actually get a fix.

Hannes De Smet (17:32)
Yes.

Yeah, so here, cursor

So it made the changes for me. Again, down here it seems to be the same issue, which wasn't getting flagged. I don't see anything saying about lines 81 to 83, but cursor seemed smart enough to fix that. A suggestion.

for some dates.

Jonny Daenen (17:55)
And here it does give a potential change.

Hannes De Smet (17:56)
Neary actually does

give proposed changes. But again, I think I have to...

Quick fix. So here if I do the quick fix, I can actually apply the fix - baf.

Jonny Daenen (18:05)
This was confusing, there were many quick fixes.

Hannes De Smet (18:08)
Yes, so it seems to be...

I'm not sure why, it is a bit confusing targeting the same... No, I think it's always duplicate it seems. Yes, see it's always...

Jonny Daenen (18:11)
Yeah, in terms of user experience, this is not ideal.

And if we look at this compared to Code Rabbit, seems like Code Rabbit was actually a bit more advanced.

Hannes De Smet (18:24)
In general, I thought the integration was better. the sense that I don't know if CodeRabbit by then had to fix it with AI button. I don't think it had it. So it was back then only the quick fix or ignore but not a fix with AI. Which that was the part I really liked about the integration.

Jonny Daenen (18:37)
Click Fix.

okay.

Hannes De Smet (18:46)
One cool thing it had,

It had the option to create diagrams, which was very cool.

So it has the context of the open file. the UI is maybe not the best.

Jonny Daenen (18:54)
okay, yes.

Okay, nice.

Hannes De Smet (18:58)
So that is quite cool.

Jonny Daenen (19:00)
this one is actually quite useful, I think. I mean, it's still very minimal, but I can imagine what it could do.

Hannes De Smet (19:02)
Yes. ⁓

Yeah,

I really liked the diagram. this was quite fun to play with, but I don't know how useful is.

Jonny Daenen (19:13)
you press

Do you have a rough idea on the cost and the cost models?

Hannes De Smet (19:17)
CodeRabbit it's per developer as well.

Sourcery pricing, let's quickly check, Pro, seat So all it's per seat so depending on the size of your organization, except if it's enterprise,

Jonny Daenen (19:29)
Okay, so they're all charging per seat and it's roughly in the 20, 30 dollar range, more or less.

Hannes De Smet (19:35)
Yes.

Jonny Daenen (19:36)
But so if I understand correctly, CodeRabbit and Sourcery are quite similar. They have this cursor integration. They also have this dashboard and GitHub integration. So they're direct competitors.

Hannes De Smet (19:48)
Yeah, yeah, exactly. I think CodeRabbit is the biggest player out there in general. I think this, to be fair, having this for free in open source projects, yeah, count me in I would say.

I am curious to see at least in the space we're in, if it's gonna change.

and how much it will improve. because if you need context over different repos, then it becomes, I think, very difficult for AI, especially in data engineering, platform engineering. There's a lot of integrations, a lot of glue between components. You still need to have that context.

Jonny Daenen (20:21)
You mean also between different projects and repositories? but we have a video for that. Emil demoed Cursor workspaces, which allows you to go beyond your project and beyond your code base.

Hannes De Smet (20:24)
Yes, yes.

Yeah, that's very well hidden promotion. It's good, it's good. I like it, I will check it out actually.

Jonny Daenen (20:36)
Sorry, sorry.

How would you summarize your experience with all these AI code assistants or pull request assistants?

Hannes De Smet (20:41)
well.

I would say, first of all, it was fun playing around with. So if you are curious, try it out for yourself, see what works. If you have a team that maybe wants to use it, at least give it a chance. find out what works for you.

Is it a preflight check? Is it having both Sourcery and CodeRabbit on the same repo and doing it all? go ahead and play with it.

Jonny Daenen (21:09)
Yeah. Okay. My takeaway on all of this is basically you showed me that you have two variants. You have this local variant and the preflight check is always a good idea before you push. And then just like with pre-commit hooks, you could also run something on your PRs so that everybody can benefit from that in a centralized way. We saw several different tools. The space there is getting a bit crowded it seems.

I think this is very nice to automate also some review systems and to get feedback fast.

Hannes De Smet (21:39)
Yes.

And one thing I might forgot to say is that I showed you two UI integrations. So both Sourcery and CodeRabbit here via UI extension. But for example, CodeRabbit released, a CLI tool. So you can also use it in the CLI, which makes, for example, I'm also a PyCharm user. I don't only use Cursor. And there you don't have an extension for CodeRabbit, so that it becomes a bit more difficult. So there I could, for example, use the CLI, which is maybe a nice thing to know.

Jonny Daenen (21:56)
Yeah.

All right, good. Hannes, thanks a lot for showing us around in the space of the AI pull request reviewers.

Hannes De Smet (22:11)
Very happy to do so.

Jonny Daenen (22:12)
All right, let's wrap it up. I want to thank everybody for watching. Thank you all and I hope to see you next time. Bye bye.

Hannes De Smet (22:18)
See ya.