AI Security Ops

In this episode of BHIS Presents: AI Security Ops, the team breaks down a growing problem in cybersecurity: AI-generated bug bounty “slop” overwhelming the system.

What started as a powerful way to crowdsource vulnerability discovery is now hitting a breaking point. Programs like cURL’s bug bounty and platforms like HackerOne are seeing a massive surge in submissions — but fewer and fewer of them are actually valid.

The result? Security teams spending hours reviewing reports that go nowhere, while real vulnerabilities risk getting buried in the noise.

We dig into:
• Why cURL shut down its bug bounty program after years of success
• How valid reports dropped from 1-in-6 to 1-in-20
• What “death by a thousand slops” actually looks like in practice
• How AI is flooding programs with low-quality vulnerability reports
• The difference between “theoretical” vs. exploitable vulnerabilities
• Why reviewing findings is now harder than generating them
• How HackerOne is responding to the surge in submissions
• Whether AI can be used to filter AI-generated noise
• The role of reproducibility and proof-of-impact in triage
• Why human expertise still matters in vulnerability validation

This episode explores a critical shift in security operations: when vulnerability discovery becomes cheap and automated, validation and triage become the real bottleneck.

⸻

📚 Key Concepts & Topics

Bug Bounty Programs & Triage
• Submission quality vs. volume imbalance
• Signal-to-noise challenges in vulnerability pipelines
• The growing burden of manual validation

AI in Vulnerability Discovery
• Automated scanning vs. real exploitability
• AI-generated findings and false positives
• The “editor’s dilemma” — review vs. generation

AI Security Risks
• Lower barrier to entry for vulnerability discovery
• Over-reliance on AI without domain expertise
• Flooding systems with low-quality submissions

Defensive Strategy
• Requiring reproducible steps and proof-of-impact
• Using AI to pre-filter vulnerability reports
• Combining human expertise with AI tooling

Industry Impact
• cURL bug bounty shutdown
• HackerOne submission pause
• Shifting economics of vulnerability research

#AISecurity #BugBounty #CyberSecurity #LLMSecurity #ArtificialIntelligence #InfoSec #BHIS #AIAgents #AppSec
----------------------------------------------------------------------------------------------

  • (00:00) - Intro: Bug Bounty Burnout & AI Noise
  • (01:14) - cURL Kills Its Bug Bounty Program
  • (02:05) - “Death by a Thousand Slops” Explained
  • (03:42) - AI vs Vulnerability Scanners: Signal vs Noise
  • (04:38) - HackerOne Pauses Submissions & Industry Impact
  • (05:41) - Can AI Filter AI? Proposed Solutions
  • (07:49) - Why Humans Still Matter in Validation
  • (12:55) - Final Takeaway: AI as a Tool, Not a Replacement

Click here to watch this episode on YouTube.


Brought to you by:
Black Hills Information Security 
https://www.blackhillsinfosec.com

Antisyphon Training
https://www.antisyphontraining.com/

Active Countermeasures
https://www.activecountermeasures.com

Wild West Hackin Fest
https://wildwesthackinfest.com

🔗 Register for FREE Infosec Webcasts, Anti-casts & Summits
https://poweredbybhis.com


Creators and Guests

Host
Brian Fehrman
Brian Fehrman is a long-time BHIS Security Researcher and Consultant with extensive academic credentials and industry certifications who specializes in AI, hardware hacking, and red teaming, and outside of work is an avid Brazilian Jiu-Jitsu practitioner, big-game hunter, and home-improvement enthusiast.
Host
Bronwen Aker
Bronwen Aker is a BHIS Technical Editor who joined full-time in 2022 after years of contract work, bringing decades of web development and technical training experience to her roles in editing pentest reports, enhancing QA/QC processes, and improving public websites, and who enjoys sci-fi/fantasy, Animal Crossing, and dogs outside of work.
Host
Derek Banks
Derek is a BHIS Security Consultant, Penetration Tester, and Red Teamer with advanced degrees, industry certifications, and broad experience across forensics, incident response, monitoring, and offensive security, who enjoys learning from colleagues, helping clients improve their security, and spending his free time with family, fitness, and playing bass guitar.
Guest
Ethan Robish
Ethan Robish has worked with Black Hills Information Security (BHIS) since 2008 — first as an intern and then as a full-time Security Consultant starting in 2012. In his current role as a Threat Hunter, Ethan is involved with customer engagement, research, working with Active Countermeasures’ AC-Hunter, as well as improving BHIS HTOC and SOC offerings. Previously, he implemented defensive security solutions for the Exchange Online security team as a Microsoft intern. While in college, he competed in the International Collegiate Programming Competition (ICPC) World Finals. In his time off, he enjoys cooking, playing the piano, and reading fantasy novels.

What is AI Security Ops?

Join in on weekly podcasts that aim to illuminate how AI transforms cybersecurity—exploring emerging threats, tools, and trends—while equipping viewers with knowledge they can use practically (e.g., for secure coding or business risk mitigation).

Brian Fehrman:

Hey, everyone. Welcome to this week's episode of AI Security Ops. So imagine that you wake up and you've got a bunch of new security vulnerability reports that were filed overnight. You spend your entire morning reading through them, checking code paths, trying to reproduce the issues, and you come up empty handed. Find out every single one was just sloppily submitted to you without extra review, and that's exactly a problem that some of our bug bounty program partners are facing out in the industry today, and it's something we'll talk about in this episode.

Brian Fehrman:

But before we get started, let's talk about Black Hills information security. If you or your organization are in need of any security services, whether that's external testing, internal testing, AD testing, web apps, physical pen testing, wireless, security operation center. Any security needs you might have, check us out at blackhills info sec dot com. Additionally, we have a training branch, anti siphon, where our practitioners who are doing these things day in and day out package up their skills and knowledge in a very digestible way to put out to the public at a very affordable freight price. Check check them out at antisiphontraining.com.

Brian Fehrman:

So let's dive into this a little bit. So just to kick it off, first story is back in January 2026, cURL, Daniel Stenberg, the creator of cURL, actually killed this bug bounty program that he had out. And the six years that he had it out, it paid out about a $100,000 for 87 vulnerabilities that have been submitted. So it's a very successful program for them. But then things started to go downhill recently where the valid submissions went from a one in six ratio to about a one in 20 ratio.

Brian Fehrman:

And his breaking point was that he had seven submissions in a sixteen hour window, and none of those were real vulnerabilities, which he called death by a thousand slops. So let's talk about what's contributing to something like this. And what do you guys think? Since we're on the AI Tech podcast, what do you think is a driving factor here?

Derek Banks:

Well, if you believe my x feed, the security has been solved by everybody and their brothers at GentekAI platforms. Platforms. So I don't know. Don't know. The whole cybersecurity industry is is now solved, I guess.

Brian Fehrman:

We're all done. We can go home.

Derek Banks:

Yeah. So I'll credit John Strand for this, but it it's kind of akin to, you know, you're running vulnerability scans and working on them, doing basically un auth vulnerability scans, and then all of a sudden you decide one day, hey. I'm gonna give this thing creds and, like, log in to stuff and find vulnerabilities, and your vulnerability report went from 200 pages to 200,000 pages overnight, and then you said, the hell am I gonna do with all this? So I think there's a difference between finding technically a vulnerability, technically a vulnerability, versus actually being able to take advantage of that vulnerability and exploit it and deliver some kind of impact. Right?

Derek Banks:

And so I think that still takes, obviously, a human, apparently. I mean, I'm not saying, like you said, AI seems to be finding some legit vulnerabilities. Apparently, the the latest hotness, the copy fail thing was found by some some kind of AI, excellent or something. Right? But, yeah.

Derek Banks:

One in 20 is a pretty big ratio when you have to, like, spend your time manually looking through all that stuff. Right?

Brian Fehrman:

Yeah. Yeah. I mean, that that takes it takes real time. Right? I mean, to to tell the AI to go out and to do something is about I I think the scanner analogy is perfect.

Brian Fehrman:

It's about the same amount of effort that it takes to tell a scanner to go out and do something at this point. I mean, maybe, like, marginally more depending on how they're using the AI, but actually going through the output, that's that can take a lot of time. That that that's where a lot of real work can can happen. And I think that's where a lot of people are getting bogged down right now. It's easy to generate content.

Brian Fehrman:

It's a lot harder and more time consuming to actually review the content.

Bronwen Aker:

So Editor's dilemma.

Brian Fehrman:

Yeah. And so curl's not alone in this. I think they were actually part of the hacker one program, which is what we'll mention now. I think that's where they withdrew their their bug bounty program from, if I'm not mistaken. But hacker one as a whole, as of March 27 and as still of this recording, I I checked that their program is actually still paused for new submissions, because they're finding that they are spending just an exorbitant amount of time going through these vulnerabilities that vulnerability reports that are getting submitted by people, and it just been overloaded with the volume.

Brian Fehrman:

And so they're having to work through how are they going to filter out, basically, the signal from the noise. Right? And that's that's a very that's a difficult problem that I think a lot of people are currently trying to to figure out in the industry as a whole is how do we effectively how do we strike that balance between, keeping up with the pace of being able to quickly find and triage vulnerabilities, but yet still be able to separate out what's not important from that. And and it's and it's kind of a tough problem. Right?

Brian Fehrman:

What are your what are your guys' thoughts, opinions?

Bronwen Aker:

Well, the hacker one story came up in the the regular newscast for BHIS, and I know that we did a lot of discussion about the fact that the bug bounty program rewarded people for finding vulnerabilities, but there isn't a bounty program or a reward system for the people who are performing the actual fixes. And now that a little more time has passed, it seems kind of like a silly, obvious solution would be to have develop AI to reality check the submissions before then handing it off to a human for resolution. That would help take a legitimate load off of the humans. It would also help speed up the pipeline of getting legitimate issues into the hands of people who can actually fix them. But, you know, the the challenge there is gonna be training a new model to identify what's a a valid vulnerability versus what is a well, let's call it vulnerability slop.

Ethan Robish:

The if AI is the problem, the solution is more AI.

Derek Banks:

That's typically how a data scientist would look at it. Yes. Exactly. How do we protect AI? Well, we put more AI in front of it.

Derek Banks:

Sounds like a plan.

Ethan Robish:

No. But I think, Bronwen, you're you're onto something, not necessarily like better training being the whole picture, but giving like, if if you are fielding bug bounty requests for a specific application, giving the AI access to, you know, a staging environment, a testing environment with your application that can take the reports and try to reproduce it, like, this even able to be exploitable? And that's like an immediate validation. But I mean, that requires setting up more tooling. But I mean, realm of possibility, right?

Ethan Robish:

Yeah.

Derek Banks:

I mean, I guess it's not just the model, it's also the harness. Right? And I think at the end of the day, I'm still in the camp of it really a human to determine whether or not this is truly, like, legitimate vulnerability. I think AI is a tool, and we shouldn't lose sight of that as technology workers. Right?

Derek Banks:

It's just a really fancy pattern matcher. It's very good, but, you know, the more you learn about how AI works, the more you realize it's not a person, it's not an entity, it's a really fancy pattern matcher matcher. There's a lot of math going on, but, I that's really, at the end of the day, what's happening. And so I I still think that, you know and I've spent a lot of time recently, you know, you trying to develop custom agentic solutions to do, you know, penetration testing work, and I've been really successful. But also, sometimes you look at what the quote AI found.

Derek Banks:

Right? And, you think, man, a human would have never thought this was an issue. This is not an issue kind of thing. So, I mean, I I think, you know, yes, you know, models are going to get better, harnesses are gonna get better, but, yeah, I think at the end of the day, as long as we're using transformers, like, is this is gonna be an issue that, you know, there's just gonna be a whole bunch of people who are pointing AI at stuff and going, see it, found a vulnerability, and and not using what we would call, like, domain knowledge, like, how much do you know about this thing, you, and applying that to it and being you know, figuring out is this for real or not.

Bronwen Aker:

Okay. So I have a question because I've never participated in a bug bounty program. If I submitted to hacker one, do I have to tell them how to replicate the error? Because if so again, Derek, you you hit the nail on the head. AIs are tools.

Bronwen Aker:

They're good at following sort of directions. And maybe if they have that roadmap to follow, then they can go, yes, the steps worked. No, it didn't. If they get a yes, that can bump it up whatever weighting scale can be used to get things through the queue faster. Just a thought.

Brian Fehrman:

Yeah. So typically, I've only done bug bounties a couple times, but usually, they they ask for reproduce, like, steps to reproduce so that, to make it quicker for their their people to go through.

Derek Banks:

Yeah. I think the the better your write up, the better the chances you're gonna get paid. Right? It's like a little mini

Brian Fehrman:

pen test. Right? That's that's Yeah. What it was supposed supposed to be about. And, yeah, certainly, so I I think that's, you know, a good point is a is a first filter level is, like, you know, only only taking reports that actually have the steps, the commands that are in them that allow for the AI to go through and reproduce without, like, error to try to bump up, you know, bump up certain submissions higher than others.

Derek Banks:

Yeah. If I was gonna do it, like, right now, I'll just say our AI platform. Right? You find something, it actually outputs everything for every single finding. Here are the commands that a human should use to go and verify this.

Derek Banks:

So the human should go and use, you know, those commands to go verify it. So to me, that that's like the starting point, not the ending point for the vulnerability. Right? And so I feel like, yes. Okay.

Derek Banks:

Now a human has verified it. Can you actually cause, like, more impact with this? Can you, even though the AI even though I try and have our platform tie stuff together, does this, like, have, like, more tie ins to things? And and so I think to me, like, you have, like, a human pen tester and an AI pen tester that, you know, depending on what you're looking for, you might think that one's better than the other, but both of them together are better than either one by themselves, in my opinion.

Ethan Robish:

That's what I was going to say too.

Derek Banks:

Oh, damn it. Sorry, Ethan. Doesn't have to be

Ethan Robish:

no. No. No. I'm I'm agreeing with you. It doesn't have to be adversarial.

Ethan Robish:

You mentioned earlier, like, AI is a tool. So I I mean, yeah, we should use it to to make us better at our jobs. And as humans are really good at taking new tools and try to use them in ways they're not meant to be or finding out the limitations, like you have a hammer, everything starts looking like a nail.

Derek Banks:

You once told me about that I don't with Jupyter Notebooks, and I'll have you know that I

Ethan Robish:

still do use Jupyter Notebooks. Nice. Yeah. And I I don't think we've found the limits yet, right, of, like, where AI can stop being helpful. Like we've we've found limits, but not like the absolute, you know, here's we can't progress any further or whatever.

Ethan Robish:

Like we're still it's still changing at super rapid pace of both the innovations in the fundamental models and, like, what people are able to do with them.

Brian Fehrman:

Yeah. I I completely agree. So, yeah, I think I think that'd be probably a good place to, wrap up and close and and just say that, I mean, a AI is definitely finding, like, real bugs. I mean, it is it is an extreme extremely useful tool. Like we've said, it's great for augmenting, you know, many many different tasks, but it's important that we're still understanding that that humans are still needed and, professional judgment is still needed in this space, to be able to, so we can augment one another, not try to, you know, replace or, completely shift off, tasks and responsibilities.

Brian Fehrman:

So, yeah, with the with that, any other closing thoughts?

Derek Banks:

No. I'll just say with that, keep on prompting.

Brian Fehrman:

Keep on prompting.