Subscribe
Share
Share
Embed
In this episode of BHIS Presents: AI Security Ops, the team introduces a new approach to offensive security:
Fusion AI Pentesting.
https://www.blackhillsinfosec.com/fusion-penetration-testing/
As AI continues to reshape cybersecurity, one question keeps coming up — is AI replacing pentesters, or just changing how they work?
This episode answers that directly.
Rather than replacing human expertise, Fusion combines AI-driven discovery with human-led validation and exploitation, creating a workflow that’s faster, more scalable, and far more effective than either approach alone.
The result isn’t just more findings — it’s better findings, faster, with real-world impact.
We dig into:
• What “Fusion AI Pentesting” actually means in practice
• Why AI alone isn’t enough for real security testing
• How human + AI collaboration outperforms either independently
• The difference between finding vulnerabilities and proving impact
• Where AI excels in offensive security workflows
• Where human intuition and experience still matter most
• How this approach scales continuous testing and red teaming
• Why traditional pentesting models are starting to break down
• How organizations should think about integrating AI into security testing
• What this means for the future of offensive security
This episode highlights a key shift in cybersecurity: AI doesn’t replace the pentester — it changes what a great pentester looks like.
⸻
📚 Key Concepts & Topics
Fusion AI Pentesting
• Combining AI discovery with human validation
• Augmenting—not replacing—pentesters
• Faster, more scalable offensive workflows
AI in Offensive Security
• Automated vulnerability discovery
• Pattern matching vs real-world exploitation
• Limits of AI-only approaches
Human + AI Collaboration
• Human intuition and domain expertise
• Chaining vulnerabilities for real impact
• Validating and prioritizing findings
Security Testing Evolution
• Continuous testing vs point-in-time pentests
• Red teaming with AI-assisted workflows
• Changing expectations for coverage and speed
Defensive Implications
• Better signal vs noise in findings
• Faster identification of real risk
• Preparing for AI-augmented attackers
#AISecurity #CyberSecurity #Pentesting #ArtificialIntelligence #LLMSecurity #InfoSec #BHIS #RedTeaming #AIAgents
----------------------------------------------------------------------------------------------
About Melisa Wachs - https://www.blackhillsinfosec.com/team/melisa-wachs
About Brian Fehrman - https://www.blackhillsinfosec.com/team/brian-fehrman/
About Bronwen Aker - https://www.blackhillsinfosec.com/team/bronwen-aker/
About Derek Banks - https://www.blackhillsinfosec.com/team/derek-banks/
About Ethan Robish - https://www.blackhillsinfosec.com/team/ethan-robish/
About Ben Bowman - https://www.blackhillsinfosec.com/team/ben-bowman/
Click here to watch this episode on YouTube.
Brought to you by:
Fusion AI Pentesting.
https://www.blackhillsinfosec.com/fusion-penetration-testing/
As AI continues to reshape cybersecurity, one question keeps coming up — is AI replacing pentesters, or just changing how they work?
This episode answers that directly.
Rather than replacing human expertise, Fusion combines AI-driven discovery with human-led validation and exploitation, creating a workflow that’s faster, more scalable, and far more effective than either approach alone.
The result isn’t just more findings — it’s better findings, faster, with real-world impact.
We dig into:
• What “Fusion AI Pentesting” actually means in practice
• Why AI alone isn’t enough for real security testing
• How human + AI collaboration outperforms either independently
• The difference between finding vulnerabilities and proving impact
• Where AI excels in offensive security workflows
• Where human intuition and experience still matter most
• How this approach scales continuous testing and red teaming
• Why traditional pentesting models are starting to break down
• How organizations should think about integrating AI into security testing
• What this means for the future of offensive security
This episode highlights a key shift in cybersecurity: AI doesn’t replace the pentester — it changes what a great pentester looks like.
⸻
📚 Key Concepts & Topics
Fusion AI Pentesting
• Combining AI discovery with human validation
• Augmenting—not replacing—pentesters
• Faster, more scalable offensive workflows
AI in Offensive Security
• Automated vulnerability discovery
• Pattern matching vs real-world exploitation
• Limits of AI-only approaches
Human + AI Collaboration
• Human intuition and domain expertise
• Chaining vulnerabilities for real impact
• Validating and prioritizing findings
Security Testing Evolution
• Continuous testing vs point-in-time pentests
• Red teaming with AI-assisted workflows
• Changing expectations for coverage and speed
Defensive Implications
• Better signal vs noise in findings
• Faster identification of real risk
• Preparing for AI-augmented attackers
#AISecurity #CyberSecurity #Pentesting #ArtificialIntelligence #LLMSecurity #InfoSec #BHIS #RedTeaming #AIAgents
----------------------------------------------------------------------------------------------
About Melisa Wachs - https://www.blackhillsinfosec.com/team/melisa-wachs
About Brian Fehrman - https://www.blackhillsinfosec.com/team/brian-fehrman/
About Bronwen Aker - https://www.blackhillsinfosec.com/team/bronwen-aker/
About Derek Banks - https://www.blackhillsinfosec.com/team/derek-banks/
About Ethan Robish - https://www.blackhillsinfosec.com/team/ethan-robish/
About Ben Bowman - https://www.blackhillsinfosec.com/team/ben-bowman/
- (00:00) - Intro: A Different Kind of AI Sec Ops Episode
- (01:59) - Introducing Fusion AI Pentesting
- (03:34) - Why AI Alone Isn’t Enough
- (05:59) - Human vs AI: Strengths and Limitations
- (09:12) - Finding vs Exploiting Vulnerabilities
- (11:43) - How Fusion Improves Speed and Coverage
- (15:06) - Scaling Offensive Security with AI
- (18:12) - Final Takeaways: The Future of Pentesting
Click here to watch this episode on YouTube.
Brought to you by:
Black Hills Information Security
Antisyphon Training
Active Countermeasures
Wild West Hackin Fest
Creators and Guests
Host
Brian Fehrman
Brian Fehrman is a long-time BHIS Security Researcher and Consultant with extensive academic credentials and industry certifications who specializes in AI, hardware hacking, and red teaming, and outside of work is an avid Brazilian Jiu-Jitsu practitioner, big-game hunter, and home-improvement enthusiast.
Host
Derek Banks
Derek is a BHIS Security Consultant, Penetration Tester, and Red Teamer with advanced degrees, industry certifications, and broad experience across forensics, incident response, monitoring, and offensive security, who enjoys learning from colleagues, helping clients improve their security, and spending his free time with family, fitness, and playing bass guitar.
What is AI Security Ops?
Join in on weekly podcasts that aim to illuminate how AI transforms cybersecurity—exploring emerging threats, tools, and trends—while equipping viewers with knowledge they can use practically (e.g., for secure coding or business risk mitigation).
So hello, and welcome to a different type of episode for AI Security Ops, where we have a special guest from Black Hills Business Capture team. We don't have a sales team. We have business capture. We're a little bit different in that regard, which is an interesting thing, but not what this podcast will be about. And we're gonna talk about a new service offering that we have Black Hills, who, as if you've been following the podcast, you know that Black Hills Information Security brings you this podcast.
Derek Banks:They're our sponsors. We also work for them, and where we will do any kind of offensive security that you're looking for. We have a SOC that, you know, will will manage your security posture on the inside and monitor for threats. We have continuous red teaming and pen testing where we will test you all year long like real threat actors and traditional pen testing as well, you know, web apps and such. And then also anti siphon training where you wanna learn how to be a penetration tester or do forensics or learn about the intersection of AI and cybersecurity and many, many other security topics in an anti siphon is your place.
Derek Banks:Thanks for coming on, Melisa. We love it when we have special guests because it's usually just us droning on and on and on about things. Well,
Melisa Wachs:we do it often. Now we're just doing it where
Derek Banks:So
Melisa Wachs:like, so
Derek Banks:Yeah. I mean, I I think I spend most of my days staring into the the camera and talking either on Teams meeting or on YouTube. And it's funny. It always reminds me of when my oldest who's about to go off to college, she found out that I was on YouTube, like, way back in the day. And she was like, oh, that's so cool.
Derek Banks:And then she started watching one of my YouTube videos and then never talked about it ever again.
Melisa Wachs:That's funny.
Derek Banks:Like, I don't even know what you're talking about. I think you have a couple of questions for us about what we're going to be calling the Fusion AI Pen Testing Suite at Black Hills.
Melisa Wachs:Yeah. And I think the reason why I have some questions is because being on the business capture team, we're definitely in the front lines of answering customer questions. So to bring just a few questions forward, first of all, like, what is it?
Derek Banks:That's a fantastic question because everybody and their brother is doing AI. If you are on X or Twitter or whatever, I think everybody's got a new red team agentic solution that's going to kill all pen testing or something something something. Right, Brian?
Brian Fehrman:Yeah. That's that's what we keep hearing. Everyone's out of business. Close the doors. AI's AI's gonna do it all.
Brian Fehrman:Right?
Derek Banks:Yeah. So, you know, back in in in I'd say November, you know, the AI world kinda started changing. And then after the Christmas break of 2025 into 2026, people started really talking about agentic AI and coding agents and personal AI agents. And it really started to blow up. Brian and I, at least I think this was the genesis of it, I think we were in DC with John Strand and Erica at dinner at a Chinese restaurant, and they challenged us to make our own AI powered external penetration testing offering.
Derek Banks:And, you know, that that was kinda like the general strategic direction from John Strand. We could see which way the wind was blowing with traditional pen testing. And, you know, and as you know, Melisa, talking about, you know, internally with data and tests all the time and trends, we were seeing certain trends. And so they asked us to create our own AI powered pen testing suite. Really, we kind of started there.
Derek Banks:Now, I don't know, like if you're just tuning into this podcast, I think, you know, Brian and I have been doing AI stuff for a while. I went and got a master's over the pandemic in data science. And then Brian one upped me and went and got a doctorate. And what is your doctorate in?
Brian Fehrman:Data science and engineering.
Derek Banks:Yeah, so it seemed like a good fit. I could see why John and Erica were asking us. And so we started essentially with Claude code and Claude code skills kind of testing the waters and it quickly grew into a custom coded agentic platform that basically ingests external scan results and then spawns agents to go off and investigate those results, which, you know, is kind of generically like I think what a lot of folks are are doing out in in, you know, in in the industry, but we wanted to do it in a way where we built in Black Hills institutional knowledge. Essentially, like, how would we go about doing an external penetration test, and can we get the AI to do that?
Brian Fehrman:Yeah. And I think that that's a that's a key point. I mean, we spent a lot of time looking through reports, seeing what works, what doesn't, and trying to make sure that we capture that that expertise and the methodology and what makes Black Hills, so successful and keeps people coming back. Because, you know, it's like, just because you have a hammer doesn't mean that you're a master carpenter. Right?
Brian Fehrman:Like, it it's you gotta you gotta have the expertise behind it, to be able to make a good product. You can't just say, hey. We're using we're using this technology, and suddenly, it's great. I mean, we we saw this, I mean, almost a decade ago when, before this round of AI hype when AI was, being integrated more into the defensive products, the EDR type products, and comp so many companies were popping on. They're like, hey.
Brian Fehrman:We got AI in here. And it's like, okay. Well, it doesn't necessarily mean that that it's good. And so we we took the time to make sure that, like, that this is gonna be a good product, that this is, something that we feel good about and really encompasses that domain knowledge that we've built up at this company throughout the years.
Derek Banks:Yeah. So here's a really good example. Like so I'm sure that if you're watching this podcast and you've been following Black Hills and John Strand, he's talked about, how you look at vulnerability results, like scanner results. Like what do you do if you're a person and you're looking for scanner results? And we have preached for a long time, just don't look at the criticals and the highs, right?
Derek Banks:There are things in the mediums and lows and informationals and a scanner result that could be really important. And so we built our agents to do that, to look at, you know, mediums, lows, and informationals and figure out can they be chained together to do something more critical, to have more impact? And so there's actually what we call chaining algebra, like, built into the actual, like, you know, platform where it it'll go and and find, you know, things that could be combined together.
Melisa Wachs:Wow. Yeah. I would not know that. That's
Brian Fehrman:Yeah. And and Go ahead.
Melisa Wachs:I was just gonna say, I think the thing that I probably should have started with, the thing that's exciting to me about you two doing this, is because of all the people that have worked here the longest. I've been here since day one. But you two have both been here from the grounds up. And so that's what's exciting to me is I I feel like our roots are are leading this. So go ahead, Brian.
Brian Fehrman:Oh, I, yeah. No. That's that's a great point, and thank thank you. I appreciate that. So, yeah, going on and then also adding adding on, what also makes us special is that what we get out of the, the AI component is you can see exactly what it did and how it did it, how it came to the conclusions that it did.
Brian Fehrman:So it's not just you get information like, hey. You have these vulnerabilities, and then you're left wondering like, okay. Well, how did you find it? How did you validate it? How did you confirm it?
Brian Fehrman:I mean, that's always that's always it's it's been an issue in AI in general, you know, interpretive interpretability. How did you come to that conclusion? Well, with what we built, you see all of that. You get you get all the steps. You get all the commands.
Brian Fehrman:You get everything that you need to see. How did it come to these conclusions, and how can I reproduce this if I need to?
Melisa Wachs:Okay. So you probably answered this question, but my next question is is why was there a need for this service? And I think you already talked about it a little bit of listening to the environment and everything. Is there anything else? Can you give me a case study of why you think we finally took this on?
Derek Banks:Yeah. I think, like, I I mean, I've been using AI to help me do offensive security work and defensive security work since chat GPT came out, basically. And even before then, was dabbling in some more traditional, like, you know, machine learning type stuff. And I guess, you know, threat actors caught on too. And one of the things that we've always tried to do at Black Hills is be leading the security industry and kind of mimicking what threat actors do and to do the same types of things.
Derek Banks:And so in, I think it was October 2025, Anthropic actually released a report on how a Chinese threat actor was abusing their services to essentially run, you know, hacking campaigns, successful hacking campaigns. And they put out a paper that actually had a 30,000 foot view of how they were doing that, which I, you know, kind of use as like, oh, look, this is like kind of a blueprint. Because even back last fall, we were talking about this kind of stuff, right? And I think that, you know, one of the needs is just the world is changing, right? I mean, we have threat actors who are using AI to discover vulnerabilities, to do penetration testing, to automate their work.
Derek Banks:And so we wanted to build a platform that essentially enabled our testers to be better, like not to replace them, because that's not really going to happen in the short term in my opinion. Like, I mean, AI taking our jobs, if anything, has made more work for us to be honest. And so I think that I basically honestly, I think it was necessity. To keep up and to do what other folks and threat actors are doing, we almost had to do it. And it's actually kind of funny at the beginning, like say, I'd say February and March of this year, I actually had folks, peers from other firms like, reaching out going, oh, crap.
Derek Banks:I guess I shouldn't ignore this AI thing. What do I need to do to get started kind of thing? And I was like, oh, you're calling me? Oh, that's flattering. Thank you.
Brian Fehrman:So Yeah. It's it's funny with the the quick turnaround, and we I mean, we just we discussed this just within the last week or so. And, you know, what our what our, business capture team is is is seeing is that, you know, about a year ago, companies were like, hey. We absolutely don't want you to be using AI in the engagements. And now, like, a year later, they're like, hey.
Brian Fehrman:Why are you using AI in your engagements? Because we really wanna see that.
Derek Banks:Hey. Y'all got any more of that AI stuff in there? Yeah.
Melisa Wachs:Dark change. That's for sure. Which has made us really, really stand up and pay attention to what the customers are saying and the customer concerns in that regard. My next question is in regards to the timing. I know internally in the discussions, there was a lot of discussion about should we do this?
Melisa Wachs:How do we do this? But there was a there was a pause. Right? And I know there was a lot of trial and error. And you can probably tie in my next question with which is, what were your apprehensive?
Melisa Wachs:What were you apprehensive about? And then, what did you find that you were surprised was surprisingly successful? So this is like a three way question. I'm sorry. So what why did it take it took a while, because here we are six months later.
Melisa Wachs:What were you apprehensive about, or what did you have expected expected hurdles or that that you knew were going to be there? And then what surprised you in in its success?
Brian Fehrman:Yeah. So I think, I think all the all that kinda ties in in in together, I would say, as you as you said. So, you know, taking taking the time to build it out as we discussed earlier, that also kinda ties into what we're a little bit apprehensive about, is just ensuring that at the end of this that we have a product that we can feel good about, that it's not just it's not just putting out garbage that doesn't make sense. It's not hallucinating, things that that findings that aren't there. It's not missing things that should be clearly found and that the results are usable and actionable.
Brian Fehrman:And so that's something that we spent quite a bit of time on, iterating through, testing against our own company, testing it testing against some of our continuous customers, who wanted to kinda pilot this technology and we're okay with it, and just iterating through this to ensure that at the end of it, that when we run this and we see the results that we go through, we look at it, we're like, okay. This looks good. Some of the surprising successes that we've had out of of it are some of the interesting things that it found that, that our testers probably wouldn't norm might not have found not because of a skill level, but just because of time constraints. And I think that that's where this can really help augment some of that traditional pen testing is that there's only so much time to look at at everything in the customer's environment. And one particular example that we came across recently when we ran this is it actually found a customer website that had been compromised.
Brian Fehrman:But the way that it found it was that the threat actors within the HTML code of the website embedded links to shady gambling sites. And the reason that, we expect they did that is so that it would rank up it would rank their sites high on up on, SEO because that was a reputable website otherwise, so that information gets scraped off. Yeah. And now suddenly, their shady websites are appearing at the top, and that's something that likely wouldn't have been found because typically, we're not going line by line through HTML source code of every single web page to to check for it.
Derek Banks:Yeah. It was a a three day engagement. Like, I I will I will be honest. I'll I'll tell everybody this. If I was on a three day external engagement for a a a decent sized, like, environment, it's like a medium sized kind of thing for a three day.
Derek Banks:Three days are kind of our smaller size, but they had a number of web services. There's no way I would have looked for that. Like there's no way. So I would agree with Brian. I think the coverage, like we started out, Erica, our CFO, wanted to like say, you know, we talk about saving money and efficiency, can AI do that for externals?
Derek Banks:And I think the answer is, well, it depends on how you measure that. Certainly, there's an aspect to it, but I was surprised at the coverage that we could get. That really surprised me in terms of that kind of thing. Mean, we found a by the way, that one particular one was a it also found the exploit that probably led to that compromise. It's found a critical for, you know, someone that we that, you know, the tester missed.
Derek Banks:But on the converse side, right, it's it's also kind of I mean, I'll just say, not hallucinated, confabulated some stuff. Like, has trouble still like with fitting into our findings, so to speak. And so that's where we still need a tester to come in and go, okay, is this really a high? Because that happens a lot. It'll rank something like a high or a critical and then we'll look at it and say, I don't think so really.
Derek Banks:And so I still, you know, human is still in the loop and the AI report like in the data doesn't go to the customer, it goes to one of our testers and then they go and look through and verify. In fact, if you see one of the reports, it'll have what the AI found and it'll give the human like all the commands to go verify. So it's like all like right there, you know. And so I think, you know, the coverage aspect kind of surprised me a bit. Oh, and the first part of it too, is that what reason it took so long is because I think both of us, we wanted an enterprise class code base.
Derek Banks:Like we wanted to make sure that if we were leaning on it to be like a pillar of what we were gonna do moving forward for Black Hills, that it was something that not only like if we had to go do something else and somebody had to take it over, that it was a mature code base, but that it was done in kind of like, I'll say like state of the art type of way. And in fact, I actually just had AI go through and look at it and do like kind of a red team comparison and see like, you know, is this really like, you know, how would you gauge this versus the state of the art? And I think what it said was it's solid production mind, React style engineering, a deterministic workflow orchestration plus autonomous tool using workers. I was like, oh, that's really cool. Yeah, Brian, I would give that to you.
Derek Banks:I just did that this morning. So I'll send that to you. It also came up with a bunch of stuff we should fix. But hey, I mean, is an ongoing thing. I think what we've told John or tried to tell John in management that like, okay, we've built this as an MVP.
Derek Banks:This isn't like now it's done. Like it's something that's going to need like tuning and maintaining. So, you know, I think that that's hopefully we're going to keep, you know, working on that and then branching out into other types of tests. Think probably web apps are next.
Melisa Wachs:Okay. Great. So as two of our longest standing employees, I think you guys have been here I've been here all eighteen years. So I think you guys are right there. I know I know you, Fehrman, used to run the internship program.
Brian Fehrman:Mhmm. Yeah. Half years ago.
Melisa Wachs:Well, yeah. You I definitely feel much more confident hearing you explain it, knowing that you were putting the BHIS name to it, but maintaining the quality as well, and me moving forward cautiously, intelligently, and and with with purpose. Me, on the on the business capture side, I just wanna say, I personally am excited to not isolate the mid and small customers. Of course, anybody would be able to use this. Right?
Melisa Wachs:But of course, this does, you mentioned efficiency, this does cut the cost. And so that's what I'm excited about, are those customers who, quite frankly, we never wanna isolate any of our customers. But I'd we're not just for the big customers. You know, we're there for the people who who need it. And and our best customers are the ones who actually want it.
Melisa Wachs:Does that does that make sense? The ones who want it. And so I feel like this is, for me personally, I feel like this is filling a gap that we hear all the time from customers that they want they want security, and and we're just a little bit out of reach in price. So that's all I have.
Derek Banks:Yeah. No. I think it's a really good point because, you know, we've already started selling this, and it's about the third of cost of our normal, like, penetration test, like, cost. Right? And it does take less time, human on the keyboard time, right?
Derek Banks:And so I think that's a really important thing is that we can get something out there for smaller companies who normally In fact, I think that's one of the customers we have right now. They don't have a very large attack surface, right? And so it makes sense for those customers, but then also, like the way we've set this up very soon, I mean, actually it could happen now, right? Testers will be able to essentially go run this on their externals, full externals that they're running. They can go use this plus their manual stuff.
Derek Banks:So it's like kind of two different things. We have the AI only offering, but also on our normal externals, testers are going be using this to make sure that they're getting coverage. And it just has a data point to like, what else did I miss something, right? And so
Melisa Wachs:I
Derek Banks:think the goal is unless the customer specifically opts out is to run this essentially on all externals, and it also, I think the plan is to essentially integrate it into upcoming changes to CPT offerings too. Okay.
Brian Fehrman:Yeah, certainly, and then we get that mix of automated testing and manual testing together.
Melisa Wachs:Yep. It's that it's definitely I mean, now I get the name, the fusion of the automated AI with the human aspect. And that because I did go to that ROE call, and one of the things that I noticed the customer was receptive to was the confirmation that they still had access to, at any time, a a BEJIS tester, a BEJIS penetration tester. I think that's a that's something to keep in mind too, is that this is not a this is not a cruise control situation. And I could
Brian Fehrman:see Yeah. Yeah.
Melisa Wachs:Yeah. I could see when it's Yeah.
Brian Fehrman:I agree. Like
Melisa Wachs:their lights the you know, their faces open up and go, oh, we got it. Because that's what they wanted. So
Brian Fehrman:Yeah. Yeah. I agree. Like, that's one thing I noticed too. Like, at the beginning of the call, they seemed a little guarded, like, little bit reserved.
Brian Fehrman:And by the end of it, they're like, alright. You guys seem on top of things. We're excited about this.
Derek Banks:So I think, if that's all your questions, that was really fun. I was expecting, like, a really, like, know, gotcha questions. No no gotcha questions.
Melisa Wachs:I don't know. Now I'm now I'm disappointed in myself.
Derek Banks:I know. Right? No. I'm just I'm just kidding. Well, thanks for coming on, Melisa.
Derek Banks:You're welcome. Anytime you wanna come hang out on our our podcast.
Melisa Wachs:Yeah. Well, it was nice. It was nice. It was different. It's different than just staring at emails all day because never customer facing.
Melisa Wachs:And I so this was nice. So
Brian Fehrman:Well, well This is great.
Melisa Wachs:Thanks so much. Alright.
Derek Banks:Well, I guess I'll the tagline. Keep on prompting. Keep on prompting.