Author: Kevin Hager, Senior Security Specialist

Marketing vs. Reality: The Risk of AI-Only Testing

We are seeing a risky trend: organizations are being told that automated penetration tests are a faster, cheaper, and "sufficient" replacement for traditional testing. Lured by the aggressive marketing of autonomous agents, leaving many companies with massive blind spots in their defenses.

The reality is that not all AI tools are created equal. While basic AI-driven pentests are excellent for high-speed external vulnerability scanning, they operate strictly on rigid checklists and known software bugs. They lack the capacity to pivot.

On the other hand, advanced autonomous penetration testing, like DRT Cyber’s Tanuki, excel because it doesn't just scan for known flaws; it actively makes decisions and chains multiple minor vulnerabilities together to simulate a real-world attack. Tanuki dynamically adapts its tactics based on the live environment, allowing it to continuously hunt for complex network flaws without requiring constant human intervention.

But even the most advanced automation cannot replace human intuition. While autonomous AI is a fantastic tool for continuous baseline testing between cycles, human-led penetration testing currently remains essential. True security still requires human engineers at the wheel to spot critical business-logic flaws that software alone simply cannot see.

Real-World Case Study: The Hidden Vulnerability

A mid-sized client recently came to us after receiving a "clean" security report from an AI-driven pentesting vendor. The automated tools found only minor, low-risk issues, which gave the client a false sense of security.

During the engagement we leveraged DBG’s External Network Testing methodology and discovered a vastly different story:

The Oversight: It was found that the client’s AI testing tool lacked a thorough discovery. The external team performed a comprehensive discovery then performed password spraying against Microsoft 365, resulting in a successful guess for a VERY common password.

The Entry Point: By manually analyzing the client’s VPN SSO configuration, we identified a misconfiguration that allowed access to the client’s network—bypassing multifactor authentication!

The Critical Miss (False-Negative): False-negatives are a liability that leave a client dangerously overconfident. By applying our manual methodology, we identified two critical flaws (CVE-2025-25231 and CVE-2025-25229) in the client’s MDM solution. An unauthenticated attacker can chain these exploits to achieve Remote Code Execution (RCE) on the appliance, resulting in network access. To prioritize the client's stability, the team stopped once we confirmed the exploit path was viable and immediately contacted them to remediate this high-impact finding.

Client’s Comment After Seeing Our Work

“These all (AI pentests) seem to be gimmicky. I think they will be better but these are not real solutions from what I have seen. At least not yet… We stopped using them because they seemed a bit lacking. It missed the MDM vulnerability that you caught. Since we were using it for external scanning, this seemed like a significant hole/risk.”

Our Position: Tools are Not Testers

As testers, we use AI as a force multiplier for rapid scripting and deeper enumeration. However, human-led testing remains our foundation.

Just as running a Nessus scan does not count as a full vulnerability assessment, an AI-driven test is a diagnostic tool—not a replacement for a deep-dive penetration test.

How Digital Boundary Group (DBG) Can Help

AI-driven security testing is a powerful tool for identifying vulnerabilities quickly, but it can't replace human expertise. The most critical risks—such as business-logic flaws and complex attack paths—often require experienced penetration testers who can think like real-world attackers.

At DBG (a division of DRT Cyber), we combine advanced testing tools with expert-led penetration testing and red team exercises to uncover the risks automation alone can miss. The strongest security strategy leverages both AI-driven insights and human validation.

Ready to see how your environment stands up against real-world attacks?

Contact us to schedule a penetration test or red team engagement today >

Disclaimer: AI and technology are changing every day. The content of this article reflects our current understanding and perspective at the time it was written. As new advancements, research, and regulations emerge, some of the information and viewpoints shared here may evolve over time.