AI took the lead in an attack
Yesterday, we witnessed a historic moment in cybersecurity. Anthropic reported that AI systems weren’t just assisting human attackers, they became the operators themselves. In this case, AI models, applications and agents performed 80–90% of a complex cyber espionage campaign, from reconnaissance to exploitation to credential harvesting, with humans only providing minimal oversight.
This isn’t a sci-fi scenario. It’s here. It’s today. And it changes everything about how we should think about enterprise AI risk.
Key takeaways
- AI isn’t just a tool. It can now act autonomously, performing sophisticated tasks at machine speed, including malicious attacks.
- Enterprise AI models, applications, and agents are part of the attack surface. Vulnerabilities are specific to each system.
- Guardrails alone are insufficient to prevent misuse. Automated red teaming and real-time AI defense are essential.
- Protecting your AI protects your entire enterprise, including traditional systems that AI can access.
- Red teaming for good is no longer optional. It’s how you level the playing field against AI-enabled attackers.
Not just chats, AI now has real operational power
Forget the days of prompting chatbots to “sell a truck for $1.” The AI of today can write exploit code, scan networks, and manipulate systems at machine speed. Misuse is no longer a novelty, it’s a business-impacting reality.
The Anthropic report provides a stark illustration of the new reality. Attackers jailbroke the model using techniques such as prompt injection, not with a single master prompt, but by breaking down their attacks into "small, seemingly innocent tasks that the model would execute without being provided the full context of their malicious purpose." By chaining these tasks together, the AI agent performed a full-cycle espionage campaign, from reconnaissance to data exfiltration, as an autonomous, multi-turn operation.
If you think your enterprise is safe because you aren’t building attack tools, think again. AI can be leveraged by attackers to reach deep into your infrastructure faster than any human team ever could.
Don’t blame the model: powerful tools can be misused, including yours
Let’s be clear: in the recent disclosure by Anthropic, Claude isn’t the problem. It’s a highly capable tool built for utility. The issue is when powerful tools fall into the wrong hands or are misused internally.
If you are building AI models, applications, or agents in your enterprise, you are now creating incredibly powerful systems. And with that power comes great responsibility. You need to make sure these systems cannot be weaponized against your organization or anyone else.
AI models, apps, and agents are now part of the attack surface
Every AI system you deploy is a potential vector for abuse. Vulnerabilities are unique to each model, application, and agent. Even a perfectly designed enterprise app can be tricked, misled, or exploited if it interacts with external data, tools, or users.
AI doesn’t exist in a vacuum. It touches your infrastructure, your APIs, your workflows. If compromised, it can give attackers unprecedented speed and scale, reaching systems traditional defenses alone can’t keep up with.
Guardrails alone won’t save you
Built-in guardrails are necessary, but they’re not enough. Each AI system behaves differently and has specific vulnerabilities. That’s why automated red teaming is critical. Automated red teaming performs the following essential functions:
- Tests each model, agent, and application in realistic adversarial scenarios.
- Identifies where guardrails can be bypassed or misused.
- Surfaces risks before attackers do.
And here’s the key insight: the same tools you use to test AI can protect your traditional infrastructure. Attackers will leverage AI to target it, using emerging agentic tools that interact with these system components. Automated red teaming lets you simulate attacks at the same scale and speed as AI-enabled adversaries.
How to fight back: automated red teaming + real-time AI defense
There are two levers every enterprise must pull to defend against this new class of threat. These include automated red teaming and real-time AI defense.
Automated red teaming
Automated red teaming requires that you continuously probe your AI systems for exploitable behaviors. Models, apps, and agents need to be thoroughly tested now before attackers are able to exploit them. To effectively red team models, agents, and apps in this new era, we must use new techniques. This includes leveraging techniques like using computationally generated attacks. These scenarios use predefined but randomized scripts to construct a multi-turn conversation. This method is ideal for systematically testing for known patterns of attack at scale across a model's entire surface. This enables far greater coverage for real-world threats than a traditional prompt response evaluation approach. Additionally, using LLM-generated approaches can increase coverage as well, which requires using a controlled LLM to generate malicious behavior to simulate real-world scenarios that attackers use.
Real-time AI defense
As AI models, apps, and agents are deployed, abnormal or malicious activity needs to be detected as it happens. This is especially true for AI-generated jailbreaks or commands that could indicate an attack in progress.
In this new era of AI-weaponized attacks, models and agents collaborated for malicious purposes. This challenge deepens in the emerging landscape of autonomous agents powered by Model Context Protocol (MCP) and its expanding ecosystem of tools. While agentic systems promise efficiency, they also introduce complexity and additional risk from rogue MCP servers, unapproved tools, and shadow infrastructure that slip beyond governance. Achieving visibility and preventing threats in real time across this stack is essential to secure AI systems.
Together, automated red teaming and real-time AI defense give you visibility, resilience, and control. They allow you to leverage AI for defense before attackers can use it for offense, thus leveling the playing field.
Level the playing field on attackers
AI is no longer a novelty. It is a powerful operational tool that can be misused at machine speed. Every enterprise AI system is part of your attack surface. Protecting it is no longer optional.
Generic guardrails alone won’t keep you safe. Red teaming and real-time AI defense are the defenses you need now, for both your AI systems and the traditional infrastructure they touch.
The future is not about fear. It’s about preparation, resilience, and using AI for defense as fiercely as attackers use it for offense. The question isn’t “will AI be weaponized?” It’s “how prepared are you?” when it is.
How TrojAI can help
TrojAI delivers security for AI. Our mission is to enable the secure rollout of AI in the enterprise. Our comprehensive security platform for AI protects AI models, applications, and agents. Our best-in-class platform empowers enterprises to safeguard AI systems both at build time and run time. TrojAI Detect automatically red teams AI models, safeguarding model behavior and delivering remediation guidance at build time. TrojAI Defend is an AI application firewall that protects enterprises from real-time threats at run time. TrojAI Defend for MCP secures agentic AI workflows at runtime.
By assessing the risk of AI model behavior during the model development lifecycle and protecting it at run time, we deliver comprehensive security for your AI models, applications, and agents.
Want to learn more about how TrojAI secures the largest enterprises globally with a highly scalable, performant, and extensible solution?
