Last week I had the opportunity to attend JFrog’s 2025 swampUP conference in Napa, California. As a JFrog Partner, it was a great way to meet with various teams across DevOps, DevSecOps, Security, and Compliance, and showcase our TrojAI integrations with JFrog Artifactory and Evidence.
As enterprises race to operationalize AI, securing the AI software supply chain is emerging as a critical priority. AI models are increasingly being treated as first-class software artifacts — packaged, versioned, and distributed alongside traditional applications. As it’s being more frequently surfaced, AI models, and the applications and agents they are being embedded in, introduce unique security risks: data leakages, adversarial prompts, bias and hallucinations, and model extraction threats.
Protecting and mitigating against these AI risks as early as possible in the model management and security lifecycle is where the integration of TrojAI and JFrog delivers real value, combining TrojAI’s AI Security platform with JFrog’s DevSecOps platform.
Managing and securing the model lifecycle
JFrog Artifactory is the single source of truth for binaries and artifacts, including AI/ML models. As most AI practitioners know, models can be versioned, stored, and distributed. This lifecycle requires the same security and governance applied to more traditional artifacts, like container images, Helm charts, and packages. By treating models as artifacts, organizations gain traceability, consistency, and reproducibility.
Knowing where your models are is an important first step. The next piece is making sure those same models are adequately tested and secured. For this, JFrog Xray natively provides security scanning for AI models, which helps uncover CVEs in model files, such as data poisoning risks or backdoors during model serialization or deserialization processes.
Complementing model scanning is TrojAI’s automated red teaming capabilities. Since AI is more than just code – it’s data and weights – securing AI requires comprehensive understanding of the behavior of the AI itself, plus making sure it operates as intended from a security and safety perspective. That’s why our integration with JFrog helps enable automated red teaming on models using TrojAI Detect, with the ability to connect stored models in JFrog Artifactory into the TrojAI platform.
TrojAI Detect automates the red teaming of AI models, applications, and agents during the build and pre-deployment stages. Leveraging both single and multi-turn attack techniques, as well as a variety of algorithmic, LLM-generated, and agentic workflows, TrojAI Detect systematically assesses models for vulnerabilities such as prompt injection, data and PII leakages, and toxic, unwanted, or otherwise harmful content. Attack datasets are out of the box, with the ability to load custom datasets for bespoke application and agent use cases, providing comprehensive capabilities to automate model red teaming.
After selecting and running your tests, you can review the results. TrojAI Detect offers granular views to surface model behavior gaps and vulnerabilities, exposing how the model responds and whether it’s in line (Pass/Fail) with expectations, with context.
Bringing the integration back full circle, these red teaming results can then be passed back into JFrog for visibility as Artifactory Evidence. JFrog Evidence helps enable compliance and audit readiness for security teams by providing attestation records across the AI software supply chain. By surfacing TrojAI red teaming and risk results inside JFrog Evidence, organizations get auditable proof that their AI models were tested and secured prior to deployment. This is especially critical for organizations preparing for compliance.
Why this matters to your DevSecOps and AI security teams
AI is part of the supply chain now. Models should be treated as artifacts, subject to the same DevSecOps controls as packages and containers. Security leaders need confidence that every model checked into Artifactory has been scanned, tested, and validated.
Automated red teaming at scale helps provide confidence that enterprise models are safe and secure. Manual AI red teaming can be slow, expensive, and inconsistent. By integrating TrojAI Detect directly into the JFrog pipeline, organizations can auto red team models at ingestion.
With AI regulations emerging, it won’t be enough to say “we tested our models.” Auditors will want proof. By surfacing TrojAI’s results in JFrog Evidence, security and compliance officers can show a chain of custody and testing history for every AI model.
By plugging TrojAI into the JFrog platform, practitioners don’t need to swivel between tools. Model risks show up in the same dashboards where they already track software packages and vulnerabilities. It’s easy and transparent.
The JFrog - TrojAI integration represents a paradigm shift in security for AI. By treating models as artifacts in the AI supply chain, scanning them for vulnerabilities, automatically red teaming them, and capturing results as audit evidence, organizations can reduce and address AI attack surfaces as early as possible in the model pipeline.
Building on the momentum
This is the first step in our strategic partnership with JFrog, and we’re excited to do more and dive deeper. Among the various announcements at swampUp was the JFrog AI Catalog. The JFrog AI Catalog is a central registry for a wide variety of models that can now be discoverable, versioned, and governed in JFrog.
This offers an expanded starting point for our integration, acting as a new endpoint to register models to TrojAI. Downstream, these models can be automatically red teamed by TrojAI Detect, with evidence-based insights returned to JFrog. AI Catalog provides frictionless access to the models enterprises are using – open source (e.g., NVIDIA Nemotron), frontier models from major service providers including OpenAI and Anthropic, as well as internally developed models or third-party ones like those from Hugging Face or MLflow registries.
In a similar vein to our integration today, TrojAI Detect can initiate automated red teaming on these models and capture evidence-based insights back into JFrog. We’ll also integrate TrojAI Defend – our GenAI runtime defense capability – so that JFrog users can continuously monitor and protect deployed models, feeding runtime threat insights back into JFrog’s ecosystem.
There’s a lot to explore and we can’t wait to continue our work with JFrog! Stay tuned for more information as we evolve and continue this better together journey.
How TrojAI can help
Our best-in-class security platform for AI protects AI models, applications, and agents both at build time and run time. With support for agentic and multi-turn attacks, TrojAI Detect automatically red teams AI models to safeguard model behavior and deliver remediation guidance at build time. TrojAI Defend is our GenAI Runtime Defense solution that protects enterprises from threats in real time.
By assessing model behavioral risk during development and protecting it at run time, we deliver comprehensive security for your AI models, applications, and agents.
Want to learn more about how TrojAI secures the world's largest enterprises with a highly scalable, performant, and extensible solution?
Learn more at troj.ai now.
