What Does an AI Penetration Test Cover? Models, RAG, Agents, and APIs
Published September 23, 2026
By Kelvin O. Medina
“AI penetration test” can describe very different engagements. A useful scope starts with the deployed product: what users can reach, what the application sends to a model, which documents retrieval can access, which actions an agent can take, and which web or API controls stand between those paths and sensitive data.
The goal is not to declare an AI system “secure.” It is to test agreed trust boundaries and abuse cases, manually validate what is reproducible, and explain the technical and business consequences clearly enough for the product and engineering teams to act.
Start with the deployed system and its trust boundaries
The diagram below is illustrative. Some products call a hosted model directly; others add retrieval, documents, agents, function calling, or an MCP gateway. A test should include only the components that exist, are authorized, and can be exercised safely under the rules of engagement.
Four layers buyers commonly need to scope
| Component | What to test | Access the client supplies | Example finding or business consequence |
|---|---|---|---|
| Model interface | Instruction boundaries, prompt injection paths, unsafe output handling, sensitive-data exposure, and abuse controls visible through the product. | Test accounts, supported workflows, model and system-prompt context where available, rate limits, and prohibited actions. | A crafted input overrides intended instructions and causes the application to reveal restricted context. |
| RAG / retrieval | Document authorization, tenant separation, retrieval filters, ingestion trust, citations, and context leakage. | Representative documents, user roles or tenants, ingestion paths, index boundaries, and expected access rules. | A user retrieves another tenant’s document fragment because filtering occurs after retrieval. |
| Agents / tools / MCP | Tool selection, argument validation, authorization, approval gates, indirect prompt injection, and action boundaries. | Tool definitions, MCP servers where present, test integrations, safe test data, roles, and permitted side effects. | Untrusted content causes an agent to invoke a business action outside the user’s authorization. |
| Application / API | Authentication, object and tenant authorization, session handling, data flows, file processing, API abuse, and business logic around AI features. | Application and API endpoints, test users for relevant roles, architecture notes, environments, and workflow expectations. | An API authorization flaw exposes prompts, conversations, or generated artifacts belonging to another account. |
Practical prerequisites and rules of engagement
Before testing, the client and provider should agree on target environments, accounts and roles, tenants, APIs, model endpoints, retrieval sources, enabled tools, test data, allowed techniques, rate limits, prohibited actions, escalation contacts, test window, evidence handling, and stop conditions. Production testing may require narrower techniques than a dedicated test environment, especially when tools can change records or contact third parties.
Architecture and trust-boundary review should come first. It identifies where user input, retrieved context, model output, tool requests, tool results, and application data cross authorization or tenant boundaries. Targeted adversarial cases can then be chosen for the actual design rather than copied from a generic checklist.
What is not automatically included
Testing a third-party model’s exposed behavior and its integration into an application is not the same as auditing the model provider. Inaccessible weights, proprietary training data, the provider’s internal training or evaluation pipeline, and controls inside the provider’s environment are outside a standard application penetration-test scope. Reviewing those areas requires separate access, authorization, scoping, and potentially different specialists.
The current official OWASP GenAI LLM Top 10 for 2026 is a useful risk reference when selecting test cases. It is not a certification, and citing it does not promise complete coverage of every category. The NIST AI Risk Management Framework can provide separate governance context, but a penetration test does not establish AI RMF compliance.
What a useful report should contain
The report should identify the tested components, environment, roles, dates, assumptions, exclusions, and techniques. Findings should include reproducible evidence, affected trust boundaries, severity, plausible business impact, and remediation guidance. It should also distinguish confirmed vulnerabilities from observations that need design or governance work.
An optional retest can validate agreed fixes against the original reproduction steps. It should state what was retested and whether the specific finding was resolved; it should not imply that every part of the system was tested again or that future changes remain secure.
Five questions to ask a provider
- Will scope start from our deployed architecture, trust boundaries, roles, and data flows?
- How will you test model behavior together with retrieval, tools, web, and API authorization?
- Which actions, environments, data, and third-party services are excluded or require special approval?
- Will findings include manual validation, reproducible evidence, business impact, and practical remediation?
- What will an optional retest cover, and how will its result be documented?
ControlSolid’s AI & LLM penetration testing service scopes the deployed application, model interface, retrieval paths, agents or tools where present, and the surrounding web and API controls. Prefer to describe your needs in writing? Contact us.
This article provides general guidance. A responsible test scope depends on the system, authorization, environment, data, integrations, and agreed rules of engagement.