Skip to main content
Research / Journal / Archive
PROTOCOL.READ / 9 min read

AI Security Audit Checklist: 20 Controls to Verify Before Shipping an LLM Application

A practical, control-by-control checklist for auditing LLM applications and AI agents: prompt injection, data leakage, tool permissions, retrieval security, logging, and incident response.

Why LLM Applications Need Their Own Security Audit

A conventional application security review covers authentication, injection, and dependencies. An LLM application adds a new kind of component: a model that follows natural-language instructions from anyone who can get text in front of it. That includes users, web pages, emails, documents, and tool outputs. An audit has to test that boundary directly.

Use this checklist before launch and again after every significant change to prompts, tools, models, or data sources.

1. Inputs and Prompt Injection

  1. Map every input channel. List each place text can reach the model: chat, uploaded files, retrieved documents, emails, web content, tool results.
  2. Label provenance. Mark which content is trusted instructions and which is untrusted data, and keep them structurally separate in the prompt.
  3. Test direct injection. Try instruction overrides, role-play framing, and encoding tricks against the live system.
  4. Test indirect injection. Plant instructions inside a document or web page the agent will read, and check whether it obeys them.
  5. Limit input size and format. Cap length, strip hidden text, and reject unexpected file types.

2. Tools, Actions, and Permissions

  1. Inventory every tool the model can call and what each one can change.
  2. Apply least privilege. Each agent gets only the scopes its job needs, with separate read and write tools.
  3. Validate tool arguments in code against a schema and policy before executing, never trusting the model’s output.
  4. Require human approval for irreversible or high-impact actions such as payments, deletions, and external messages.
  5. Use short-lived credentials issued per task. Never embed long-lived admin keys in prompts or agent configuration.

3. Data Protection

  1. Classify the data the model can see. Remove secrets and unnecessary personal data from context.
  2. Control egress. Allow-list outbound domains, and block rendering of untrusted remote images or links that could carry data out.
  3. Scan outputs for credentials, personal data, and internal identifiers before they reach the user.
  4. Check provider terms for data retention and training use, and confirm they match your compliance needs.

4. Retrieval and Memory

  1. Enforce access control at query time. A user must only retrieve documents they are allowed to read.
  2. Control ingestion. Know who can add content to the knowledge base and record the source of every chunk, because poisoned documents are easier to prevent than to find.

5. Monitoring, Testing, and Response

  1. Log every prompt, tool call, and policy decision with enough context to reconstruct an incident, while protecting the logs themselves.
  2. Alert on anomalies: unusual tool usage, repeated policy violations, spikes in refusals.
  3. Maintain a red-team regression suite of injection and jailbreak attempts and run it on every prompt or model change.
  4. Write an incident runbook that can disable a tool, an agent, or the whole feature quickly.

How to Run the Audit

Work through the list with the people who own each area: engineering for tools and permissions, data owners for classification, and security for logging and response. For each control, record the evidence: a test result, a configuration, a screenshot. A control without evidence is a control you cannot claim.

Prioritize by blast radius. An agent that can only draft text is lower risk than one that can move money or modify records. Fix the controls that limit the worst-case outcome first.

What a Good Result Looks Like

You do not need every box ticked on day one, but you do need to know which are open, who owns them, and what the compensating control is in the meantime. Security for AI systems is a continuing process, because models, prompts, and attacks all change.

Get Help With Your Audit

FalconicLab runs AI and LLM security assessments and security audits and penetration tests. For background, read our article on zero trust for autonomous AI models, see the guardrail enclave experiment, or book a consultation.