Generative & Agentic AI Security: Prompt Injection, Tool Abuse, Data Leakage, Excessive Agency, and Runtime Controls

 

Generative and agentic applications introduce security risks that appear when natural-language inputs influence models, retrieval, tools, and actions. The main question is not whether a model can be perfectly trusted; it is whether the surrounding system can constrain what untrusted input is allowed to cause.

Treat prompts and retrieved text as untrusted input

User messages, uploaded documents, web content, and retrieved knowledge may contain instructions that conflict with application policy. Prompt injection occurs when such content changes model behavior in an unintended way.

AI security has to cover the full application rather than prompt wording alone; AI-102 solution architecture shows how model calls, data, retrieval, services, and application logic form one attack surface.

Separate data from privileged instructions

System instructions should not be mixed casually with untrusted content. Use clear boundaries, structured fields, and application logic so documents are treated as data rather than policy.

This helps reduce accidental instruction following, but it is not a complete security control.

Authorization belongs outside the model

A model must not decide whether a user is allowed to access a database record or execute a privileged tool. Enforce identity and scope at the actual service boundary.

The principles in zero trust security apply directly: access should be verified at each protected resource, not inferred from conversational context.

Tool abuse creates real-world impact

An agent with write-capable tools can create tickets, change records, send messages, or trigger infrastructure actions. Validate tool arguments and use least-privilege credentials.

For high-impact actions, require deterministic policy checks or human approval before execution.

Excessive agency increases blast radius

Giving an agent broad permissions, many tools, and long autonomous runs increases the number of ways a manipulated or mistaken plan can cause harm.

Start with narrow capabilities and explicit stopping conditions. Expand autonomy only after the control model is demonstrated.

Retrieval can leak sensitive information

A RAG system may retrieve documents that are relevant but unauthorized for the current user. Apply permissions before content enters model context.

Identity, encryption, and information boundaries remain enforceable requirements in AI applications; AWS data protection separates those controls from the model itself.

Logs can become a privacy problem

Prompt traces, retrieved passages, tool arguments, and model outputs may contain sensitive data. Logging everything by default can create a second uncontrolled data store.

Redact secrets, define retention, restrict access, and log enough metadata for investigation without unnecessarily copying sensitive content.

Model output should not be trusted as code or policy

Generated SQL, scripts, commands, or configuration should be validated before execution. Use allowlists, parsers, sandboxes, and review where appropriate.

Model output still crosses ordinary trust boundaries, and the SC-100 architecture overview places those boundaries inside a wider security architecture.

Rate limits and budgets are security controls

Limit tool calls, tokens, concurrent tasks, and total execution time. Budgets reduce denial-of-wallet risk and prevent loops from consuming resources indefinitely.

They also create predictable failure behavior when an agent cannot complete safely within its limits.

Secrets should never enter prompts unnecessarily

API keys, passwords, signing material, and privileged tokens belong in secret-management systems. Tool execution should use managed identities or protected credentials rather than revealing secrets to the model.

The same identity, network, configuration, and monitoring controls appear in AZ-500 security overview, showing that AI workloads inherit the surrounding cloud-security posture.

Test adversarial workflows

Create cases where documents contain malicious instructions, users request unauthorized data, tool arguments exceed permitted scope, or an agent tries to continue after a denial.

Security evaluation should verify the entire trajectory, not only the final text.

Monitor policy decisions

Record which tools were requested, why calls were denied, which identities were used, and whether human approval occurred. Correlate model traces with application-security events.

Logging, incident response, and cloud security operations become more important as AI gains access to sensitive data and tools; the AWS security specialty path develops those operational responsibilities.

Use deterministic controls for deterministic rules

If a rule says “never transfer more than this amount” or “only owners may read this record,” enforce it in code or policy. Do not ask the model to remember the rule.

AI systems remain engineered applications with services and controls around the model; the Azure AI Engineer path makes that implementation boundary explicit.

Security is layered containment

No single prompt, filter, or model setting can secure an agentic application. The reliable design combines input boundaries, identity, authorization, tool validation, least privilege, logging, evaluation, and human control for consequential actions.

The goal is not to make the model infallible. It is to keep a model mistake from automatically becoming a security incident.