Prompt Injection

Description

Prompt Injection occurs when untrusted input influences an AI system to disregard or override its intended instructions, policies, or constraints. Malicious instructions may be supplied directly through user prompts or indirectly through external content, documents, retrieved data, or tool outputs. Depending on the application design and available capabilities, successful prompt injection may cause unintended information disclosure, misuse of connected tools, unauthorized actions, or circumvention of application safeguards.

Recommendation

Treat user input, retrieved content, external documents, and tool outputs as untrusted data. Do not rely on system prompts alone as a security boundary. Enforce authorization and access controls outside the model, restrict tools to the minimum permissions required, validate tool arguments, and require explicit approval for sensitive or consequential actions. Separate trusted instructions from untrusted content where possible, and test the application against direct and indirect prompt injection scenarios.

References

• OWASP Top 10 for LLM Applications
• Model Context Protocol: Security Best Practices