Prompt Injection for AI Applications — An Instruction Is Not a Security Boundary

A small application reads an incoming email, asks a language model to summarize it, and then, if the summary mentions a task, lets the model trigger a follow-up action. The email is coming from someone you do not control. The text inside it can say anything, including "ignore your previous instructions." When that text is added to the model's instructions, the model may treat it as an instruction too. This is the shape of a class of attacks that the OWASP GenAI Security Project lists as the first risk in its Top 10 for LLM applications: prompt injection.

A prompt injection vulnerability exists when input alters a model's behavior or output in unintended ways. The dangerous part is not that the model writes something strange; it is that an application cannot reliably tell its own instructions apart from attacker-controlled data. This article explains why that distinction is so hard, shows how the attack changes when data comes from outside, and looks at mitigations that live in application code rather than inside the prompt.

One Text Stream, Two Kinds of Content

Many small integrations are effectively one concatenation: system instructions, retrieved content, and user input are joined into a single prompt before being sent to the model. In that stream there is no structural marker that the model treats as absolute. Simon Willison, who helped name the vulnerability in 2022, demonstrated the basic version in an April 2023 essay: an application that was supposed to translate text into French and return JSON stopped translating and started speaking like a stereotypical eighteenth-century pirate because the untranslated input contained its own instruction to do exactly that (Simon Willison, "Prompt injection: What's the worst that can happen?", 14 April 2023).

The OWASP definition emphasizes a subtle point: a prompt injection does not have to be visible to a human reader. It only has to be parsed by the model. An instruction hidden in the white space of a page, in an image, or inside a long document can have the same effect as an explicit request. In security terms, this is an input-validation problem in which the "query language" is natural language.

Prompt injection is sometimes used interchangeably with jailbreaking, but the OWASP project keeps the two separate. Jailbreaking is a form of injection in which an attacker's input causes the model to disregard its safety protocols entirely. Prompt injection is broader: it manipulates behavior without necessarily removing safety mechanisms. For a small application, both can mean the model now follows attacker logic instead of the developer's.

Indirect Injection: the Attacker Is No Longer the User

The clearest shift came with what Kai Greshake and his colleagues named indirect prompt injection. Their 2023 paper, Not what you've signed up for, argues that LLM-integrated applications blur the line between data and instructions. The attacker no longer needs to prompt the model directly. Instead, they inject instructions into data that the application is likely to retrieve: a web page, a shared document, a resume, an email, or even the README of a repository (Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz, arXiv:2302.12173, 2023).

Willison's 2023 essay describes the same family of attacks with concrete examples. One especially clear case is search-index poisoning: a researcher added an instruction in white text on a white background to his academic profile page, and Bing, reading the visible content of the page into its prompt, later described him as having that specific claimed expertise. The hidden text was not meant for humans; it was meant for the retrieval pipeline that feeds pages into a model (Willison, 2023).

Greshake's paper maps a range of impacts that go beyond a single awkward reply: data theft, worming (spreading an instruction to other systems the application touches), and contamination of the broader information ecosystem. The group demonstrated practical attacks against real deployments, including a GPT-4-powered chat assistant embedded in a browser and code-completion engines. What matters for a small automation is that the vulnerable property is not exotic: retrieve data, splice it into a prompt, and the model will sometimes follow what the data says.

Why a Stronger Prompt Is Not the Answer

When people first encounter injection, they usually propose prompt-level defenses: tell the model to ignore attacks, wrap retrieved text in delimiters, add a warning to the system prompt, or ask the model to distinguish trusted from untrusted content. These are worth trying, but they are not a boundary.

Willison is equally direct about prompt-level filtering: in his 2023 essay he describes plenty of "95% effective" solutions based on filtering input and output, and warns that the remaining five percent is exactly the window an adversarial attacker will look for. Even a separated system prompt is bypassable. In the same essay he showed that GPT-4, which introduced the separate system prompt concept, still followed instructions placed inside the user input that asked it to change behavior.

The 2025 OWASP entry makes a related point about retrieval and tuning: techniques such as retrieval-augmented generation and fine-tuning are meant to make outputs more relevant, but research has not shown that they fully mitigate prompt injection. And Willison's 2025 commentary, written while quoting The Economist's coverage, summarizes the persistent situation bluntly: there is a lethal trifecta of conditions that opens AI systems to abuse, namely access to private data, exposure to untrusted input, and the ability to act (Simon Willison, "Why AI systems might never be secure", 23 September 2025). The Economist piece he links cites a January 2024 episode in which a delivery firm turned off its AI customer-service chatbot because customers could command it to reply with foul language. That was harmless and cheap; the uncomfortable truth is that the industry keeps shipping the same combination.

The conclusion from all of these sources is consistent: prompt-level text is not a security boundary. Treating a paragraph of system instructions as a firewall puts the application in the position of defending itself with the same language the attacker controls.

Mitigations That Live Outside the Prompt

If the text layer cannot be trusted, controls must move to the layers the attacker does not control: the code that decides what the model may do, and the person or policy that approves the outcome.

The OWASP entry lists practical mitigation categories that are deliberately application-level. Among them:

• Least privilege. Give the application its own API tokens, handle functions in code instead of handing them wholesale to the model, and restrict the model's access to the minimum needed for its task.
• Human approval for high-risk actions. Keep a human in the loop for privileged operations such as sending messages, changing files, or calling administrative endpoints.
• Deterministic output validation. Specify output formats and validate them with code, not with another prompt.
• Segregate untrusted content. Clearly separate external content so its influence on the core instructions is limited.
• Treat the model as an untrusted user. Run regular tests of the trust boundaries assuming the model can be persuaded.

A compact way to express this in code is a small policy table for the actions a model is allowed to propose. The code below is an illustration and was syntax-checked with php -l; adapt the tools and meanings to your own application.

<?php

declare(strict_types=1);

final class ToolPolicy
{
    private const TOOLS = [
        'search_knowledge_base' => ['risk' => 'read',     'confirm' => false],
        'create_draft'          => ['risk' => 'write',    'confirm' => false],
        'send_email'            => ['risk' => 'external', 'confirm' => true],
        'run_shell_command'     => ['risk' => 'execute',  'confirm' => true],
    ];

    public static function proposal(string $tool, array $arguments): array
    {
        if (!isset(self::TOOLS[$tool])) {
            return ['ok' => false, 'reason' => 'tool_not_allowed'];
        }

        $allowedArguments = [
            'search_knowledge_base' => ['query'],
            'create_draft'          => ['title', 'content'],
            'send_email'            => ['to', 'subject', 'body'],
            'run_shell_command'     => ['command'],
        ];

        $unexpected = array_diff(array_keys($arguments), $allowedArguments[$tool] ?? []);
        if ($unexpected !== []) {
            return ['ok' => false, 'reason' => 'unexpected_argument'];
        }

        return [
            'ok'        => true,
            'tool'      => $tool,
            'confirm'   => self::TOOLS[$tool]['confirm'],
            'reason'    => 'proposal recorded for review',
        ];
    }
}

Even if an indirect injection succeeds and the model proposes send_email with an attacker-chosen recipient, the boundary holds: the action is not in the application's allowlist without a confirmation step, so the code refuses to treat it as final. Injection manipulates the proposer, not the gatekeeper. The gatekeeper is ordinary, boring, reviewable code belonging to the developer.

This does not mean confirming everything. Read-only actions with a narrow argument allowlist can proceed; write actions that reach outside the application are where confirmation earns its cost. The exact split depends on what the application does, which is a decision for its operator, not for the model.

Uncertainty and Open Questions

Be honest about the limits of what this article claims. Prompt injection has no generally accepted, guaranteed defense; every mitigation reduces exposure and raises the attacker's cost rather than closing the problem. Willison wrote in 2023 that he had not seen a robust defense guaranteed to work one hundred percent of the time, and his 2025 commentary indicates that nothing has fundamentally changed. Treat any framework, this one included, as a starting point for your own threat model.

Two open questions are worth holding in mind. First, how much agency should an automation receive? Each additional tool is a new way for an injected instruction to produce an effect; the cheapest mitigation is often to remove the tool entirely. Second, is sandboxing the real answer? If the model is treated as untrusted software, containing it in a restricted environment with no privileges beyond those of a disposable worker changes the worst-case outcome from "full access" to "one contained step." Neither question has a universal answer, which is exactly why they belong in a design conversation rather than a prompt.

Conclusion

Prompt injection is not a bug that a better instruction set fixes. It is the consequence of an application that lets text act as both data and instructions at once. The honest response is architectural: separate the two, keep the model powerless outside its declared role, validate output in deterministic code, and require someone other than the model to approve anything that matters. The phrase "an instruction is not a security boundary" is the summary of that position. The prompt proposes; the application decides.

References

1. OWASP GenAI Security Project — LLM01:2025 Prompt Injection. OWASP Foundation. Accessed 23 September 2026. Primary source (community security standard).
2. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz — Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. arXiv:2302.12173 (2023). Accessed 23 September 2026. Primary source (peer-adjacent academic paper).
3. Simon Willison — Prompt injection: What's the worst that can happen?. 14 April 2023. Accessed 23 September 2026. Secondary/technical expert.
4. Simon Willison — Why AI systems might never be secure. 23 September 2025 (link post quoting The Economist). Accessed 23 September 2026. Secondary/technical expert.