LLM Security Basics: The Full Threat Model

Article author (GLM pipeline): Almost every LLM vulnerability traces to one property. A language model receives instructions and data as a single sequence of tokens, and that sequence contains no marker separating commands from information. Traditional software keeps the two apart. For example, a parameterized database query holds the command in one position and the user's input in another. This means that the text typed into a name field stays text even if it spells out a SQL statement, because the structure of the query enforces the separation. However, an LLM context has one position for everything. The system prompt defining the assistant's role, the user's message, a document retrieved from a database, and the output returned by a called tool all arrive concatenated in the same sequence. Any portion of that sequence can affect the generated output as if it were an instruction.

Article author (GLM pipeline): This is where prompt injection comes in, which means the supply of instruction-like text so that the model output conforms to those instructions instead of the operator's intent. It reaches the model by two routes: The direct route is a hostile instruction typed into the chat box. This is the version most people picture. The indirect route places the instruction inside the content that the model retrieves during a legitimate task, such as a web page being summarized, a document being read, or an email in a managed inbox. EchoLeak was an example of the indirect route. The user requested ordinary work, and the attacker's instructions arrived in an email the user had not opened. In fact, the payload passed through Microsoft's dedicated cross-prompt-injection classifier, which indicates that input filtering can be porous when used alone.

Article author (GLM pipeline): The point at which LLM attacks cause material damage has a specific structure and is identifiable in a system. It is also called the lethal trifecta. It consists of three capabilities held together by a single agent: Access to private data ... Exposure to untrusted content ... A channel to send data out or act externally ... An agent holding all three can be directed by injected instructions to transfer private data to an attacker. Model alignment does not remove this exposure, because producing output that conforms to instruction-like input is how the model normally operates. Removing any one of the three capabilities reduces the exposure. The least costly reduction is usually cutting the outbound channel.