Prompt injection reaches a model by two routes — a direct route (hostile instructions typed into the chat box) and an indirect route (instructions embedded in content the model retrieves during a legitimate task) — and the EchoLeak incident demonstrates the indirect route passing through a dedicated cross-prompt-injection classifier.

This is where prompt injection comes in, which means the supply of instruction-like text so that the model output conforms to those instructions instead of the operator's intent. It reaches the model by two routes: The direct route is a hostile instruction typed into the chat box. This is the version most people picture. The indirect route places the instruction inside the content that the model retrieves during a legitimate task, such as a web page being summarized, a document being read, or an email in a managed inbox. EchoLeak was an example of the indirect route. The user requested ordinary work, and the attacker's instructions arrived in an email the user had not opened. In fact, the payload passed through Microsoft's dedicated cross-prompt-injection classifier, which indicates that input filtering can be porous when used alone.

Read full article ↗excerpt · fair-use quotation