ToolHazard: Scaling Adversarial Environments for Security Evaluation and Alignment of LLM-based Agents

The Core Finding

LLM-based agents that use external tools—like APIs, databases, and file systems—can be tricked through "indirect prompt injections": attackers embed malicious instructions in environmental data that the agent reads and unknowingly executes. Current security research relies on hand-crafted test scenarios, limiting how thoroughly we can evaluate these risks at scale. This paper shows that automated synthesis of adversarial environments can uncover widespread vulnerabilities and generate training data that hardens agents against attacks without breaking their normal functionality.

How ToolHazard Works

The framework has three main moving parts:

Environment Simulator generates stateful, executable environments from seed domains (e-commerce, banking, etc.). Rather than relying on stochastic LLM-based simulation of tools, this creates real environments with actual state—orders, accounts, files—that agents can interact with. This grounds attacks in plausible, complex scenarios.

Attacker Agent systematically discovers where and how to inject malicious prompts. It probes the environment to find viable injection points (fields an agent must read) and crafts domain-specific payloads that exploit those points. Crucially, the attacker adapts its strategy to the particular environment, rather than using predefined injection locations across all test cases.

User Simulator constructs realistic, multi-step tasks that agents attempt to complete. By embedding attacks within these long-horizon workflows, the framework tests whether agents fall for injections amid genuine work—a harder, more realistic problem than isolated injection detection.

Together, these components automate what previously required extensive manual engineering: creating diverse, complex attack scenarios across multiple domains.

Why This Matters

The experiments reveal two critical insights. First, agents show substantial vulnerabilities—standard LLM agents fail at non-trivial rates when facing these synthesized attacks. Second, injection timing and placement matter: an attack is only effective if the agent reads the malicious data at the right moment in a workflow. This temporal dimension is easy to miss in hand-crafted tests.

The practical payoff: ToolHazard-generated attack data can be used to fine-tune agents for better robustness. The authors show that this synthetic alignment data improves security on both their own ToolHazard-Bench benchmark and on AgentDojo, a separate evaluation suite. Importantly, defensive training doesn't degrade benign task performance—agents remain useful while becoming harder to trick.

The framework also scales: additional seed domains and compute can expand the benchmark, enabling researchers to push agent vulnerabilities further without proportional increases in manual effort. This is essential for keeping pace with LLM capabilities and ensuring that security research isn't a bottleneck for responsible agent deployment.