Chat2Scenic: An Iterative RAG-Based Framework for Scenario Generation in Autonomous Driving

The Core Problem and Solution

Testing autonomous vehicles demands diverse scenarios that comply with safety regulations—but these regulations are written in human language, while testing systems need executable code. Existing methods hit a wall: you can either assemble scenarios from retrieved pieces (which compiles but doesn't scale), or have language models generate full scripts directly (which fails to compile most of the time). Chat2Scenic breaks this trade-off by making scenario generation iterative, grounding it in both regulatory knowledge and code syntax through retrieval-augmented generation (RAG).

The payoff is substantial: Chat2Scenic achieves a 76.42% compilation success rate—more than double the retrieval-assemble baseline (30.08%) and nearly five times better than full-script generation (16.26%). On semantic accuracy, it reaches 58.17% compared to 11.03% and 10.86% respectively.

How It Works

Chat2Scenic treats scenario generation as an interactive dialogue rather than a one-shot task. A user describes a regulation or safety requirement in natural language; the system generates code in SCENIC, a domain-specific language for autonomous driving scenarios. The key innovation is combining RAG with iterative refinement.

RAG works by retrieving relevant regulatory knowledge and existing code examples before prompting the language model. Instead of asking an LLM to invent syntax from scratch, the framework feeds it concrete examples of valid SCENIC scripts and the specific regulation the scenario must satisfy. This anchors generation in ground truth rather than learned patterns that might hallucinate incorrect syntax.

The iterative part matters equally. When the generated code fails to compile, Chat2Scenic doesn't discard it—the system feeds the compilation error back into the dialogue loop, allowing the LLM to fix syntax mistakes and refine the scenario. This mimics how a human developer works: write, test, debug, repeat. A chatbot interface lets users also provide feedback and refine scenarios interactively, steering the system toward their intent.

Why This Matters

Autonomous driving validation is genuinely bottlenecked by scenario generation. Regulations like NHTSA standards or UN Vehicle Regulations describe edge cases and safety requirements in prose, but simulators need executable code. Manual translation is labor-intensive and error-prone. Prior automated approaches sacrificed either correctness (full-script generation) or diversity (assembly methods that recombine fixed pieces).

Chat2Scenic's 76% compilation rate suggests the problem is now tractable at scale. The 58% framework accuracy—meaning scenarios that not only compile but semantically match the regulation—indicates the system grounds itself in actual regulatory intent, not just syntactic legality. This combination unlocks practical validation pipelines.

The authors also released a benchmark of 123 real regulatory scenarios from NHTSA and UN standards, letting the community measure progress beyond their specific approach. The open-source code release further lowers the barrier to adoption and iteration.

The implication is clear: iterative LLM-based generation, when properly anchored by retrieval and interactive feedback, outperforms both brittle assembly methods and naive end-to-end approaches. For safety-critical systems that must be thoroughly tested, moving from manual scenario authoring to semi-automated, human-in-the-loop generation represents a meaningful capability gain.