- Source
- arXiv
- Published
- Runtime
- 0:00
- Snippets
- 4
A conversation between
SpecFirst: Behavioral Specification Elicitation as a First-Class Step in Agent-Based Program Synthesis from Scratch
§02
Snippets
-
Behavioral specification elicitation should be a first-class phase preceding implementation, not conflated with documentation reading and code synthesis.
Separating specification from synthesis improved test pass rates by 6.9%–21.3% and binary exploration by 9.4%–18.5% across all tested models.
-
Single-pass synthesis causes agents to probe insufficiently, lose behavioral intent as context drifts, and propagate early misinterpretations into implementation.
This fragmentation explains why frontier models solve fewer than 1% of from-scratch programming tasks despite handling code-in-context well.
-
A dedicated spec agent probes the binary and combines observations with documentation into a structured specification; a code synthesis agent then drives implementation from this spec.
Resolving ambiguities and locking in behavior before coding prevents misinterpretations from cascading into the final implementation.
-
Inspired by classical requirements engineering, behavioral specification becomes a stable reference throughout synthesis that survives context drift.
This engineering discipline transferred to LLM workflows bridges the gap between understanding a problem and solving it in code.
§03
Synthesis
The Problem: Building Code from Nothing Is Brutally Hard
LLMs can modify or extend existing code reasonably well, but creating a program entirely from scratch—given only documentation and a working binary they can test against—remains nearly impossible. On ProgramBench, a benchmark of 200 such tasks, state-of-the-art models solve fewer than 1% of instances. The core issue: agents try to read docs, explore the binary's behavior, and write code all in one messy pass. They don't probe enough, they forget what they learned as context fills up, and early wrong guesses contaminate the entire solution.
The Insight: Separate Specification from Implementation
The authors' key observation comes from classical requirements engineering: build a detailed behavioral specification first, then code against it. This decomposition does two things. First, it resolves ambiguities in the documentation by systematic exploration before any implementation attempt begins. Second, it gives the code synthesis phase a stable, reference specification that prevents drift and misinterpretation.
How SpecFirst Works
The framework runs in two stages:
Spec agent: Probes the execute-only binary systematically, tests it with various inputs, and combines those observations with the documentation into a structured behavioral specification. This agent focuses purely on understanding what the program should do—not how to build it.
Code synthesis agent: Uses the specification as a guide to write the actual implementation, without needing to simultaneously figure out what the spec is.
The authors don't detail the exact specification format in the abstract, but the concept is straightforward: explicit requirements before code, not implicit assumptions discovered during coding.
Results: Consistent, Significant Wins
SpecFirst was tested on all 200 ProgramBench instances across four LLM models (two families, spanning a wide capability range). On every model, it outperformed the single-pass baseline:
- Test pass rates improved by 6.9% to 21.3%
- Binary exploration coverage improved by 9.4% to 18.5%
- All gains were statistically significant
The improvements held across weak and strong models alike, suggesting the benefit is fundamental, not just a scaling artifact. Analysis of the generated code showed that agents with an explicit prior specification constructed code earlier and more consistently, rather than meandering and backtracking.
Why It Matters
From-scratch program synthesis is a critical capability gap for agent-based AI development. Most real-world software engineering will eventually require building new systems, not just editing existing ones. This work shows that borrowing a 50-year-old idea from requirements engineering—separate specification from implementation—yields immediate, substantial improvements.
The result is also methodologically important: it demonstrates that better problem decomposition can matter more than raw model capability. A weaker model following a structured process beats a stronger model working haphazardly.
Mine your own.
Lode is a workbench, not a feed. Paste a YouTube URL. The model proposes a transcript, a set of quote-grounded snippets, a synthesis essay, and the fan-out. You decide what stays.