A Vocabulary for Multi-Agent Automated Research Systems

The Problem with Describing AI Research Systems

Current multi-agent automated research systems—tools that use AI agents to conduct research tasks autonomously—lack a shared language for describing how they work. When researchers build these systems, they make many structural decisions (Should agents communicate? Can they learn new skills mid-run? Should information persist across attempts?) but have no systematic way to compare their choices. The authors propose a vocabulary that standardizes how we talk about these systems, making design tradeoffs explicit and testable.

Eight Core Questions, One Framework

The vocabulary answers eight fundamental questions about any multi-agent research system. It specifies: (1) which agents exist and their roles; (2) what operations they can perform; (3) who can call which operations; (4) how agents send information to each other; (5) what data is visible within a single run versus across multiple runs; (6) the decision rule for choosing the next action; (7) how a run starts; and (8) how outputs get evaluated.

This framework is flexible enough to describe systems ranging from tightly orchestrated pipelines to loosely coupled multi-agent teams. A single run—called a trajectory—traces one path from input task to final output. Because agents and operations can be stochastic, the same task generates a distribution of different trajectories rather than one fixed behavior.

Two Kinds of Taste, Two Solutions

A key insight separates a fuzzy complaint ("these systems lack taste") into two measurable failures. Generative taste measures novelty: how many distinct trajectories does a system explore before consulting any scoring signal? High generative taste means the system tries diverse approaches. Evaluative taste measures fidelity: how well does the proxy score used during a run match the true quality of the output? If a system optimizes for a weak proxy, it may improve reported scores while actual research quality stalls.

This split is important because they require different fixes. Improving generative taste means better initialization, sampling, or action selection logic. Improving evaluative taste means better evaluation—the vocabulary treats the evaluator as a system component whose design matters as much as the agents themselves.

Why It Matters

The vocabulary transforms vague design questions into testable choices. Should agents communicate continuously or only at checkpoints? Does carrying information across runs help or cause overfitting to specific task patterns? When should a system gain or lose capabilities? These become empirical questions with clear answers once you can describe and measure them consistently.

By instantiating the vocabulary on recent autoresearch systems, the authors demonstrate it captures real diversity in system design. The framework doesn't prescribe the right architecture—it maps the space of possibilities, making it easier to spot which design choices actually drive performance gains versus which are cargo cult decisions copied from prior work.

This is foundational infrastructure for the field: before you optimize a system, you need a shared language to say what you're optimizing.