AutoIndex: Learning Representation Programs for Retrieval

The Core Claim

Rather than treating document representation as a fixed preprocessing step, AutoIndex learns executable transformation programs that reshape documents before indexing—and does so automatically. The result: across eight diverse retrieval tasks, the learned programs boost recall by 8.4% on average (up to 30.5% on individual tasks) while keeping the underlying retriever (BM25) unchanged.

This reframes the optimization problem. Instead of tuning a retriever or reranker, AutoIndex optimizes what documents look like when they enter the system.

How It Works

AutoIndex uses an iterative, validation-guided search process. At each step:

1. Diagnosis. An agent examines where the current program fails—which documents aren't being retrieved when they should be.

2. Synthesis. Based on failures, the agent generates candidate program updates. These updates can slice documents into smaller chunks, enrich them with metadata, normalize text, reweight fields, or reorganize their structure.

3. Validation. Only updates that improve retrieval quality on a validation set are retained. The program evolves incrementally.

The programs are executable code, not just parameters. This means transformations can be complex and data-dependent—not confined to tuning a handful of hyperparameters.

Why It Matters

The authors' key insight is that document representation is an underexplored lever in retrieval. Most work focuses on better models (neural rankers), better retrievers (dense embeddings), or better reranking. But how you chop up and present documents shapes what any retriever can do.

By automating this search, AutoIndex sidesteps manual trial-and-error. The learned programs adapt to task-specific structure: one task might benefit from splitting long documents into passages, another from extracting and promoting key fields. The framework discovers these patterns automatically.

The evaluation on CRUMB—a benchmark of heterogeneous tasks—strengthens the claim. Because tasks vary widely, a one-size-fits-all representation is suboptimal. AutoIndex learns task-specific programs, explaining why gains are modest on average (+8%) but dramatic on some tasks (up to +43.6% nDCG@10).

Crucially, the baseline retriever stays fixed. This isolates the value of representation learning from confounding improvements in ranking algorithms. It also makes the approach practical: teams can apply AutoIndex without retraining their production retrievers.

The reproducible code and concrete numbers (8 tasks, specific recall/nDCG gains) support a credible empirical contribution. The work opens a path toward treating document representation not as a solved preprocessing problem, but as a first-class optimization target—one that, for many teams, may yield larger returns than the marginal improvements available from tweaking rankers alone.