The Embedder's Dilemma: When Better Isn't Worth It

Large language models (LLMs) have displaced specialized text-embedding models in many pipelines, but this paper reveals a hard truth: they're not actually better—and they cost vastly more. On 37 real-world embedding tasks, the best LLM and best embedding model tie at roughly 77.5 points, yet the LLM costs up to 1,431× more money and runs 2.5–736× slower on the same hardware.

The Results

The authors benchmarked ten LLMs (including Gemini, GPT, and open models) and 26 embedding models (118M to 14B parameters) across six task families: classification, semantic textual similarity (STS), clustering, pair classification, and retrieval. The headline finding is a dead heat in aggregate performance—but task-level breakdown tells a richer story.

LLMs excel at reasoning-heavy retrieval tasks, where they can leverage chain-of-thought reasoning to understand complex queries. Embedding models dominate on classification and match LLMs on clustering, STS, and pair classification. In other words, LLM superiority isn't universal; it's narrowly concentrated.

The economic case is devastating. A single benchmark pass costs USD 154 for Gemini 3.1 Pro versus USD 0.11 for a comparable embedding model—a 1,400× difference. Open-source LLMs on GPU are slower: processing tokens at 2.5–736× the latency of embedding inference. This matters in production: slower inference means higher serving costs and worse user-facing latency.

Why LLMs Cost So Much: Reasoning Tokens

A key insight emerges from ablation studies: LLMs burn 28–81% of their inference cost on reasoning tokens (the intermediate steps in chain-of-thought reasoning). But here's the surprise—reducing reasoning budgets preserves or improves retrieval quality for most models. This suggests LLMs are over-reasoning on many tasks, burning money for no gain.

Practical Guidance

The authors identify a Pareto frontier: the decision boundary where you stop getting better performance for your money. It consists of leading embedding models and only one LLM, Gemini 3.1 Pro, which justifies its cost for truly reasoning-intensive use cases.

Their recommendation is a division of labor: deploy embedding models for similarity, classification, and clustering tasks—the workhorses of most NLP pipelines. Reserve LLMs for retrieval scenarios that genuinely demand reasoning, like multi-hop question answering or complex query interpretation. This hybrid approach avoids both the sunk cost of replacing your embedding pipeline and the hubris of assuming one model solves everything.

The authors have released code, datasets, and full results publicly, making this more than a cautionary tale—it's an audit of a field-wide assumption that bigger and slower always means smarter.