Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages

## The Gap: Python-Only Benchmarks Hide Real Coding Ability

LiveCodeBench has become the standard way to test whether large language models (LLMs) can write code. It uses real competitive programming problems, releases fresh ones regularly, and dates problems carefully to avoid testing on code the model has already seen during training. But it only tests Python. This leaves a blind spot: can LLMs actually code in C++, Java, Go, Rust, or ten other languages? Or does their strong Python performance hide the fact they've overfit to that one language?

Multi-LCB answers this by expanding LiveCodeBench from Python alone to twelve languages—Python, C++, Java, C#, JavaScript, TypeScript, Go, Rust, Kotlin, PHP, Swift, and Bash. The authors didn't just translate problems word-by-word; they carefully converted each Python problem into semantically equivalent tasks in other languages while keeping the same test cases, performance constraints, and contamination safeguards that make LCB trustworthy.

## How It Works

The method is straightforward in concept: take each Python problem from LCB, rewrite it in each target language to preserve the logic and difficulty, and run the same evaluation. Because Multi-LCB maintains LCB's format and structure, it automatically benefits from future LCB updates without needing manual re-translation.

The crucial step is ensuring equivalence. A problem isn't just a prompt—it includes specific input-output examples and performance requirements. The authors ensured these constraints held across languages. For instance, a sorting problem that must run in under one second in Python must meet the same timing on C++ or Go. This prevents accidentally making some languages artificially easier or harder.

## What They Found

The team tested 24 LLMs across Multi-LCB. The results break down into three key findings:

**Python overfitting is real.** Models perform noticeably better on Python than on other languages, even after controlling for language difficulty. This suggests LLMs have absorbed more Python training data or learned Python-specific patterns that don't transfer.

**Language-specific contamination exists.** Some models perform suspiciously well on particular languages, hinting that those specific solutions appear in training data—a risk the original LCB was designed to detect for Python but couldn't address for other languages.

**Performance gaps are massive.** A model strong on Python might be mediocre on Go or weak on Rust. No single model excels uniformly. This exposes how current LLMs lack true multilingual coding robustness—they're specialists, not generalists.

## Why It Matters

Real software engineering happens in many languages. Benchmarks that only measure Python hide critical limitations. Multi-LCB transforms a single-language evaluation tool into a multilingual one without losing LCB's rigorous contamination controls. It directly addresses LCB's primary weakness and provides the first systematic way to measure whether LLMs can actually generalize coding skills across language boundaries. For anyone shipping LLM-based coding assistants, Multi-LCB is now the benchmark that reveals what these models can and cannot do in practice.