IndicTalk: A Large-Scale Persona-Based Multilingual Conversational Corpus for Indic Languages

The Problem: Indic Languages Get Left Behind

Most conversational AI systems work well in English and a handful of major languages. But speakers of Indian languages—Hindi, Tamil, Telugu, Bengali, and others—don't speak in pure language. They code-mix: they switch between their native tongue, English, and sometimes use Roman script instead of native script. No large, high-quality dialogue datasets existed that captured this reality. IndicTalk fills that gap with over 1.3 million conversations across 18 language variants of 9 Indic languages.

How They Built It

The authors didn't manually transcribe conversations. Instead, they built a fully automated pipeline:

1. News grounding: They anchored conversations in real-world news events. This gives dialogue natural context and topics—personas discussing actual events rather than generic small talk.

2. Persona-conditioned generation: Using multilingual LLMs, they generated multi-turn conversations where characters with defined personas respond to news. This ensures diversity in voice and perspective.

3. Automatic quality filtering: Not all LLM output is good. They applied automatic validation to catch fluency issues, coherence breaks, and unnatural code-mixing patterns before including conversations in the corpus.

The key insight here is automation. Manual annotation for 1.3 million conversations across 18 variants would be prohibitively expensive. By grounding in real events and using LLMs as generators with automated filters, they scaled to a size that would be impossible by hand.

Why This Matters

Representation of code-mixing: Indic language speakers naturally mix scripts and languages mid-conversation. Most datasets ignore this. IndicTalk preserves both Romanized (e.g., "namaste") and native-script (नमस्ते) variants, reflecting how real speakers actually communicate.

Persona-based dialogue: Rather than generic Q&A, conversations come from characters with distinct personas responding to grounded events. This creates richer, more realistic conversational turns—closer to how people actually talk when they have stakes and viewpoints.

Underrepresented languages: The Indic language NLP community is small. Releasing this dataset publicly (via Hugging Face) democratizes access to training data for building local conversational AI, potentially enabling chatbots and voice assistants that actually work for hundreds of millions of speakers.

What They Validated

The authors didn't just release raw data. They ran three layers of evaluation:

• Linguistic analysis: Measuring code-mixing patterns, script diversity, and dialogue coherence.
• Automatic metrics: Standard NLP evaluation to ensure fluency and relevance.
• Human evaluation: Native speakers rated fluency, naturalness, and code-mixing appropriateness.

The result: conversations that feel natural, not like they were machine-generated. For a dataset built entirely through automation, that's the non-obvious win—humans found the code-mixed dialogue believable.