EduPanel: A Three-Agent LLM Judge for Teaching Videos -- Reliability, Complementarity, and Human Trust Calibration

The Problem: Teaching Videos Need Context-Aware Grading

Teaching quality isn't a universal property—it depends on who's learning. A video explaining calculus might be perfect for advanced high schoolers but useless for middle schoolers. Yet current automatic evaluation systems treat pedagogical quality as a fixed, objective score. They also ignore multimodal signals: tone of voice, visual clarity, pacing, and interaction all matter, but single-modality systems miss most of it. This creates a bottleneck: evaluating teaching videos at scale requires human experts, who are expensive and inconsistent.

How EduPanel Works: Divide and Conquer

The authors built a three-agent LLM judge that breaks evaluation into specialized roles. Each agent focuses on a different aspect of teaching quality—clarity, engagement, pacing, accessibility—and produces interpretable reasoning rather than a black-box score. The key innovation is learner conditioning: the system takes a learner persona (e.g., "beginner with no prior knowledge") as input and adjusts its assessment accordingly. Imagine asking different teachers to grade the same video but reminding them who the intended audience is—that's the core insight.

The system grounds its judgments in a rubric, so evaluations follow explicit criteria rather than arbitrary standards. This makes outputs traceable and contestable, not opaque.

Results: Useful But Not Autonomous

EduPanel achieved median-level human expert reliability across multiple evaluation studies. More importantly, its role isn't replacement—it's augmentation. When experts used EduPanel's feedback to guide their own scoring, their evaluation accuracy improved measurably: mean absolute error (MAE) dropped from 0.87 to 0.73 on a typical scoring scale.

Crucially, experts were able to distinguish reliable from unreliable outputs with AUC = 0.77, meaning the system provided useful confidence signals. Experts didn't blindly trust the agent's answers; they used them as informed second opinions. This is the opposite of automation bias.

The architecture ablations confirmed that specialization matters: using multiple agents beat single-agent baselines, and learner-conditioning improved assessment fit for specific student profiles.

Why This Matters

Teaching video evaluation is a real bottleneck in online education. Platforms like Coursera, Khan Academy, and corporate training all need scalable quality control. Current options are either hire humans (slow, inconsistent) or use generic automatic metrics (miss pedagogical nuance). EduPanel occupies a pragmatic middle ground: it reduces expert workload while preserving human oversight and decision authority.

The multimodal, learner-conditioned approach also sets a template for other educational AI tasks—tutoring feedback, curriculum design, assessment design all have similar properties (multimodal, learner-specific, rubric-driven). The emphasis on human-AI complementarity over full automation is a rare and welcome framing in a field often chasing agent autonomy.

The system's interpretability (through rubric grounding and agent decomposition) makes it auditable and trustworthy in a domain where pedagogical stakes are high.