Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challenges

The Core Challenge

Multimodal Large Language Models (MLLMs) struggle with humor because jokes don't work through literal description—they operate through incongruity, cultural references, and unstated assumptions. A meme's punchline might hinge on knowing a celebrity's persona or recognizing that an image contradicts its caption. This survey maps the landscape of how AI systems can understand and generate humor across memes, cartoons, and comics, revealing that current approaches have fundamental gaps.

How the Field Organizes Itself

The authors structure humor AI research into three tiers of capability. Recognition is the simplest: detecting whether an image is funny. Interpretation and reasoning requires understanding why—unpacking non-literal meaning, spotting the intended incongruity, and drawing on background knowledge. Generation is the frontier: creating novel humorous content, which remains nascent.

This hierarchy matters because it clarifies what each benchmark actually measures. Many existing datasets conflate these levels, making it hard to isolate what systems genuinely understand versus what they guess.

The field has evolved tactically over time. Early work built task-specific models that fused visual and textual features through hand-designed architectures. Recent progress leans on two trends: multimodal alignment (training models so that images and text occupy compatible semantic spaces) and evidence-grounded reasoning (forcing systems to cite which parts of an image justify their conclusions). This shift reflects a broader lesson—general-purpose models with explicit reasoning chains outperform narrow, opaque systems.

Why Current Evaluation Fails

The authors identify four critical barriers. First, shortcut-prone evaluation: benchmarks often reward systems that exploit statistical patterns without genuine understanding. A model might learn that certain words correlate with humor without grasping the underlying mechanism. Second, limited cultural and narrative scope: most datasets center English-language Western humor, so systems never learn to navigate the specific conventions of other cultures' comedy. This is not a minor limitation—humor is deeply cultural.

Third, weak evidence grounding: current systems rarely explain which visual elements triggered their interpretation. Without this transparency, it's unclear whether an MLLM understood the joke or simply pattern-matched. Fourth, safety and ownership issues: generation systems risk amplifying offensive stereotypes or infringing on the intellectual property of original creators, yet the field has no consensus mitigation strategies.

Why This Matters

Humor understanding is a genuine intelligence test. It requires systems to model intent, deploy abstract reasoning, and navigate ambiguity—all things that literal scene description sidesteps. Progress here has spillover benefits for sarcasm detection, dialogue understanding, and other domains where surface meaning diverges from actual meaning.

The survey's main practical takeaway is that bigger models and better alignment help, but they're not sufficient. The field needs harder evaluation metrics that penalize shortcuts, datasets spanning diverse cultural traditions and narrative structures, and generation frameworks that include interpretability and safety by design. Without addressing these gaps, AI humor systems will remain party tricks rather than genuine demonstrations of flexible reasoning.