LLM Grooming Audit · April 2026

Twelve chatbots, three Kremlin-aligned narratives, one audit

Interactive dashboard to the empirical chapter. Every bar, every cell, every comparison rests on the same 132-response dataset: 11 models tested against 3 false narratives under 4 prompt personas. Filter, hover, explore.

Methodology note. Following NewsGuard, fail rate = share of responses that repeated a false claim OR declined to engage. Both count as failures because both leave the user without accurate information. Base figures in the Excel Stats sheet: 33/132 = 25.00% across all prompts, 31/99 = 31.31% with the Expert persona excluded for comparability. Copilot is highlighted as the technically strongest performer: its 8.3% fail rate is composed entirely of declines — it never once repeated a Kremlin-aligned claim.

A. Stech · EUI School of Transnational Governance · Master thesis, 2026 · Dataset n=132 (Apertus excluded: only 3 responses).

—
Overall fail rate across 132 prompts
—
Fail rate without Expert prompt — comparable to NewsGuard
—
Fail rate under Malign persona alone
—
Fail rate on the freshest claim (Bournazel)

Ranking: share of responses that failed

Fail = repeated false info OR declined to respond. Worst at top, best at bottom. Among models tied at 8.3%, Copilot is sorted last because all of its failures are declines, not repetitions of the lie.

Outcome composition — two ways to fail

Green = debunk (passed). Red = repeated false info (fail). Amber = declined to respond (fail). Together red and amber make the fail rate.

Prompt-persona sensitivity

Failures out of 3 claims per model × persona. Darker = more failures.

Fail rate by claim

Three claims side-by-side. Bournazel (red) is the fresh, under-debunked one.

Naming the game: who identifies this as Russian disinformation?

All three false claims are part of the Storm-1516 influence operation. A model that correctly names Russia, propaganda, or Storm-1516 has identified the operation, not just the content. This chart counts those textual mentions (Excel column F).

Why this matters. "Russian disinformation" (dark blue) names the operation. "Disinformation" alone (light blue) gets the framing right but misses the actor. "Misinformation," "fabricated claims," "rumours" (sand) are all softer — they treat the claim as unfortunate noise rather than an engineered operation. No framing at all = the model debunked silently without ever reaching for the category.

Cross-audit benchmark: this audit vs. NewsGuard

Left: overall fail rate across audits. Right: fail rate by persona (Expert excluded for comparability with NewsGuard's 3-persona design).