When a Debate’s Richness Becomes a Point of Vigilance

Metamorfon’s purpose — setting several models against each other dialectically on demanding questions — produces exchanges of unusual conceptual density: in just a few turns, models invoke dozens of attributions, statistics, and historical milestones. That richness is the very point of the exercise; it is also its point of vigilance. The more precise references a model multiplies, the more it exposes a known limit of its probabilistic logic: the plausible but inaccurate citation. Source Verification mode addresses exactly this — not to fact-check ideas, but to audit, where the material warrants it, the attributions and figures advanced over the course of a debate.

What this mode checks — and what it doesn’t

One misunderstanding needs clearing up first: Source Verification is not general-purpose fact-checking, and it does not rule on the “truth” of the ideas being debated. An epistemological thesis (“is homogenization best understood at the population level or the level of individual outputs?”), a normative projection (“AI development should be deliberately engineered for cognitive biodiversity”), a forward-looking interpretation (“convergence will become irreversible once models become the primary medium of knowledge transmission”) — these are not facts an external source can verify; they are conceptual positions. The mode explicitly excludes them from its scope.

What it audits are verifiable claims: those resting on a named author, a dated work, an explicit figure (a percentage, a ratio, a value). In short, it checks who said what, and on what basis — not whether the idea is right. The distinction looks modest, but it’s decisive: it confines the mode to what it can actually control, and protects it from the impossible promise of arbitrating substance.

How it works

As the debate unfolds, the mode extracts verifiable claims into a registry, assigning each a stable identifier. For every claim it retains, it formulates a query and runs a real web search to check the cited reference against accessible sources. The result is a nuanced verdict — confirmed, partially correct, misattributed, incorrect, not checked, out of scope — accompanied by an explanatory note and the sources consulted.

Two properties matter here, because they determine how much weight the result can bear:

• The audit depends on a search actually having been run. If the model being queried did not execute a search, no verdict can be treated as reliable: the mode flags this explicitly and suggests rerunning with a different model. A verification built solely on a model’s memory would simply reproduce the risk it’s meant to correct.
• Coverage is incremental and bounded. The search budget is limited: each pass prioritizes a subset of five claims and honestly flags the ones it couldn’t check. An audit never claims exhaustiveness; it proceeds in successive layers.

This prioritization isn’t arbitrary: the mode targets first the claims that are both most verifiable and highest-risk — attributions to a named author or work, and precise figures advanced without a source (a percentage, a ratio). Purely interpretive statements are set aside, and lower-stakes claims are deferred to a later pass. With a limited budget, the audit concentrates its effort where a wrong citation would carry the most weight.

This rigor has a cost: because it runs genuine web searches and re-reads the entire debate to extract and then verify claims, often across several passes, Source Verification is noticeably more expensive than other analysis modes — one more reason to reserve it for sessions where the factual material justifies it.

A worked example: a debate on cognitive monoculture

To make this concrete, take a session built around a fittingly self-referential question: does reasoning through a handful of large models produce a monoculture of thought? Two models, deepseek-v4-pro and mistral-large-latest, argue the mechanism, scope, and reversibility of this convergence — and lean heavily on machine-learning literature to do it: RLHF papers, model-collapse studies, domain-reweighting techniques. The registry extracts 44 verifiable claims; the audit works through them in successive passes of five, producing the full spread of verdicts — which is exactly what makes it instructive.

A confirmed reference. The attribution of “model collapse” and the loss of tail distributions to Shumailov et al. (2023) is confirmed: the paper is real, correctly dated, and says what it’s cited as saying. The same holds for Bommasani et al. (2021) on the homogenization risks of foundation models — a citation that checks out cleanly on author, year, and substance.

A right paper, a finding it never made. DeepSeek cites Bai et al. (2022), the Constitutional AI paper, for the claim that models with different base architectures and identical reward signals converge on “indistinguishable refusal boundaries.” The paper is correctly identified — author, year, topic all check out — but it reports a training methodology applied to Anthropic’s own models; it runs no cross-architecture comparison of refusal behavior at all. Verdict: partially correct — right source, finding invented.

A finding inverted, on an untraceable figure. Mistral cites Kandpal et al. (2023) for the claim that “even small adjustments (e.g., 10% upweighting of niche data) can restore tail distributions without harming performance.” The paper exists and is exactly where cited — but its actual conclusion runs the other way: it shows that large language models struggle to learn long-tail knowledge regardless of training distribution. The “10% upweighting” figure has no traceable source anywhere. Verdict: misattributed. The claim doesn’t just overstate the paper; it contradicts it, around a number the model appears to have constructed for its own reasoning. This is where the mode renders its clearest service: distinguishing the attested figure from the merely plausible one.

A reference that doesn’t turn up as cited. DeepSeek cites “Liu et al. (2023)” for a “debate-style RLHF” technique said to broaden output distributions. The search surfaces several different Liu et al. (2023) papers — Chain of Hindsight, rejection sampling — none matching this description. Verdict: incorrect. Not a wrong date or a stretched scope, but a citation that appears to correspond to no real paper under that description.

A cautious abstention. A claim attributed to “Li et al. (2024), On the Diversity of Large Language Models” — said to find that output variance increases with model size — cannot be located: the search returns related work on LLM output diversity but nothing matching this exact title and finding. Rather than cry fabrication, the mode records not checked, with an explicit caveat: this is an unconfirmed absence, not a proven one — the paper may exist but wasn’t surfaced, and the search budget was exhausted before a firmer query could run. That restraint says a lot about the tool’s discipline: it flags what it couldn’t establish without condemning beyond what it actually checked.

Nuance as the result, not the shortfall — and a reflexive twist

One might expect a verification tool to render binary yes/no verdicts. Experience shows otherwise. Across the session, “partially correct” and “misattributed” dominate by a wide margin, with confirmations a clear minority and a handful of “incorrect” and “out of scope” at the edges. Far from a failure, that distribution says something accurate: real sources keep turning up, but the specific findings attached to them keep drifting from what those sources actually say. A mode that returned clean “confirmed” verdicts everywhere would be suspect; here, the nuance is the mark of a serious audit, not of indecision.

What makes this particular session worth dwelling on is the topic it was arguing, and where the drift concentrates. The citations that confirm cleanly tend to be the broad, structural ones — foundation models incentivize homogenization, model collapse is real. The citations that drift — stretched, inverted, or unfindable — are disproportionately the ones doing the actual argumentative work, the specific findings each model leans on to press its case. And DeepSeek and Mistral converge on much of the same supporting literature — Bai et al., Ouyang et al., invoked by both in service of a shared thesis about AI-induced convergence in human reasoning. That agreement between the two models is, in miniature, the very phenomenon they’re debating; and the audit shows part of it resting on citations that don’t withstand the scrutiny their own argument calls for. The asymmetry the mode exposes — between how readily the models agree and how reliably their shared sources hold up — is the audit doing its job.

Acknowledged limits

Source Verification is an aid to scrutiny, not an authority. Its limits need to stay in view:

• It depends on the quality and accessibility of web sources; paywalled articles are often only reachable through secondary sources.
• By construction, it covers only part of a dense debate’s claims (bounded budget, prioritization).
• It’s only useful for sessions with high factual density: a purely conceptual or normative exchange offers little to audit — hence the case for using it selectively, in proportion to the verifiable material at hand.
• Its verdicts are indicative: they reflect a limited pass, not an exhaustive certification.

The user remains the final judge

Friction between models often reveals asymmetries in factual reliability; Source Verification makes them visible and documented. But it never substitutes for human judgment. It surfaces the points worth corroborating — an attribution stretched too far, a figure with no source, a distorted title — and leaves the conclusion to the user. For sessions with high factual stakes, cross-checking several angles of verification remains sounder than relying on a single source: the same logic of epistemic plurality that drives Metamorfon’s confrontation between models applies to verification itself.

That’s probably the clearest way to understand this mode: not as a final verdict on truth, but as an instrument that makes the work of verification faster, more targeted, and better equipped — without ever claiming to replace it.

By confronting models not only against each other but against external sources, Source Verification gives Metamorfon a vantage point on how AI behaves under debate: a place where their reasoning patterns, their blind spots, and now their factual reliability become visible and documentable.