Auditing the Factual Supports of a Debate
Metamorfon’s purpose — confronting several models on demanding questions — produces exchanges in which arguments, figures, attributions, bibliographic references, empirical examples, and historical landmarks can accumulate quickly.
This density is part of the value of the exercise. It also creates a particular problem: the same answer may contain claims that are well supported, others that are only approximate, and still others backed by a plausible reference that does not quite say what the model claims it says.
This heterogeneity is well documented in the evaluation of long-form language-model outputs. FActScore, for example, starts from precisely the observation that a text can combine supported and unsupported information, making a single global judgment of factuality inadequate. Its method therefore decomposes a generation into atomic facts that can be assessed separately.
Source Verification addresses a related problem in the specific context of Metamorfon: it does not attempt to declare an entire answer “reliable” or “unreliable,” but to audit the factual supports that actually carry the argument.
A Deliberately Limited Scope
Source Verification is not a general-purpose fact-checking system, and it does not judge the “truth” of the ideas under debate.
An epistemological thesis — “does diagnosis describe a reality or produce it?” —, a normative proposition — “psychiatry should become reflexive” —, or a prospective interpretation — “uninhabitability will become systemic” — are not, in that form, facts that an external search can simply confirm or refute.
The mode therefore leaves them outside its scope.
It focuses on claims for which confrontation with external sources is meaningful: an attribution to an author or institution, the content of a study, a figure, a date, an order of magnitude, an empirical case, a mechanism presented as established, or an example used as evidence.
This distinction protects the mode from a promise it could not keep: settling the substance of an intellectual debate through documentary search.
But being verifiable is not enough for a claim to receive priority. A peripheral date and an empirical premise on which an entire conclusion depends do not carry the same argumentative weight.
The central question therefore becomes: if this claim were false or materially distorted, how much of the argument built upon it would remain?
Verification at the Claim Level
The mode extracts verifiable claims into a registry. It does not treat an answer as an indivisible block.
This choice reflects an important development in methods for factual evaluation: verification becomes more robust when it operates on claims precise enough for supporting or contradicting evidence to be searched for. FActScore follows this logic by decomposing long-form generations into atomic facts before testing them against a knowledge source.
Metamorfon does not implement FActScore and does not attempt to reproduce its score. It retains the same general principle of granularity: one identifiable claim, one identifiable verification task.
Each registry entry is linked to its turn and, when the attribution can be established with sufficient confidence, to its speaker.
If several voices within the same turn could have produced the wording — for example after cross-quotation or an ambiguous reprise — the report does not force an attribution. The turn and quotation are sufficient.
A missing attribution is preferable to an invented one.
Source Verification therefore does not replace Argumentative Evaluation. It assesses neither the strength of an objection, interpretive charity, nor the overall coherence of a line of reasoning. It audits the factual elements on which that reasoning relies.
Prioritising Argumentative Load
The model selected for the analysis conducts the entire chain: it builds the registry, selects the priority claims, and then verifies them. What changes between stages is not the model, but whether web search is available.
Once the registry has been built, and before external search begins, the analysis model rereads the debate and retains at most ten load-bearing propositions, ranked by decreasing argumentative weight.
The first is the claim whose failure would do the greatest damage to the conclusions reached in the debate.
Claims likely to receive high weight include:
- a central empirical premise;
- a figure that establishes the scale of the problem;
- a case presented as evidence;
- a causal mechanism on which a conclusion depends;
- a study or institution invoked to settle a disputed point;
- a decisive factual element introduced within a rebuttal.
By contrast, a contextual date, an illustrative reference, or a background citation carries little weight unless an important argument actually depends on it.
A source does not receive additional weight merely because it is named.
The mode does not artificially fill the selection to ten items. It also excludes unanchored normative judgments and assessments that concern argumentative coherence alone.
This hierarchy is a design rule specific to Metamorfon. It does not claim to identify every factual statement worth checking; it concentrates verification effort on the claims on which the argumentative architecture of the session depends most heavily.
External Search and Evidence Traceability
Once the propositions have been selected, web search is activated and the queue is audited in batches of five.
When that queue is exhausted, another run triggers a new selection: the next propositions are chosen, again by decreasing argumentative weight, without returning to those already audited.
Only when no load-bearing proposition remains can an additional safety-net pass check bibliographic references that were left outside the main queue.
If the session has continued in the meantime, or if the analysis model is changed, the selection is rebuilt.
The report states how many claims were examined during the pass, which remain without a conclusion or outside the scope of the mode, and how many are still waiting in the queue.
The same figure taken up by several voices or across several turns can be marked as a circulating claim. This is neither a new independent check nor necessarily a disagreement.
The search queries actually executed are displayed below the report, together with the number of searches used.
Most importantly, no strong verdict — “confirmed,” “incorrect,” or “misattributed” — is accepted without an external source retrieved during that verification pass. The model’s memory is not sufficient. Nor is the mere fact that a reference was cited in the debate.
This distinction between the presence of a citation and actual support for a claim matters. ALCE, a benchmark for generated answers with citations, evaluates precisely the relationship between the claims produced and the sources that are supposed to support them. Its results illustrate why the appearance of sourcing does not by itself guarantee complete evidential support.
A reference is therefore not evidence merely because it is present: it must actually support the statement to which it is attached.
A Conservative Threshold for “Confirmed”
Metamorfon deliberately sets a strict threshold for the verdict “confirmed.”
It is assigned only when a passage from the source, reproduced verbatim in the source’s original language, actually states the fact being checked — rather than a neighbouring proposition from which the verifier could reasonably infer it.
This requirement is not presented as a universal standard for fact-checking. It is a design choice: it prevents the verifying model from turning its own inference into external evidence.
The report itself remains in the language of the session; only the evidential quotation retains the source’s original language.
When a source broadly supports a claim but differs in scope, year, wording, magnitude, or attribution, the verdict should preserve that difference rather than erase it.
The closed vocabulary therefore includes, among others:
confirmed, partially correct, incorrect, misattributed, not verified, non-verifiable, out of scope.
Graduated Verdicts Rather Than True or False
Serious verification often produces something other than a binary verdict.
The FEVER dataset, one of the foundational works on automated claim verification against textual evidence, already distinguished between Supported, Refuted, and Not Enough Info. For supported and refuted claims, annotators were also required to identify the sentences serving as evidence.
Metamorfon uses a different and more detailed vocabulary, but the underlying principle is similar: a lack of sufficient evidence should not be silently transformed into a refutation.
A study may exist without supporting exactly the figure attributed to it. An author may defend a closely related position without having made the precise claim assigned to them. A result may be correct for one population and become misleading when generalised to another.
In such cases, “partially correct” is not a weak or indecisive verdict. It describes a specific relationship between the claim and the available evidence.
Likewise, “not verified” does not mean “false.” It may simply indicate that the searches performed so far have not provided enough evidence to decide.
Nuance is therefore not a residual flaw in verification: it is part of the result.
Cost, Coverage, and Dependence on the Analysis Model
This method requires more resources than a purely internal analysis.
The first pass must reread the debate, build the registry, rank the load-bearing claims, and perform real external searches. Later passes may be faster when they draw from an existing queue; a new selection — because the queue has been exhausted, the session has continued, or the model has changed — requires part of that work to be performed again.
Each pass audits at most five propositions, and each selection retains at most ten. A factually dense session may therefore require several runs to achieve broad coverage.
The selection itself is also a judgment made by the analysis model. It is performed before web search: the model decides which factual claims appear to carry the greatest argumentative load.
That hierarchy is contestable. Two analysis models might not rank exactly the same propositions in the same order.
The report should therefore not be read as an exhaustive inventory of every factually important point in the debate, but as a prioritised audit produced from a particular corpus by a particular analysis model.
The mode also depends on documentary accessibility. Some publications can be consulted directly; others may be available only through an abstract, an institutional page, or secondary reporting. Paywalls, vanished sources, or insufficient documentation can limit what the analysis is able to establish.
Only models whose web search produces usable sources can conduct this mode reliably; for that reason, some models may be marked as recommended within the application.
And not every debate warrants the cost. In a primarily conceptual or normative session, Source Verification may have very little relevant material to audit.
An Aid to Scrutiny, Not Certification
Source Verification does not turn Metamorfon into a factual authority.
Its verdicts remain the result of a limited pass, based on accessible sources interpreted by a model. They are neither an exhaustive certification of the session nor a guarantee that every source retrieved is itself error-free.
The same caution applies to assistance provided by language models during verification.
An experimental study by Si et al., Large Language Models Help Humans Verify Truthfulness – Except When They Are Convincingly Wrong, found that explanations generated by LLMs can help people verify claims more effectively, but can also induce overconfidence when the model’s explanation is itself wrong.
A model’s ability to produce a convincing justification should therefore not be confused with the reliability of that justification.
This is precisely why the report exposes its sources, search queries, evidential quotations, and limitations rather than asking the user to trust an opaque verdict.
Human Judgment Remains Decisive
The function of the mode is to make certain points in the debate more inspectable.
It can show that an attribution is too broad, that a figure is supported only within a narrower scope, that a frequently cited study does not quite support the proposition attributed to it, or that no usable evidence has yet been found.
It does not decide for the user what that correction means for the argument as a whole.
A partially weakened premise may be enough to collapse one conclusion and have only marginal consequences for another. Assessing that effect belongs to argumentative analysis and, ultimately, to the reader.
For sessions with high factual stakes, cross-checking sources is also preferable to excessive dependence on a single reference. A primary source can be complemented by a replication, review, institutional dataset, or methodological critique when the issue warrants it.
Plurality does not automatically guarantee truth, but it makes the difference more visible between a claim corroborated by several independent pieces of evidence and a proposition supported only through a single chain of repetition.
Source Verification therefore extends Metamorfon’s general logic onto another terrain: after confronting positions with one another, it confronts some of their factual supports with material external to the debate.
Its value is not to create an additional appearance of certainty, but to sharpen the boundary between what the dialogue asserts, what the available sources actually support, and what still remains to be established.
References
- Sewon Min et al., FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation, EMNLP 2023 — decomposition of long-form generations into atomic facts and separate evaluation of their factual support.
- Tianyu Gao et al., Enabling Large Language Models to Generate Text with Citations, EMNLP 2023 — citation quality, claim coverage, and the distinction between the presence of a reference and actual evidential support.
- James Thorne et al., FEVER: a Large-scale Dataset for Fact Extraction and VERification, NAACL 2018 — claim verification against textual evidence and the distinction between supported, refuted, and insufficiently evidenced claims.
- Chenglei Si et al., Large Language Models Help Humans Verify Truthfulness – Except When They Are Convincingly Wrong, NAACL 2024 — benefits of LLM explanations for human verification and the risk of overconfidence when those explanations are wrong.