Evidence Engineering for Retrieval-Augmented Generation: Recall Errors, Citation Attribution, and End-to-End Trustworthiness

1. How parametric memory and external documents divide their roles

The RAG study by Lewis and colleagues combined a generative model with a retrievable, nonparametric document store. RAG-Sequence and RAG-Token use different approaches to modeling how retrieved documents participate in generation. This mechanism allows the model to use external text, but the retrieval repository’s content, coverage, and version still determine the boundaries of available evidence; connecting a model to retrieval does not automatically give it complete knowledge of the world.[1] Retrieval-Augmented Generation for Knowledge-Intensive

2. DPR optimizes the retrieval stage

DPR uses a dual encoder to represent questions and passages as vectors and finds candidate texts by similarity. This allows retrieval to go beyond literal matching, but vector proximity may reflect only topical proximity. The system must still distinguish whether a passage mentions the same entity or actually answers the same question, and whether its content is semantically relevant or applicable in terms of time, region, and conditions.[2] Dense Passage Retrieval for Open-Domain Question Answer

3. How to faithfully replot published results

Table 1 in the original RAG paper reports exact-match scores of 41.5, 44.1, and 44.5 for DPR, RAG-Token, and RAG-Sequence, respectively, on the Natural Questions test. This is a comparison of systems listed in the same historical table, with differences in both training and components; the entire score difference cannot be attributed to the causal benefit of a single module. An earlier version replotted that table; Figure 2 in this version instead presents an analytical calculation of the supporting evidence chain and no longer plots those historical scores as a current experiment.[1] Retrieval-Augmented Generation for Knowledge-Intensive

4. Putting evidence in the context can still fail

Lost in the Middle observed that the position of relevant information affected the performance of the models studied on multi-document question answering and key–value retrieval tasks. This finding reminds us that fitting documents into an input window and reliably using information at every position are different capabilities. It did not establish that all subsequent models degrade by the same amount; new models, prompts, and tasks require new measurements.[3] Lost in the Middle: How Language Models Use Long Contex

5. Citation links and logical support require separate verification

An answer may place a correct link after an incorrect conclusion, or generalize a result with narrow conditions to all cases. This review recommends retaining a “claim—evidence-span location—source version” mapping for each key fact and checking whether the evidence actually entails the conclusion. Comparisons, dates, and numbers require checking denominators, units, and conditions of applicability; inferences should be explicitly labeled as analysis in the text.作者分析与复核建议 / Author analysis

6. An actionable decomposition of evaluation

First construct a set containing both answerable and unanswerable questions, retaining authoritative answers and evidence locations. Measure candidate-evidence recall, final-answer correctness, citation-target correctness, and the claim-support rate separately, and examine abstention independently. Do not combine these metrics into an undefined “accuracy.” If evaluators consider only linguistic fluency, the system may receive inflated scores when evidence is missing.作者分析与复核建议 / Author analysis

7. How to record updates and conflicts

When sources are revised or retracted, or different institutions’ accounts conflict, first preserve the access time and specific version, then explain which account is adopted and why. Retrieval results are only leads for discovery; checking the original text is the foundation of traceable facts. For information that remains unconfirmed, the correct output may be to explain a gap or defer a conclusion, rather than assemble an apparent consensus from several web pages that republish one another.作者分析与复核建议 / Author analysis

Giving “support” an operational definition

For a claim such as “a method outperforms a baseline on a dataset,” adequate evidence should specify at least the method version, data split, baseline, and scoring metric. If the original paper evaluated only an English-language test set, the reading guide must not omit the language condition; if values come from a development set, they must not be rewritten as performance on an independent test set. The more precisely the scope of source support is specified, the easier it is for subsequent readers to identify real disagreements rather than debate several differently meant versions of “better performance.”作者分析与复核建议 / Author analysis

8. Success rates should be formulated as conditional events, not multiplied arbitrarily

Let R denote that the retrieval results contain sufficient evidence, and G denote generating a correct, supported conclusion from that evidence set. The joint success rate is P(R)P(G|R), without assuming that R and G are independent. Multiplication is meaningful only when the second term is explicitly a conditional success rate. If the model can still answer correctly from parametric memory when retrieval misses evidence, final-answer correctness also includes P(correct|not R)P(not R) and cannot be directly equated with evidence-supported success. Evaluation should record both whether the answer is correct and whether sufficient, attributable evidence exists, avoiding the treatment of a fortuitously correct answer as evidence of a reliable retrieval chain.[1] Retrieval-Augmented Generation for Knowledge-Intensive [2] Dense Passage Retrieval for Open-Domain Question Answer

9. Citation correctness, relevance, and sufficiency are distinct judgments

A citation may genuinely exist and discuss the same topic without supporting the immediately adjacent claim. Sentence-level attribution requires, at a minimum, checking whether the source is accessible, whether the located passage contains supporting information, whether the text preserves the conditions of applicability, and whether multiple sources cover every part of a combined claim. Contradictory materials should have their conflict explicitly presented, not erased by selecting only an agreeable passage. The system should preserve document versions, retrieved passages, and the context actually visible during generation; retaining only the final link cannot reconstruct the evidence conditions at the time.[1] Retrieval-Augmented Generation for Knowledge-Intensive [3] Lost in the Middle: How Language Models Use Long Contex

10. Locating failing components through substitution experiments

Hold the question and generator fixed, and separately provide sufficient gold-standard evidence, current retrieval results, a distractor-only set, and an empty context. Failure even with gold-standard evidence indicates that the problem extends beyond recall; failure with current retrieval but success with gold-standard evidence suggests that retrieval or context selection deserves priority inspection. Key passages should also be moved within the same evidence set to avoid conflating increased length with changes in evidence position. This experiment is a design awaiting execution; measured differences between stages can be reported only after actual inputs, model versions, and outputs have been recorded.[2] Dense Passage Retrieval for Open-Domain Question Answer[3] Lost in the Middle: How Language Models Use Long Contex

11. Index updates are also part of reproducibility

Retrieval results change with crawling, segmentation, deduplication, embedding models, and nearest-neighbor indexes. Different answers to the same question from the same language model do not necessarily indicate a change in the generator. Complete verification requires an index snapshot or a manifest enabling reconstruction, chunking rules, document dates, and ranking configurations; it should also record how document deletions and corrections propagate. For time-sensitive questions, accessibility and the dates on which facts are valid must also be preserved separately. Deduplication reduces repeated exposure but cannot replace judgments about source quality and conflicts.[4] Deduplicating Training Data Makes Language Models Bette[1] Retrieval-Augmented Generation for Knowledge-Intensive

Core equations and derivations

Method and evidence workflow

Analytical plots and numerical examples

Recomputable worked example

If the supporting-evidence retrieval rate is 0.8 and the grounded-generation rate after successful retrieval is 0.9, the joint rate is 0.72. This is not the accuracy of all answers; correct guesses without evidence and errors despite evidence must be counted separately.

Method comparison

Verification and experimental protocol

1. Preserve the document collection and index snapshot; record the embedding model, segmentation, deduplication, nearest-neighbor search, and reranking configurations; specify a data cutoff.
2. Annotate the necessary evidence and indispensable conditions of applicability for each question; archive disagreements between annotators separately.
3. Use gold-standard, actual-retrieval, distractor, and empty contexts separately, keeping the generator and sampling budget consistent.
4. Place the same key evidence at different context positions while controlling total length to distinguish positional effects from changes in information quantity.
5. Evaluate answer correctness, support sufficiency, citation localization, and conflict disclosure separately, without compressing multiple dimensions into an opaque score.
6. Rerun the same request after correcting or deleting a source to confirm that invalidated evidence is not silently supplied by an old index or cache.

Scope and limitations

• Annotation of sufficient evidence depends on the task and domain.
• The probability curves are only analytical illustrations of conditional events.
• Retrieval systems do not automatically verify the truthfulness of original sources.
• No new retrieval benchmarks or model-comparison scores are provided.

Mapping primary sources to claims

1. Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksLewis et al. · 2020-05-22v4; section 2 and Table 1
2. Dense Passage Retrieval for Open-Domain Question AnsweringKarpukhin et al. · 2020-11EMNLP 2020 abstract and paper
3. Lost in the Middle: How Language Models Use Long ContextsLiu et al. · 2023-07-06v3 abstract; multi-document QA and key-value retrieval
4. Deduplicating Training Data Makes Language Models Better, , , , , , · 2021-07-14Abstract; 原文方法、理论和实验章节见作者原始论文链接