Distinguish candidate generation, reranking, and context packing.
Compute rank-sensitive metrics.
Diagnose exposure, redundancy, and context-budget effects.
PrerequisitesW03 run files and error taxonomy.Bridge note on reciprocal rank and NDCG.
Builds on
W03 run files and error taxonomy. · Bridge note on reciprocal rank and NDCG.
Investigates
Why does an eligible source enter the context?
Feeds
L03 retrieval sandbox II.
02
Before class
Arrive with a prepared artifact
Reading route
Core PAPER-17/PAPER-22; Extend PAPER-18; inspect A04 and candidate repositories G03/G04.
Viewing route
Open the complete text-first lecture packageSix-minute worked NDCG example with ties and unjudged documents. The current equivalent is notes, slide script, worked case, and no-video transcript; no recording is claimed.
Readiness check
Explain why reranking cannot recover a relevant document missing from the candidate set.
Bring
A metric card specifying relevance scale, cutoff, aggregation, and unjudged policy.
03
Explanation
Concepts, assumptions, and boundary
Ranking is a sequence of decisions
A retrieval stack often uses an inexpensive first stage to create candidates and a more expensive reranker to reorder a fixed set. Context packing then selects passages under a token or document budget. Improvement at one stage cannot repair candidates that were never generated. Conversely, a better candidate set can look unchanged if the reranker suppresses the new items. Experiments therefore preserve the candidate set when testing reranking and preserve the ranking when testing packing, making the changed factor interpretable.
Exposure competes with coverage
Rank-sensitive metrics value where relevant evidence appears, but context selection also faces redundancy, diversity, source concentration, passage length, and coverage trade-offs. An apparently high-quality slate may repeat the same claim from near-duplicate sources and crowd out complementary evidence. The course treats attention visualizations as mechanism clues only: they can help inspect an open model, but they are not a universal explanation of closed-platform selection or generated attribution.
04
Primary visual
Inspect the mechanism or evidence structure
MechanismRanked-slate inspection route
Figure design. Five equal-format rows name the candidate, reranking, span-selection, redundancy, and capacity records needed to diagnose a retrieval pipeline. Shapes are categorical and encode no score. Equal-sized marks encode stages, not measured scores. Hold one stage fixed when diagnosing another.Long description
Each row connects an input record to the observation required at that pipeline stage: candidate membership, ordering, inclusion, overlap, or capacity. Equal-sized marks prevent the schematic from implying measured values.
05
Interactive check
One action, one feedback state
Action
Reallocate a fixed context budget across candidate passages.
Feedback
The result reports relevance coverage, source diversity, and redundancy separately rather than collapsing them into one score.
Accessible alternative
A worksheet gives four predefined allocations and the same metric table.