# W04 No-Video Transcript — Ranking and Context Selection

## Status and use

No W04 recording exists. This document is a readable and recordable no-video equivalent, not a transcription of a completed lecture. The chapter durations below are **instructional planning budgets**. They have not been verified through rehearsal, recording, caption production, or learner use. A future media artifact must replace these estimates with observed duration and must be checked for synchronized captions, speaker identification, mathematical voicing, audio description, and navigable chapters.

The route contains the core propositions needed for the W04 outcomes. A learner should be able to follow it without seeing the slide deck. Whenever a diagram would appear on screen, the narration describes its structure and states the corresponding data. All examples are synthetic. No commercial platform internals are represented.

## Planned chapter budget

| Chapter | Planned minutes | Core purpose |
|---:|---:|---|
| 1 | 4 | Separate candidate, rank, context, use, and citation |
| 2 | 4 | Establish fixed-candidate and fixed-order invariants |
| 3 | 4 | Prove candidate identity and define the scored unit |
| 4 | 4 | Interpret reranker scores and labels |
| 5 | 4 | Calculate reciprocal rank, DCG, and NDCG |
| 6 | 4 | Read topic-level gains, regressions, and unchanged results |
| 7 | 4 | Model context as finite evidence allocation |
| 8 | 4 | Conduct a single-factor packing ablation |
| 9 | 5 | Separate packing from answer use, citation, and outcomes |
| 10 | 5 | Apply reading ceilings and write the bounded conclusion |

**Total planned route: 42 minutes.**

## Chapter 1

Welcome to W04, ranking and context selection. The essential question is: why does an eligible source enter the context? We will answer that question in an inspectable teaching setting, and we will also learn when the same question cannot be answered from a closed surface.

Begin with five separate events. First, a candidate representation enters a pool. Second, a reranker places the members of that pool in an order. Third, a context allocator admits passages or excerpts under a finite budget. Fourth, a generator may use some of that text in an answer. Fifth, an interface may display a citation. Beyond these is a production outcome such as a visit, inspection, sign-up, or purchase.

The central discipline is simple to say and difficult to maintain: candidate does not mean highly ranked; highly ranked does not mean packed; packed does not mean used; used does not mean cited; cited does not mean that a user acted. A single word such as “selected” hides too many different records. Throughout the lesson, use the narrow verbs *retrieved*, *reranked*, *packed*, *used*, and *cited*.

Imagine that a public answer displays a citation to Source S. The citation is an observed presentation event under the recorded query, time, locale, interface, and account state. It may make several earlier paths plausible, but it does not expose the candidate list, score, reranker features, packed span, or causal contribution to answer text. Those fields remain unknown unless another trace supplies evidence.

Now imagine an open sandbox. We can freeze a pool of candidate passages, record every ID and hash, apply a transparent ordering rule, and preserve the exact result. We can then construct a context ledger with passage costs and a budget. The open sandbox supports mechanism learning because intermediate variables are visible. It does not disclose how an unrelated commercial surface operates.

This distinction controls how we use the readings. PAPER-17 is a route into conditional contribution and fair context attribution. PAPER-22 is a bounded route into attention and exposure. PAPER-18 extends the discussion of evidence composition and model response. The local catalog records all three as preprints whose venue status was not confirmed. PLAT-01 is official documentation for a named product. Each route has a claim ceiling. None reveals a universal context selector.

Pause and answer this question: when you see a citation, which one of the five events is directly visible? The answer is citation presentation. Candidate entry, order, context admission, and answer use require separate evidence.

## Chapter 2

We now turn from vocabulary to experimental design. A stage is not isolated because a software component has a convenient name. It is isolated because the records show that other relevant factors were held fixed.

For reranking, define a candidate pool for query q. Call it C sub q. It contains stable passage identifiers, not merely URLs. The baseline order and treatment order may differ, but each must contain exactly the same members and versions. The treatment is a pure reranking comparison only if it permutes that fixed pool.

Why does this matter? Suppose the treatment contains a relevant passage absent from baseline. The improvement might arise from candidate construction, a new fetch, different segmentation, or a larger candidate depth. Even if the treatment software is called a reranker, the observed delta is not attributable to ordering alone.

A fixed pool creates a useful invariant. At the full candidate depth, the count of relevant members cannot change through a permutation. Recall at that full depth is therefore constant. Recall at a shallower cutoff can change because a relevant member can cross the cutoff. NDCG can change because it discounts position. This combination—constant membership but changing shallow rank metrics—is exactly what a reranking study can expose.

Context packing uses a different invariant. Freeze the ordered slate, passage text and costs, material-claim annotations, origins, and usually the budget. Then change one allocation rule. You might add a redundancy penalty, an origin cap, or a claim-coverage reward. If the treatment also changes the reranked order or budget, the allocation difference has multiple possible causes.

A budget ablation is valid when budget is the one manipulated factor. In that case, keep the order, passages, costs, annotations, and allocation rule fixed. Report the results across preregistered budgets. Do not look at the curve and then choose the budget range that makes a preferred rule appear favorable.

Every comparison should have three explicit lists. The manipulated list names the one factor allowed to change. The held-fixed list names identities and conventions that must match. The not-evaluated list names downstream stages absent from the data. For W04, answer generation, answer use, citation, and production outcomes usually belong in that final list.

Consider a proposed experiment in which treatment uses a new reranker and increases candidate depth from five to ten. Is that a pure reranking comparison? No. Candidate depth changed. Split the work: first create one common ten-item pool; then apply baseline and treatment orderings to that same pool. Preserve the original five-item result as a different analysis rather than overwriting it.

## Chapter 3

The phrase “same candidates” needs a precise unit. A URL is usually insufficient. A reranker may score a document, passage, chunk, image–text pair, product row, or synthesized evidence card. Two rows sharing a URL can contain different excerpts. Two rows on different domains can contain the same syndicated text.

A candidate identity record should include a stable candidate ID, parent source, capture or version, passage offsets or segmentation rule, content hash, origin cluster, recovery route, original rank, and any truncation before scoring. The content hash detects silent drift. Offsets or segmentation explain which evidence entered the scorer. An origin cluster warns that apparent source diversity may be duplicated dependence.

Suppose baseline chunks documents into five-hundred-token passages and treatment uses two-hundred-and-fifty-token passages. The document IDs might be the same, but the scorer inputs are different. The comparison combines representation and ordering. Either regenerate one shared passage set or label the design as a multi-stage comparison.

The fixed-pool proof should include at least four checks. First, candidate ID sets match. Second, counts and uniqueness match. Third, each ID resolves to the same content hash and truncation. Fourth, evaluation gains remain the same. If judgments are revised after results are observed, create a new version and retain the first.

Set equality alone is not enough because sets discard duplicates. A list with C1 twice and no C5 might collapse deceptively under careless processing. Validate uniqueness and expected count before set comparison. Also preserve order separately: equality establishes common members, while the rank list establishes the treatment effect.

Deduplication deserves its own record. Exact hashes identify identical content under the declared normalization. Near duplicates require a method, threshold or human rubric, and uncertainty. Two mirrored pages do not become independent evidence merely because they have different domains. Yet aggressive clustering can also erase a distinct exception embedded in mostly repeated prose.

The L03 sandbox offers a clean local example. It creates a hybrid top-five candidate pool and applies a transparent token-coverage reranker only to those members. The frozen dense score stream has model identity none; the values are authored teaching data. A04 is a background paper route. G03 and G04 are candidate implementation routes for a separately pinned extension. We must not use their names to invent an identity for the fixture.

Here is the checkpoint. Two runs have identical document IDs and gains, but one truncates passages at a different length. Has candidate identity been preserved at the reranker input? No. The scored representations differ, so the comparison does not isolate ordering.

## Chapter 4

Once candidate identity is fixed, we can interpret the ordering function. A reranker can be pointwise, pairwise, or listwise. A pointwise method produces a value for each candidate with limited direct dependence on other candidates. A pairwise method expresses preference between candidates. A listwise method considers the slate, so removing a competitor can alter the positions of otherwise unchanged items.

Record the method or transparent rule, checkpoint, input template, truncation, prompt or features, output parser, candidate depth, tie policy, score direction, and determinism controls. These fields are not optional metadata. They determine what another analyst would need to reproduce the order.

Do not treat a reranker score as a calibrated probability unless calibration has been established for the declared setting. A value of zero point eight for Query A need not be comparable with zero point eight for Query B. Even within one query, score scale can change with model version or list composition. The score is never automatically a probability that a passage is true, authoritative, packed, cited, or commercially effective.

Labels define the target. Consider two annotation tasks. One asks whether a passage is topically related. Another asks whether it is fit to support a dated regulatory claim. A promotional summary might score highly on the first and poorly on the second. A long primary document might show the reverse pattern. NDCG cannot resolve the mismatch; it faithfully rewards whichever labels we supply.

Create a gain card. State the scale and meaning of every grade, the annotation unit, assessor instructions, adjudication procedure, unjudged policy, and limitations. The worked case uses gains from zero through three. Zero means not useful for the authored question. One means partly useful. Two means directly useful. Three means that the passage contains the central audit contract. These labels are constructed for teaching, not estimates from a population.

Ties can change the cutoff. Declare whether ties retain original order, sort by stable ID, or use a seeded random rule. A stable ID rule supports deterministic reproduction but has no semantic meaning. If displayed scores are rounded, evaluate with full precision unless the protocol explicitly defines ties after rounding.

Attention also needs an identity card. In an open model, an attention statistic depends on layers, heads, tokens, and aggregation. It can supply a mechanism clue under that system. It does not automatically equal passage utility, causal answer use, displayed citation, or another platform’s exposure. PAPER-22 is assigned so we can examine a bounded relationship, not so we can use attention as an all-purpose explanation.

## Chapter 5

We now calculate three rank-sensitive quantities. The purpose is not to memorize symbols. The purpose is to attach the arithmetic to a declared candidate pool, labels, cutoff, and judgment policy.

Reciprocal rank looks only at the position r of the first relevant item. The value is one divided by r. A first hit at rank one gives one. A first hit at rank two gives one half. A first hit at rank five gives one fifth. If no relevant item appears, many protocols assign zero; that convention should be stated. Reciprocal rank ignores later relevant items and graded relevance.

Discounted cumulative gain, or DCG, uses graded gains. In the convention for this package, transform gain g at position i into two to the power g minus one, then divide by log base two of i plus one. Sum through cutoff k. The exponential transform gives larger separation to higher grades, and the logarithm discounts later positions.

Take observed gains three, zero, and two. The first contribution is seven divided by one. The second is zero. The third is three divided by two. DCG at three is eight point five.

Normalized DCG compares that order with the ideal order under the same judgments. Sort gains into three, two, zero. The ideal DCG is seven plus three divided by log base two of three. That is approximately eight point eight nine two eight. Divide the observed eight point five by the ideal value to obtain approximately zero point nine five five eight.

What does this number mean? Under the declared exponential gain and logarithmic discount, the observed top-three order is close to the ideal order supplied by the judgments. It does not show that an answer is correct or that a source was used. Different gain definitions, ties, or unjudged policies can change it.

Now use the five-candidate worked case. Baseline order is C1, C2, C3, C4, C5. Reranked order is C2, C4, C1, C5, C3. Gains are one, three, zero, two, and one for C1 through C5 respectively. IDs, hashes, and gains are identical across conditions.

Baseline top-three gains are one, three, zero. Its DCG is approximately five point four one six five. The ideal top-three gains are three, two, one, with ideal DCG approximately nine point three nine two eight. Baseline NDCG at three is approximately zero point five seven six seven. Reranked top-three gains match the ideal, so reranked NDCG is one.

Four candidates have positive gain. Baseline Recall at three is two divided by four, or one half. Reranked Recall at three is three divided by four. Recall at the full five-candidate depth is four divided by four for both. That final equality is the membership invariant made visible.

## Chapter 6

Macro metrics are summaries, not diagnoses. A treatment can raise the mean while harming an important topic. Preserve every topic-level outcome: improvement, regression, and unchanged.

Start with candidate membership. If the relevant unit is absent, classify a candidate miss and return to first-stage retrieval or representation. A reranker cannot repair it. If the unit is present but falls below less relevant candidates, consider an ordering error. If the scored passage lacks the useful sentence found elsewhere in its parent document, consider representation loss. If the useful sentence lies after the scorer’s maximum input, record truncation loss.

Label mismatch occurs when judgments fail to represent the task. A general relevance scale might omit source fitness, freshness, or necessary qualifications. Duplicate crowding occurs when near-identical candidates occupy scarce high ranks. Unjudged ambiguity occurs when a moved candidate has no reliable label. Tie instability occurs when arbitrary ordering changes a cutoff result.

The error name is not chosen from intuition alone. It follows the earliest boundary with evidence. Candidate IDs and recovery traces establish membership. Passage hashes and offsets establish representation. Score inputs and rank lists establish ordering. An annotation guide establishes what the gain meant. When the evidence does not distinguish two explanations, record both and mark the cause unresolved.

Negative results are legitimate. In the default L03 fixture, selected macro NDCG remains unchanged between hybrid and reranked outputs at reproduced precision. The transparent reranker changes score values, but the reported top-k metric does not improve. That does not invalidate the component or the exercise. It teaches us to inspect whether order changed, whether top-k gains changed, and whether the candidate pool remained fixed.

Avoid narrating every adverse result as a flaw. A lower NDCG can indicate a mismatch between the reranker and labels, but labels themselves can be incomplete. A diversity rule might demote the strongest primary source. A source-quality requirement might reduce topical relevance. Report the axis that moved, the topic examples, and the limitation.

When reading PAPER-17, PAPER-18, and PAPER-22, retain the same stage discipline. A conditional contribution result is not a retrieval result. A fixed-evidence composition study does not establish organic acquisition. A relationship involving attention in one system does not establish a closed platform’s context rule. The scientific value comes from understanding the specified estimand, not expanding it.

Here is the chapter check. Suppose the macro score rises, but a passage containing a safety qualification moves below cutoff for one topic. Your report must preserve that topic. Investigate whether the gain scale represented the qualification. Do not allow the macro mean to erase it.

## Chapter 7

Reranked passages may exceed the available context. Packing is a separate allocation problem. It selects passages or excerpts, assigns an order, and may truncate them. A high rank can help a passage reach this decision, but it does not make admission inevitable.

Build a passage ledger. Each row needs a stable ID, source-origin cluster, token cost, relevance value, material-claim coverage, redundancy relations, qualification or contradiction flags, selected span, position, and exclusion reason. A cost must name its tokenizer or be labeled as an authored fixture value.

One abstract objective rewards passage relevance and unique claim coverage, penalizes redundant pairs, and restricts total cost to a budget B. An optional origin cap can limit domination by one dependency cluster. This equation is an experimental design, not a platform blueprint. Coefficients encode choices that can create tradeoffs.

Report at least four axes separately. First is relevance under the declared passage labels. Second is coverage of preregistered material claims. Third is diversity of dependency origins, not merely domain count. Fourth is redundancy, with a distinction between repeated dependence and useful independent corroboration.

Five domains can reproduce one syndicated statement. Counting five sources would overstate diversity. On the other hand, two independent sources supporting the same material fact can strengthen corroboration. A blanket penalty for repeated claims can discard useful evidence. Redundancy is task- and provenance-dependent.

Greedy rank packing scans the ordered slate and admits a passage when it fits. It is simple and auditable. With variable lengths, however, a long early passage can prevent several complementary passages from fitting. A knapsack-style rule may better optimize a declared objective, but it introduces coefficient, label, and computation choices. There is no neutral allocator without an objective.

Rejected passages must remain visible. Record whether each was excluded for budget, duplication, origin cap, missing rights, failed quality threshold, or another reason. Otherwise, a polished context hides the evidence allocation and makes later diagnosis impossible.

Our worked ledger uses the reranked order C2, C4, C1, C5, C3. Authored costs are ninety-five, seventy-five, eighty, fifty-five, and forty units. C2 covers material requirements A and B. C4 repeats A and belongs to the same mirrored dependency cluster. C1 covers C. C5 covers D. C3 covers none of A through D. The budget is one hundred eighty.

## Chapter 8

First apply greedy rank packing. C2 costs ninety-five and fits, leaving eighty-five. C4 costs seventy-five and fits, leaving ten. C1, C5, and C3 do not fit in the remaining space. The baseline context is C2 followed by C4.

The total cost is one hundred seventy of one hundred eighty. The context covers A and B, or two of four material requirements. It represents one dependency cluster because C4 mirrors the origin behind C2. The selected pair is declared redundant for this task. These are authored annotations, not inferred facts about an external system.

Now conduct one single-factor ablation. Keep the order, budget, costs, coverage labels, and origins fixed. Add one rule: if a later passage is a near duplicate in the same dependency cluster and adds no new material claim, skip it.

The treatment selects C2. It skips C4 with the reason redundant, same cluster, no new claim. C1 then fits for eighty units. The treatment context is C2 followed by C1, with cost one hundred seventy-five. It covers A, B, and C, or three of four requirements. It represents two dependency clusters and contains no declared redundant pair.

What changed? Material-claim coverage increased by one of four. Dependency-cluster count increased by one. One declared redundant pair was removed. What did not change? The fixed order, budget, passage identities, costs, and labels. Requirement D remains absent.

Do not simply call the treatment better. It loses C4, which might offer useful corroboration. If review discovers that C4 contains a unique qualification, its redundancy label is wrong for this task. Version the ledger and repeat the analysis. If the origin clusters are uncertain, the diversity result is uncertain. If a different tokenizer changes costs, create a representation version rather than silently substituting numbers.

Changing the budget at the same time would invalidate the redundancy attribution. A separate budget ablation can hold the rule fixed and compare preregistered values. The conclusion would remain conditional on the authored passages and segmentation. It would not reveal the context budget of a commercial product.

The correct packing conclusion is narrow. In this five-passage ledger, under a fixed order and one-hundred-eighty-unit budget, the declared redundancy rule replaced C4 with C1 and increased authored material-claim coverage from two of four to three of four. No generator has been run, so answer use and citation remain not evaluated.

## Chapter 9

We have arrived at a packed context. We have not arrived at an answer-use result. Context admission is an input event. To study influence, we need a separate generation experiment.

One route is leave-one-passage-out analysis. Hold the query, model, prompt, decoding configuration, and all other passages fixed. Generate with the full context and with one passage removed. Segment answer claims and inspect changes. With stochastic generation, repeat under a declared design rather than relying on one output.

A difference can support conditional influence in that particular context and configuration. It does not prove unique authorship, legal ownership, or global importance. Equivalent information elsewhere can mask influence. Removing a passage can change ordering or token positions as well as content. The utility function and the definition of a source player matter for contribution measures. PAPER-17 helps frame these assumptions.

Citation is another stage. An interface might produce citations jointly with text, resolve them from passage metadata later, or attach them through a presentation component. A source can be packed and not cited. A citation can be displayed next to a claim it does not support. A visible citation cannot reveal the source’s earlier rank.

Evaluate citation at the claim level. Segment the answer into atomic claims. Resolve the cited source and exact passage. Ask whether the passage entails the adjacent claim, whether material claims receive appropriate evidence, and whether the source is fit for the claim. Raw citation count is not a substitute for correctness, completeness, or source quality.

Production outcomes require a user–task design. A citation is not a visit, and a visit is not a purchase. If the outcome is referral, define exposure, eligible population, interface, time window, event logging, missing events, privacy controls, and concurrent changes. W04 does not conduct this study.

On a closed surface, record only what is visible or lawfully instrumented. A system card can include query, locale, account state, date, response, citations, and collection method. Candidate membership, score, reranker, context order, and answer-use cause remain unknown unless exposed by the system or identified by a controlled design.

PLAT-01 can support a statement about the named product documentation. It cannot fill hidden telemetry or establish a cross-engine context policy. A paper result, repository, blog, or diagram likewise does not convert a visible response into a mechanism observation.

Apply the stage table to C5 in the worked case. It entered the pool and ranked fourth. It was not packed by either authored rule because it did not fit. It was not “ignored,” because there is no generator or user in the case. Answer use, citation, and outcomes are not evaluated. Precise language preserves what the trace actually supports.

## Chapter 10

We end with evidence ceilings and the bounded W04 conclusion. A strong conclusion does not merely add a limitation sentence after an inflated claim. Its grammar is built around the estimand.

First, name the setting and unit. Was this the ten-document L03 fixture, the five-passage worked case, or a separately governed study? Name the query set, candidate or passage unit, and version.

Second, state the comparison and invariants. For reranking, prove the fixed candidate pool. For packing, state the fixed order, budget, costs, and annotations. Name the one manipulated factor.

Third, define measurement. Include gain scale, cutoff, aggregation, tie handling, and judgment policy. For packing, include budget and the separate coverage, diversity, and redundancy measures.

Fourth, report topic-level or passage-level results before the macro summary. Keep gains, regressions, and unchanged cases. Fifth, state at least one plausible failure or sensitivity. Sixth, list the downstream events not evaluated.

Here is a bounded L03 statement. In the ten-document, three-topic synthetic fixture, the transparent coverage reranker operated only over each fixed hybrid top-five pool. Candidate identity was preserved. Under the declared qrels and cutoff, selected macro NDCG at three remained unchanged at reproduced precision, although score values changed. The script did not evaluate context packing, answer generation, citation, or production behavior.

Here is the bounded packing statement. In the five-passage authored ledger, the redundancy-aware rule used the same order, one-hundred-eighty-unit budget, passage costs, and coverage labels as greedy rank packing. It replaced one same-cluster near duplicate with a complementary passage and changed authored material-claim coverage from two of four to three of four. It did not evaluate answer use, citation, or any named platform.

Now apply reading ceilings. PAPER-17, PAPER-22, and PAPER-18 offer bounded research settings and mechanisms to inspect. Their local status remains preprint / venue not confirmed. Do not merge their estimands into a universal rule. A04 provides dense-retrieval background. G03 and G04 are implementation routes when pinned and separately validated. PLAT-01 is authoritative for its named product documentation only.

The final exit question is: where did the evidence first disappear? Your answer must name the stage and the record. A candidate miss requires a membership trace. An ordering loss requires a fixed-pool rank comparison. A packing exclusion requires a ledger and budget. Answer use requires a controlled generation study. Citation requires a visible source object and claim-level support analysis. If the record is absent, write unknown.

Submit two sentences. The first must name the setting, fixed object, manipulated factor, metric or allocation measure, cutoff or budget, and result. The second must name the strongest downstream unknown. Add the unit and evidence source to every substantive noun. If a second analyst cannot reconstruct the claim without asking what “selected” means, revise it.