# W04 Worked Case — One Pool, Two Stage-Isolated Comparisons

## Case status

This is an original, synthetic, hand-auditable teaching case. It contains no commercial query, platform trace, paper figure, copied prose, customer data, or model output. Candidate scores, relevance gains, passage costs, material-claim labels, and redundancy relations are authored fixtures. The case teaches two different comparisons: first, reranking a fixed candidate pool; second, packing a fixed ordered slate under one fixed budget. It does not evaluate acquisition, a named model, answer generation, answer use, citation, referral, or production behavior.

The reading identifiers PAPER-17, PAPER-22, and PAPER-18 are conceptual routes only. Their catalog status remains preprint / venue not confirmed. PLAT-01 is a named-platform documentation route whose authority does not extend to the hidden mechanics represented by this synthetic case.

## 1. Frozen question and stop conditions

**Authored question:** “Which records are needed to audit a source-selection experiment?”

**Candidate unit:** one versioned passage record, not a URL or whole document.  
**Pool identity:** `C-W04-01`, containing exactly five candidate IDs `C1` through `C5`.  
**Gain scale:** `0 = not useful for the authored question`, `1 = partly useful`, `2 = directly useful`, `3 = directly useful with the central audit contract`.  
**Unjudged policy:** none; every pool member has an authored gain.  
**Cutoff:** `k = 3`.  
**Tie policy:** descending score, then stable candidate ID.  
**Packing budget:** `B = 180 authored token units`. These costs are fixture values, not counts from a platform tokenizer.

Stop the reranking comparison if candidate IDs, passage hashes, or gains differ between runs. Stop the packing comparison if the order, passage costs, coverage labels, or budget changes in addition to the declared redundancy rule. Mark an uncertain redundancy label unresolved rather than selecting the interpretation that produces a favorable result.

## 2. Candidate ledger and fixed-pool proof

| ID | Passage summary | Version hash prefix | Gain | Baseline score | Rerank score |
|---|---|---|---:|---:|---:|
| C1 | Names stable IDs and content hashes | `41a0c1e2` | 1 | 0.91 | 0.76 |
| C2 | Defines manipulated, held-fixed, and not-evaluated fields | `73b2e40a` | 3 | 0.84 | 0.97 |
| C3 | Gives a broad history with no audit procedure | `f02d91c8` | 0 | 0.78 | 0.42 |
| C4 | Specifies cutoff, gains, ties, and judgment policy | `ab117e05` | 2 | 0.66 | 0.89 |
| C5 | Requires negative-result and limitation reporting | `2c85b0d4` | 1 | 0.59 | 0.64 |

Baseline order: `C1, C2, C3, C4, C5`.  
Reranked order: `C2, C4, C1, C5, C3`.

The fixed-pool proof is set equality plus count and version equality:

`set(baseline IDs) = set(reranked IDs) = {C1, C2, C3, C4, C5}`

Both lists contain five unique IDs. Each ID resolves to the same version-hash prefix and gain in both conditions. The treatment is therefore candidate preserving for this fixture. This proof would fail if `C6` appeared only after treatment, if C2’s excerpt changed, or if a later capture reused C4’s identifier.

The scores are ordering values inside this authored case. Their decimal form does not make them calibrated probabilities. No score is interpreted as factuality, citation probability, or commercial exposure.

## 3. Baseline metric calculation

At `k=3`, the baseline gains are `[1, 3, 0]`. The case uses exponential gain and logarithmic discount:

`DCG@3 = sum((2^gain_i - 1) / log2(i + 1))`.

Therefore:

`DCG@3_baseline = 1/1 + 7/log2(3) + 0/log2(4)`  
`DCG@3_baseline = 5.416508275`.

The three highest gains in the fixed pool are `[3, 2, 1]`, so:

`IDCG@3 = 7/1 + 3/log2(3) + 1/log2(4)`  
`IDCG@3 = 9.392789261`.

Thus:

`NDCG@3_baseline = 5.416508275 / 9.392789261 = 0.576666646`.

There are four candidates with positive gain: C1, C2, C4, and C5. At rank three, baseline retrieves two of those four, C1 and C2. Under the case’s binary threshold `gain > 0`, `Recall@3_baseline = 2/4 = 0.50`. At the full fixed-pool depth, `Recall@5_baseline = 4/4 = 1.00`.

## 4. Reranked metric calculation

At `k=3`, the reranked gains are `[3, 2, 1]`. They match the ideal order for the fixed pool, so:

`DCG@3_reranked = 9.392789261`  
`NDCG@3_reranked = 1.000000000`.

The top three now contain C2, C4, and C1, all positively judged. Therefore `Recall@3_reranked = 3/4 = 0.75`. At the full candidate depth, `Recall@5_reranked = 4/4 = 1.00`, unchanged. The shallow recall change comes from order across the cutoff, not recovery of a new item.

The interpretable result is narrow: under the authored gain labels, tie rule, and fixed five-passage pool, the treatment ordering increases NDCG@3 by `0.423333354` and Recall@3 by `0.25`. It cannot recover evidence absent from C-W04-01. The size of this constructed delta has no external population interpretation.

## 5. Topic-level and candidate-level diagnosis

C2 moves from rank two to rank one because the authored rerank score favors its explicit experimental-invariant contract. C4 moves from rank four to rank two because it supplies the metric specification. C3 moves from rank three to rank five because its broad history is topically related but does not satisfy the audit task. C1 remains above the cutoff at rank three. C5 remains below it despite positive gain.

This is an ordering diagnosis, not an answer-quality finding. If C5 contained a material warning missing from all other passages, its below-cutoff position might reveal that the relevance scale underweights limitations. That would be a label-design problem rather than proof that the reranker is generally poor. The case preserves that question for error analysis instead of hiding it in the perfect NDCG value.

## 6. Fixed-order context ledger

The context comparison uses the reranked order `C2, C4, C1, C5, C3` without alteration.

| ID | Authored cost | Material claims covered | Dependency origin | Redundancy relation |
|---|---:|---|---|---|
| C2 | 95 | A: invariants; B: stage ceiling | O-A | Primary member of cluster R1 |
| C4 | 75 | A: invariants | O-A mirror | Near-duplicate of C2 for this task; cluster R1 |
| C1 | 80 | C: identity and version trace | O-B | No declared near duplicate |
| C5 | 55 | D: negative-result and limitation record | O-C | No declared near duplicate |
| C3 | 40 | none of A–D | O-D | Topical background only |

The four material requirements A–D were frozen before selection. “O-A mirror” is treated as the same dependency cluster as O-A; two domains would not create two independent origins in this ledger. The redundancy relation is task-specific: C4 may contain useful wording, but after C2 is present it adds no new material requirement.

## 7. Baseline packing: greedy by rank

The baseline scans the fixed order and selects a passage when it fits in the remaining budget. It selects C2 for 95 units, leaving 85. It then selects C4 for 75, leaving 10. C1, C5, and C3 do not fit.

Baseline context: `C2, C4`.  
Total cost: `170/180`; unused budget: `10`.  
Material-claim coverage: `{A, B}` = `2/4 = 0.50`.  
Dependency-origin diversity: one cluster, O-A/R1.  
Declared redundant pair admitted: `(C2, C4)`.

The baseline has high ranked relevance in a mechanical sense but spends 75 units repeating requirement A. It lacks the identity/version trace C and the negative-result requirement D. These are fixture annotations, not claims about a generator’s eventual answer.

## 8. Single-factor packing ablation

The treatment keeps the order, 180-unit budget, costs, coverage labels, and origins fixed. It adds one rule: after selecting a passage, skip any later passage declared a near duplicate in the same dependency cluster when it adds no new material claim.

The treatment selects C2 for 95. It skips C4 with reason `R-REDUNDANT-NO-NEW-CLAIM`. It then selects C1 for 80, leaving 5. C5 and C3 do not fit.

Treatment context: `C2, C1`.  
Total cost: `175/180`; unused budget: `5`.  
Material-claim coverage: `{A, B, C}` = `3/4 = 0.75`.  
Dependency-origin diversity: two clusters, O-A/R1 and O-B.  
Declared redundant pairs admitted: none.

The single-factor delta is +0.25 in material-claim coverage, +1 represented dependency cluster, and −1 admitted redundant pair. Requirement D remains missing. The treatment is not declared globally superior: it loses the corroborating mirror C4, and the result depends on the correctness of the near-duplicate and coverage annotations.

## 9. Sensitivity and adverse interpretation

If review concludes that C4 contains a unique qualification for requirement A, the redundancy label must change. Under that revision, skipping C4 may remove important evidence, and the treatment’s apparent coverage advantage may not represent better allocation. If the budget increases while the rule remains fixed, C5 might fit; that would be a separate budget ablation. If C1 is shortened, representation changes and must receive a new passage identity and hash.

The correct adverse interpretation is retained: redundancy control can suppress useful corroboration or qualification. Origin diversity can also be misleading when dependency clusters are unknown. This case supplies complete cluster labels because it is synthetic; a production audit often cannot.

## 10. Stage diagnosis table

| Question about C5 | State in this case | Evidence |
|---|---|---|
| Entered the candidate pool? | Yes | Member of C-W04-01 |
| Position after reranking? | Rank 4 | Frozen reranked list |
| Packed under baseline? | No | 10 units remained; cost 55 |
| Packed under treatment? | No | 5 units remained; cost 55 |
| Used in an answer? | Not evaluated | No generation step exists |
| Displayed as a citation? | Not evaluated | No citation interface exists |
| Changed a production outcome? | Not evaluated | No users or production system exist |

This row-by-row structure blocks the statement “C5 was ignored.” We know only that it was not packed under two authored rules. “Ignored” would imply a generator or user state absent from the case.

## 11. Reproduction route

1. Copy the five candidate IDs, version prefixes, gains, and scores into a fresh table.
2. Verify unique IDs and equality of the baseline and reranked sets.
3. Sort by each score using stable ID for ties.
4. Calculate baseline and reranked DCG@3, IDCG@3, NDCG@3, Recall@3, and Recall@5.
5. Copy the fixed reranked order into the passage ledger without reordering.
6. Execute greedy rank packing under `B=180`.
7. Reset the ledger, keep `B=180`, and add only the declared redundancy rule.
8. Report selected IDs, cost, unused budget, A–D coverage, dependency clusters, redundant pairs, and exclusion reasons.
9. Preserve answer use, citation, and production outcomes as `not evaluated`.

The package validator checks the exact numerical tokens, fixed candidate equality language, budget, selected contexts, and boundary phrase. It does not independently validate whether the authored labels are pedagogically optimal.

## 12. Bounded conclusion and learner self-check

In the synthetic five-passage pool, candidate-preserving reranking changed the top-three order and improved the declared rank metrics. In the separate fixed-order, fixed-budget packing comparison, adding one redundancy rule replaced C4 with C1 and increased authored material-claim coverage from `2/4` to `3/4`. Both results are deterministic consequences of the supplied fixture and conventions.

This case does not evaluate answer generation, answer use, citation, referral, or production behavior. It does not identify the reranker, context budget, or packing rule of any closed platform.

Before submitting, the learner must answer yes to all five questions: Are candidate identities fixed in the reranking comparison? Are order and budget fixed in the packing comparison? Is every metric convention written? Are adverse interpretations retained? Are downstream claims explicitly outside scope?