# W03 Worked Case — From Synthetic Crawl Ledger to Candidate Recall

## Case status

This is an original instructional case built around the CC0 synthetic inputs in `course-labs/L03_retrieval_reranking_sandbox`. The acquisition ledger below is a constructed pre-index trace for teaching; it is not a record of requests to the public web. The ranked lists and metrics reproduce the deterministic L03 command with `k=3`, candidate depth 5, hybrid alpha 0.55, rerank weight 0.25, BM25 `k1=1.2`, and `b=0.75`.

The case estimates no behavior of a named crawler, model, search engine, answer product, or commercial platform.

## 1. Frozen question and stop conditions

**Research question.** In an immutable ten-document synthetic corpus, which documents enter the top-three candidate set for three declared topics under the L03 BM25 route, and where do the supplied opaque and hybrid lists change candidate recall or ordering?

**Primary event.** A qrel-positive document appears in the top three for its topic.

**Units.** Document–topic pairs for retrieval; topic for macro averaging.

**Judgment policy.** Seven positive graded qrels are supplied. Unlisted pairs receive gain zero inside this closed fixture only.

**Stop conditions.** Stop and mark the run invalid if the corpus, topics, qrels, tokenizer, run metadata, parameters, or input hashes differ without a documented branch. Do not revise qrels after inspecting rankings. Do not attach a named model to the authored opaque scores. Do not convert a retrieval result into a claim about citation, absorption, traffic, or production visibility.

## 2. Synthetic discovery and acquisition ledger

Imagine a lawful internal publishing exercise with twelve synthetic source locators, `S-001` through `S-012`. No network request occurs. The instructor-authored acquisition trace is:

| Stage | Count | Record |
|---|---:|---|
| Declared synthetic locator population | 12 | S-001…S-012 frozen before inspection |
| Discovered by the fixture manifest | 11 | S-012 intentionally absent as a negative discovery control |
| Authorized fetch attempts | 11 | all are local fixture reads, not public requests |
| Usable responses | 10 | S-011 returns a synthetic unsupported-media condition |
| Parsed document representations | 10 | one title plus body text per record |
| Admitted to index snapshot `IDX-L03-001` | 10 | D-001…D-010 |

The stage ratios answer different questions. Discovery coverage is 11/12 under this artificial population. Usable-response rate conditional on attempted local reads is 10/11. Index admission conditional on parsed representations is 10/10. None of those ratios is candidate Recall@3, which is query-conditioned and uses qrels rather than source locators as its denominator.

**Boundary check.** If a learner reports “83.3% visibility” because 10 of 12 locators enter the index, the label is rejected. The valid statement is “10 of 12 declared synthetic locators produced representations in the constructed acquisition trace.” It does not describe a generated-answer event.

## 3. Representation and index contract

Each admitted record has `doc_id`, `title`, `text`, `source_type`, and `license`. Title and text are concatenated for retrieval. The tokenizer is the shared standard-library regular expression `[a-z0-9]+(?:[-'][a-z0-9]+)?`, applied case-insensitively, with tokens lowercased. The method is deliberately English-oriented.

The ten tokenized title-plus-text lengths are:

| Document | Tokens | Document | Tokens |
|---|---:|---|---:|
| D-001 | 21 | D-006 | 18 |
| D-002 | 20 | D-007 | 21 |
| D-003 | 20 | D-008 | 20 |
| D-004 | 19 | D-009 | 24 |
| D-005 | 21 | D-010 | 21 |

The mean document length is 20.5 tokens. The inverted index stores the document identity and within-document count for each token. For topic T-03, the exact query tokens are:

`query`, `sampling`, `strata`, `negative`, `controls`, `held`, `out`

Their corpus document frequencies and positive-IDF values are:

| Token | Document frequency | IDF |
|---|---:|---:|
| query | 6 | 0.526093 |
| sampling | 1 | 1.992430 |
| strata | 2 | 1.481605 |
| negative | 1 | 1.992430 |
| controls | 1 | 1.992430 |
| held | 1 | 1.992430 |
| out | 1 | 1.992430 |

This table explains why a document matching several rare T-03 terms can outrank a document matching only `query`. The values describe rarity in this ten-document snapshot, not social importance or source authority.

## 4. One BM25 trace

Use the positive-IDF convention:

\[
IDF(t)=\log\left(1+\frac{N-n_t+0.5}{n_t+0.5}\right),
\]

\[
BM25(q,d)=\sum_{t\in q}IDF(t)
\frac{f(t,d)(k_1+1)}{f(t,d)+k_1(1-b+b|d|/avgdl)}.
\]

For D-010 on T-03, the only query token present is `query`, once. With `N=10`, `n_query=6`, `|D-010|=21`, `avgdl=20.5`, `k1=1.2`, and `b=0.75`:

\[
IDF(query)=\log(1+4.5/6.5)=0.526093.
\]

The length factor is `1 - 0.75 + 0.75 × 21/20.5 ≈ 1.01829`. The denominator is therefore approximately `1 + 1.2 × 1.01829 = 2.22195`. The saturated term-frequency factor is `2.2/2.22195 ≈ 0.99012`. Multiplying by the unrounded IDF produces the recorded D-010 score `0.520895689`.

D-001 also has length 21 and contains only `query` among the T-03 terms, so it ties at `0.520895689`. D-008 is slightly shorter at 20 tokens and also matches only `query`, producing `0.531395266`. That small length difference places D-008 above D-001 and D-010 under this convention. The tie policy then orders equal scores by document ID.

By contrast, D-005 matches `query`, `sampling` twice, `strata`, `held`, and `out`, yielding `8.654282550`. D-006 matches `query`, `strata`, `negative`, and `controls` twice, yielding `7.047062981`. The full BM25 T-03 order begins:

1. D-005 — 8.654282550
2. D-006 — 7.047062981
3. D-008 — 0.531395266
4. D-001 — 0.520895689
5. D-010 — 0.520895689

**Interpretation.** D-010 is relevant with qrel gain 1, but its lexical representation shares only the common query token and falls below the top-three cutoff. That is a candidate miss at cutoff three. It is not evidence that D-010 is absent from the corpus or universally irrelevant.

## 5. Candidate recall calculation

The positive relevant set for T-03 is `{D-005, D-006, D-010}`. BM25’s top three are `{D-005, D-006, D-008}`. Therefore:

\[
Recall@3(T\text{-}03,BM25)=\frac{2}{3}=0.666667.
\]

For T-01 and T-02, BM25 recovers both qrel-positive documents in the top three, so each topic has Recall@3 of 1.0. Macro Recall@3 gives each topic equal weight:

\[
(1+1+2/3)/3=0.888889.
\]

The macro value does not show which topic missed a relevant item. The topic-level table is the required diagnostic.

| Topic | Relevant documents | BM25 top 3 | Miss | Recall@3 | NDCG@3 |
|---|---|---|---|---:|---:|
| T-01 | D-001, D-002 | D-001, D-004, D-002 | none | 1.000000 | 0.955831 |
| T-02 | D-003, D-004 | D-009, D-003, D-004 | none | 1.000000 | 0.665315 |
| T-03 | D-005, D-006, D-010 | D-005, D-006, D-008 | D-010 | 0.666667 | 0.946768 |

T-02 demonstrates why recall and ordering must remain separate. Both relevant documents are present, but gain-zero D-009 occupies rank one. Candidate coverage is complete; ordering under the graded qrels is not ideal.

## 6. Supplied opaque and hybrid comparison

The supplied opaque run places D-005, D-010, and D-006 in the T-03 top three. Its Recall@3 is 1.0. Its metadata says the values are authored and `model_identity` is `none`. The result supports a list-analysis exercise, not a statement about an encoder.

L03 min–max normalizes BM25 and supplied opaque scores within each topic, then computes a weighted interpolation with `hybrid_alpha=0.55`. The hybrid T-03 top three are D-005, D-006, and D-010, also giving Recall@3 of 1.0. Across the three topics:

| Method | Macro Recall@3 | Macro NDCG@3 |
|---|---:|---:|
| BM25 | 0.888889 | 0.855971 |
| Supplied opaque | 1.000000 | 0.880035 |
| Hybrid | 1.000000 | 0.888438 |
| Transparent fixed-set reranker | 1.000000 | 0.888438 |

The reranker does not change the top-three order in this parameterization, so its metrics equal the hybrid metrics. That unchanged result is valid. The reranker operates only within each hybrid top-five candidate set; no new document enters that set.

## 7. Stage diagnosis table

| Observation | Stage diagnosis | What is held fixed | What cannot be claimed |
|---|---|---|---|
| D-010 is BM25 rank 5 for T-03 | lexical candidate miss at cutoff 3 | corpus, query, tokenizer, index, qrels, parameters | D-010 is absent from all indexes |
| D-010 is hybrid rank 3 | candidate membership changes under declared fusion | same frozen task and component score files | hybrid retrieval is generally superior |
| D-009 ranks first for T-02 | ordering problem under supplied qrels | relevant set and cutoff | the source is false in every possible claim |
| Reranked metrics equal hybrid | no top-three metric change under this reranker/weight | fixed hybrid top-five set | reranking never helps |
| Macro Recall@3 rises by 0.111111 | average change over three synthetic topics | topic set and aggregation | expected production lift of 11.1 percentage points |

## 8. Reproduction route

From the workspace root, run:

```bash
python3 course-labs/L03_retrieval_reranking_sandbox/scripts/run_sandbox.py \
  --corpus course-labs/L03_retrieval_reranking_sandbox/data/corpus.jsonl \
  --topics course-labs/L03_retrieval_reranking_sandbox/data/topics.jsonl \
  --qrels course-labs/L03_retrieval_reranking_sandbox/data/qrels.tsv \
  --dense-run course-labs/L03_retrieval_reranking_sandbox/data/frozen_dense_run.tsv \
  --dense-metadata course-labs/L03_retrieval_reranking_sandbox/data/frozen_dense_run_metadata.json \
  --output /tmp/geo-l03-w03-case \
  --k 3 --candidate-depth 5 --hybrid-alpha 0.55 --rerank-weight 0.25
```

Inspect `run_manifest.json` before the macro metrics. Then compare `runs/bm25.tsv`, `runs/frozen_dense.tsv`, `runs/hybrid.tsv`, `topic_error_analysis.csv`, and `stage_boundary.json`. A screenshot of one score is not an adequate submission because it omits inputs, parameters, and boundaries.

## 9. Bounded conclusion

In the immutable L03 ten-document CC0 synthetic corpus, with three authored topics, seven positive graded qrels, lowercase English-oriented tokenization, positive-IDF BM25 (`k1=1.2`, `b=0.75`), cutoff three, and unlisted pairs treated as gain zero, BM25 achieves macro Recall@3 of 0.888889. It misses relevant D-010 at cutoff three for T-03 because D-010 matches only the relatively common query token under this representation. The supplied opaque and hybrid lists include D-010 in T-03’s top three and achieve macro Recall@3 of 1.0, but the opaque values are authored teaching data and the model identity is `none`. T-02 retains a gain-zero record at rank one, demonstrating that complete Recall@3 does not imply ideal ordering.

This case does not evaluate public crawling, a production index, a named dense model, context packing, answer generation, citation correctness, source absorption, referral, user action, or current platform visibility.

## 10. Learner deliverable and self-check

Submit one page containing:

1. the frozen task identities and parameter line;
2. the T-03 BM25 term-frequency trace for D-010;
3. Recall@3 with numerator, denominator, and judgment policy;
4. one candidate-miss diagnosis and one ordering diagnosis;
5. one sentence on the authored opaque-run ceiling; and
6. one explicit downstream or closed-platform non-claim.

Self-check: Did you use the phrase “dense model” anywhere? If so, replace it with “supplied opaque run” unless you separately executed and documented an identified model. Did you describe the 0.111111 macro recall delta as production lift? If so, remove the extrapolation. Did you state cutoff, qrels, and corpus scope? If not, the metric statement is incomplete.