Generated from the same controlled README shipped in the offline package. Source SHA-256: 0da5072fe24091bbb34245dc7e61ce1cdcad950f7ea5d389887687f8dc32e750.
Linked course weeks: W03–W04
Expected effort for the full assignment: 7 hours
Core path: offline, deterministic, Python standard library only
Research question
Which stage changes when retrieval quality changes—and which downstream claims remain untested?
Learning objectives
You will learn to:
- distinguish first-stage candidate generation, score fusion, fixed-set reranking, context selection, and generation;
- reproduce BM25, frozen-run, hybrid, and reranked lists over one immutable corpus;
- compute topic-level and macro Recall@k and NDCG@k with explicit qrels;
- diagnose missed relevant documents and non-monotonic metric changes; and
- state why retrieval results do not establish citation, answer quality, absorption, referral, or production visibility.
data/corpus.jsonl: ten synthetic documents following ../shared/schemas/document.schema.json.data/topics.jsonl: stable topic_id and exact query text.data/qrels.tsv: topic_id, doc_id, and integer relevance >= 0; unlisted pairs are unjudged/treated as zero only inside this closed fixture.data/frozen_dense_run.tsv: frozen topic_id, doc_id, and score records.data/frozen_dense_run_metadata.json: declares that the supplied run is a synthetic, opaque teaching fixture. It is not evidence from a named embedding model.
All fixtures are CC0 synthetic. Do not replace them with scraped documents unless license, redistribution, privacy, and temporal provenance have been reviewed.
Run
python3 scripts/run_sandbox.py \
--corpus data/corpus.jsonl \
--topics data/topics.jsonl \
--qrels data/qrels.tsv \
--dense-run data/frozen_dense_run.tsv \
--dense-metadata data/frozen_dense_run_metadata.json \
--output /tmp/geo-l03 \
--k 3 --candidate-depth 5 --hybrid-alpha 0.55 --rerank-weight 0.25
BM25 uses k1=1.2, b=0.75, lowercase English-oriented tokenization, and Robertson-style IDF. Hybrid scores are per-topic min–max normalized before interpolation. The transparent reranker rewards query-token and adjacent-token coverage only inside the fixed hybrid top five. It is a teaching function, not a learned production reranker.
Output contract
Table from the controlled L03 lab handout| Artifact | Contract |
|---|
runs/*.tsv | ranked lists for bm25, frozen_dense, hybrid, and reranked |
metrics.json | per-topic and macro Recall@k/NDCG@k, with denominator policy |
topic_error_analysis.csv | relevant, retrieved-relevant, and missed IDs for every topic/method |
metric_deltas.csv | change relative to BM25; negative values are retained |
stage_boundary.json | stages manipulated, held fixed, and not evaluated |
run_manifest.json | hashes, parameters, and deterministic/offline flags |
A successful run may show that a method gets worse. Never alter qrels, topics, or the held-out split after observing a metric.
Full assignment procedure
- Verify corpus license, snapshot date, topic set, qrels, and environment before running.
- Reproduce all supplied baseline metrics within a declared numerical tolerance.
- Compare topic-level lists before reading the macro score.
- Build a hybrid run only after documenting score normalization.
- Rerank a fixed candidate set and prove no new document entered it.
- Run one ablation: remove dense scores, change fusion weight, or remove the reranker feature.
- Classify each error as candidate miss, ordering error, incomplete judgment, or representation mismatch.
- Submit run files, environment record, metric table, ablation chart, and bounded interpretation.
Low-compute and no-network route
Use the supplied frozen dense scores and analyze them without downloading a model. BM25 and all metrics run in milliseconds on one CPU. An optional real dense route belongs in a separate environment lock and must preserve the same corpus/topics/qrels; it is not required for core completion.
Negative results and interpretation
- A lower NDCG after reranking is a valid result and should trigger topic-level diagnosis.
- Higher Recall@k with lower NDCG@k can mean more relevant candidates but worse ordering.
- Missing qrels make unjudged documents ambiguous; this closed fixture simplifies them to zero and records that policy.
- The opaque frozen run cannot support a claim about any named encoder or current product.
- Retrieval metrics do not measure whether answer text cites, entails, absorbs, or sends traffic to a source.
Ethics and stop conditions
Do not redistribute copyrighted corpora, infer personal interests from query logs, or issue automated live-product requests. Stop if corpus rights, qrel provenance, or train/test separation are unknown. Never optimize against a held-out topic after inspecting its results.
Acceptance
python3 scripts/run_sandbox.py --corpus data/corpus.jsonl --topics data/topics.jsonl --qrels data/qrels.tsv --dense-run data/frozen_dense_run.tsv --dense-metadata data/frozen_dense_run_metadata.json --output /tmp/geo-l03 --k 3 --candidate-depth 5 --hybrid-alpha 0.55 --rerank-weight 0.25
python3 -c 'import json; m=json.load(open("/tmp/geo-l03/metrics.json")); assert set(m["methods"]) == {"bm25","frozen_dense","hybrid","reranked"}'
See RUBRIC.md.