Generated from the same controlled README shipped in the offline package. Source SHA-256: bc0e764999a764d4522dde8f055ff372b2bbea4b303bc8fb6d22c491c31fbb73.
Linked course week: W06
Expected effort for the full assignment: 5 hours
Core path: offline, deterministic, Python standard library only
Research question
Does the query bank represent a declared universe rather than a desired result?
Learning objectives
You will learn to:
- define a population before writing prompts;
- stratify queries by intent, entity class, locale, provenance, and development/held-out role;
- add counterfactual and negative-control prompts without contaminating the target set;
- detect exact and near duplicates with a transparent lexical rule;
- detect desired-answer leakage and preserve exclusions; and
- state the representativeness limits of synthetic or convenience-sampled queries.
data/sampling_frame.json declares the population, unit, exclusions, strata, allocation, locale, time boundary, and sampling limitations.data/query_bank.csv follows ../shared/schemas/query.schema.json. query_id is stable and unique. Boolean values are lowercase strings because CSV has no native Boolean type.- Every row names
intent_stratum, entity_class, locale, provenance, is_synthetic, and split.
The checked-in bank is a small demonstration fixture, not the required 60-query assignment. It intentionally contains one near duplicate and one desired-answer prompt so the exclusion workflow is observable.
Run
python3 scripts/engineer_queries.py \
--frame data/sampling_frame.json \
--input data/query_bank.csv \
--output /tmp/geo-l02 \
--similarity-threshold 0.80
Near-duplicate similarity is token-set Jaccard similarity. It is intentionally inspectable, English-oriented, and not a semantic-equivalence oracle.
Output contract
Table from the controlled L02 lab handout| Artifact | Contract |
|---|
accepted_query_bank.csv | valid, non-leaking records retained in input order |
query_report.json | counts, coverage, exclusions, warnings, and method boundary |
duplicate_pairs.csv | exact/near-duplicate candidates with score and decision |
exclusion_log.csv | every excluded ID with a machine-readable reason |
coverage.csv | accepted counts by intent stratum, locale, and split |
run_manifest.json | hashed inputs and threshold |
Contract violations such as duplicate identifiers or unknown vocabularies fail the run. Quality findings such as a near duplicate, leakage, insufficient sample size, or an empty declared stratum are reported without turning a scientifically honest negative result into a software error.
Full assignment procedure
- Freeze the sampling frame and target allocation before authoring the bank.
- Create at least 60 English queries across at least four substantive intent strata.
- Reserve a held-out split before tuning any retrieval, response, or intervention method.
- Add labeled counterfactuals and negative controls.
- Record naturalistic, transformed, and synthetic provenance separately.
- Run the tool; adjudicate near duplicates rather than blindly accepting the rule.
- Preserve every exclusion and its reason. Never replace a difficult prompt merely because it hurts a metric.
- Analyze any additional language/locale as a separate sampling frame before pooling.
Low-compute and no-network route
The lexical audit processes thousands of short queries on one CPU and uses no model, embedding, external corpus, API, or personal search history. If semantic review is unavailable, perform a blinded human adjudication of the candidate pairs in duplicate_pairs.csv.
Negative results and interpretation
- Poor stratum coverage means the bank is not yet a valid measurement instrument.
- High duplicate rates may reveal a narrow authoring process, not merely a cleaning problem.
- Desired-answer leakage can invalidate an apparent brand or method advantage.
- Lexical similarity can miss paraphrases and can flag legitimate controlled variants. Report both error directions.
- A balanced synthetic bank is not automatically representative of real demand.
Ethics and stop conditions
Do not use private search logs, personal queries, protected attributes, or deanonymizable free text without approved governance and de-identification. Never ask annotators to infer sensitive intent. Brand comparisons must not embed fabricated accusations or endorsements. Stop if provenance cannot be established.
Acceptance
python3 scripts/engineer_queries.py --frame data/sampling_frame.json --input data/query_bank.csv --output /tmp/geo-l02 --similarity-threshold 0.80
python3 -c 'import json; r=json.load(open("/tmp/geo-l02/query_report.json")); assert r["status"] == "PASS" and r["excluded_count"] >= 2'
See RUBRIC.md.