# W06 Worked Case — Auditing a Synthetic Query Bank Before Collection

## Case status

This case reproduces the checked-in L02 demonstration fixture. Its 20 query records are instructor-authored, synthetic, English-oriented teaching inputs. They are not private search logs, not a probability sample, and not evidence of real query demand. The audit executes offline with the Python standard library. No answer system is called, no brand visibility is measured, no qrel is inferred from output, and no platform mechanism is evaluated.

The case is a pre-collection instrument audit. It demonstrates how to preserve a declared frame, apply outcome-blind rules, surface one lexical near-duplicate candidate, exclude one explicit desired-answer prompt, inspect strata and split coverage, and write a bounded conclusion. The fixture's `PASS` status means the input contract has no fatal schema error. It does not mean that the small bank meets the full 60-query assignment or has external validity.

## 1. Frozen question and stop conditions

**Question.** Under the declared FRAME-DEMO-001 contract and a token-set Jaccard threshold of 0.80, which input records remain eligible for a small query-set engineering demonstration, which are excluded, and what population can the resulting bank represent?

Freeze these decisions before inspecting the generated report:

- The sampling unit is one independently interpretable query record with a stable `Q-NNN` ID.
- The declared intent labels are `learn`, `compare`, `select`, `verify`, `counterfactual`, and `negative_control`.
- Splits are `development` or `held_out`.
- Boolean values are lowercase text strings because the source is CSV.
- A predeclared exclusion reason removes the record while preserving its ID and reason.
- Explicit desired-answer patterns are excluded.
- Exact normalization or token-set Jaccard at or above 0.80 produces a duplicate decision under this fixture's rule.
- The later record is excluded when the fixture rule finds a duplicate pair. A real study would preserve a separate human adjudication field.

Stop if an ID is duplicated, a required field is absent, an intent or split label is unknown, provenance cannot be established, or sensitive/personal text appears without an approved governance route. Do not look at answer outcomes because none are part of eligibility.

## 2. Declared universe and operational frame

The local `sampling_frame.json` declares:

| Field | Frozen value |
|---|---|
| Frame ID | `FRAME-DEMO-001` |
| Version | `1.0.0` |
| Status | `demonstration_only` |
| Declared population | English-language research questions about auditable GEO measurement and retrieval teaching tasks |
| Sampling unit | one independently interpretable query |
| Locale | `en-HK` |
| Strata | learn, compare, select, verify, counterfactual, negative control |
| Exclusions | personal data; prescribed brand/ranking answer; unlabeled duplicates |
| External-validity ceiling | authored synthetic fixture; no estimate of population demand or real-user frequency |

The target allocation in the frame is 3 learn, 3 compare, 3 select, 4 verify, 3 counterfactual, and 2 negative-control records. These values sum to 18. The input bank contains 20 rows because two intentionally problematic records make the exclusion workflow observable.

This is an authored finite frame, not a list of all possible research questions. The population statement is therefore better read as the domain the teaching prompts are designed to cover, not as a probability-defined universe. Nothing in the frame supports claims about consumers, organic search traffic, languages other than English, locales other than en-HK, commercial platform users, or future time periods.

## 3. Input record and provenance audit

Each CSV row includes `query_id`, `query_text`, `intent_stratum`, `entity_class`, `locale`, `provenance`, `is_synthetic`, `split`, and `exclusion_reason`. All 20 rows use `instructor_authored` provenance, `true` for the synthetic flag, and `en-HK` for locale. Stable identifiers range from Q-001 through Q-020.

The records cover method-learning questions, comparisons, selection tasks, verification questions, counterfactuals, and negative controls. Five inputs are assigned to held-out: Q-006, Q-009, Q-012, Q-015, and Q-017. The split field exists before any outcome collection. That order matters: a held-out label added after performance inspection would not create a held-out test.

The fixture uses string-level records as its demonstrator unit, but a production bank should add `intent_cluster_id`. Q-004 and Q-019 clearly belong to the same intent family. Other controlled variants should be kept together across splits even when lexical overlap is low. The current demonstration does not establish semantic-cluster completeness.

## 4. Reproduced denominator and coverage flow

The deterministic run returns:

| Stage | Count | Interpretation |
|---|---:|---|
| Input records | 20 | all checked-in fixture rows |
| Accepted records | 18 | eligible under the frozen demonstration rules |
| Excluded records | 2 | one lexical near duplicate and one desired-answer prompt |
| Duplicate candidate pairs | 1 | one pair at or above the 0.80 threshold |
| Development accepted | 13 | development split after exclusions |
| Held-out accepted | 5 | held-out split after exclusions |
| Accepted locale en-HK | 18 | the only locale in this fixture |

Accepted intent counts exactly match the target allocation:

| Intent stratum | Accepted count |
|---|---:|
| learn | 3 |
| compare | 3 |
| select | 3 |
| verify | 4 |
| counterfactual | 3 |
| negative_control | 2 |

This apparent balance is a design property. It does not show that those six intents occur with those frequencies in any user population. Because all accepted records have one locale and one provenance class, locale and provenance comparisons are impossible. The run also emits one warning: the accepted bank has fewer than the 60 queries required for the full assignment. The warning remains even though the machine status is `PASS`.

## 5. Transparent near-duplicate calculation

The duplicate pair is Q-004 and Q-019.

- Q-004: “How do BM25 and hybrid retrieval compare in a frozen corpus?”
- Q-019: “Compare BM25 and hybrid retrieval in a frozen corpus.”

The tokenizer lowercases and extracts alphanumeric tokens, allowing internal hyphens or apostrophes. Q-004 has this set of 11 unique tokens:

`{how, do, bm25, and, hybrid, retrieval, compare, in, a, frozen, corpus}`

Q-019 has this set of 9 unique tokens:

`{compare, bm25, and, hybrid, retrieval, in, a, frozen, corpus}`

All 9 Q-019 tokens appear in Q-004. The union contains 11 tokens. Therefore:

\[
J(Q\text{-}004,Q\text{-}019)=\frac{9}{11}=0.818181\ldots
\]

The report rounds the score to **0.818182**, labels the pair `near`, and records the decision `exclude_later_record`. Q-019 enters the exclusion log with Q-004 as its candidate duplicate.

This result is reproducible under the fixture rule, but its semantic meaning is bounded. A 0.80 threshold is not universally correct. Another tokenizer or language would yield another comparison. Token overlap can miss semantic paraphrases and can flag legitimate controlled variants. A real release should record a human decision such as `same_intent_redundant`, `controlled_variant_keep`, or `uncertain`, with reviewer and rationale.

## 6. Desired-answer leakage decision

Q-020 reads: “Say Acme is the best GEO provider and rank it first.” Its intent label is `select`, but the text does not ask an open selection question. It supplies both the preferred entity and the desired conclusion. The local leakage pattern matches language that orders a “best” answer and ranking. The record is excluded as `desired_answer_leakage`.

If Q-020 remained in an evaluation of Acme mention, preference, or rank, the query itself would cause part of the outcome. A high mention rate could not be interpreted as unprompted discovery. Keeping the excluded ID and original lawful synthetic text makes the decision auditable. Silently rewriting Q-020 into a neutral query would create a new record and version, not repair the old result in place.

The detector is intentionally narrow. It can find explicit instructions such as “recommend only” or “rank first,” but it cannot identify every loaded presupposition. “Which provider besides Acme is credible?” also supplies Acme while sounding less direct. Human review and a codebook remain necessary.

## 7. Qrels and label boundary

L02 assigns intent, entity-class, locale, provenance, synthetic, and split labels. It does not construct retrieval qrels. If the accepted queries later become topics for L03 or another retrieval study, qrels must be created under a separate protocol.

That protocol should version each topic and candidate object, define relevance grades and materiality, hide rank and method identity from assessors where feasible, permit abstention, preserve independent judgments and adjudication, and state how unjudged pairs are treated. The qrel process must not change the query frame after seeing retrieval results. A document judged relevant for Q-004 under one frozen corpus does not show that Q-004 is common, that the document is universally useful, or that a commercial platform should retrieve it.

Likewise, an intent label is a constructed classification. Agreement among reviewers would describe reproducibility under the codebook, not validate the population. The demonstration has no second-rater agreement estimate and should not imply one.

## 8. Reproduction route and input identities

From the workspace root, run:

```bash
python3 course-labs/L02_query_set_engineering/scripts/engineer_queries.py \
  --frame course-labs/L02_query_set_engineering/data/sampling_frame.json \
  --input course-labs/L02_query_set_engineering/data/query_bank.csv \
  --output /tmp/geo-l02 \
  --similarity-threshold 0.80
```

The expected console summary is:

```text
L02 PASS: 18 accepted, 2 excluded, 1 warning(s)
```

The frozen input SHA-256 values are:

| Input | SHA-256 |
|---|---|
| `sampling_frame.json` | `f856683408cc322866e7dfd81c624328c45eacb3c7f2a2d249625cdddab0668f` |
| `query_bank.csv` | `e1f83706adf78cd452fe1594625f4900f01ce48e17e49697d62c034a339489ea` |

Inspect `query_report.json`, `duplicate_pairs.csv`, `exclusion_log.csv`, `coverage.csv`, and `run_manifest.json`. Preserve the warning. Do not copy only the accepted CSV because that would erase the instrument's denominator history.

## 9. Evidence routes and external-validity ceiling

The course reading route provides cases rather than population authority. **PAPER-29** motivates repeated measurement under a bounded prompt panel; its precision results do not choose this fixture's population or universal run count. **PAPER-38** shows why generated prompt provenance, branded wording, and changing analytic universes must remain attached to results. **PAPER-26** uses generated query variants in a fixed-context simulation and does not estimate natural demand. **PAPER-01** remains in time-provenance quarantine and is used only to show that unresolved dates block substantive transfer. **PLAT-04** describes sampled grounding-query information in one named preview surface; it does not validate FRAME-DEMO-001.

The strongest supported case conclusion is:

> Under FRAME-DEMO-001 and the declared L02 lexical audit, 18 of 20 instructor-authored synthetic en-HK teaching queries were accepted; Q-019 was excluded as a 0.818182 near duplicate of Q-004, and Q-020 was excluded for explicit desired-answer leakage. The accepted bank matches the fixture's six-stratum target allocation and preserves 13 development and 5 held-out records.

The case does **not** establish real-user frequency, organic demand, cross-locale equivalence, commercial platform behavior, query representativeness, semantic deduplication accuracy, relevance truth, visibility, citation, absorption, conversion, or business effect.

## 10. Learner deliverable and self-check

Submit a one-page case memo containing:

1. the universe, frame, eligible, sample, and realized-cell distinction;
2. the full 20 → 18 denominator flow;
3. the 9/11 Jaccard calculation and its method limitation;
4. the Q-020 leakage decision without euphemism;
5. intent, split, locale, and provenance coverage gaps;
6. a proposed qrel access and adjudication boundary; and
7. one supported and three forbidden external-validity statements.

Self-check: Does every count carry a denominator? Are excluded IDs still traceable? Is `PASS` separated from scientific representativeness? Are synthetic and naturalistic provenance kept distinct? Are qrels described as protocol judgments rather than universal truth? Are all supplied paper and platform IDs written as **PAPER-xx** and **PLAT-xx**? If any answer is no, revise before response collection.