Generated from the same controlled README shipped in the offline package. Source SHA-256: b297f0c960a3edcafe685fb8f4b976c7209773e655438d1fb9d5b9a534553c13.
Linked course weeks: W07 and W13
Expected effort for the full assignment: 8 hours
Core path: frozen synthetic panel, offline, deterministic, Python standard library only
Research question
Is an observed difference larger than repeated-sampling variation and time drift?
Learning objectives
You will learn to:
- model query, surface, repetition, and time as separate panel dimensions;
- preserve missing/refused responses rather than silently changing denominators;
- estimate mention, citation, entailment, absorption, and referral as distinct outcomes;
- compute transparent binomial intervals and time-block deltas;
- inspect sensitivity to query intent mix; and
- state why a frozen synthetic panel cannot prove a live cross-engine or causal effect.
data/panel.csv follows ../shared/schemas/measurement-event.schema.json and contains exactly 360 frozen synthetic events: 20 queries × 2 teaching surfaces × 3 repetitions × 3 time blocks.
- One row exists for every designed combination, even when
response_status is missing or refused.
- The five outcome fields are independent columns with values
0 or 1 only for complete responses.
- All outcome cells are empty for missing/refused responses.
captured_at is an ISO-8601 timestamp with offset; event and combination keys are unique.synthetic=true is mandatory for the bundled path.
The values were deterministically authored to exercise interval, drift, missingness, and query-mix logic. They are not observations of named products.
Run
python3 scripts/analyze_panel.py \
--panel data/panel.csv \
--output /tmp/geo-l06 \
--minimum-queries 20
Intervals are 95% Wilson score intervals. They summarize binary variation under the declared denominator; they do not correct dependence among repeated observations or justify causal comparisons.
Output contract
Table from the controlled L06 lab handout| Artifact | Contract |
|---|
tidy_panel.csv | normalized full panel including explicit missing rows |
metric_estimates.csv | surface × time × metric numerator, denominator, rate, and Wilson interval |
time_drift.csv | last-minus-first block rate by surface and metric |
query_mix_sensitivity.csv | estimates by intent stratum, surface, and metric |
missingness.csv | designed event, complete, missing/refused counts and response rate |
data_dictionary.json | types, allowed values, and outcome-specific denominator policy |
validity_boundary.json | design coverage, dependence warning, synthetic ceiling, and excluded causal claims |
run_manifest.json | input hash and declared minimum design |
Full assignment procedure
- Freeze at least 20 queries, two surfaces, three repetitions, and three time blocks—or justify an information-equivalent panel before collection.
- Record product/surface version, locale, account state, time, and collection mode in an external system card.
- Preserve failures, refusals, unavailable citations, and collection interruptions as explicit event states.
- Lock metric definitions and outcome-specific denominators before viewing comparisons.
- Estimate surface/time outcomes with intervals and inspect individual query trajectories.
- Repeat estimates by query stratum; compare pooled and stratified interpretation.
- Distinguish within-block repetition variation from between-block drift.
- Report collection deviations, cost, terms/rate constraints, and every missingness decision.
Low-compute and no-network route
The bundled 360-row panel is already frozen. Analysis uses CSV, arithmetic, and the standard-library normal distribution only. No account, paid API, browser automation, model call, notebook service, or network connection is required.
Negative results and interpretation
- Overlapping intervals do not prove equality; non-overlap is not a substitute for a prespecified statistical model.
- A larger surface difference than time drift can still be confounded by product version, session, or query composition.
- Missingness can be outcome-related; complete-case rates may be biased.
- Mention, citation, entailment, absorption, and referral must not be substituted for one another.
- One screenshot or one run cannot support a repeatability claim.
- Synthetic panel effects are known fixture patterns, not empirical findings.
Ethics, cost, and stop conditions
Core completion never requires a paid account. Optional collection requires authorization, terms/rate review, a budget cap, privacy review, and manual/API provenance. Stop on personal data, hidden automation, uncontrolled cost, account-risk signals, product-policy conflict, or inability to preserve missing events.
Acceptance
python3 scripts/analyze_panel.py --panel data/panel.csv --output /tmp/geo-l06 --minimum-queries 20
python3 -c 'import json; b=json.load(open("/tmp/geo-l06/validity_boundary.json")); assert b["event_count"] == 360 and b["design"]["query_count"] == 20'
See RUBRIC.md.