Distinguish design, adjustment, and untestable assumptions.
PrerequisitesW07 metric card.Bridge note on potential outcomes, controls, and randomization.
Builds on
W07 metric card. · Bridge note on potential outcomes, controls, and randomization.
Investigates
Can the change be attributed to the intervention?
Feeds
Pre-analysis plan clinic feeding L05/L06.
02
Before class
Arrive with a prepared artifact
Reading route
Core PAPER-10/PAPER-23; audit PAPER-32 for benchmark-size, test-set selection, and run-uncertainty gaps; inspect V03 as a design case.
Viewing route
Open the complete text-first lecture packageEight-minute estimand and DAG walkthrough. The current equivalent is notes, slide script, worked case, and no-video transcript; no recording is claimed.
Readiness check
Write treatment, control, unit, outcome, and target population for one proposed test.
Bring
Draft pre-analysis plan without outcome data.
03
Explanation
Concepts, assumptions, and boundary
An effect requires a comparison world
An intervention effect is a contrast between potential outcomes under treatment and control for a defined unit and population. Before seeing outcomes, the analyst states the estimand, treatment assignment, control condition, observation schedule, exclusion rules, and analysis. Simple before/after changes are vulnerable to platform releases, query drift, seasonality, concurrent edits, and regression to the mean. Randomization, blocking, matched controls, staggered timing, or interrupted-series designs can reduce particular threats but cannot make every assumption observable.
Validity is a boundary, not a badge
A causal diagram makes assumed relationships and adjustment choices visible. Interference matters when one page or source ecosystem affects another unit. Sequential peeking inflates false-positive risk unless stopping rules are planned. Power depends on variance, clustering, treatment size, and feasible sample size—not just the number of screenshots. Even a strong internal design may generalize only to the tested systems, queries, dates, locales, and interventions.
04
Primary visual
Inspect the mechanism or evidence structure
Conceptual modelCausal diagram and validity boundary
Figure design. A DAG shows intervention, content quality, query mix, platform state, exposure, and outcome; a shaded frame encloses the tested population and period. The graph records assumptions; it does not prove them.Long description
Directed paths identify confounding, mediation, and drift. Adjustment candidates and unobserved variables use different line styles. A boundary box states exactly where the estimand applies.
05
Interactive check
One action, one feedback state
Action
Toggle confounding and interference paths, then choose a design response.
Feedback
Feedback distinguishes blocking, randomization, measurement, adjustment, sensitivity analysis, and assumptions that remain untestable.
Accessible alternative
A decision table presents every graph state and defensible response.