# W08 Planned Transcript — From Observation to Identification

## Status and use

No W08 recording exists. This is a planned no-video equivalent, not a transcript of a delivered class. The chapter durations are instructional planning budgets. The seminar, studio, and transcript route have not been taught, rehearsed, timed, or piloted. No clock-style timestamps are included. A future recording must preserve the evidence boundaries, receive edited captions, and be checked against this canonical route.

All assessed concepts are expressed in words. Diagram relationships are described directly, so the route can be completed without seeing slides. Public references use `PAPER-10`, `PAPER-23`, `PAPER-32`, and `PLAT-04`.

## Planned chapter budget

| Chapter | Topic | Planned minutes |
|---:|---|---:|
| 1 | After is not because | 4 |
| 2 | Estimands and potential outcomes | 5 |
| 3 | DAG paths and adjustment decisions | 5 |
| 4 | Randomization, blocking, and matching | 5 |
| 5 | Staggered, switchback, DiD, and ITS designs | 5 |
| 6 | Clustering, power, missingness, and sequential rules | 5 |
| 7 | Interference, validity, preregistration, and deviations | 4 |
| 8 | L05 and L06: valid but disconnected fixtures | 4 |
| 9 | W08 pair-blocked synthetic reproduction | 4 |
| 10 | Source audit and identification verdict | 4 |
|  | **Total planned route** | **45** |

## Chapter 1 — After is not because

Imagine that a citation rate is zero point three before a page edit and zero point four two afterward. The observed change is plus zero point one two. That subtraction is correct. The word “effect” is not yet justified.

The missing object is the post-period outcome under control for the same target units and system state. We did not observe what would have happened after the date if the page had remained unchanged. The baseline is not automatically that counterfactual. Time moved.

During that time, a platform release may have changed retrieval or presentation. The index may have absorbed unrelated source updates. The query sample may include more favorable intents. Seasonal demand may shift. Competitors may edit or disappear. A cache or propagation delay may alter treatment uptake. The retry policy may change. Missing responses may be dropped. The judge or parser may be updated. Every process can make the after value differ from the before value.

Our first task is therefore classification. A versioned observation says: under recorded conditions, the measured rate changed from one value to another. An association says: observed treatment and control groups, or observed periods, differ. An identified effect adds a design and assumptions that connect the observed contrast to a counterfactual target. A transported effect adds an argument that the result applies beyond the tested units, state, and treatment version.

Here is the learner check. Rewrite “the edit improved citations by twelve percent.” A bounded version is: “In the recorded panel, citation incidence was zero point three before the edit date and zero point four two afterward, a difference of twelve percentage points; the comparison does not isolate the edit from concurrent time-varying processes.” Notice the unit: twelve percentage points, not twelve percent. The relative change would be forty percent, which is a different contrast.

Now list three alternative explanations. Then write the comparison unit you would need. If treatment can be assigned across query-intent clusters, concurrent controls may be possible. If only one time series exists, an interrupted design may be considered, but the identification burden becomes heavier.

The W08 habit is to ask “what comparison world?” whenever causal language appears. A positive number is an observation. A counterfactual claim is a design achievement.

## Chapter 2 — Estimands and potential outcomes

An estimand is the target contrast. It is not a model name and not a result. Build the estimand card before selecting an estimator.

The card begins with treatment and control versions. Store complete content, markup, source links, rendered state, timing, and hashes. “Added a table” is not a treatment version if other text, sources, or layout changed. A bundle identifies a bundle effect unless the design separates components.

Next name the unit. It may be a page, domain, query intent, user task, or time block. The unit of assignment determines the independent allocation. Repeated generated answers under one intent are measurements inside that unit unless assignment occurs separately and without contamination.

Then name the outcome, eligibility, and scale. “Visibility” is not enough. State mention incidence, citation correctness, absorption, referral, or another defined event. State which responses are eligible, what missing means, and over what window the outcome is summarized.

Name the target population or finite set and versioned system state. A result on twenty frozen synthetic queries is an effect on that set only if identified. It is not automatically an effect on market demand. A result at one model or index state does not transport to a future state without evidence.

Finally, specify the contrast, weights, treatment uptake, interference, and missingness. Is the target an intent-to-treat assignment effect, a treatment-on-treated effect, or a cluster-average contrast? Are query strata weighted equally or by a target distribution?

Potential outcomes make the missing comparison explicit. Let `Y_i(1, S_t)` be the outcome for unit i under treatment at system state t and `Y_i(0, S_t)` under control. We observe only the outcome corresponding to the assigned state. An average effect is the mean difference between these two potential outcomes over declared units. We do not observe both for one unit in one period.

An estimator is the rule that maps observed data to an estimate. A randomized difference in cluster means is an estimator. Difference-in-differences is an estimator under a temporal design. The estimate is a realized number such as zero point zero six two five.

Learner check: classify these phrases. “Average assignment effect across eight query clusters” is an estimand. “Difference in equal-cluster changes” is an estimator. “Six point two five percentage points” is an estimate. Then ask whether the estimator is justified by assignment and assumptions. Correct arithmetic can still target the wrong object.

Potential-outcome notation does not identify the contrast. It records the question. Assignment, uptake, measurement, and assumptions create the bridge.

## Chapter 3 — DAG paths and adjustment decisions

A directed acyclic graph is an assumption map. It should name treatment, outcome, pre-treatment causes, mediators, selection, measurement, time state, and interference where relevant.

Consider treatment D, outcome Y, baseline content quality C, query mix Q, platform state P, response observability R, and measurement rule M. If stronger pages are more likely to receive treatment, C points to D. If stronger pages also perform better, C points to Y. The path D left-arrow C right-arrow Y is a backdoor path. Randomization can break the association between C and D over its assignment. Blocking can ensure balance on important observed C strata. Observational adjustment can close the measured path under stronger assumptions.

A mediator lies on the causal pathway. Suppose D changes readability, and readability changes comprehension. Adjusting for readability removes part of the total effect and targets a direct-effect question. That may be intentional, but it must match the estimand.

A collider is caused by two variables. Suppose treatment affects whether a citation panel appears, and answer quality also affects whether the response is captured. Conditioning on complete observed panels can open an association between treatment and quality. Complete-case analysis is not automatically neutral.

Time needs explicit nodes. Platform state can affect both treatment uptake and outcome. A parser update can change observed Y without changing the underlying event. Query composition can shift across periods. Draw these pathways rather than writing “time fixed effects” as a ritual.

Interference adds arrows between assignments and neighboring outcomes. If treating source A displaces source B from a finite context, `D_A` points to `Y_B`. A no-interference assumption is then implausible.

Use line style to distinguish observed and latent nodes. Put a small table beside the graph: path, design response, diagnostic, remaining assumption, and verdict if the diagnostic fails. A balance check can reveal assignment problems. Multiple pre-periods can reveal nonparallel trajectories. Negative controls can expose some residual pathways. None proves the DAG complete.

Learner check: draw content quality as a confounder, readability as a mediator, and response availability as a collider. For a total treatment effect, adjust for pre-treatment content quality if needed, do not automatically adjust for readability, and inspect selection rather than conditioning blindly on R.

The critical statement is “DAG does not equal proof.” It communicates assumptions so reviewers can challenge them. It cannot manufacture a missing comparison.

## Chapter 4 — Randomization, blocking, and matching

Randomization is often the strongest available route when units can receive versions concurrently, treatment is authorized, and interference is manageable. It provides a known assignment mechanism. It does not ensure uptake, eliminate measurement error, prevent spillover, or create external validity.

Choose the real assignment unit. If treatment is applied to an entire page, repeated queries against that page do not create independent assignments. If semantically related queries can contaminate one another through strategy selection, assign at the intent cluster. If a domain-wide template changes, the domain may be the cluster.

Block randomization on prognostic pre-treatment variables. Query intent, locale, source type, and baseline rate can form blocks. Randomize treatment and control within each block. Declare block weights. Blocking is performed before outcomes; it supports balance and precision. It does not justify post hoc blocks chosen because they make the result favorable.

Cluster randomization reduces within-cluster contamination when interference stays inside clusters. It requires enough independent clusters. A study with hundreds of responses but four domains has four assignment clusters if domains receive treatment. More calls do not repair that limitation.

Matching is useful when assignment cannot be randomized but treated and untreated units overlap on rich pre-treatment features. Freeze the distance, caliper, replacement, ratio, tie policy, and target population before post outcomes. Preserve matched pairs or sets in analysis.

Matching is not randomization. It balances measured variables under the matching design. Unmeasured causes remain. Units outside common support are excluded, narrowing the target. If every high-baseline treated page lacks a comparable control, the study cannot estimate that region by matching.

Randomization and adjustment can work together. A prespecified covariate-adjusted estimator may improve precision, while the unadjusted assignment estimate remains a transparent primary sensitivity. A complicated mixed model does not fix assignment failure or too few clusters.

Learner check: thirty query intents are blocked by four intent strata and assigned within strata. Each is run five times. How many assignment units exist? Thirty. How many measurements? One hundred fifty. Which number belongs in the primary independence argument? Thirty.

Now choose between individual and cluster randomization. If prompts share a source page and treatment changes that page for all prompts, query-level treatment may be impossible. Assignment belongs at the page or a version-routing unit that can reliably deliver separate versions without contamination.

The design should follow controllable exposure. A prestigious estimator cannot compensate for an assignment that never reached the unit.

## Chapter 5 — Staggered, switchback, difference-in-differences, and interrupted series

Sometimes every unit must eventually receive treatment, but adoption timing can be assigned. A staggered rollout randomizes or otherwise schedules cohorts. It creates not-yet-treated comparisons. The design must address anticipation, wash-in, dynamic effects, heterogeneous cohorts, and changing platform state. Already-treated cohorts are not automatically valid controls for newly treated cohorts.

A switchback alternates treatment and control over time. It is attractive when treatment is reversible and short-lived. It fails when index propagation, caching, or user learning carries treatment into the next period. Define washout and period length from mechanism knowledge, not convenience. Balance order because mornings and evenings may differ.

Difference-in-differences compares treated change with control change. The formula is treated post minus treated pre, minus control post minus control pre. Its identification condition is not the formula. It requires a credible untreated parallel trend, no anticipation, stable measurement and composition, and appropriate interference handling.

Plot several pre-periods. Test placebo intervention dates. Inspect whether controls face the same state changes. Pre-trend similarity supports plausibility but cannot prove what would happen after treatment. If trends diverge before the intervention, redesign or report a model-dependent association.

Interrupted time series studies level and slope change around a dated interruption. It can be appropriate when no control exists and a long stable series is available. It requires sufficient pre and post observations, an explicit outcome, autocorrelation treatment, seasonality handling, stable measurement, and a concurrent-event log. A product release on the same date prevents clean attribution unless it is part of the treatment.

Matched concurrent controls can strengthen a time series. Controlled ITS or synthetic comparison methods still depend on donor relevance and no simultaneous differential shocks. Complex weighting does not make poor donors equivalent.

Learner check: a citation rate has two pre points and one post point. Is ITS credible? No. There is insufficient history for slope, seasonality, or placebo diagnostics. A simple line break would overstate information.

Now imagine four rollout cohorts, and the platform changes between cohort two and cohort three. Record the state change. Cohort-specific contrasts may target different states. Pooling them as one invariant effect needs a transport or stability assumption.

Design selection is a failure-mode choice. Randomization addresses assignment confounding. Blocking improves balance. Matching addresses observed comparability. Staggering addresses rollout constraints. DiD addresses common temporal change under parallel trends. ITS uses longitudinal structure but remains vulnerable to concurrent shocks. None solves every threat.

## Chapter 6 — Clustering, power, missingness, and sequential rules

Power begins with the independent units, outcome distribution, and minimum worthwhile effect. Include allocation, cluster sizes, intraclass correlation, repeated-run variance, time drift, attrition, multiplicity, and guardrail precision.

For roughly equal cluster size m and intraclass correlation rho, the diagnostic design effect is one plus m minus one times rho. If m equals five and rho equals zero point two five, design effect is two. The five correlated rows contain about half the independent information implied by a naive row count under that approximation. Unequal clusters and few clusters require design-specific simulation or conservative analysis.

Choose a minimum effect based on the decision, not on what makes the available sample look powerful. Simulate the actual assignment, binary base rate, missingness, and analysis. Power important harm guardrails as well as the primary outcome. If resources cannot distinguish a worthwhile effect, call the study a pilot or feasibility audit.

Missingness changes the target. A refusal, timeout, parser failure, unavailable citation panel, and policy block are different states. Decide whether each is an outcome, bounded retry, censoring state, or protocol failure. Preserve every designed cell. Never retry until a desired output appears.

Complete-case estimates condition on observability. If treatment and outcome influence observability, selection bias can result. Report missingness by arm, block, and time. For binary outcomes, show transparent worst cases: set all missing treated outcomes to zero and control outcomes to one, then reverse. These bounds do not solve missingness, but they show sensitivity.

Sequential peeking is another source of hidden selection. If the plan is a fixed horizon, do not stop at the first favorable result. If continuous monitoring is required, declare a sequential boundary, cadence, minimum information, and terminal actions. Preserve every interim look.

Safety stops are distinct. A factuality or accessibility breach can trigger immediate rollback regardless of efficacy inference. Declare which stops protect participants or integrity and which support a confirmatory efficacy claim.

Multiplicity matters because GEO work often measures mention, citation, entailment, absorption, referral, prominence, fidelity, and risk. Choose one primary outcome or a justified formal composite. Mark secondary outcomes and guardrails. Control confirmatory error or call the search exploratory.

Learner check: a report examined eight outcomes after every twenty responses and highlighted the first significant one. Identify two problems: multiple outcomes and repeated looks. The repair must be designed before collection; a retrospective label cannot fully restore confirmatory status.

## Chapter 7 — Interference, validity, preregistration, and deviations

Interference occurs when one unit’s assignment affects another unit’s outcome. Retrieval and context capacity make this plausible. Treating source A can displace B. A shared template can modify many pages. An adaptive strategy learner can use earlier treatment outcomes when choosing later interventions.

Write potential outcomes as depending on own assignment, neighboring assignments, and system state. Define direct, spillover, total, or displacement effects. Cluster randomization may contain interference if clusters align with the network. If not, use an exposure mapping or report the limitation. Measure target gain and displaced-source harm.

Validity has four useful boundaries. Internal validity concerns attribution within the studied units and state. Construct validity concerns whether treatment and outcome represent the intended concepts. Statistical conclusion validity concerns precision, dependence, multiplicity, and model adequacy. External validity concerns transport across systems, queries, locales, dates, and versions.

Statistical significance cannot rescue internal validity. Randomization cannot prove a dashboard count measures source influence. High internal validity in an open reconstruction does not guarantee transport to a closed product. State each boundary separately.

Preregistration freezes the estimand, versions, assignment, blocks, sample, outcomes, weights, missingness, retries, exclusions, analysis, uncertainty, guardrails, stopping, multiplicity, sensitivity, and decision rule before outcome access. It creates an auditable decision path. It does not make assumptions true.

The deviation log is append-only. Record the planned rule, departure, detection time, whether outcomes were visible, affected units, reason, corrective action, analysis consequence, approver, and status. Keep the original plan. A propagation delay may shift a window under a frozen rule. A parser change may require reprocessing all raw artifacts. Control contamination may invalidate a pair.

Learner check: midway through collection, the platform exposes a new citation panel. Do not quietly pool. Apply the preregistered comparability rule: pause, rebaseline, stratify by regime, or invalidate. Record when the change was noticed and whether outcomes had been reviewed.

End every report with an estimand-to-action map. Retain, revise, rollback, and inconclusive should depend on the primary interval plus fidelity, accessibility, and risk guardrails. A positive average cannot override severe subgroup or guardrail harm.

## Chapter 8 — L05 and L06: valid but disconnected fixtures

Now inspect the local evidence. L05 is a controlled content-intervention package. It freezes three claims and three source identities. Control and treatment HTML are identical except for the stylesheet name. The stylesheets differ only in the declared evidence-block rule. Six independent equivalence checks pass. Rollback hashes are preserved.

The deterministic command prints: “L05 PASS: three locked claims, one changed factor, zero errors.” The factor is evidence-layout proximity. The hypothesized stage is human comprehension. The card explicitly states that crawler, retrieval, ranking, citation, traffic, and business effects are not measured.

L05 therefore validates treatment-version integrity. It does not collect a response outcome. It does not estimate an effect.

L06 is a repeated-measurement panel. It contains 360 synthetic events: twenty queries, two teaching surfaces, three time blocks, and three repetitions. There are 354 complete events and six noncomplete events. Every designed cell remains. Five outcomes stay separate.

The deterministic command prints: “L06 PASS: three hundred sixty events, twenty queries, two surfaces, three blocks.” It reports descriptive Wilson intervals, time changes, query-mix sensitivity, and missingness. It warns that repeated events for one query are not independent. No content treatment is assigned.

The identification failure is now visible. L05 has a treatment version without an outcome panel. L06 has an outcome panel without the L05 assignment. The unit registries are not joined. There is no uptake record. `SYN-A` and `SYN-B` are teaching surfaces, not treatment and control versions.

Learner check: can we say L05’s treatment caused an L06 difference? No. Placing two valid artifacts beside one another does not create a data-generating link. A valid joined design would assign L05 versions to eligible units and write that assignment, exposure, timing, and outcomes into one preregistered panel.

This negative result is important. Reproducibility of separate components is necessary but not sufficient for causal identification.

## Chapter 9 — W08 pair-blocked synthetic reproduction

The package includes a separate eight-cluster arithmetic case. Four matched pairs contain one treated and one control cluster. Each cluster has twenty binary opportunities at baseline and twenty post-period opportunities. Assignment occurs by cluster.

Pair one has treated Q-01 with eight baseline successes and eleven post successes. Its rates are zero point four and zero point five five, a change of zero point one five. Control Q-02 changes from eight to nine successes, a rate change of zero point zero five.

Pair two has treated Q-03 changing from ten to twelve successes, or zero point one. Control Q-04 changes from ten to eleven, or zero point zero five.

Pair three has treated Q-05 changing from six to nine, or zero point one five. Control Q-06 changes from six to seven, or zero point zero five.

Pair four has treated Q-07 changing from twelve to thirteen, or zero point zero five. Control Q-08 makes the same count change and rate change.

Average the four treated cluster changes equally: zero point one five plus zero point one plus zero point one five plus zero point zero five, divided by four, equals zero point one one two five. The four control changes all equal zero point zero five, so their mean is zero point zero five. Difference-in-differences is zero point one one two five minus zero point zero five, equal to zero point zero six two five.

Report that as six point two five percentage points on the binary-rate scale. Do not call it a six point two five percent relative effect.

What does this prove? The manifest arithmetic is deterministic. It proves that these counts produce this estimator under equal-cluster weighting. A causal interpretation still requires that the pair assignment was implemented before outcomes, treatment was taken up, untreated trends would be parallel, measurement stayed stable, interference was negligible, and the synthetic state remained comparable.

There are only eight assignment clusters. The case has no claim of adequate power or population inference. No p-value is reported. Its purpose is to connect estimand, unit, estimator, assumptions, and verdict.

Learner check: why not analyze 320 binary opportunities as independent? They share eight assignments and cluster characteristics. Which assumption is threatened if treated Q-01 displaces control Q-02? No interference. Which design evidence could address baseline prognosis? Pair blocking and within-pair assignment, if implemented and preserved.

The correct conclusion is a synthetic design contrast, not a live GEO effect.

## Chapter 10 — Source audit and identification verdict

W08 routes `PAPER-10`, *CC-GSEO-Bench*. It supplies a content-centric benchmark case with query–article organization and multiple influence dimensions. We use it to ask what the unit is, how article-level aggregation works, and how exposure, credit, and causal impact differ. Its findings remain bounded to its benchmark construction and systems.

`PAPER-23`, *SAGEO Arena*, supplies a reconstructed end-to-end pipeline case. It supports stage-specific evaluation and the question of how baseline target selection, corpus, retrieval, reranking, generation, and reported uncertainty affect claims. An open reconstruction does not identify a current closed product.

`PAPER-32`, *From Experience to Skill*, is audit-only. Learners inspect benchmark size, test selection, reusable strategy adaptation, attribution metric construction, run uncertainty, and interference. The course does not adopt headline improvements as causal findings.

`PLAT-04` is official interface documentation for a named platform preview. It can define citations, cited pages, sampled grounding queries, and trends for the named surfaces and date. It does not create treatment assignment, counterfactual outcomes, causal source-use evidence, or transport.

The final identification verdict has five fields: estimate, design evidence, assumptions, diagnostics, and transport boundary. It routes to one of four states: identified within scope, association, inconclusive, or invalid.

Here is a complete W08 conclusion. “L05 reproducibly validates a one-factor, fact-locked local treatment package, and L06 reproducibly validates a 360-event descriptive synthetic panel; they are not one intervention study. In the separate eight-cluster W08 teaching table, the equal-cluster difference-in-differences contrast is zero point zero six two five outcome units. The arithmetic supports a design exercise. Causal interpretation is conditional on assignment integrity, uptake, parallel untreated trends, stable measurement, and no interference; no live platform or population effect is estimated.”

Final learner check: take one before/after result and produce an estimand card. Draw the backdoor and time paths. Choose a design based on feasible assignment. Name the unit. Write the missingness and stop rules. State one falsification check and one external-validity boundary. Then issue the strongest verdict supported.

The goal is not to call every study causal. The goal is to know exactly when causal language is earned, when an association is the honest answer, and when the comparison is too damaged to interpret. Identification quality begins with a reconstructible comparison world.

Your submitted verdict should remain useful even if the point estimate changes sign during independent reproduction. That means the version identities, unit registry, assignment evidence, missingness table, code, and decision thresholds carry the argument—not a preferred direction. If another analyst can alter the conclusion merely by discovering which arm was expected to win, blinding, preregistration, or outcome governance was inadequate. Preserve null, adverse, heterogeneous, and inconclusive patterns with the same care as favorable ones.