# W13 Lecture Notes — Field Experiments and Monitoring

## 1. The platform is part of the time series

A field observation occurs inside a changing environment. Models, indexes, interfaces, retrieval policies, product aliases, source corpora, account features, rate limits, and collection code can change while a study is running. A response record without system state therefore invites a false story: an observed difference can be attributed to an intervention even when the measurement regime changed at the same time.

The minimum observation is not merely query plus answer. It joins a frozen query ID, intended surface, configuration label, alias or explicit status, time block, repetition, captured time with offset, session and account state, locale, collection mode, collection order, response status, raw artifact pointer, parser version, and known release event. When a field is unavailable, record `unknown`; do not replace an unobservable backend variable with a plausible guess.

Blocking makes some variation visible. LAB-L06 uses 20 queries, two synthetic teaching surfaces, three time blocks, and three repetitions. It preserves one row for every designed combination. That structure separates repeated-run behavior inside a block from descriptive differences between blocks. It does not make time a randomized treatment or identify why a block changed.

Monitoring is part of identification because it records alternative explanations. A release event, account-state change, locale drift, parser revision, or collection interruption can break comparability. The log does not eliminate confounding. It tells the analyst which assumptions no longer hold and when to continue, pause, rebaseline, stop, or rollback.

The W13 core route is supplied and offline. No learner needs a paid account, browser automation, model call, or network connection. That is not merely convenient: it prevents account access, provider terms, rate limits, and unapproved costs from becoming hidden prerequisites.

## 2. Freeze the observation plan and system card

Before looking at outcomes, freeze the unit, panel, time blocks, configuration fields, primary metric, guardrails, missing-state policy, retry rule, planned analysis looks, change-point alerts, decision thresholds, budget, authorization owner, and rollback target. Give the plan a version and hash. A later change becomes a deviation event rather than a silent edit.

A system card belongs to each recorded surface/configuration state. It includes public product or fixture label; alias or explicit identifier; declared version when available; interface or API route; locale; account and subscription state; session isolation; search or browsing state; collection method; known release date; parser; and unobservable fields. For an instructor-provided snapshot, it also states synthetic status and source-file hash.

Session matters because prior turns, cookies, cached state, personalization, and account features can affect outputs. `fresh-session` is not a universal technical fact unless the collection method verifies it; record what was cleared and what remains unknown. Account state can control product access or surface behavior. Locale combines language, region, and sometimes location. A prompt written in one language is not sufficient evidence that the interface used a particular locale.

Collection order is a design variable. If all SYN-A records are collected before SYN-B, surface and time can be confounded. Counterbalance or randomize order within the authorized plan, preserve the realized order, and record interruptions. LAB-L06 timestamps its authored repetitions, but the package does not infer how a real collector should schedule calls.

Failures remain events. A timeout, parser error, refusal, unavailable citation panel, authorization denial, and rate-limit response have different meanings. Preserve the first attempt, the allowed retry reason, retry count, delay, and terminal status. Never retry until a favorable answer appears.

## 3. Dynamic aliases and explicit configurations

A dynamic product alias is convenient because it tracks a provider’s current offering. That same convenience weakens longitudinal identity: the provider can change routing, snapshot, tools, safety policy, index, or interface behavior without changing the alias. An explicit configuration improves traceability when the provider exposes one, but even an explicit model label can leave backend routing, corpus state, caching, or retrieval settings unobserved.

Record alias and explicit routes separately. If possible, compare them in the same time block, with matched queries, locale, session, account, order, and collection mode. The comparison documents output differences conditional on the recorded labels. It does not reveal which hidden component caused those differences.

This package maps LAB-L06 `SYN-A` to an authored alias-route system card and `SYN-B` to an authored explicit-route system card. That mapping is teaching metadata, not a claim about real products. In block B-02, the fixture mention rates are 25/59 = 0.423729 for SYN-A and 31/60 = 0.516667 for SYN-B, a descriptive difference of 0.092938. A same-time alias difference does not identify a hidden backend change. The authored panel could embody any combination of deterministic fixture choices; no backend exists to discover.

If a provider announces that an alias changed its mapping, log the official event and date. That event can justify a new measurement regime. It still does not prove that a concurrent outcome difference was caused by the mapping change. If no version information is available, the correct field is `backend_state: unknown`, not a reverse-engineered assertion.

PAPER-29 reinforces this boundary in its own setting: repeated web-interface observations cannot isolate token sampling from retrieval, index, routing, cache, and product drift. Its study-specific counts are not a universal schedule. PAPER-38 likewise does not permit inference about training, ingestion, memory, or retrieval causes from black-box outputs.

## 4. Time blocks, state, order, failures, and releases

A time block is a declared comparability window, not simply a date label. Define why observations inside it are treated as comparable, how long it lasts, and what event closes it. A block may close after a planned batch, a release, a configuration change, a parser revision, a collection outage, or a protocol deviation.

The configuration log has two kinds of entries. Known-state entries cite an official release, a controlled fixture change, or an approved collector revision. Unknown-state entries record an observed discontinuity without assigning cause. Keep `release_observed`, `release_effective`, and `release_retrieved` dates separate when possible.

Counterbalanced collection order reduces systematic time confounding, but it does not freeze the platform. Record the realized order and duration. An outage halfway through a batch can make early and late query strata differ. Preserve whether the interrupted cells were retried, rescheduled, or left noncomplete. If order cannot be reconstructed, flag the block rather than treating it as exchangeable.

Account, session, locale, and privacy state belong in each observation or a referenced system card. A shared account can introduce personalization or history. A learner must never use someone else’s credentials or personal data for a course exercise. The account-free supplied route keeps those risks outside the core task.

Known releases enter as candidate regime changes. An outcome-based change detector enters as an alert. These are not equivalent. A documented version change can require rebaseline even without a large metric shift because the comparability assumption changed. A large shift with no documented release is an unexplained change point; it may require pause and investigation, not a causal announcement.

## 5. Append-only event and deviation ledger

The event ledger connects plan, observations, releases, failures, deviations, decisions, and rollback. Append-only means a prior row is never overwritten to make the record look cleaner. A correction adds a new row that refers to the mistaken row. Sequence numbers, parent IDs, timestamps, and a canonical hash chain make mutation detectable.

Minimum fields are `event_id`, `sequence`, `occurred_at`, `recorded_at`, `event_type`, `block`, `system_card_id`, `actor_role`, `severity`, `facts`, `evidence_pointer`, `deviation_code`, `decision`, `reason_code`, `parent_event_id`, `previous_hash`, and `row_hash`. Personal data and secrets do not belong in the ledger. Store only permissible identifiers and access-controlled evidence pointers.

Distinguish three record classes. A planned event, such as block completion or a scheduled look, follows the protocol. A deviation is a departure from the plan, such as order change, parser hotfix, or extra retry. An incident threatens authorization, privacy, cost, rate-limit, measurement integrity, or evidence preservation. A deviation can become an incident, but the terms are not synonyms.

The canonical hash rule in the W13 fixture serializes each event without `row_hash`, with sorted keys and compact UTF-8 JSON. `previous_hash` equals 64 zeroes for the first row and the preceding `row_hash` thereafter. The row hash is SHA-256 of that canonical serialization. The validator checks unique IDs, contiguous sequence, monotonic recorded time, valid parents, allowed event types, decision precedence, and final chain hash.

An append-only log does not make the facts true. It makes the declared history tamper-evident under the chosen file and hash process. Access controls, independent review, and secure time sources remain separate controls.

The ledger must also distinguish event time from knowledge time. `occurred_at` records when the event is said to have happened; `recorded_at` records when the study learned or wrote it. A release discovered after a block cannot be backdated into the earlier ledger sequence. Append it at the current sequence, cite the earlier effective date, and reopen affected decisions if the rule requires. This difference exposes retrospective knowledge without pretending the team acted on information it did not yet have.

Deviations need disposition. Use states such as `open`, `accepted-with-boundary`, `requires-reanalysis`, `invalidates-regime`, and `closed`. A reviewer records which outcomes, blocks, or decisions are affected. A deviation ledger that merely lists surprises without connecting them to analytic consequences is an archive, not a monitoring control.

## 6. Planned sequential looks and conservative interpretation

Repeatedly inspecting outcomes and stopping when a preferred difference appears increases researcher freedom. Freeze the analysis-look schedule. The W13 synthetic plan permits one outcome look after each complete block: B-01, B-02, and B-03. It makes no efficacy or success stop claim. The final outcome interpretation remains descriptive.

Operational hard-gate monitoring is different. Authorization expiry, privacy exposure, evidence loss, policy conflict, rate-limit warning, or budget-cap risk must be checked as events arrive. A safety stop does not wait for a scheduled outcome look. Conversely, continuous operational monitoring does not authorize continuous outcome peeking.

If an unplanned outcome look occurs, append a deviation. Do not delete it. Mark subsequent confirmatory language as exploratory unless a prespecified sequential method supports it. A formal sequential design would need an estimand, information times, stopping boundaries, multiplicity control, and rules for missing and delayed observations. W13 does not invent an alpha-spending result for the synthetic fixture.

Predefine what each look can decide. A scheduled outcome look may continue the panel, flag an exploratory pattern, or activate a prespecified quality threshold. It cannot reinterpret the primary metric, change the denominator, drop a difficult block, or silently alter query weights. A review committee can override a rule for safety, but the override becomes a dated ledger event with reasons.

PAPER-38 includes observational repeated tracking and proposed future protocols; unrun protocols remain proposals. Its vendor-defined carry-forward and cohort choices are not imported as universal monitoring rules. PAPER-29 provides study-specific evidence about repeated observation, while its finite sample and exclusions remain attached.

## 7. Change points and measurement regimes

A change point is a boundary after which the old and new observations may not share a comparable measurement regime. It can be triggered by known state or by an outcome alert. Do not collapse those paths.

Known-state triggers include an explicit configuration change, alias remapping announced by the provider, interface or citation-surface revision, parser version change, query-bank revision, locale change, account-state change, or collection-mode change. The rule can rebaseline immediately because the recorded design object changed. Rebaseline means close the old regime, preserve it, start a new baseline, and prohibit unqualified pooling across the boundary.

Outcome alerts can use a prespecified descriptive threshold. For example, flag an absolute block-to-block primary-rate shift of at least 0.10 or a noncomplete proportion above 0.10. Such a flag says “investigate the measurement regime.” It does not say “the platform updated.” Seasonality, query mix, order, missingness, parser behavior, authored fixture design, or chance variation remain alternatives.

Change-point methods have tuning parameters, minimum history requirements, and false-alarm behavior. A short three-block fixture cannot validate a production detector. W13 therefore uses deterministic thresholds to teach governance, not to estimate detector performance.

After rebaseline, maintain both regimes in the archive. Report pre- and post-boundary estimates separately. If a scientifically justified bridge analysis is later performed, label its assumptions. Rebaseline is not permission to discard inconvenient earlier data.

Change alerts should include negative controls when a design has them. A parser-only metric, collection-latency measure, or fixture field known not to respond to the intervention can help reveal operational discontinuity. A negative control does not automatically identify the cause, and choosing it after the alert weakens its value. Freeze the control, expected relationship, threshold, and missing-state rule alongside the primary metric.

Decision latency matters too. Record when an alert fired, when evidence was preserved, when an owner acknowledged it, and when collection paused. A threshold with no response-time commitment can fail operationally even if its arithmetic is correct. The W13 fixture reproduces decisions immediately for instruction; it does not validate human response performance.

## 8. Continue, pause, rebaseline, stop, and rollback

Decision states need definitions and precedence.

**Continue** means the current authorized protocol and measurement regime remain valid, all hard gates pass, quality indicators remain below thresholds, and the next planned action may proceed. It does not mean the outcome is favorable.

**Pause** means temporarily suspend collection while preserving state and investigating a potentially recoverable operational problem: outage, rate-limit warning, missingness spike, unknown configuration state, parser anomaly, or approaching cost threshold. Pause has an owner, expiry, and resume criteria.

**Rebaseline** means a documented system or protocol change breaks comparability. Close the previous regime, start a new regime, and avoid unqualified pooling. Rebaseline can accompany pause.

**Stop** means end the authorized collection path because a hard gate failed or recovery conditions were not met. Authorization failure, privacy exposure, prohibited automation, evidence-preservation failure, persistent policy or rate-limit conflict, and cost-cap breach are stop conditions.

**Rollback** is a tested authorized action that restores or contains a specific prior approved state. It can accompany stop or a severe guardrail event. It names target version, affected artifacts, actor, verification, and follow-up. Rollback does not erase the incident or collected evidence that may be lawfully preserved.

Use precedence: a stop condition overrides continue, pause, and rebaseline; required rollback attaches to stop or containment. If no stop applies, pause overrides rebaseline and continue while diagnosis is active. If no pause applies but comparability changed, rebaseline. Otherwise continue. Never let a favorable outcome override authorization or privacy.

## 9. Cost, rate limits, authorization, and privacy are design fields

Collection cost belongs in the plan: maximum total, expected per block, warning fraction, remaining budget, approver, and action. Uncontrolled cost is not a statistical nuisance. Stop before exceeding an approved cap. If provider pricing changes, pause and re-estimate rather than hoping the run completes.

Rate limits are product constraints and harm controls. Do not bypass, distribute, or conceal automation to evade them. Log warning and denial states. A bounded retry rule must respect provider terms and cannot select preferred answers. Persistent rate-limit conflict triggers stop or redesign.

Authorization has subject, scope, method, dates, account, data, and revocation path. Access to a user interface does not automatically authorize automated collection. The W13 core needs none. Optional collection requires a separate approval record; absence or expiry is a stop condition.

Privacy review asks whether prompts, responses, account metadata, or logs contain personal or confidential information. Do not place secrets in the append-only ledger. If unexpected personal data appears, stop, preserve only what policy authorizes, restrict access, notify the responsible owner, and follow the approved incident process.

PAPER-12 and PAPER-23 illustrate controlled or reconstructed testbeds, not authorization for live calls. Their artifact and configuration limits remain explicit. PLAT-04 documents a named preview interface and date, not a data-collection license or cross-engine measurement standard.

## 10. Deterministic four-incident exercise

The incident overlay leaves LAB-L06 unchanged. It is authored metadata processed by a fixed precedence rule.

**INC-01 — quality within tolerance.** After B-01, four of 120 designed events are noncomplete, or 0.033333. The continue threshold permits a noncomplete proportion at or below 0.05 when hard gates pass. Decision: `CONTINUE`. This is a protocol action, not a favorable-outcome judgment.

**INC-02 — configuration regime change.** Before B-02, an authored release card changes the explicit configuration record while the dynamic alias still exposes no stable backend identity. No authorization or safety gate fails. Decision: `REBASELINE` at B-02. The old and new regimes remain separate. The B-02 alias/explicit mention difference of 0.092938 remains descriptive and does not identify a hidden backend cause.

**INC-03 — simulated outage.** In a supplied 20-cell mini-batch, six cells receive an authored unavailable state, proportion 0.30, above the pause threshold of 0.10. Authorization and privacy remain valid. Decision: `PAUSE`, preserve the interrupted order, diagnose, and resume only under a new ledger event.

**INC-04 — authorization and budget hard gate.** The supplied authorization expires before restart, and the projected 12-unit cost exceeds the approved remaining eight units. Decision: `STOP_AND_ROLLBACK` to snapshot-only mode. Unapproved rows are not admitted. No live request occurs.

The exercise includes a plan event, block event, release/deviation event, incident decisions, rollback verification, and closure. Hash-chain reproduction establishes ledger consistency, not real incident-response effectiveness.

## 11. Reading boundaries and bounded conclusion

PAPER-29 supports repeated-observation questions in its exact systems, language, verticals, dates, exclusions, and metrics. Its recommended counts are not universal, and same-window differences do not isolate token-generation stochasticity. PAPER-38 describes one vendor platform and unreleased commercial data; it is not independently reproduced, its cohorts and provider policies differ, and its future protocols are not findings.

PAPER-12 is a locked v2 fixed-candidate testbed: effects condition on cached candidate inclusion and do not establish live retrieval, sales, or future model behavior. PAPER-23 is a locked v2/KDD 2026 reconstructed pipeline: stage-specific findings remain conditional on selected targets, corpus, pipeline, and date. It does not disclose a commercial engine. PLAT-04 is bounded official interface documentation; aggregate preview counts are not rank or authority.

LAB-L06 establishes a deterministic synthetic panel: 360 events, 354 complete, six noncomplete, five distinct outcome fields, and descriptive Wilson intervals. SYN-A mention changes from 0.482759 in B-01 to 0.400000 in B-03, a descriptive difference of −0.082759. SYN-B changes from 0.534483 to 0.491525, or −0.042958. Neither identifies a mechanism.

The W13 conclusion is procedural: under the frozen synthetic rules, INC-01 continues, INC-02 rebaselines, INC-03 pauses, and INC-04 stops and rolls back. The ledger is reproducible. No named platform, hidden backend transition, causal intervention effect, production detector performance, user outcome, or business result is estimated.

Exit check:

1. Are system, alias/explicit status, time, session, account, locale, order, failure, and release fields recorded?
2. Are planned looks separated from hard-gate monitoring?
3. Does every change point state whether its cause is known or unknown?
4. Are thresholds and precedence frozen before incident outcomes?
5. Are authorization, privacy, rate limits, cost, and evidence preservation hard gates?
6. Is the ledger append-only and hash-linked?
7. Does rollback name an approved target and verification step?
8. Does the conclusion avoid inferring a hidden backend from alias differences?

If any answer is no, the monitoring decision is not ready for publication.