Specify numerator, denominator, unit, and aggregation for a metric.
Separate within-time variation from drift.
Report intervals, missingness, and sensitivity to query mix.
PrerequisitesW06 query bank.Bridge note on proportions, bootstrap intervals, and clustered observations.
Builds on
W06 query bank. · Bridge note on proportions, bootstrap intervals, and clustered observations.
Investigates
When is a visibility change repeatable?
Feeds
L06 protocol dry run.
02
Before class
Arrive with a prepared artifact
Reading route
Core PAPER-29/PAPER-33; Extend PAPER-38; candidate synthesis R07 and interface case PLAT-04.
Viewing route
Open the complete text-first lecture packageSeven-minute metric-card construction and denominator audit. The current equivalent is notes, slide script, worked case, and no-video transcript; no recording is claimed.
Readiness check
Rewrite a vague “visibility increased” statement as a complete metric specification.
Bring
One metric card with formula, unit, denominator, time window, and uncertainty.
03
Explanation
Concepts, assumptions, and boundary
Metric names are not specifications
Mention rate, citation rate, share, rank, and visibility score can hide incompatible denominators. A metric card must state the event counted, eligible unit, repeated-run rule, query weighting, missing-response treatment, aggregation, and uncertainty method. Mention, citation, entailment, absorption, and referral belong to different stages and should remain separate before any justified composite is considered.
Variance and drift occupy two time scales
Repeated runs close together reveal stochastic or session-level variation; observations across time can reveal platform, corpus, or query-population drift. Pooling them without a time model can make intervals too narrow or changes look stable when they are not. Resampling must respect clustering by query and time block. Missing responses are not automatically zero because failure, refusal, outage, and unavailable citation UI may represent different mechanisms.
04
Primary visual
Inspect the mechanism or evidence structure
Conceptual modelMetric ladder with two time scales
Figure design. Five event/eligible-unit rows separate mention, citation, entailment, absorption, and referral. Brackets distinguish within-block repetitions from between-block drift; no values or confidence intervals are encoded. This is a denominator map, not an estimate. Within-block repetition and between-block drift require different intervals.Long description
Five separate event metrics occupy distinct rows. Three equal circles represent repeated observations without assigning values. Two labeled brackets distinguish nearby repetitions from comparisons across time blocks. The figure is a denominator and sampling map, not observed data.
05
Interactive check
One action, one feedback state
Action
Vary repetitions, query weights, time blocks, and missing-data rules on a frozen panel.
Feedback
Intervals and estimates update with a warning whenever independence or denominator assumptions change.
Accessible alternative
A static sensitivity table reports every predefined configuration.