Generated from the same controlled README shipped in the offline package. Source SHA-256: ba995959c9afb901329a7394cab4570093e7848718e2e33ef2e77a86621e8e23.
Linked course weeks: W01–W02
Expected effort for the full assignment: 4 hours
Core path: offline, deterministic, Python standard library only
Research question
What can a second analyst actually reproduce from one answer-surface observation?
Learning objectives
By the end of this lab, you should be able to:
- distinguish a visible surface event from an inference about crawling, indexing, retrieval, ranking, generation, or training;
- preserve the system, surface, locale, account state, time, exact query, response artifact, and visible citations as separate fields;
- resolve every cited source to a stable registry identity;
- generate a hash-addressed observation package another analyst can audit; and
- write a bounded conclusion that names what the observation cannot establish.
data/observation.json follows ../shared/schemas/observation.schema.json. Required fields are:
observation_id, system, surface, locale, account_state, and an ISO-8601 observed_at value;- the exact
query and a relative response_path;
- unique
visible_citations that resolve in the supplied source registry; and
hidden_stage_statements, each labeled not_observed, hypothesis, or externally_supported.
The sample is a synthetic surface observation and deliberately includes a tempting retrieval inference labeled not_observed. Replace it only with an instructor-approved public or synthetic observation. Never store cookies, account identifiers, tokens, personal history, or unredacted screenshots.
Run
python3 scripts/audit_surface.py \
--observation data/observation.json \
--sources ../shared/fixtures/source_registry.csv \
--output /tmp/geo-l01
Output contract
The command creates:
Table from the controlled L01 lab handout| Artifact | Contract |
|---|
audit_report.json | status, hard errors, cautions, visible-citation count, and invalid-inference count |
system_card.json | normalized context plus SHA-256 of the frozen response artifact |
source_manifest.csv | one resolved row per visible citation |
run_manifest.json | input paths, hashes, parameters, and offline/deterministic flags |
The script refuses missing response artifacts, duplicate or unresolved citations, malformed timestamps, undeclared account states, and hidden-stage statements without an admissible status. A PASS means the package is internally reconstructable; it does not validate the answer or reveal a hidden mechanism.
Full assignment procedure
- Obtain permission for the observation surface and freeze one response artifact.
- Redact personal/session data before placing any artifact in the package.
- Record the observation fields without filling unknown fields from memory.
- Resolve visible citations by source identity, not just display label.
- List each tempting hidden-stage inference and assign an epistemic status.
- Run the auditor and correct contract failures without deleting inconvenient observations.
- Submit the generated artifacts, a screenshot/file manifest, and a 500-word boundary note.
- Ask a peer to reconstruct the context and locate every source from the package alone.
Low-compute and no-network route
Use the checked-in text snapshot. No browser, OCR, model, network request, or external package is needed. A real screenshot can remain outside the repository if privacy or licensing requires it; store a redacted manifest entry and hash instead.
Negative results and interpretation
- No citations is a valid observation. Record an empty list; do not invent sources.
- An unresolved displayed source is a finding. Record it in the boundary note and fix the registry only when identity is established.
- A peer who cannot reconstruct locale, account state, time, or response version has discovered a reproducibility failure.
- Surface evidence cannot establish that a source was crawled, indexed, retrieved, reranked, absorbed, or used for training.
Ethics and stop conditions
Stop if collection would require bypassing access controls, automated load, deception, personal data, secret recording, or violation of interface terms. Do not infer sensitive attributes from account state. Redact identifiers before hashing because hashes do not anonymize predictable personal data.
Acceptance
python3 scripts/audit_surface.py --observation data/observation.json --sources ../shared/fixtures/source_registry.csv --output /tmp/geo-l01
python3 -c 'import json; assert json.load(open("/tmp/geo-l01/audit_report.json"))["status"] == "PASS"'
See RUBRIC.md for grading.