Generated from the same controlled README shipped in the offline package. Source SHA-256: 5bd3cde92ca7fde1b3f02d6637187dfb495ac08925b07d313a8d61a3c7e44090.
Linked course weeks: W02 and W05
Expected effort for the full assignment: 6 hours
Core path: offline, deterministic, Python standard library only
Research question
Which answer claims are supported, contradicted, merely cited, unsupported, or plausibly source-distinctive?
Learning objectives
You will learn to:
- segment compound answer text into atomic, auditable claims;
- keep citation presence, entailment, citation correctness, source quality, and absorption as separate labels;
- represent support, contradiction, merely-cited, and unsupported states as an explicit CES graph;
- double-code a subset, compute agreement, and preserve disagreements before adjudication; and
- produce an unsupported-claim report with a defensible evidence ceiling.
data/responses.jsonl contains five frozen synthetic responses with stable IDs and visible-source lists.data/annotations_coder_a.csv contains 25 atomic claims and follows ../shared/schemas/ces-edge.schema.json.data/annotations_coder_b.csv independently double-codes a ten-claim subset.../shared/fixtures/source_registry.csv and source_snapshots.jsonl resolve every non-empty source_id.
Key invariants:
unsupported claims have no source edge and use not_applicable source/citation labels;contradiction requires entailment=contradicted;support requires a resolvable source and entailed or partial entailment;- a merely visible citation is not silently converted into support; and
- each claim ID is unique within the primary annotation table.
Run
python3 scripts/build_ces_graph.py \
--responses data/responses.jsonl \
--annotations data/annotations_coder_a.csv \
--second-coder data/annotations_coder_b.csv \
--sources ../shared/fixtures/source_registry.csv \
--snapshots ../shared/fixtures/source_snapshots.jsonl \
--output /tmp/geo-l04
Output contract
Table from the controlled L04 lab handout| Artifact | Contract |
|---|
ces_graph.json | typed response, claim, and source nodes plus containment/evidence edges |
annotation_summary.json | relation counts, support coverage, source-quality counts, agreement, and evidence ceiling |
unsupported_claims.csv | unsupported, contradicted, and merely-cited claims requiring review |
disagreement_log.csv | every differing double-coded dimension before adjudication |
relation_matrix.csv | relation × citation-correctness counts for a static diagnostic figure/table |
run_manifest.json | hashed inputs and offline/deterministic flags |
Raw agreement and Cohen's kappa are descriptive diagnostics for the double-coded subset. Ten claims are too few for a strong reliability claim; report the sample size and class distribution.
Full assignment procedure
- Freeze five approved responses and lawful source snapshots or links plus hashes.
- Segment at least 25 atomic claims before judging them.
- Resolve source identity, then label relation, entailment, correctness, source quality, and absorption independently.
- Have a second annotator code a predeclared subset without seeing the first labels.
- Run the graph builder and inspect contract failures.
- Review every row in the disagreement log; preserve pre-adjudication labels.
- Adjudicate with a written rule change or case-specific rationale.
- Submit the graph, annotation table, disagreement log, unsupported-claim report, and one diagnostic figure.
Low-compute and no-network route
All response and source snapshots are synthetic and checked in. Graph construction, agreement, and diagnostics use no model, browser, database, or graph package. Students may draw the diagnostic figure from relation_matrix.csv with any offline tool; the table itself is the accessible alternative.
Negative results and interpretation
- Low agreement is a valid indication that the codebook or claim boundary needs revision.
- Many unsupported claims may reveal an answer-quality problem or an overly coarse segmentation rule.
- A source can entail a statement yet remain low-authority evidence for a scientific conclusion.
- A cited source can contradict the claim it is attached to.
- Absorption cannot be established merely from citation presence or lexical overlap;
plausible remains weaker than source_distinctive.
Ethics and stop conditions
Do not redistribute source snapshots without permission. If redistribution is not lawful, store a resolvable link, access date, and content hash rather than the text. Redact personal data before annotation. Do not ask annotators to infer protected attributes or adjudicate brand accusations without source and legal review. Stop when source identity cannot be resolved.
Acceptance
python3 scripts/build_ces_graph.py --responses data/responses.jsonl --annotations data/annotations_coder_a.csv --second-coder data/annotations_coder_b.csv --sources ../shared/fixtures/source_registry.csv --snapshots ../shared/fixtures/source_snapshots.jsonl --output /tmp/geo-l04
python3 -c 'import json; s=json.load(open("/tmp/geo-l04/annotation_summary.json")); assert s["claim_count"] == 25 and s["double_coded_count"] == 10'
See RUBRIC.md.