# W05 Worked Case — Diagnosing Citation, Support, Absorption, and Use in L04

## Case status

This is a worked teaching diagnosis of the checked-in synthetic L04 Claim–Evidence–Source fixture. It is reproducible as a data-processing case: the inputs are local, the graph builder uses the Python standard library, and the declared run yields stable counts. The claims, source snapshots, and primary labels were authored for instruction. They have not received independent scholarly annotation at publication scale. Therefore the case supports reasoning about objects, invariants, denominators, and disagreements; it does not estimate the reliability of an annotation population or the quality of a public generative system.

The relevant course sources are `PAPER-24`, `PAPER-33`, and the extension `PAPER-13`. They motivate stage diagnosis, separation of selection from absorption, and adversarial scrutiny, respectively. No paper figure or prose is reused. `PLAT-04` appears only as a reminder that an official citation counter is a surface-specific measurement definition, not evidence of claim entailment or causal source use.

## 1. Frozen question and evidence ceiling

The case asks: **Which claims in five frozen responses are supported, contradicted, merely cited, unsupported, or plausibly source-distinctive?** It treats source presence, source use, absorption, displayed attribution, entailment, and answer correctness as separate objects. The frozen objects are:

- five synthetic responses, `R-01` through `R-05`;
- twenty-five primary claim records, `C-001` through `C-025`;
- eight synthetic sources and matching snapshots, `S-001` through `S-008`;
- ten independently double-coded claims; and
- six annotation dimensions: source identity, relation, entailment, citation correctness, source quality, and absorption.

Every response lists visible source identities. The case can therefore inspect whether an annotation edge refers to a visibly listed source. It cannot inspect a real retrieval candidate set, effective generator context, attention pattern, citation resolver inside a product, or answer-generation trace. The strongest causal source-use claim is deliberately out of scope. Even a `source_distinctive` absorption label means that distinctive verified content is present under the fixture codebook; it does not prove which hidden process produced the text.

Stop if a source identity cannot be resolved, a claim span changes between coders, or an annotation combination violates the declared schema. Mark substantive ambiguity unresolved rather than forcing agreement.

## 2. Input identities and run route

The command is:

```bash
python3 course-labs/L04_claim_evidence_source_graph/scripts/build_ces_graph.py \
  --responses course-labs/L04_claim_evidence_source_graph/data/responses.jsonl \
  --annotations course-labs/L04_claim_evidence_source_graph/data/annotations_coder_a.csv \
  --second-coder course-labs/L04_claim_evidence_source_graph/data/annotations_coder_b.csv \
  --sources course-labs/shared/fixtures/source_registry.csv \
  --snapshots course-labs/shared/fixtures/source_snapshots.jsonl \
  --output /tmp/geo-l04
```

The checked input hashes are:

| Input | SHA-256 |
|---|---|
| responses | `fd6ab45e5b54048c278a677c12397c4b23c0a70d44989f3e4771d11d03739502` |
| coder A | `394006259c7ad7c53e1ac020c9652729b7d35bc079db788862da9ee06979355c` |
| coder B | `dcc7df747689a6faec24eb6d318cea5abb2ab6137db54d1a94a7b5fd4908a58b` |
| source registry | `f8594d39b231365f26de6dfbbac8a6e2a2356934141e0f912e4fba0432ecacc8` |
| source snapshots | `caa73ad46da65973b90ca153f5ecea9956ce978cf43f2fe2ce6fc3cabe60c81d` |

The program validates each row before building the graph. Unsupported claims must have no source edge and must use non-applicable source/citation fields. Contradiction requires `entailment=contradicted` and `citation_correct=false`. Support requires a resolved source plus `entailed` or `partial`. Merely cited requires a visible source but no support. These are fixture rules, not a universal ontology.

## 3. Reconstruct one compound response

Response `R-04` reads as a series of statements about citation evaluation and a marketing claim. Its visible sources are `S-004` and `S-006`. Coder A segments it into five claims:

| Claim | Short proposition | Source | Relation | Absorption |
|---|---|---|---|---|
| C-016 | citation correctness requires entailment | S-004 | support | source-distinctive |
| C-017 | citation presence equals absorption | S-004 | contradiction | none |
| C-018 | all answer claims are completely cited | S-004 | merely cited | none |
| C-019 | source-distinctive absorption is automatic | S-004 | contradiction | none |
| C-020 | the marketing page says its method always ranks first | S-006 | support | source-distinctive |

The segmentation matters. If C-016 through C-019 were one compound claim, the support for the definition could mask two contradictions and one unsupported completeness assertion. If C-020 were rewritten as “the method always ranks first,” S-006’s marketing statement would not establish effectiveness. The meta-claim “the page says X” can be entailed by a low-quality source even when X itself remains unevaluated. That is the difference between passage entailment, source fit, and answer correctness.

## 4. Compute relation and coverage counts

The primary table yields exactly `14 support`, `4 contradiction`, `3 merely_cited`, and `4 unsupported` relation labels:

| Relation | Claim count | Citation correctness state |
|---|---:|---|
| support | 14 | true |
| contradiction | 4 | false |
| merely cited | 3 | false |
| unsupported | 4 | not applicable |

There are 25 claims total. Twenty-one carry resolvable source edges; the four explicitly unsupported claims carry none. Eleven claims enter the review report because they are contradicted, merely cited, or unsupported.

Do not call `14/25 = 0.56` a platform “accuracy rate.” The response set was intentionally constructed to contain instructional failure modes and has no target-population sampling frame. A bounded statement is: **56% of coder A’s frozen L04 claim rows have the relation label `support` under the fixture codebook.** That describes this table.

We can also illustrate metric denominators. In this fixture, every supported edge is marked citation-correct and every contradiction or merely-cited edge is marked false. Among the 21 displayed source-edge records, 14 are supported, so the unweighted edge correctness under this authored convention is `14/21 ≈ 0.667`. Among all 25 claims, 14 have at least one supporting visible edge, so unweighted completeness is `14/25 = 0.56`. The figures differ because the first denominator excludes unsupported claims without an edge. Neither evaluates current truth outside the snapshots.

## 5. Diagnose C-018: edge placement changes the label family

Claim C-018 says that all answer claims are completely cited. Coder A retains a source edge to S-004 and labels it `merely_cited`: S-004 is visible, but its general annotation guidance does not supply response-level evidence that every claim is covered. Coder B assigns no source and labels the claim `unsupported`. This single decision produces five dimension-level differences:

| Dimension | Coder A | Coder B |
|---|---|---|
| source ID | S-004 | empty |
| relation | merely_cited | unsupported |
| citation correctness | false | not applicable |
| source quality | medium | not applicable |
| absorption | none | not assessed |

Both coders agree that the claim is not supported. They disagree about the placement policy: does the visible S-004 citation create a claim–source edge even when it only provides general topic context? A defensible adjudication must consult the frozen interface-placement rule. If the anchor is locally attached to C-018, retain `merely_cited`. If S-004 appears only in a global source list with no published resolver mapping it to the claim, prefer `unsupported` or create a separate “visible at response level” relation. The case should not decide this by choosing the label that improves a metric.

## 6. Diagnose C-020: support does not settle absorption

Both coders agree that S-006 entails the meta-claim that the marketing page states its method always ranks first. Both assign low source quality and `citation_correct=true` for that report-about-page relation. They disagree only on absorption. Coder A labels the claim `source_distinctive`; coder B labels it `plausible`.

This disagreement asks whether the wording is distinctive enough and whether plausible alternatives were ruled out. The exact marketing superlative may be uniquely present in S-006 within the synthetic registry, but similar claims are common in promotional materials. The annotation guide needs an alternative-source test: search the frozen registry for equivalent statements, identify derivative origins, and decide whether “source-distinctive” refers to wording, proposition, or both. If uniqueness is established only inside a tiny registry, the final label must retain that scope.

Notice what agreement would still not prove. Even a source-distinctive label plus visible S-006 does not show a generator causally used S-006. A controlled context comparison or another justified influence design would be needed. The answer could reflect a duplicate, parametric knowledge, or an authored fixture decision.

## 7. Read agreement by dimension

The double-coded subset contains ten claims. The reproduced raw agreement and Cohen’s kappa values are:

| Dimension | Raw agreement | Cohen’s kappa |
|---|---:|---:|
| source ID | 0.90 | 0.880952 |
| relation | 0.90 | 0.848485 |
| entailment | 1.00 | 1.000000 |
| citation correctness | 0.90 | 0.841270 |
| source quality | 0.90 | 0.803922 |
| absorption | 0.80 | 0.714286 |

The correct interpretation is descriptive: under the frozen ten-claim subset, the two authored coder files agree most on entailment and least on absorption. The sample is too small and deliberately selected for instruction. Kappa is influenced by class prevalence. These values do not establish population reliability, multilingual validity, or expert consensus. Their useful contribution is locating codebook pressure: placement for C-018 and distinctiveness for C-020.

The program prints exactly: `L04 PASS: 25 claims, 10 double-coded, 6 label disagreement(s)`. “PASS” means row invariants and output construction succeeded. It is not a peer-review result.

## 8. Design the missing source-use test

Suppose we want to test whether S-006 contributed to C-020 in a controlled generator. Construct two context conditions. Condition A includes the frozen ordered context with S-006. Condition B removes S-006 but leaves a same-length empty spacer, preserves every other passage and position, uses the same query and instructions, freezes the model/checkpoint, and applies a predeclared paired repetition schedule. Do not insert a replacement unless replacement is itself part of the treatment definition.

Predeclare outcomes: faithful presence of the exact promotional meta-claim; unsupported adoption of the superlative as fact; qualification language; and answer correctness under the external evidence policy. A positive difference in faithful meta-claim presence would support conditional contribution under this constructed context. A positive difference in unsupported superlative adoption might indicate harmful contribution. A null difference remains compatible with redundancy, memorized content, insensitive outcomes, or noise.

This ablation cannot be run on the L04 fixture because no generator or effective context exists. That absence is an evidence boundary, not a deficiency to conceal. The present case diagnoses annotations; the proposed experiment tests a different estimand.

## 9. Bounded conclusion

The frozen L04 route reproduces five responses, twenty-five primary claims, eight sources, ten double-coded claims, and six dimension-level disagreements. Coder A assigns fourteen support, four contradiction, three merely-cited, and four unsupported relations. The C-018 disagreement concerns whether visible general guidance creates a claim-level edge; the C-020 disagreement concerns the alternative-source threshold for source-distinctive absorption. These findings justify revisions to placement and distinctiveness rules.

The case does not establish hidden retrieval, context selection, generator attention, causal source use, public-platform citation policy, general annotation reliability, or answer quality in a target population. It does not turn `PAPER-24`, `PAPER-33`, `PAPER-13`, or `PLAT-04` into universal mechanisms. Any broader claim requires a new design with a declared population, system state, independent labels, repeated observations, and appropriate controls.

## 10. Learner deliverable and self-check

Submit a one-page diagnostic memo containing:

1. the frozen input identities and hash status;
2. one segmented response with exact claim IDs;
3. separate entailment and completeness denominators;
4. a C-018 adjudication or unresolved decision with codebook rationale;
5. a C-020 alternative-source test;
6. one controlled source-use experiment that names at least six held-fixed variables; and
7. a two-sentence bounded conclusion.

Self-check every sentence:

- Did I call a visible source “used” without a contribution test?
- Did I call topical overlap entailment?
- Did I treat a source-quality label as truth?
- Did I hide an unresolved citation or coder difference?
- Did I report a denominator and unit for every rate?
- Did I preserve the synthetic, frozen, unpiloted boundary?

If any answer is yes, revise. The desired result is not an impressive score. It is a graph in which every observation, judgment, uncertainty, and unsupported inference is attached to the correct object.