1. The boundary is a governed judgment
Optimization becomes system abuse when an intervention crosses authorized scope, undermines evidence or system integrity, creates disproportionate risk, harms affected parties, or evades a control boundary. Intent alone is insufficient. A creator may intend to clarify while publishing a misleading implication. An adversary may use individually true statements to fabricate consensus. A defensive researcher may cause harm through an unauthorized “test.”
Use a seven-part boundary test: truth, provenance, disclosure, system integrity, competitive fairness, authorization, and proportionality. Truth asks whether explicit and implied claims are supported. Provenance asks whether origin and dependency are traceable. Disclosure asks whether material interests are visible where needed. System integrity asks whether content remains data rather than an attempt to control the consuming system. Competitive fairness asks about impersonation, fabricated independence, and unsupported harm. Authorization fixes what may be tested. Proportionality asks whether defensive evidence justifies risk and whether a less intrusive method exists.
This chapter deliberately omits operational details for live poisoning, evasion, credentials, unauthorized load, harmful payloads, cloaking, and third-party targeting. Scenario-class descriptions are sufficient for threat modeling and control design. Technical possibility is not permission.
2. Model the system before naming threats
A threat model begins with a versioned system, not a list of attack names. Record scope, components, data flows, assets, actors, affected parties, capabilities, entry points, trust boundaries, exclusions, and accountable owners. An asset is something whose compromise matters: source identity, claim-evidence lineage, privacy, accessible evidence, governance-status accuracy, answer fidelity, attribution, fair exposure, or user autonomy.
Actors differ from stereotypes. L07 names an authorized learner, an independent reviewer, and an untrusted corpus contributor. Capabilities are constrained permissions: contribute an inert record, execute a local validator, review a frozen output, or approve a course decision. Never infer that an actor can access credentials, modify configuration, contact a live target, or persist an executable artifact unless authorization explicitly says so.
Entry points are places where data or decisions enter: ingest, normalization, retrieval, annotation, and release. Trust boundaries mark changes in control or confidence. L07 has one boundary from the untrusted corpus to ingest and another from analysis outputs to release. Harm is then linked to an asset and affected party. A dependency cluster can distort a reviewer’s evidence count; a broken accessible relation can exclude an assistive-technology user; a collapsed standards label can mislead a decision maker.
The threat model is an evidence object. Give it a version and hash. If the pipeline, actors, or permissions change, the model is stale even if the diagram still looks plausible.
3. Capabilities and entry points define possible paths
Write capabilities as verb-object-scope triples: “submit one course-owned synthetic record to the inert corpus,” not “corpus access.” Add frequency, time, and approval constraints where relevant. This grammar prevents a read permission from silently becoming write access and a local fixture from becoming a live-service test.
For each entry point, ask four defensive questions. What may enter? Who can introduce it? Which checks occur before the next trust zone? What evidence proves the checks ran? Avoid describing how to defeat the checks. A threat model can state “source identity laundering” or “instruction/content boundary failure” without supplying a recipe.
Paths should terminate in harms and controls, not success tips. For example: an unresolved source identity reaches the claim graph, harming citation integrity; prevention is an allowlisted stable-ID registry, detection is a failed join, response is quarantine, the stop rule blocks a public claim edge, and rollback restores the prior registry snapshot and rebuilds the graph. This is a defensive control trace.
Mark unknowns. If coverage of a transformation is unknown, do not draw a protective shield. If a third-party subsystem is outside authorization, treat it as an external dependency and stop at its boundary.
4. Authorization is part of the method
Authorization is a concrete record, not a general belief that research is beneficial. It identifies the accountable owner, system and asset versions, allowed actors, exact capabilities, approved scenario classes, dates, locations, data classes, test methods, evidence retention, notification route, stop conditions, and rollback authority. It also lists exclusions.
The L07 authorization is narrow: course-owned synthetic files, local fixture ingestion, retrieval, annotation, and release review, with no external system or user traffic. Live answer engines, credentials, accounts, real personal data, and executable adversarial payloads are excluded. That scope makes the exercises auditable and reversible.
Authorization can expire. A prior approval does not automatically cover a new model, target, payload class, network, data source, or persistence mechanism. A toolkit’s ability to execute a check does not grant permission. G08 may inform test-matrix structure, assertions, and CI concepts, but the course does not install or connect it, and its generated scores would still need audited judges and denominators.
When authorization is missing or ambiguous, the correct decision is stop and use the synthetic route. Recording the stop is a research result, not administrative failure.
5. Proportionality selects the least intrusive test
Proportionality compares expected defensive evidence with potential harm. Ask whether the question is important and specific, whether an inert static record can answer it, whether the test is reversible, who could be affected, what uncertainty it reduces, and what lower-risk alternative exists. The minimum necessary scope is preferred.
A high-risk live test cannot be justified because it might produce interesting data. A synthetic case may be enough to test schema joins, control coverage, release rules, evidence preservation, or reviewer calibration. If no bounded method can answer the question safely, state the evidence gap.
Use a proportionality card with: defensive question, protected asset, proposed method, incremental evidence, affected parties, maximum plausible harm, reversibility, alternative methods, authorization owner, stop condition, and decision. Reassess when scope changes. The decision can be allow, sandbox, mitigate, stop, or disclose; it need not force every case into “test” or “ignore.”
Responsible disclosure is also proportional. Notify the party able to contain or correct the problem, preserve confidentiality, provide enough evidence to reproduce defensively, omit weaponizing detail, and follow applicable policy, contract, law, and qualified guidance. Public accusation is not the default.
6. Prevention is not detection
Prevention acts before the harmful transition: allowlists, stable identity, dependency registration, separation of source content from operator configuration, synthetic-only inputs, accessible structured alternatives, least privilege, and controlled status vocabularies. Evidence must show that the preventive gate ran on the relevant version.
Detection observes a signal after or during exposure. It has a target family, observation point, threshold, coverage, latency, and error profile. A case labeled detected in L07 was not necessarily prevented. AB-02 enters the retrieval review and is flagged; aggregate evidence is then withheld. AB-06 reaches a status review and is corrected before memo generation. Preserve this temporal difference.
Layering controls does not permit double counting. Two detectors trained on related features may share failures. A provenance check and a source allowlist may depend on the same registry. Document dependencies, common-mode failures, bypassed states, and manual review.
No control is a universal shield. A passed synthetic case covers the tested configuration and scenario only. Unknown threats, model drift, incomplete logs, implementation errors, and affected-party impacts remain residual risks.
7. Detector coverage needs a denominator
A detector report starts with the intended threat family and domain. Give the number of positive and negative examples, sampling route, ground truth process, threshold-selection data, held-out design, repetitions, and uncertainty. Report true positives, false negatives, true negatives, false positives, recall, false-positive rate, and the units of analysis. Add worst-family and worst-domain results.
PAPER-37 is core because it exposes the contrast between high results for some known manipulation templates and poor results for several new black-box families in the reported evaluation. One table reports an aggregate zero false-positive rate, while an independent small table reports a nonzero false-positive rate. The lesson is not which number wins. It is that sample, family, and denominator belong beside the rate.
PAPER-41 extends the detector question to GEO-optimized web-content style. Its benchmark results support evaluation within that dataset. A style label is not proof of falsehood, malicious intent, policy violation, or legal wrongdoing. Live citations lack a complete GEO ground truth, and metadata/coverage limitations remain. Detector output should route evidence to review, not trigger automatic public accusation or irreversible punishment.
Base rates matter. Even a low false-positive rate can produce many false alerts in a low-prevalence population. When prevalence is unknown, report conditional test behavior rather than positive predictive certainty.
Coverage is a vector rather than a single percentage. A useful detector card lists threat family, content domain, language, modality, model or parser version, transformation stage, sample route, and time. A high average can hide a family with near-zero recall. A strong text result says nothing about images unless cross-modal evidence exists. A result on known templates says nothing about an unseen strategy unless the held-out design tests it. Report abstentions and unscorable cases rather than forcing them into the denominator silently.
Threshold choice also belongs to governance. Lowering a threshold may raise recall and false-positive burden simultaneously. The cost of a false alert differs by action: routing a record to private review is not equivalent to publishing an accusation, suppressing a source, or permanently blocking an account. Connect the threshold to a proportional response. High-impact dispositions require stronger evidence, independent review, an appeal or correction path, and an accountable owner.
When labels depend on human judgment, record instructions, reviewer expertise, independence, disagreements, and adjudication. A model judge can assist but is not ground truth. Check calibration and subgroup burden. Do not select the threshold on the same cases used to advertise performance. Retain a genuinely held-out family or time slice and freeze the selection rule before evaluation.
8. PAPER-40 and evidence aggregation under controlled poisoning
PAPER-40 is core for a different object: fact-verification evidence aggregation under constructed GEO-style poisoning. It shows that increasing contamination can degrade reported verification performance and that no aggregator dominates across every attack condition in the study. The evidence supports a bounded robustness question, not an operational attack playbook.
The claim ceiling is strict. Configurations appear to have limited repetitions, and reporting of data dates, annotation agreement, implementation, and uncertainty is insufficient for exact universal robustness rankings. W14 therefore uses the paper to demand per-condition failure matrices, repeated runs, provenance of evidence items, independence analysis, and abstention rules. It does not reproduce poisoning mechanics.
Evidence aggregation must account for dependency. Several pages derived from one upstream source are not independent corroboration. Source-family clustering, provenance graphs, and leave-one-family-out sensitivity help identify synthetic consensus. The safe objective is not merely to retain a final label; it is to preserve which evidence items, families, conflicts, and unknowns drove it.
PAPER-13 provides a bounded citation-vulnerability perspective, while PAPER-07 provides bounded adversarial search evidence. Both motivate threat questions. Neither establishes that every production system shares the vulnerability or authorizes live manipulation.
9. Mechanisms and dynamics do not erase external validity
PAPER-34 studies persuasion-related internal pathways in controlled tasks and open models. It can motivate a mechanistic hypothesis about limited components in those settings. It does not prove a complete causal explanation of free-form generation or proprietary systems, and it does not authorize intervention on live models.
PAPER-11 models longer-run adversarial dynamics. Local theoretical results can be non-monotonic: reducing an attack-success parameter need not automatically increase cooperation under every assumption. Use this to teach parameter sensitivity, equilibrium assumptions, and unintended incentives. Do not turn it into a policy proof.
PAPER-39 studies recommendation-agent risks and defenses in a bounded benchmark. Its target rates, constraint violations, and utility outcomes belong to the evaluated cases, models, variants, and single-turn setting. It motivates hard-constraint monitoring and permission boundaries, not universal prevalence or a guaranteed defense.
Together, these extension sources expand questions across behavior, mechanism, dynamics, citation, recommendation, and detection. They remain a defense matrix, not one leaderboard.
10. Response begins with safe triage
Use five triage dispositions:
- Allow: the ordinary, authorized activity may continue because required controls are evidenced and residual risk is accepted by the accountable owner.
- Sandbox: the defensive question may be tested only in the declared inert environment, with frozen inputs and no external effect.
- Mitigate: repair a control or evidence gap before reassessment; do not treat mitigation as completed until tested.
- Stop: halt the current test, transition, or release when authorization, privacy, security, integrity, accessibility, proportionality, logging, or rollback fails.
- Disclose: initiate a scoped notification or disclosure assessment under policy and qualified review; preserve confidentiality and avoid unsupported public allegations.
These are not mutually exclusive over time. A case can be sandboxed, reveal a gap, require mitigation, stop release, and trigger owner notification. Record the sequence and decision rights.
Detector scores are inputs, not dispositions. A high score with poor family coverage may lead to sandbox review, not a public label. A confirmed exposed personal identifier leads to stop, containment, privacy-owner notification, and recovery regardless of a detector’s aggregate performance.
The word allow needs discipline. It means the specified activity may proceed under the current scope; it is not a declaration of zero risk. Record the accepted residual risk, owner, review date, monitoring trigger, and conditions that revoke the decision. Sandbox similarly names a bounded environment, not a safe content class. A sandbox requires isolation, synthetic inputs, output controls, logging, and cleanup evidence.
Mitigate is incomplete until the changed control is tested against the frozen case and relevant neighboring cases. A patch that fixes AB-05 might break another accessible relation or alter source identity. Preserve the before/after artifacts and run regression checks. Stop identifies what halts—the current test, one transition, or the whole release—and who can resume it. Disclose identifies the audience and basis: internal owner notification, coordinated vendor report, affected-party communication, regulator notice where applicable, or public correction. These paths have different evidentiary and legal requirements.
A safe triage record contains case ID, observed evidence, immediate harm, authorization status, control state, detector confidence and ceiling, chosen disposition, containment, preservation locator, notification owner, recovery criterion, residual risk, approver, and next review. It never embeds an executable payload. It can link to a restricted evidence store when qualified reviewers need more detail, while the course artifact retains only inert abstractions.
11. Preserve evidence before changing state
Incident response follows detect, preserve, triage, contain, correct, notify, recover, learn. Preservation precedes destructive cleanup. Record the exact artifact or answer, hashes, URLs or local identifiers, timestamps, locale, account and tool state, source versions, model/parser versions where exposed, control outputs, reviewer observations, and permissible logs. Minimize personal or sensitive content while retaining the evidence needed for the owner.
Containment prevents further harm: quarantine a record, withhold aggregate evidence, pause release, disable an unauthorized path, or restore an accessible artifact. Correction must propagate to dependent records, public pages, structured representations, caches within scope, course materials, benchmark labels, and errata. A local edit is not recovery if downstream copies remain wrong.
Notification identifies who must know, why, by when, and under which policy or applicable requirement. It may include the system owner, privacy owner, accessibility owner, evidence owner, release owner, or affected party. W14 does not provide legal advice; qualified review determines external duties.
Preservation has a ceiling. A hash proves identity, not truth. A log can be incomplete. A screenshot can omit state. Record missing evidence and chain-of-custody limits.
12. Rollback and recovery require verification
Rollback is an authorized restoration action with a trigger, owner, exact target, preserved pre-state, method, success criterion, verification evidence, and residual effects. “Restore the last snapshot” is incomplete without the snapshot identity, dependency rebuild, and post-restore checks.
In L07, each control includes rollback. CTRL-01 restores the prior registry and rebuilds the graph. CTRL-02 restores the pre-dedup run while retaining both audit views. CTRL-03 restores the clean corpus and parser configuration. CTRL-04 restores an authorized redacted snapshot. CTRL-05 restores the last accessible control artifact. CTRL-06 restores the verified crosswalk. These are course fixture specifications, not evidence that an external rollback works.
Recovery validation checks the protected asset, not only tool exit status. Confirm source joins, evidence independence, instruction/data separation, absence of unapproved identifiers, accessible claim-source relations, and accurate governance labels. Re-run relevant monitoring and verify that correction propagated.
Residual risk remains after recovery: unknown attacks, untested configurations, false negatives, false positives, stale copies, incomplete logs, inaccessible review, and future drift. The release owner either accepts a bounded residual risk with rationale or keeps the release blocked.
Test rollback before relying on it. A tabletop description can reveal missing owners and dependencies, but only an authorized synthetic exercise can provide restoration evidence. Define a recovery point and recovery time target where relevant, then inspect data integrity, access permissions, monitoring continuity, and downstream propagation. If the rollback erases incident evidence, the design is defective; preservation and restoration must coexist.
Recovery is not closure. Close an incident only after the corrective action is verified, affected owners have acknowledged the result, required notifications are complete, the detection or process gap has an owner, dependent artifacts are corrected, and residual risk is recorded. Reopening criteria should be explicit when later evidence changes the conclusion.
13. Frameworks, tools, and policies keep their status
G08 is an open-source evaluation and red-team toolkit candidate. It can inspire test matrices, assertions, and CI workflows. Tool execution does not validate the question, fixture, judge, threshold, or score. W14 uses no G08 network or live-target function.
S01, NIST AI Risk Management Framework 1.0, is a voluntary government framework. S02, the NIST Generative AI Profile, is voluntary cross-sector guidance. They can organize Govern, Map, Measure, Manage activities and risk/action categories. They are not laws, certifications, conformity assessments, or proof that a named control was implemented. Applicability requires context and status refresh.
The local course policy applies to course artifacts and labs. It prohibits live manipulation and requires evidence preservation, notification, tested rollback, recovery validation, and residual-risk recording. It is not external platform authorization or legal advice.
PAPER-37, PAPER-40, PAPER-07, PAPER-11, PAPER-13, PAPER-34, PAPER-39, and PAPER-41 are research routes with setting-specific evidence. G08, S01, and S02 have candidate roles. Public identifiers keep these namespaces visible; internal citation keys do not belong in learner-facing files.
14. L07 release reasoning and exit standard
L07 contains six inert cases: unresolved source identity, dependency amplification, instruction/content boundary failure, synthetic identifier detection, accessible claim-source loss, and governance-force collapse. Three outcomes are blocked, two detected, and one escaped. The deterministic audit status is PASS. Release is BLOCKED because AB-05 has high residual severity and likely residual likelihood after the supplied control.
This is the correct result. PASS describes execution of the audit contract. BLOCKED applies the release rule. The escaped accessibility case remains visible and cannot be averaged away. No general safety, compliance, certification, legal, or production-system conclusion follows.
A final W14 memo names the versioned system, authorization, assets, actors, capabilities, entry points, trust boundaries, harms, control evidence, detector coverage and false-positive limits, response owner, evidence preservation, notification decision, recovery validation, and residual risk. It assigns one of the five triage dispositions with rationale.
For the exit test, write three sentences. First, state what separates the proposed defensive test from abuse. Second, state why a detector score is not a release decision. Third, explain why L07 audit PASS and release BLOCKED are compatible. Use all controlled routes—PAPER-37, PAPER-40, PAPER-07, PAPER-11, PAPER-13, PAPER-34, PAPER-39, PAPER-41, G08, S01, and S02—without exceeding their roles.