Distinguish robust testing from harmful manipulation.
Select prevention, detection, response, and rollback controls.
PrerequisitesW11 agentic boundary.Responsible-use policy and synthetic-lab agreement.
Builds on
W11 agentic boundary. · Responsible-use policy and synthetic-lab agreement.
Investigates
When does optimization become system abuse?
Feeds
L07 adversarial review.
02
Before class
Arrive with a prepared artifact
Reading route
Core PAPER-37/PAPER-40; Extend PAPER-07/PAPER-11/PAPER-13/PAPER-34/PAPER-39/PAPER-41; candidates G08 and S01/S02.
Viewing route
Open the complete text-first lecture packageSeven-minute threat-model walkthrough. The current equivalent is notes, slide script, worked case, and no-video transcript; no recording is claimed.
Readiness check
Name asset, actor, capability, harm, and control for one synthetic case.
Bring
Initial threat model and prohibited-action list.
03
Explanation
Concepts, assumptions, and boundary
Optimization creates adversarial incentives
When visibility, citation, or recommendation has value, actors may manipulate source text, metadata, links, prompts, or retrieval corpora. Threat modeling names assets, actors, capabilities, entry points, trust boundaries, and harms before choosing a test. Defensive evaluation uses synthetic corpora and bounded cases. It does not deploy live poisoning, evasion, credential misuse, unauthorized load, or harmful payloads.
Controls form a lifecycle
Prevention includes authorization, input isolation, provenance, access controls, and safe defaults. Detection needs meaningful signals and known false-positive limits. Response defines triage, containment, evidence preservation, notification, and recovery. Rollback must be tested rather than merely promised. Detector scores never replace policy or human review, and residual risk remains explicit after controls are applied.
04
Primary visual
Inspect the mechanism or evidence structure
Threat modelThreat-model reasoning graph
Figure design. A six-node directed graph links asset, actor, trust boundary, attack path, control, and residual risk. A dashed edge marks the assumption-sensitive control relationship. All cases are synthetic; prevention, detection, response, and recovery are distinct.Long description
Six labeled nodes form a directed reasoning graph. Paths move from an asset and actor through a trust boundary and attack path toward a control and residual risk. The graph is a conceptual checklist for the synthetic cases; it does not encode attack counts, likelihoods, or measured control effectiveness.
05
Interactive check
One action, one feedback state
Action
Triage synthetic cases into allow, sandbox test, mitigate, stop, or disclose.
Feedback
Feedback cites authorization, harm, reversibility, evidence value, and disclosure considerations.
Accessible alternative
A printable case matrix and rationale reproduce the interaction.
Add an evidence-preservation and rollback test to the incident plan.
Low-compute route
All cases are static and synthetic; no model attack or external system access is required.
Five retrieval questions
01What separates defensive testing from abuse?
Authorization, bounded synthetic scope, proportionality, reversibility, harm controls, evidence need, and responsible disclosure.
02Why is a detector not a complete control?
Detectors have coverage and error limits and need policy, human review, response, and recovery procedures.
03What must rollback evidence contain?
Trigger, authorized owner, exact restoration action, preserved evidence, validation of restored state, and residual risk.
04What is the evidence boundary for W14?
Attack papers support bounded threat understanding. The course omits operational details that would enable live abuse and uses only synthetic fixtures.
05What must you submit or revise after this week?
L07 adversarial review. Add an evidence-preservation and rollback test to the incident plan.