# W03 Transcript Equivalent — Discovery, Indexing, and Retrieval

## Status and use

No W03 recording exists. This document is a stand-alone, readable/recordable equivalent planned for approximately 35–45 minutes at a clear instructional pace, including short pauses. The chapter durations below are **instructional planning budgets**, not timestamps from media and not evidence from a rehearsal, classroom, or learner pilot. All scientific propositions required for the W03 core route appear in text. If audio or video is later produced, its actual duration, captions, synchronization, audio description, and chapter timing must be measured from that artifact.

The transcript uses verbal descriptions of every essential visual. A listener can pause at each “learner check,” write an answer, and continue. Equations are read in words as well as shown symbolically elsewhere in the package.

## Planned chapter budget

| Chapter | Topic | Planned spoken/reading budget |
|---:|---|---:|
| 1 | Eligible is not retrieved | 4 minutes |
| 2 | One funnel, several denominators | 4 minutes |
| 3 | Discovery, fetch, representation, index | 5 minutes |
| 4 | Inverted-index intuition | 4 minutes |
| 5 | BM25 bridge and micro-calculation | 6 minutes |
| 6 | Qrels, Recall@k, and the candidate ceiling | 5 minutes |
| 7 | Supplied opaque and hybrid routes | 4 minutes |
| 8 | L03 topic-level case | 5 minutes |
| 9 | Open-sandbox evidence and closed-platform uncertainty | 4 minutes |
| 10 | Bounded conclusion and handoff | 3 minutes |

Total planned route: 44 minutes including the stated checks and pauses. A continuous read with no pauses may be shorter. The package makes no claim that this budget has been achieved in delivery.

## Chapter 1 — Eligible is not retrieved

Welcome to W03: discovery, indexing, and retrieval. The essential question is, “Can the system obtain the source at all?” That question comes before any claim about ranking, citation, or downstream action.

Imagine a public page. It returns a successful response to an authorized request. It appears in a publisher’s sitemap. Its content is clear to a human reader. Now imagine that a generated answer does not cite it. What have we learned? We have learned something about the page and about one visible response. We have not learned which internal stage excluded it.

The page might not have been discovered by the external system. It might have been discovered but not fetched under that system’s policy and schedule. A fetched representation might have been parsed differently from what a browser shows. A representation might be eligible for one index but absent from the index used for this surface, locale, or time. It might be indexed but fail to enter the candidate cutoff for this query. It might enter the candidate set and rank below a later selection budget. It might enter context yet make no contribution to generated text. It might contribute but receive no visible attribution. The final omission does not identify which transition failed.

For W03 we use a stage vocabulary: discover; fetch; parse and represent; index; retrieve; and select. Later weeks examine generation, attribution, absorption, and user outcomes. These labels are an analytical scaffold. They help us specify what a test observes. They are not a diagram of every commercial system.

The distinction also tells us how to repair a vague question. “How do we improve AI visibility?” lacks an event, population, query distribution, surface, time, comparator, and evidence boundary. A bounded W03 question could be: “In the immutable L03 corpus, under a named tokenizer and BM25 configuration, does labeled-relevant document D-010 enter the top three for topic T-03?” That question is narrow enough to answer and broad enough to expose a candidate miss.

Here is the first learner check. Pause and write two sentences. First: one observation supported by a successful fetch in an owned test. Second: one later-stage event the fetch cannot establish. A good pair would be: “The declared request obtained these bytes at this time.” And: “That record does not establish membership in an external platform’s index.”

The key phrase is conditional access. Every stage may create an opportunity for the next, but no earlier label silently proves a later event.

## Chapter 2 — One funnel, several denominators

Picture a stepped funnel. At the top are one thousand URLs in a declared source population. A controlled crawler discovers seven hundred sixty. It attempts seven hundred authorized requests. Six hundred fifty produce usable responses. Six hundred ten become parsed representations. Five hundred seventy are admitted to a named index snapshot. For one query, a retriever returns one hundred candidates. A selector uses eight passages.

The funnel narrows, but its ratios are not interchangeable. Seven hundred sixty divided by one thousand can describe discovery coverage only if the one-thousand-URL population was frozen independently. Six hundred fifty divided by seven hundred can describe usable responses conditional on attempted requests. Five hundred seventy divided by six hundred ten can describe index admission conditional on a parsed representation. The final eight passages are conditioned on one query, one candidate list, one selection rule, and perhaps a passage rather than document unit.

It would be misleading to call eight divided by five hundred seventy a universal selection rate. Most indexed documents may have no relation to the query. The selector may operate on passages created after retrieval. One source may generate several passages. The denominator changes from corpus representations to query-conditioned candidates.

The reasons for loss also matter. A source excluded because acquisition was unauthorized should not be grouped with a source that returned a network error. A response rejected because its media type is unsupported differs from a parsed document excluded by a quality or duplication rule. A document missing from top three differs from a document never indexed. A funnel should therefore carry a ledger beside each count: unit, conditioning set, time, version, inclusion rule, exclusion reason, and whether the number is observed or unknown.

Closed products create a special temptation. A researcher observes one cited source and imagines a hidden denominator: perhaps one of one hundred candidates, or one of ten thousand indexed pages. Unless the system exposes those sets, both denominators are invented. Repeated response collection can estimate how often a visible event occurs in the sampled protocol. It does not reveal the hidden candidate universe.

Learner check: use the funnel counts and reject one invalid statement. For example, reject “The content has a 1.4 percent retrieval rate because eight of five hundred seventy indexed documents were selected.” The eight are passages under one query and a later selector; the statement changes unit, stage, and conditioning set.

The denominator is not clerical detail. It defines the scientific object. If you cannot name it, report the count and mark the intended ratio unresolved.

## Chapter 3 — Discovery, fetch, representation, index

Discovery means obtaining an address or source identity that could be considered. Possible signals include lawful links, submitted maps, feeds, curated lists, repositories, or prior authorized captures. A sitemap entry is a publisher-side discovery aid. It does not prove that a third-party system fetched or indexed the URL.

Fetching is an attempted acquisition of bytes under a request identity, time, policy, and authorization boundary. A responsible record includes the address, request configuration, relevant crawler or user-agent identity, response, redirects, content type, locale, authentication state when lawful, capture time, and content hash. “HTTP two hundred” alone is not the complete object. Content negotiation can vary the response. A client-rendered section may not exist in raw HTML. Caches and redirects can alter what is acquired.

Representation is the step many high-level diagrams omit. Acquired bytes become fields, tokens, passages, metadata, link edges, or another indexable unit. A parser may remove navigation, select canonical content, detect language, segment a long document, truncate a field, or consolidate duplicates. These choices can create a retrieval mismatch even when the human-visible page contains the desired phrase. In an open system, preserve the parser version and output hash. In a closed system, do not claim to know the representation choice from a public answer.

Index eligibility and index membership are also different. Eligibility means a representation satisfies declared conditions for possible admission. Membership means the representation is present in one named snapshot or partition. Indexes are versioned. They can be rebuilt, delayed, filtered, language-specific, or scoped to a product surface. A page can be indexed in one place and absent from another.

Retrievability adds a query and configuration. An indexed document might receive no score because none of the query terms survive tokenization, a filter removes it, or a cutoff is shallow. “Indexed” therefore cannot stand in for “retrieved.” Even “retrieved” needs a candidate depth: rank five is retrieved at depth ten but absent from top three.

Let us classify five statements. A sitemap entry is a discovery signal. A successful response captured by your own authorized crawler is a fetch observation. Removing navigation boilerplate is a representation decision. A local manifest listing D-007 is an index observation for that snapshot. A public answer omitting the page does not reveal a closed-platform stage; it is an observed surface omission with multiple possible explanations.

Learner check: complete this identity record for an owned index. You need a source or capture identity, acquisition method and time, rights, content hash, parser version, document or passage unit, tokenizer, index build identity, and exclusion reason. If a field is unknown, write `unknown`. Do not fill it from intuition.

The research advantage of this precision is diagnosis. A discovery experiment asks whether source identities enter an acquisition frontier under a policy. A representation experiment changes parsing under a frozen source. A retrieval experiment changes candidate scoring under a frozen corpus and qrels. These designs answer different questions.

## Chapter 4 — Inverted-index intuition

Now we enter the inspectable retrieval sandbox. An inverted index maps tokens to documents that contain them. Imagine three documents. Document A says, “retrieval trace retrieval.” Document B says, “citation trace.” Document C says, “retrieval boundary.”

The postings list for `retrieval` is A with frequency two and C with frequency one. The postings list for `trace` is A with frequency one and B with frequency one. The document frequency of `retrieval` is two, because two documents contain the token. Its term frequency in A is two, because A contains two occurrences.

At query time, a lexical retriever can visit postings for query terms rather than scan every document. It then combines evidence about how rare a term is in the corpus, how often it occurs within a document, and how document length compares with the corpus average. This trace makes exact lexical evidence visible.

Tokenization determines the postings. Consider `AI-search`, `AI search`, a full-width script form, and the Chinese phrase for generative search. A tokenizer may preserve a hyphen, split it, normalize case, use word pieces, segment Chinese characters, apply stemming, or separate fields. None of those choices is universally correct. The method must match language and task, and comparisons must record it.

L03 uses a transparent English-oriented regular expression. It keeps letters and digits, permits some internal hyphens and apostrophes, lowercases matches, and does not claim multilingual adequacy. The simplicity is a learning advantage because every token can be inspected. It is an external-validity limit because a multilingual or production study may require a different representation.

Candidate failure can now be classified. A relevant source can be absent from the corpus. Its relevant passage can be removed during representation. Its query and document tokens can mismatch. Its score can fall below the cutoff. These are not the same as a later ordering error among already recovered candidates.

Learner check: create the postings entry for `boundary` in the three-document example. It should contain only C with frequency one. Then answer: does that posting establish that C is authoritative about boundaries? No. The index stores lexical occurrence, not truth, expertise, or entitlement to support a claim.

This is why a lexical baseline is useful in W03. It reveals which token evidence produced access. It is not included because every production system uses it or because exact words are always best.

## Chapter 5 — BM25 bridge and micro-calculation

BM25 is a family of lexical scoring conventions. We will use the same positive-IDF convention as the Core Notes and L03. For each query term, define `N` as the number of corpus documents, `n sub t` as the number of documents containing term t, `f of t comma d` as the term frequency in document d, document length as the token count, and average document length across the frozen corpus.

The inverse document frequency is the natural logarithm of one plus the fraction: N minus n sub t plus one half, divided by n sub t plus one half. In symbols:

`IDF(t) = log(1 + (N - n_t + 0.5) / (n_t + 0.5))`.

The BM25 term contribution multiplies that IDF by a saturated term-frequency fraction. Its numerator is term frequency times `k1 plus one`. Its denominator is term frequency plus `k1` times the quantity `one minus b plus b times document length divided by average document length`. Contributions are summed over unique query terms in the L03 implementation.

What do the controls mean? As a query term appears in more corpus documents, the stated positive IDF falls. That is corpus rarity, not world importance. As term frequency increases inside one document, the contribution rises with diminishing returns. Parameter `k1` controls saturation. Parameter `b` controls the effect of document length. With `b` equal to zero, there is no length normalization in that factor. L03 uses `k1` equal to 1.2 and `b` equal to 0.75.

Let us reproduce the Core Notes micro-example. There are three documents with average length one hundred. One query term appears in two documents. The positive IDF is the logarithm of one plus one point five divided by two point five, approximately zero point four seven zero.

Document A has length one hundred and contains the term three times. Document B has length fifty and contains it once. Substituting the stated parameters gives a term contribution of approximately zero point seven three eight for A and zero point five nine one for B. A ranks above B under this exact convention.

Now state what the result does not say. It does not say A is factual. It does not say A has better provenance. It does not say a generator will cite A. It does not say repeating the term is a responsible content intervention. BM25 has no variable for truth, source independence, accessibility, or user outcome.

The implementation manifest matters. Record the tokenizer, corpus and index snapshot, title or body fields, repeated-query-term handling, IDF convention, log base, average-length scope, `k1`, `b`, tie policy, cutoff, and score precision. Raw scores are normally interpreted within one query and locked index. Comparing raw BM25 scores across unrelated queries can be misleading because different terms have different corpus frequencies.

Here is a second learner check. Suppose `n sub t` increases while `N` stays fixed. The positive IDF falls. Does that prove the term matters less in the real world? No. It only says the token appears in more documents in this index snapshot under this tokenizer.

A strong baseline statement sounds like this: “Under the declared L03 corpus and implementation, document A receives a higher lexical score than B for query q, and the difference is traceable to these term frequencies and lengths.” It does not support the broader assertion that A necessarily ranks better in an external AI-search product.

## Chapter 6 — Qrels, Recall@k, and the candidate ceiling

A qrel is a query–document relevance judgment record. It normally includes a topic identity, document identity, and relevance value. The assessor instructions, population, unit, pooling method, date, and adjudication determine what the value means. A qrel is not timeless ground truth.

The L03 fixture has three topics and seven positive graded qrels. Because the corpus is a tiny closed teaching object, the lab treats unlisted pairs as gain zero. That policy is explicit. In an open collection, unjudged often means unknown rather than irrelevant, so you would need a different evaluation treatment.

Candidate Recall at k asks what fraction of the declared positive relevant set appears among the top k retrieved candidates. The numerator is the number of top-k documents that are relevant. The denominator is the total number of positive relevant documents for that topic.

For T-03, the relevant set is D-005, D-006, and D-010. The BM25 top three are D-005, D-006, and D-008. Two of three relevant documents are recovered, so Recall at three is two thirds, or approximately zero point six six seven. D-010 is a candidate miss at that cutoff.

Candidate membership creates a downstream ceiling. If a reranker can only reorder those three BM25 candidates, it cannot promote D-010 because D-010 is not present. The reranker may change the positions of D-005, D-006, and D-008. It may change a position-sensitive metric. It cannot make Recall at three exceed two thirds without a different or deeper candidate set.

Recall ignores order among the recovered relevant documents. NDCG uses graded gains and discounts lower ranks under a declared convention. W04 gives ordering and context selection more attention. For W03, ask first whether the relevant evidence was available to the next stage.

Aggregation also needs a rule. L03 macro recall averages topic values, so each topic contributes equally. T-01 and T-02 each have Recall at three equal to one; T-03 has two thirds. The macro average is eight ninths, approximately zero point eight eight nine. The mean alone hides the T-03 miss, which is why the topic error table is required.

Learner check: a fixed-set reranker raises NDCG at three but leaves Recall at three unchanged. What changed? Ordering among existing candidates. What did not become possible? Recovery of a relevant document absent from that set.

One more boundary: complete candidate recall does not imply answer quality. T-02 will show that all relevant fixture documents can be present while a gain-zero promotional record occupies rank one. Retrieval metrics measure their declared stage. They do not measure whether a generator makes faithful claims.

## Chapter 7 — Supplied opaque and hybrid routes

Dense retrieval represents queries and documents as vectors and scores their similarity. It can help with paraphrase and conceptual match. Its identity includes model and checkpoint, pooling, truncation, normalization, language, domain, and index build. Similarity is not entailment and not source authority.

L03 contains a file named `frozen_dense_run.tsv`, but its metadata imposes a strict claim ceiling. The scores are authored synthetic teaching values. The model identity is `none`. We therefore call it the supplied opaque run. We may say which document appears at which rank in that file. We may not say a dense encoder achieved the result. No encoder was executed to create the fixture.

Hybrid retrieval combines score or rank streams. The streams can have different scales, so raw addition requires justification. L03 normalizes the BM25 and supplied opaque scores separately within each topic, then interpolates them with hybrid alpha equal to zero point five five. That is one transparent protocol choice. Other systems might calibrate scores, learn fusion, or use reciprocal-rank fusion.

Fusion is not automatically better. It can add a candidate that one component missed while degrading the order of another topic. The proper analysis compares per-topic lists and preserves negative or unchanged outcomes. A favorable macro average is not permission to hide a failure.

The transparent L03 reranker operates inside the fixed hybrid top-five set. It rewards query-token and adjacent-token coverage. It is an authored teaching function, not a learned production reranker. Its candidate-set invariance lets learners separate membership from ordering.

Learner check: correct this sentence—“The dense model improves production recall by eleven percent.” A defensible version is: “In the three-topic synthetic fixture, the supplied opaque list has macro Recall at three of one, compared with BM25 at approximately zero point eight eight nine; because the opaque values are authored and no model identity exists, the difference supports only a list and metric exercise.”

## Chapter 8 — L03 topic-level case

Let us inspect the actual deterministic fixture. It contains ten CC0 synthetic documents, three authored topics, seven positive graded qrels, and the supplied opaque score file. BM25 uses the title plus body, simple lowercase English-oriented tokenization, positive IDF, `k1` 1.2, and `b` 0.75. We evaluate at cutoff three.

For T-01, BM25 ranks D-001 first, D-004 second, and D-002 third. D-001 and D-002 are the two positive relevant documents. Recall at three is one. Because the gain-three document comes first and the gain-two document comes third, NDCG at three is approximately zero point nine five six.

For T-02, BM25 ranks D-009 first, D-003 second, and D-004 third. D-003 and D-004 are the two relevant documents, so Recall at three is one. But D-009 is an unsupported promotional record with gain zero in the fixture. It occupies rank one, so NDCG at three is approximately zero point six six five. Candidate coverage is complete; ordering is the problem.

For T-03, BM25 ranks D-005, D-006, and D-008 in the top three. The relevant set is D-005, D-006, and D-010. Recall at three is two thirds. D-010 appears at rank five, tied in raw score with D-001 and ordered after it by document ID. It matches only the relatively common query token. D-005 and D-006 match several rarer query terms and rank much higher.

The supplied opaque T-03 order begins D-005, D-010, D-006. The hybrid order begins D-005, D-006, D-010. Both include all three relevant documents in the top three, so T-03 Recall at three is one. Across topics, BM25 macro Recall at three is approximately zero point eight eight nine, and hybrid macro Recall is one. BM25 macro NDCG at three is about zero point eight five six; hybrid macro NDCG is about zero point eight eight eight.

Now apply the claim ceiling. We observed a synthetic list difference. We did not observe a named model advantage. We did not observe public crawling. We did not observe a production index. We did not observe context selection, generation, citation, or user action. The numerical delta is not an expected business or platform effect.

Notice one more result: under the standard reranker weight, the reranked top-three metrics equal the hybrid metrics. The reranker does not improve the reported scores. That unchanged outcome belongs in the report. A research workflow that accepts only positive changes encourages hidden tuning and weakens reproducibility.

Learner check: identify one candidate miss and one ordering error. The candidate miss is D-010 outside BM25 top three for T-03. The ordering error is gain-zero D-009 at rank one for T-02 under the fixture judgments.

## Chapter 9 — Open-sandbox evidence and closed-platform uncertainty

The open sandbox permits strong, specific statements. We can inspect exact corpus records, licenses, topic text, qrels, tokenizer, parameters, score files, candidate lists, aggregation, output hashes, and the stage-boundary record. A second analyst can run the command and compare artifacts. That is computational reproducibility for this fixture.

The same run cannot establish scientific external validity. Ten synthetic documents do not represent the public web. Three authored topics do not estimate a user population. A supplied opaque list with no model identity does not compare retriever families. A successful reproduction does not independently validate the relevance scheme or a production mechanism.

Closed answer surfaces offer different evidence. We can record the system surface, locale, account state, time, exact query, visible response, and visible attribution. Repeated samples can characterize the response event under that protocol. The surface record normally does not reveal full crawl state, index membership, candidate lists, scoring functions, query rewrites, context budget, or the causal reason for omission.

An owned open index can serve as a mechanism probe, but not as a hidden-platform meter. If your BM25 index retrieves a page, state that the representation is lexically retrievable in your index under the query and protocol. You may hypothesize that representation or vocabulary affects candidate access elsewhere. You may not claim that an external platform indexed or retrieved the page.

Good research can combine the designs. An open sandbox isolates mechanisms under controlled conditions. A closed-surface panel estimates observable output variability under recorded conditions. A bridge argument connects them only when assumptions are explicit. They should not be collapsed into one number called “AI visibility.”

Learner check: write one open-sandbox observation and one closed-surface observation that share a topic but not a stage. For example: “D-010 is BM25 rank five in `IDX-L03-001` for T-03.” And: “Under the recorded public surface and query at a given time, source X was not visibly cited.” The first does not explain the second.

## Chapter 10 — Bounded conclusion and handoff

We can now write a complete W03 conclusion. It needs four parts: scope, observation, diagnosis, and boundary.

Here is a model: “In the immutable L03 ten-document synthetic corpus, with three declared topics, seven positive graded qrels, lowercase English-oriented tokenization, positive-IDF BM25 using `k1` 1.2 and `b` 0.75, and unlisted pairs treated as gain zero, BM25 recovers two of three relevant T-03 documents at cutoff three. D-010 is outside the cutoff; the supplied opaque and hybrid lists include it, but the opaque scores are authored and have no model identity. The comparison does not evaluate public crawling, production indexing, context selection, generation, citation, absorption, or user outcomes.”

This conclusion is useful because it says exactly what happened and what did not. It preserves the artifact’s scientific value without turning it into a marketing promise.

Your asynchronous equivalent is to read the notes, reproduce or inspect the BM25 calculation, run the offline L03 path, compare topic-level lists, and submit a stage-boundary memo. The synchronous route reaches the same assessed artifacts through a planned 90-minute seminar and 75-minute core studio. Neither timing has yet been validated by real delivery.

Final learner check: write two sentences. Sentence one reports one L03 result with the corpus, cutoff, and denominator. Sentence two names the strongest downstream or closed-platform claim the result does not establish. If you use the word “visibility” without naming an event, revise.

W04 begins where W03 stops. Once a relevant source enters a candidate set, we ask why its position changes and how a finite context budget selects among candidates. Carry forward the candidate identity, because a reranker cannot recover evidence that never entered its set.