feat: Contextual Retrieval — Part A (dataprep contextualization) + Part B (retriever BM25+RRF hybrid)

What

Opt-in Contextual Retrieval (Anthropic-style) in the dataprep: an LLM generates a short document-context prefix per chunk, prepended before embedding + labeling, so chunks carry the document's subject.

Why

Per-chunk labeling/embedding lose document-level context. A generic chunk (e.g. irrigation) inside a cucumber guide is not retrievable by "cucumber" — label-filtered + vector retrieval miss a large fraction of a document's chunks (cucumber case: 11/34). Recurs for every document across all topics.

How

• New _apply_contextualization step between chunking and labeling, behind CONTEXTUAL_RETRIEVAL_ENABLED (default off → true no-op).
• Shared _build_vllm_client extracted (DRY with the labeling hot-path).
• Never-block: client-build guard + per-chunk fallback + 0/N ERROR summary (misconfigured model is visible).
• Original chunk preserved in metadata.chunk_text (flag-gated).
• Self-hosted vLLM only; English ingestion; response_format=json_object (validated on ibm-granite/granite-4.1-8b).

Config (documented everywhere the other labeling vars live)

CONTEXTUAL_RETRIEVAL_ENABLED · DATAPREP_CONTEXTUAL_MODEL (empty = reuse VLLM_LLM_MODEL_ID) · DATAPREP_CONTEXTUAL_DOC_BUDGET (6000) · CONTEXTUAL_RETRIEVAL_PROMPT — in env, docker-compose.yaml, deploy/ansible (env.j2 + all.yml), CLAUDE.md, install guide, labeling-strategy doc §7, dataprep README, ChoosingLLMs.

Tests

81/81 (11 new in TestContextualRetrieval); ruff check + ruff format clean; config-validator 25/25. Adversarial review (blind + edge-case + acceptance) — all findings addressed, acceptance auditor APPROVE.

Out of scope (deferred)

Part B — retriever BM25 hybrid + RRF fusion over the same contextualized text (full SOTA dense + sparse → reranker). Tracked in deferred-work.md. Flags are orthogonal: A off + B on = plain lexical hybrid; A on + B on = full SOTA.

Spec

_bmad-output/implementation-artifacts/spec-contextual-retrieval.md (incl. Suggested Review Order with clickable path:line links).

---

Part B — Retriever BM25 + RRF hybrid retrieval (NOW INCLUDED)

Note: Part B was previously listed as deferred / out-of-scope in the Part A sections above. It is now included in this MR (commit 3a1352dbc).

What: Adds an opt-in lexical (BM25) ArangoSearch channel over <GRAPH>_SOURCE.text, fused with the dense vector results via weighted Reciprocal Rank Fusion (RRF). Both channels read the same text field Part A contextualizes. Default OFF (RETRIEVER_HYBRID_RETRIEVAL_ENABLED) = true no-op (dense-only, byte-for-byte).

Why: The retriever was dense-only — exact-match, rare-term, and keyword-heavy queries that dense embeddings rank poorly were missed. BM25 + RRF recovers them.

How:

• rrf_fuse — pure function, id-normalized (_normalize_chunk_id strips COLLECTION/_key), None-guarded, intra-channel de-dup.
• _ensure_bm25_view — idempotent, best-effort (never raises), cache-on-success, lazy per graph_name; default graph ensured at init.
• _bm25_search — BM25 over the ArangoSearch view, label-filtered (same OR/AND semantics as dense), returns {"doc","score"}, never raises.
• invoke — fuses before the empty-check so BM25 can rescue dense-empty queries; OFF skips the whole block; any failure falls back to dense-only.

Config (env): RETRIEVER_HYBRID_RETRIEVAL_ENABLED (off), RETRIEVER_HYBRID_RRF_K=60, RETRIEVER_HYBRID_BM25_CANDIDATES=50, RETRIEVER_HYBRID_DENSE_WEIGHT=1.0, RETRIEVER_HYBRID_LEXICAL_WEIGHT=1.0, RETRIEVER_HYBRID_BM25_ANALYZER=text_en.

Tests: 23 new (84 retriever tests pass locally) — rrf_fuse math/dedup/dense-empty-rescue, id normalization, label filter, view idempotency + failure-not-cached + init wiring, invoke hybrid/no-op/graceful/scope-guard.

Reviews: 3 blind subagent reviews (blind hunter, edge-case hunter, acceptance auditor); findings addressed or documented — see spec Spec Change Log.

Deferred: pushing the label filter pre-limit into the BM25 AQL (recall optimization, needs live ArangoDB 3.12+ validation).

Spec: _bmad-output/implementation-artifacts/spec-hybrid-retrieval-bm25-rrf.md (status: done).