feat(rag): adaptive reranker instrumentation + offline calibrator + knowledge file

Why

The adaptive reranker drops gold chunks the retriever finds (recall 0.40 vs retrieval_recall 0.72). Tuning CONTEXT_DECAY_FACTOR / confusion formula / MIN_VALUE_THRESHOLD required redeploy + full eval per candidate value — too slow to explore the parameter space.

This MR adds the instrumentation + offline calibrator so recalibration is done offline against real per-chunk data, plus a knowledge file so the methodology survives.

Methodology (not parameter values)

The adaptive reranker keeps a chunk when value = utility − cost > MIN_VALUE_THRESHOLD, where cost = CONTEXT_DECAY_FACTOR×token_count + confusion. The tunable knobs live in the deployment (reranker env / ansible), not in this MR — their values drift over time and per chunk size.

The replay is a pure function of the logged data: the breakdown records the computed utility per candidate (fixed at eval time — depends on score + embeddings); only the cost side changes when retuning (value = utility − (factor×token_count + confusion)). Same logged data + same formula = same selection the live reranker would produce. So:

1. Run one instrumented eval → report JSON with adaptive_breakdown per candidate (real token counts + every cost term).
2. Sweep offline (calibrate.py) — replays selection for any (factor × confusion × threshold) combo. Seconds, not hours. Validity check: baseline replay must reproduce the live report's recall (confirms the replay is faithful before trusting its predictions).
3. Pick the winner by F1 (not raw recall — recall alone games toward "select everything").
4. Validate live — set winning params in the deployment, redeploy, rerun eval. Confirms the recall matches the offline prediction (guards against overfitting a small gold set).

Parameter VALUES land in a separate follow-up MR after the grid sweep picks a winner.

What (5 files)

Reranker (genieai_tei_reranker.py): adaptive_context_selection returns (selected_indices, breakdown). Each breakdown record carries score, relevance, novelty, utility, token_count, context_decay_cost, confusion_cost, total_cost, value, selected, original_index. Emitted on reranker.tei_invoke span as rag.adaptive_breakdown + the active factor/threshold. original_index maps TEI-rank order back to retrieved_docs order so the eval can score recall.

Eval harness (run_eval.py): harvests rag.adaptive_breakdown, writes it as per_query[].adaptive_breakdown in the report. Defaults genericized (no deployment-specific hostnames/container names — resolve live per stack).

Offline calibrator (calibrate.py): replays selection across the parameter grid; reports recall/precision/F1/avg-selected/empty-queries per combo; emits full grid JSON.

Knowledge file (tests/rag-benchmarks/CLAUDE.md + AGENTS.md symlink): methodology — two eval paths, metric definitions, content-hash identity, CLI/run recipe (generic), adaptive parameter surface, calibration workflow, failure-mode diagnosis, position-vs-order pitfall, pitfalls log.

Verified locally

npm run lint:py ✓, npm run format:py:check ✓, pytest tests/test_reranker.py tests/test_reranker_tracing.py → 67 passed.

No production behavior change

Pure instrumentation + tooling + docs. Production recall/selection unchanged (validity check confirms the replay reproduces live selection exactly at production params).

Honest limitations

• Tunes only the adaptive cost path. Upstream changes (retriever k, label filter, contextual retrieval, score calibration) need a fresh eval.
• Gold set size matters — the live validation step guards against overfitting.