feat(chatqna): reranker-selection identity span + eval harness
Two complementary RAG eval paths fed from one query drive, plus a re-ingestion-stable chunk identity. Designed via a multi-round BMAD party-mode review (architecture locked: content-hash anchor + external LLM-judge Ragas).
Production change (chatqna): new chatqna.reranker_selection span at the RERANK node — the single junction where both candidate and selected chunk _keys are known (recovered via the existing text_to_id). No retriever/reranker change, no metadata duplication. Extracted as _emit_reranker_selection_span for isolated unit testing. Attributes: rag.candidate_chunk_ids, rag.selected_chunk_ids, rag.selected_scores, counts.
Identity: chunk _keys are auto-generated UUIDs (langchain-arangodb) → churn on every re-ingest. The anchor matches on a CONTENT FINGERPRINT (chunk_identity.content_hash = sha256 of normalized text prefix) instead. Stable across re-ingests; a chunking-param change correctly invalidates the gold set (signal to re-baseline).
Two eval paths (tests/rag-benchmarks/eval/):
-
run_eval.py --mode anchor— deterministic recall/precision/complete_recall/noise/retrieval_recall on content-hashes. No LLM, reproducible.retrieval_recallisolates retriever failure from reranker failure. -
run_eval.py --mode dump-tuples→run_ragas_eval.py— collects (question, contexts, answer, reference) tuples from the sovereign deployment, judged by a model-agnostic external OpenAI-compatible LLM via Ragas (faithfulness/context_precision/context_recall/answer_relevancy). Deployment stays sovereign; only tuples reach the judge.
Why two paths: a confident answer can hide a dropped chunk (anchor catches), and a perfect chunk set can still give a wrong answer (semantic catches). Neither sees the other's failures.
Tests: 22 unit tests (metrics + chunk_identity) run locally green. test_chatqna_selection_tracing.py (5 cases) runs in CI under genie-ai-overlay/tests/. The eval scripts live under tests/ (not genie-ai-overlay/) so CI's ruff/pytest jobs don't cover them — they are ops tooling, verified locally.
Out of scope (next steps): the labeling-recall baseline audit (Phase 0 — the gate before reranker tuning is credible), and the gold dataset itself (needs domain annotation).