feat(chatqna): rank-weighted confidence + opt-in LLM self-grade
Context
The user-facing confidence_score was an arithmetic mean of the displayed reranker scores (chatqna:1153) — the simplified early-cycle approach Roman flagged on 2026-06-23. It had three independent problems, one of them a concrete defect causing the reported intermittently "anormally low" scores:
-
D1 defect (prime cause): a failed document-metadata lookup set
score = 0and fell through (nocontinue) into the aggregation, tanking the mean whenever the doc-repo/backend metadata call failed intermittently. It also surfaced a syntheticdocument_id:"error"source. -
Uncalibrated scale:
bge-reranker-v2-m3emits relevance logits, not probabilities; the flat mean had no probabilistic meaning. -
Mean pathology: count-dependent + tail-sensitive — low-scoring-but-novel chunks kept by the
adaptivestrategy (defaultMIN_VALUE_THRESHOLD=-1.0) depressed the mean, so richer context was punished.
Changes
Retrieval confidence (always-on):
- D1 fix: failed metadata lookups skip the document entirely (no 0 injected, no fake error source).
-
Rank-weighted aggregator (
CONFIDENCE_RANK_DECAY, default0.5): exponential rank-decay weighting so the most relevant document dominates. -
Optional calibration (
RERANKER_SCORE_CALIBRATION=none|sigmoid, defaultnone): sigmoid for deployments where TEI returns raw logits. Defaultnoneis safe — sigmoid on already-normalised scores compresses them.
LLM self-grade (opt-in, OFF by default):
-
LLM_SELF_CONFIDENCE_ENABLED=1instructs the model to end its reply with a[[CONF:<0-100>]]sentinel. ChatQnA strips it before the text reaches the user or the translation pipeline (streaming + non-streaming, reusing the existing marker-stripping buffer) and exposes it as a supplementaryself_confidencemetadata field. Missing/malformed →null, never a hard failure.
is_grounded semantics unchanged.
Why self-grade is off by default
Self-reported confidence is an LLM self-assessment, not an independent groundedness measurement — it is exposed for evaluation, not as a replacement for the retrieval confidence or is_grounded. It should be surfaced to end users only after a calibration eval harness (held-out RAG set, ECE/Brier) confirms it tracks answer correctness. That harness is deferred follow-up.
Testing
- chatqna pytest: 116 passed (+16 new: D1 regression, aggregator math, calibration, sentinel parse, streaming self-confidence + contract stability).
- ruff check + format: clean.
- config-validator: 25 passed (new env vars cross-reference cleanly).
Docs
docs/architecture.md §9.4 (new) documents the scoring + the opt-in flag.
Follow-ups (not in this MR)
- Calibration eval harness (decide whether/which confidence to surface to users).
- Faithfulness judge if self-grade proves uncalibrated.
- bge temperature tuning (needs the eval set).