Skip to content

feat(chatqna): rank-weighted confidence + opt-in LLM self-grade

Jérôme Revillard requested to merge worktree-confidence-score-hybrid into main

Context

The user-facing confidence_score was an arithmetic mean of the displayed reranker scores (chatqna:1153) — the simplified early-cycle approach Roman flagged on 2026-06-23. It had three independent problems, one of them a concrete defect causing the reported intermittently "anormally low" scores:

  1. D1 defect (prime cause): a failed document-metadata lookup set score = 0 and fell through (no continue) into the aggregation, tanking the mean whenever the doc-repo/backend metadata call failed intermittently. It also surfaced a synthetic document_id:"error" source.
  2. Uncalibrated scale: bge-reranker-v2-m3 emits relevance logits, not probabilities; the flat mean had no probabilistic meaning.
  3. Mean pathology: count-dependent + tail-sensitive — low-scoring-but-novel chunks kept by the adaptive strategy (default MIN_VALUE_THRESHOLD=-1.0) depressed the mean, so richer context was punished.

Changes

Retrieval confidence (always-on):

  • D1 fix: failed metadata lookups skip the document entirely (no 0 injected, no fake error source).
  • Rank-weighted aggregator (CONFIDENCE_RANK_DECAY, default 0.5): exponential rank-decay weighting so the most relevant document dominates.
  • Optional calibration (RERANKER_SCORE_CALIBRATION=none|sigmoid, default none): sigmoid for deployments where TEI returns raw logits. Default none is safe — sigmoid on already-normalised scores compresses them.

LLM self-grade (opt-in, OFF by default):

  • LLM_SELF_CONFIDENCE_ENABLED=1 instructs the model to end its reply with a [[CONF:<0-100>]] sentinel. ChatQnA strips it before the text reaches the user or the translation pipeline (streaming + non-streaming, reusing the existing marker-stripping buffer) and exposes it as a supplementary self_confidence metadata field. Missing/malformed → null, never a hard failure.

is_grounded semantics unchanged.

Why self-grade is off by default

Self-reported confidence is an LLM self-assessment, not an independent groundedness measurement — it is exposed for evaluation, not as a replacement for the retrieval confidence or is_grounded. It should be surfaced to end users only after a calibration eval harness (held-out RAG set, ECE/Brier) confirms it tracks answer correctness. That harness is deferred follow-up.

Testing

  • chatqna pytest: 116 passed (+16 new: D1 regression, aggregator math, calibration, sentinel parse, streaming self-confidence + contract stability).
  • ruff check + format: clean.
  • config-validator: 25 passed (new env vars cross-reference cleanly).

Docs

docs/architecture.md §9.4 (new) documents the scoring + the opt-in flag.

Follow-ups (not in this MR)

  • Calibration eval harness (decide whether/which confidence to surface to users).
  • Faithfulness judge if self-grade proves uncalibrated.
  • bge temperature tuning (needs the eval set).

Merge request reports