Skip to content

feat(confidence): surface LLM self-grade transparently via confidence_score

Jérôme Revillard requested to merge worktree-confidence-display-llm into main

Context

Follow-up to !172 (merged). When LLM_SELF_CONFIDENCE_ENABLED=1, the citizen-facing confidence_score now reflects the LLM's self-assessed groundedness ([[CONF:0-100]] sentinel), falling back to the retrieval confidence when the sentinel is absent — so the existing client badge never disappears. Transparent to web + mobile clients (they already render confidence_score): enabling the feature surfaces the LLM value with no client change.

Changes

chatqna (genieai_chatqna.py):

  • _assemble_source_documents returns retrieval_confidence_score (rank-weighted).
  • New _display_confidence(self ?? retrieval) helper.
  • Metadata (stream + non-stream): always emit retrieval_confidence_score; when flag on → confidence_score = _display_confidence(...) + raw self_confidence; when off → confidence_score = retrieval (unchanged).

Backend (query-service.js): pass retrieval_confidence_score + self_confidence through the SSE metadata reconstruction (whitelist) so they persist to the DB / QueryInspector / eval.

Admin (QueryInspectorDetail.vue): show retrieval_confidence_score + self_confidence alongside confidence_score (admin-only eval view; List unchanged).

Verification

  • chatqna pytest: 131 passed (+3 _display_confidence; streaming tests updated for new semantics).
  • backend jest query-service: 122 passed (+2 reconstruction).
  • backend jest query-routes: passed.
  • frontend QueryInspector jest: 26 passed.
  • ruff + eslint + prettier: clean.

️ Product note

The LLM self-grade is an uncalibrated self-assessment — 2B models are overconfident and may omit the sentinel (→ fallback retrieval). Surfacing it to citizens is a deliberate decision (el-salvador will enable it). The raw retrieval + self values remain available to admins (QueryInspector) for calibration evaluation. Null-case (no sentinel) → confidence_score falls back to retrieval → badge never disappears.

Merge request reports

Loading