enhancement(opea): ranker logic updated with new adaptive method for document selection

Adaptive utility-cost document selection in the reranker

Adds two new reranking strategies to genieai_tei_reranker.py, on top of the existing slice / threshold / knee_threshold.

slice_threshold

Top-N capped at a score threshold. TEI returns results sorted descending, so selection breaks early once a score drops below the threshold.

adaptive — utility-cost selection (default)

For each candidate chunk, estimates a marginal value and keeps it only when value > MIN_VALUE_THRESHOLD:

value   = utility − cost
utility = relevance × novelty_weight
cost    = token_cost + confusion_cost

• relevance — reranker score, boosted above the skew-adjusted mean
• novelty — penalises redundancy with already-selected chunks (cosine similarity), MMR-style; candidates processed in descending score order
• token_cost — each chunk consumes context-window budget
• confusion_cost — low-confidence chunks risk degrading the answer

Goal: feed the LLM an optimal, non-redundant, confidence-aware context set instead of a fixed top-N.

Embedding flow

• Query embedding: produced by the embedding node, carried in the embedding field, forwarded by chatqna align_outputs from the request.
• Chunk embeddings: the retriever fetches each chunk's stored embedding from ArangoDB (adaptive only) and returns it in the SearchedMultimodalDoc metadata list. ChatQnA assembles chunk_embeddings from there and forwards to the reranker.
• If embeddings are missing or misaligned, the reranker raises a RuntimeError (hard-fail, no silent fallback to slice).

Configuration

Variable	Default	Purpose
RERANKING_STRATEGY	adaptive	slice | threshold | slice_threshold | knee_threshold | adaptive
NOVELTY_SIGMOID_A	20.0	novelty→weight logistic steepness
NOVELTY_SIGMOID_B	0.25	novelty→weight logistic midpoint
TOKEN_COST_ALPHA	0.0025	per-token context cost coefficient
MIN_VALUE_THRESHOLD	-1.0	marginal-value cutoff (keep a chunk only if value > threshold)
RERANKER_MODEL_ID	BAAI/bge-reranker-v2-m3	multilingual reranker model

Exposed in docker-compose.yaml (reranker + chatqna services), the env template, Ansible env.j2 + README, and docs/architecture.md §9.3.

Model change

Default reranker model switched to BAAI/bge-reranker-v2-m3 (multilingual, ~2.3 GB). The lighter cross-encoder/ms-marco-MiniLM-L-6-v2 is documented as the T4 / constrained-GPU alternative. T4 deployments must override RERANKER_MODEL_ID.

Tests (437 total, ruff clean)

• test_reranker.py: adaptive helpers, adaptive_context_selection, slice_threshold, adaptive strategy (incl. alignment + hard-fail regressions), all existing strategies unaffected.
• test_retriever.py: adaptive chunk-embedding fetch + skip when non-adaptive.
• test_chatqna.py: contract test verifying chunk embeddings flow via the SearchedMultimodalDoc metadata list.
• conftest.py: numpy un-mocked (real declared test dependency).

Code review

Passed — 0 Critical, 4 Important issues fixed (doc/code consistency), 4 Minor noted for follow-up.