Skip to content

fix(reranker): handle TEI 429/non-list responses + raise concurrency

Jérôme Revillard requested to merge fix/reranker-tei-429-contract into main

Problem

When the TEI reranker on the GPU node is overloaded, it returns HTTP 429 with a JSON error OBJECT:

{"error": "Model is overloaded", "error_type": "Overloaded"}

The old reranker code called .json() (which succeeds on valid JSON), then iterated the result directly. Iterating a dict yields its string keys, so downstream r["index"] raised a confusing TypeError: string indices must be integers that completely hid the real upstream 429.

Reproduction

With k=20 (RETRIEVER_ARANGO_K=20, the prod-tested default), the reranker batches 20 chunks per chat request. TEI's default --max-concurrent-requests 8 saturates under this load -> 429. With k=4 the load stayed under the limit, so the bug appeared k-bound (masking the real cause).

Proven via instrumentation: TEI http=429 ct=application/json resp_type=dict resp_repr={"error":"Model is overloaded","error_type":"Overloaded"}

Fix (3 commits)

1. Code — genieai_tei_reranker.py: validate TEI response contract before consuming it.

  • HTTP != 200 -> raise RuntimeError naming status + body
  • non-list body -> raise RuntimeError naming type (contract violation)
  • list length != ndocs -> warn (score alignment risk)

A 429 now surfaces as a clear RuntimeError: TEI reranker call failed: HTTP 429 ... Response: {'error': 'Model is overloaded', ...} instead of a cryptic TypeError deep in a listcomp.

2. Config (compose): raise 128 (was 8). Single source of truth aligned with prod-tested value.

3. Config (ansible): tei_reranking_max_concurrent_requests: "128" in itu_rtx_gpu_api/vars.yml.

Tests

test_reranker.py:

  • create_mock_aiohttp_session now takes status=200
  • test_tei_429_overloaded_raises_with_status_and_body — pins HTTP 429 -> RuntimeError naming status
  • test_tei_200_non_list_response_raises — pins 200 + non-list -> RuntimeError naming type

Guard logic validated standalone (all 3 cases: 429-dict, 200-non-list, valid-list pass).

Validation

Deployed on el-salvador-contextual. Before fix: chatqna HTTP 500 (cryptic TypeError) for k=20. After fix: clear HTTP 429 error surfaced, then TEI concurrency raise -> HTTP 200. Full 32-query eval ran clean (n_missed=0), retrieval_recall 0.54 -> 0.72 (+34%).

Ruff: clean.

Merge request reports