Skip to content

feat(retriever): push BM25 label filter pre-LIMIT (hybrid Part B recall optimization)

Jérôme Revillard requested to merge feat/hybrid-retrieval-bm25-prelimit into main

Push BM25 label filter pre-LIMIT (hybrid Part B recall optimization)

Supersedes the deferred item from the Part B spec (MR !199 (merged)). The v1 BM25 channel applied the label filter in Python after LIMIT @n — correct semantics, but it could starve in-category recall when cross-category docs dominated the top-N.

Change

Inject the dense path's aql_filter_clause (same clause, same doc alias) into the BM25 AQL between SEARCH and BM25(), before LIMIT. The Python _chunk_passes_label_filter stays as a defense-in-depth safety net (now a no-op when the AQL filter is active).

Live validation — el-salvador (ArangoDB 3.12.4)

Validated directly against the deployed GRAPH_TEST cucumber corpus (34 chunks):

  • v1 AQL (no filter): 34 hits for cucumber
  • optimized AQL (FILTER ... "Cucumber" IN d.chunk_labels, pre-LIMIT): 17 "Cucumber"-labeled hits

Confirms FILTER-in-ArangoSearch syntax is accepted and correct on the real engine (the exact runtime risk that caused the v1 deferral).

Scope

3 files, +43/-3 lines. _bm25_search gains an aql_filter_clause param; invoke passes it (chunk start only). +1 test (test_aql_filter_clause_injected_before_limit). 85 retriever tests pass; ruff clean.

Spec

Part B Spec Change Log updated: deferred item → implemented + live-validated.

Merge request reports