fix(dataprep): scope chunk labels to document file_labels
Problem
Chunks were labeled with taxonomy labels that fall outside the document's own scope (file_labels).
Reproduced on el-salvador: tomato guide 1782731590193_d684ff9f (Guia-Centa-Tomate_no_table.md, 48 chunks) received Cucumber (chunks 37/39/40), Onion (39) and Cabbage (32/35/47) — sibling crops absent from its file_labels — on pest-management chunks whose text never mentions those crops.
Root cause
_label_with_llm prompts the LLM with the full taxonomy (all_labels) and _finalize_chunk_labels validates suggestions against that same full taxonomy, never against file_labels. As long as a label is a valid taxonomy entry (Cucumber/Cabbage/Onion are), it can land on any chunk. The flat el-salvador taxonomy (crops + topics as sibling services) amplifies this — the LLM sprinkles sibling-crop labels on generic chunks.
file_labels was used only as a fallback (empty taxonomy / 3× LLM failure).
Fix
_finalize_chunk_labels now scopes resolved labels to the document's file_labels:
- taxonomy canonicalisation (exact + synonym match) runs first, unchanged;
- any resolved label not in
file_labelsis dropped before storage; -
empty
file_labels⇒ no filtering (guard: documents without a scope keep all taxonomy-validated labels — no behaviour change for them); - dropped labels are appended to the ingestion-log entry for observability.
Surgical, ~10 lines, no change to the LLM call or prompt.
Verification
- TDD: failing test first (
Cucumbersurvives →AssertionError), then green after fix. - Full
tests/test_dataprep.py: 72 passed, 0 failures. -
ruff check+ruff format --check: clean. -
Pending: re-ingest the tomato guide on a stack and confirm 0 out-of-scope labels; re-check the cucumber doc (Cucumber ∈ its
file_labels⇒ no under-labeling regression).
Trade-off
A chunk that genuinely discusses an out-of-scope topic (e.g. a tomato chunk mentioning companion-planted cucumber) will no longer receive that label, so it won't surface in a label-filtered search for it. For monocrop-guide taxonomies this is the desired scoping; flagging for reviewer awareness.