perf(dataprep): ~37x faster labeling (response_format + concurrency + tracing) + retract log cleanup

Summary

LLM chunk-labeling was the dominant cost of document ingestion (~70% of wall time) and was completely invisible in traces. This MR makes labeling observable and ~37× faster, eliminates a JSON-output failure storm, and cleans up orphaned ingestion logs on file retract.

Problem

• One document ingestion took ~50 min; LLM labeling alone was ~35 min yet emitted zero trace spans (a blind gap in VictoriaTraces).
• ~25% of labeling calls failed with JSONDecodeError (model wrapped JSON in markdown fences/prose) → 3 retries → file-metadata fallback. 4,938 retries, 1,637 chunks fell back to inferior labels.
• Retracting a file left its ingestion_log entries orphaned.

Root causes (evidence-based)

1. Untraced, concurrency-starved labeling: one vLLM call per chunk at concurrency 5, to a remote GPU node (granite-4.1-8b). No spans → invisible bottleneck.
2. temperature unset → vLLM default ~1.0 → model appended prose → invalid JSON. Live probe: temp=1.0 → 3/4 clean; temp=0.0 → 4/4.
3. No response_format → even at temp=0, vLLM batched-inference non-determinism produced markdown-fenced/prose JSON for some chunks → JSONDecodeError.
4. docker-compose override: compose had its own DATAPREP_MAX_CONCURRENT_BATCHES:-5 default that overrode the raised code default.

Changes

1. Instrumentation + concurrency: OTel span per label call (dataprep.llm.label_chunk / .label_batch); concurrency 5→20 (code + docker-compose + ansible); labeling start/progress logs; exception-reason logging on failed attempts.
2. Deterministic JSON output: temperature=0.0 + bounded max_tokens + response_format={"type":"json_object"} (vLLM guided decoding → non-JSON impossible at token level).
3. Batching: chunks per LLM call (default 4; DATAPREP_LLM_LABEL_BATCH_SIZE, validated at 8 on granite-4.1-8b). Firmer prompt + lenient label parsing (null/string/list tolerated).
4. Retract cleanup: fileService.deleteIngestionLogs() called on retract and hard delete.

Validation (release/el-salvador, remote vLLM granite-4.1-8b, 1,718-chunk document)

Metric	Before	After
Labeling time	35 min	0.9 min (~37×)
JSONDecodeError	dominant	0
Fallback chunks	~25%	~1% (residual, graceful)
Trace visibility	blind gap	per-call spans
Total ingestion	~50 min	~10 min

Validated on release/el-salvador before promoting (workflow: test on release branch → cherry-pick to main). Phase-2 graph extraction is now the next bottleneck (separate effort). LLM-based labeling retained.

Files

dataprep (genieai_dataprep_arangodb.py + tests), docker-compose, env template, ansible all.yml/env.j2, document-repository (fileService.js, fileController.js + test). No deployment-specific config (el-salvador tuning stays on the release branch).