fix(dataprep): hash-pinned lock for reproducible cold-cache builds (#834)
What
Fixes the three independent failure layers in GitLab issue #834 (fresh dataprep image builds fail from a cold cache) by replacing the unpinned OPEA requirements.txt resolution with a uv-generated, hash-pinned lock installed via pip install --no-deps --require-hashes.
Root cause (verified)
| Layer | Cause |
|---|---|
| 1. Transitives drift | OPEA v1.3 requirements.txt pins top-level only; all transitives re-resolve against live PyPI on every uncached build. |
| 2. pip 22.0.2 resolver | Ubuntu 22.04 base ships pip 22.0.2 via apt, never upgraded. Backtracks ~56 min on trimesh[easy]/pycollada extras markers, then dies on lxml 4.6.2 sdist (no cp310 wheel, no libxml2-dev). |
| 3. docling |
docling==2.30.0 declares docling-core<3.0; docling-core 2.83.0 (2026-06-18) deleted legacy_doc.BaseText, which docling imports at startup → ImportError at service start. |
Fix
-
requirements.lock— 335 packages, full SHA256 hashes, generated byuv pip compilefor Python 3.10 / manylinux_2_31 (matches the CUDA ubuntu22.04 base). -
Dockerfile — upgrades
pip/setuptools/wheelfirst, then installs the lock with--no-deps --require-hashes. No transitive re-resolution; every wheel hash verified at install. Removes the three ad-hocpip installsteps (OPEA reqs + overlay extras + OTel) — all merged into the lock. -
docling-core==2.82.0pinned — last release with functionallegacy_doc.BaseText(removed in 2.83.0; stub-only in 2.83.1). -
Drops
openai-whisper(unused by GENIE dataprep; its sdist build needs undeclaredpkg_resourcesand blockeduv pip compile) and switchesopencv-python→opencv-python-headless(server image; resolves the wheel conflict).
Verification done locally
-
✅ uv pip compileresolves clean (the Rust resolver does not have pip 22's backtracking bug). -
✅ pip install --dry-run --no-deps --require-hashes -r requirements.lock(pip 23, python:3.10-slim) — exit 0, all 335 hashes accepted, every wheel fetchable for cp310/manylinux,docling-core-2.82.0resolved,trimesh/pycolladaresolves cleanly. -
✅ Lock is reproducible: regenerate → byte-identical past the 2-line path-embedding header. -
⏳ Full image build (CUDA base + EasyOCR model pre-download) — deferred to CIbuild:dataprep-arango.
CI gates added
-
smoke:dataprep-arango(stagescan) —docker runthe candidate image andimport docling; from docling_core.types.legacy_doc.base import BaseText; import comps.dataprep.src.integrations.genieai_dataprep_arangodb. Catches the layer-3 runtimeImportErrorthat a green build does not surface (build green ≠ service alive). -
verify:dataprep-lock(stageconfig) — re-runsuv pip compilefromrequirements.inand diffs against the committed lock (header-stripped). Fails on drift, forcingmake lock-dataprepon anyrequirements.in/OPEA change.
Regen workflow
make requirements-in-dataprep [OPEA_TAG=v1.3] # regenerate requirements.in
make lock-dataprep # recompile requirements.lock
fix_dependencies.sh is retained — retriever and reranker Dockerfiles still use it (only dataprep moves to the lock in this MR).
Out of scope (follow-up)
- OPEA v1.3 → v1.4/v1.5 bump (upstream already pins
docling-corecorrectly in v1.4+). Audited separately: mechanically viable — only the requirements-patching block needs rewriting (requirements.txt→requirements.in+ compiled-cpu/-gpu); overlay subclass contract is signature-identical. Tracked as long-term work, not this MR. - Applying the same lock pattern to
retriever/reranker(they use modern pip frompython:*-slimand are currently unaffected, but would benefit from the same determinism).
Closes layer 1+2+3 of #834 (does not fully close the issue — OPEA bump is the long-term tidy-up).
Investigation + decision rationale captured in the BMAD party-mode roundtable (Winston/Amelia/Murat) and the upstream docling-core trace.