Skip to content

Dataprep/Docling: Strategy for GPU-bound document extraction in distributed deployment

Parent: #758 (closed)

Decision

Docling becomes its own microservice on the GPU node. Dataprep runs entirely on app nodes and calls docling via HTTP API for document extraction, vLLM for labeling, and TEI for embeddings — all remote.

The GPU node serves as a shared, stateless inference cluster for multiple GENIE.AI deployments (Kubernetes, Docker Compose, Swarm, etc.). All GPU services must be idempotent — they hold no per-tenant state and can serve concurrent requests from any number of app instances.

Architecture

GPU Node (stateless inference cluster):
  ├── vllm              :8080   — OpenAI /v1/chat/completions
  ├── vllm-translation   :9031   — OpenAI /v1/chat/completions
  ├── tei               :8081   — TEI /v1/embeddings
  ├── tei_reranker      :8082   — TEI /v1/rerank
  └── docling-service    :8083   — NEW: /v1/extract (document → text)

App Node(s) (any number, no GPU):
  ├── dataprep           → calls docling-service, vllm, tei (all remote)
  ├── chatqna            → calls vllm, retriever, reranker, translation (all remote)
  ├── retriever          → calls vllm, tei (all remote)
  ├── backend, frontend, keycloak, etc.
  └── No GPU dependencies whatsoever

Part A: Docling Microservice

New service: docling-service

Create a new FastAPI service that wraps docling + easyocr:

genie-ai-overlay/docling-service/
├── Dockerfile
├── main.py              # FastAPI app
├── requirements.txt
└── README.md

API contract:

POST /v1/extract
Content-Type: multipart/form-data

Request:
  file: document (PDF, DOCX, PPTX, XLSX, image, etc.)
  options: JSON (optional) — { "ocr_lang": ["en"], "chunk_size": null }

Response:
  {
    "text": "full extracted text",
    "pages": [
      { "page": 1, "text": "...", "tables": [...], "images": [...] }
    ],
    "metadata": { "format": "pdf", "pages": 42, "size_bytes": 12345 }
  }

GET /health  → 200 OK

Docker Compose:

docling-service:
  build:
    context: .
    dockerfile: genie-ai-overlay/docling-service/Dockerfile
  environment:
    DOCLING_DEVICE: ${DOCLING_DEVICE:-cuda}
  volumes:
    - docling-models:/root/.cache/docling
  deploy:
    placement:
      constraints:
        - node.labels.gpu == true
    resources:
      reservations:
        devices:
          - driver: nvidia
            count: all
            capabilities: [gpu]
  ports:
    - "${DOCLING_HOST_PORT:-8083}:8000"

Idempotency requirements:

  • No per-tenant state — model loaded once at startup, reused for all requests
  • Concurrent request safe — docling DocumentConverter is thread-safe for read-only use
  • No persistent storage — documents processed in-memory or temp files, cleaned up after response
  • Horizontal scaling — multiple replicas can run behind a load balancer

Part B: Dataprep Code Changes

genieai_dataprep_utils.py — Replace in-process docling with HTTP calls

Currently (lines 12-65):

import easyocr
from docling.datamodel.pipeline_options import ...
# Loaded in-process at module level
reader = easyocr.Reader(["en"])
converter = DocumentConverter(...)

Change to:

DOCLING_ENDPOINT = os.getenv("DOCLING_ENDPOINT", "")

# If DOCLING_ENDPOINT is set → call remote docling microservice
# If not set → fall back to in-process docling (backward compatible)

Add async function to call docling-service:

async def extract_with_docling_service(file_path: str, endpoint: str) -> dict:
    async with aiohttp.ClientSession() as session:
        with open(file_path, "rb") as f:
            data = aiohttp.FormData()
            data.add_field("file", f, filename=os.path.basename(file_path))
            async with session.post(f"{endpoint}/v1/extract", data=data) as resp:
                return await resp.json()

Update load_with_docling() and docling_document_loader() to:

  1. Check if DOCLING_ENDPOINT is set
  2. If yes → HTTP call to docling-service
  3. If no → use in-process docling (current behavior)

Remove GPU dependencies from dataprep Dockerfile

When DOCLING_ENDPOINT is set, dataprep no longer needs:

  • nvidia/cuda base image → switch to standard Python image
  • docling, easyocr, torch pip packages → conditional or separate build target

This dramatically reduces dataprep image size and removes GPU requirement.

genieai_dataprep_arangodb.py — Already remote

LLM labeling (lines 297-419) already uses VLLM_ENDPOINT via OpenAI client. No changes needed.

Embedding (lines 421-436) already uses TEI endpoint. No changes needed.

Files to Create

File Purpose
genie-ai-overlay/docling-service/Dockerfile GPU-enabled docling service image
genie-ai-overlay/docling-service/main.py FastAPI app with /v1/extract endpoint
genie-ai-overlay/docling-service/requirements.txt docling, easyocr, fastapi, uvicorn
genie-ai-overlay/docling-service/README.md API documentation

Files to Modify

File Change
genie-ai-overlay/dataprep/genieai_dataprep_utils.py Add remote docling call, keep in-process fallback
docker-compose.yaml Add docling-service definition, update dataprep environment
genie-ai-overlay/dataprep/Dockerfile-dataprep_genie-ai Conditional GPU deps or separate build target

Acceptance Criteria

  • Docling microservice accepts documents via POST /v1/extract and returns extracted text
  • Docling microservice is stateless and idempotent (safe for concurrent multi-tenant use)
  • GET /health endpoint returns 200 when docling models are loaded
  • Dataprep calls docling-service via DOCLING_ENDPOINT when set
  • Dataprep falls back to in-process docling when DOCLING_ENDPOINT is empty (backward compat)
  • Dataprep works on non-GPU node with all endpoints remote
  • Single-node regression: DOCLING_ENDPOINT not set → current behavior unchanged
  • Multi-node: dataprep on app node, docling-service on GPU node → document ingestion works end-to-end
  • Multiple concurrent dataprep instances can call the same docling-service without errors
Edited by david-full