Skip to content

[DUPLICATE] Story 4.3: Test Dataprep Extraction Pipeline

Sprint Key: 4-3-test-dataprep-extraction-pipeline Epic: 4 PRD: testing-framework

Story 4.3: Test Dataprep Extraction Pipeline

Status: ready-for-dev

Story

As a developer, I want pytest tests for the dataprep extraction and chunking pipeline, so that document processing logic is validated without real file system or embedding services.

Acceptance Criteria

  1. AC1: Test file created — genie-ai-overlay/tests/test_dataprep.py with ITU copyright header
  2. AC2: Multi-format parsing — _load_and_chunk() routes correctly for all formats
  3. AC3: Chunking strategy — correct chunk sizes, overlaps, and empty handling
  4. AC4: Labeling logic — all 3 strategies (LLM, embedding, BM25) + fallback
  5. AC5: Embedding generation — correct cosine similarity and threshold filtering
  6. AC6: ArangoDB insertion — correct batch Document creation and graph structure
  7. AC7: Error handling — corrupted files, auto-retraction, kill switch, guardrail
  8. AC8: Retraction cascade — 5-step deletion: chunks → edges → orphans
  9. AC9: All services mocked — no real connections, passes Ruff