7.5 Deploy Observability Stack (Collector + VictoriaMetrics + Grafana)

Sprint Key: 7-5-deploy-observability-stack-collector-victoriametrics-grafana Epic: 7 PRD: testing-framework

---

Story 7.5: Deploy Observability Stack (Collector + VictoriaMetrics + Grafana)

Status: ready-for-dev

Story

As a developer, I want an OTel Collector, VictoriaMetrics, and Grafana deployed alongside the application, so that telemetry data is collected, stored, and visualized without external SaaS dependencies.

Acceptance Criteria

1. OTel Collector service in docker-compose (OTLP HTTP on 4318)
2. VictoriaMetrics service with configurable retention (default 30d)
3. Grafana service with auto-provisioned VictoriaMetrics datasource
4. Pre-configured Service Health dashboard (req rate, error rate, latency percentiles)
5. Pre-configured RAG Pipeline Trace Waterfall dashboard
6. Collector healthcheck endpoint configured
7. Grafana internal-only access (not exposed via nginx)
8. Grafana authentication from .env credentials
9. Disabled by default (ENABLE_OBSERVABILITY=true to enable)
10. Network isolation (genieai_network only)
11. Ansible support (enable_observability var + env.j2 template)
12. Env template updated (Section 12C)
13. OTel Collector config file (configs/otel/)
14. Grafana provisioning files (configs/grafana/provisioning/)
15. Docker Compose up works with --profile observability
16. Docker Swarm works with ENABLE_OBSERVABILITY=1
17. Existing services unaffected
18. All existing CI tests pass