Distributed Architecture - Expose GPU Models as Remote Endpoints and Neutralized Idempotent Container Services for Distributed Deployment

Goal

Enable fully flexible multi-node deployment of GENIE.AI by making all inter-service communication configurable via environment variables. Any container can be deployed to any node, and services find each other through .env configuration — no hardcoded Docker DNS names.

The GPU node serves as a shared, stateless inference cluster for multiple GENIE.AI deployments (Docker Compose, Swarm, and future Kubernetes).

Approach: ${VAR:-default} Pattern

Every inter-service URL in docker-compose.yaml follows this pattern:

environment:
  ARANGO_URL: ${ARANGO_URL:-http://arango-vector-db:8529}

• Default = Docker DNS name (single-node and Swarm work unchanged)
• Override = any URL (remote node, external service, Kubernetes DNS)
• No service discovery infrastructure — env vars + Ansible .env generation is sufficient
• Kubernetes mapping: Helm values set the same env vars to K8s service names — same containers, zero code changes

Architecture

Prerequisite: Shared Network Layer

All deployment modes need a network layer where services can reach each other:

Mode	Mechanism	Multi-node?
Docker Compose	Single network (automatic)	No (single host)
Docker Swarm	Overlay network (genieai_network)	Yes — DNS spans nodes
Kubernetes	Cluster DNS + Services	Yes — native
Cross-cluster	URL overrides in .env	Yes — any reachable URL

Deployment Profiles

Profile A — Single node (current default):

No .env overrides. All defaults resolve to Docker DNS names.

Profile B — Multi-node Swarm (3-label placement):

Node labels: gateway=true, genieai=true, gpu=true
Services placed on appropriate nodes. Overlay network provides cross-node DNS.
No .env overrides needed — same defaults work across nodes.

Profile C — Remote GPU node (app node + separate GPU node):

App node .env:
  VLLM_ENDPOINT=http://10.0.0.50:8080
  TEI_EMBEDDING_ENDPOINT=http://10.0.0.50:8081
  DOCLING_ENDPOINT=http://10.0.0.50:8083
  ...

Profile D — Fully distributed (any service on any node):

Each service URL independently configurable.
ArangoDB, backend, doc-repo can also be on separate nodes.

Profile E — Kubernetes (future):

Helm values.yaml sets env vars to K8s DNS names.
Same container images, zero code changes.

Current GPU Models (RTX 6000 Ada)

#	Service	Model	GPU Memory	API
1	vllm	meta-llama/Meta-Llama-3.1-8B-Instruct	17,014 MiB	OpenAI /v1/chat/completions
2	vllm-translation-guardrail	google/gemma-3-4b-it	14,384 MiB	OpenAI /v1/chat/completions
3	tei	BAAI/bge-base-en-v1.5	700 MiB	TEI /v1/embeddings
4	tei_reranker	cross-encoder/ms-marco-MiniLM-L-6-v2	540 MiB	TEI /v1/rerank
5	docling + easyocr (in-process)	Docling layout + EasyOCR	1,526 MiB	Needs new microservice

All 5 must be stateless and idempotent — safe for concurrent access from multiple app instances.

Implementation Steps (Child Issues)

Layer 1: OPEA/AI Remote Endpoints

1. #759 (closed) — ChatQnA: Add full-URL env var overrides (genieai_chatqna.py)
2. #760 (closed) — Docker Compose: Make all service endpoint URLs configurable
3. #761 (closed) — Docker Compose: Expose GPU service ports for remote access
4. #762 (closed) — Docker Compose: Fix placement constraints for non-GPU wrappers
5. #763 (closed) — Environment templates: Add remote model endpoints section
6. #764 (closed) — Docling microservice + dataprep remote call support

Layer 2: Core Service URLs

7. #765 — Core service URLs: Make ArangoDB, Backend, Doc-repo, Redis, Keycloak internal URLs configurable
8. #766 — Kong config: Make upstream host references configurable

What Already Works (No Code Changes)

Service → Target	Env Var	Already Full URL?
Dataprep → vLLM	VLLM_ENDPOINT	Yes
Dataprep → TEI	TEI_EMBEDDING_ENDPOINT	Yes
Retriever → vLLM	VLLM_ENDPOINT	Yes
Retriever → TEI	TEI_EMBEDDING_ENDPOINT	Yes
Reranker → TEI	TEI_RERANKING_ENDPOINT	Yes
Embedding → TEI	TEI_EMBEDDING_ENDPOINT	Yes
Translation → vLLM	LLM_ENDPOINT	Yes
Nginx → Kong	KONG_PROXY_HOST	Yes
Nginx → Frontend	NGINX_FRONTEND_HOST	Yes

Verification

1. Single-node regression: No .env overrides → identical behavior
2. Remote GPU test: Override model endpoints → services call remote GPU node
3. Full distribution test: Override core service URLs → services on separate nodes communicate
4. Multi-instance idempotency: Two dataprep instances calling same GPU endpoints concurrently
5. Ansible template test: ansible-playbook deploy.yml --tags deploy --check passes
6. Kubernetes readiness: Confirm all inter-service references are env-var-controlled (no hardcoded names)