Docker Compose: Fix placement constraints for non-GPU OPEA wrappers
Parent: #758 (closed)
Summary
Move placement constraints for non-GPU services off the GPU node. Only the actual model servers that load weights into VRAM should require GPU placement.
Current Problem
These services have gpu == true placement but don't need GPU:
-
embedding: Thin proxy → calls TEI via
TEI_EMBEDDING_ENDPOINT(HTTP) -
reranker: Thin proxy → calls TEI via
TEI_RERANKING_ENDPOINT(HTTP) -
translation: Thin proxy → calls vLLM via
LLM_ENDPOINT(HTTP) -
textgen: Passthrough → calls vLLM via
VLLM_ENDPOINT(HTTP)
Additionally, these services are on the GPU node but don't need GPU:
- dataprep-arango-service: Calls vLLM, TEI, and (with #764 (closed)) docling-service all via HTTP. Only the in-process docling needs GPU, which is being extracted to its own service.
- retriever-arango-service: Calls vLLM and TEI via HTTP endpoints
- chatqna-xeon-backend-server: Calls all model services via HTTP endpoints
Services that KEEP gpu == true
Only services that load models into GPU memory:
-
vllm— loads LLM into VRAM -
vllm-translation-guardrail— loads translation model into VRAM -
tei— loads embedding model into VRAM -
tei_reranker— loads reranker model into VRAM -
docling-service(NEW, from #764 (closed)) — loads docling + easyocr models into VRAM
Services that MOVE to genieai == true
embedding:
deploy:
placement:
constraints:
- node.labels.genieai == true
reranker:
deploy:
placement:
constraints:
- node.labels.genieai == true
translation:
deploy:
placement:
constraints:
- node.labels.genieai == true
textgen:
deploy:
placement:
constraints:
- node.labels.genieai == true
dataprep-arango-service:
deploy:
placement:
constraints:
- node.labels.genieai == true
retriever-arango-service:
deploy:
placement:
constraints:
- node.labels.genieai == true
chatqna-xeon-backend-server:
deploy:
placement:
constraints:
- node.labels.genieai == true
Files
| File | Change |
|---|---|
docker-compose.yaml |
Update placement constraints for 7 services |
Acceptance Criteria
-
5 model server services keep gpu == trueplacement -
7 non-GPU services use genieai == trueplacement -
Single-node deployment (both labels on same node) works unchanged -
App node runs all 7 non-GPU services, calling remote GPU model endpoints
Edited by david-full