Skip to content

ChatQnA: Add full-URL env var overrides for remote model endpoints

Parent: #758 (closed)

Summary

Add 6 new _URL environment variable overrides to genieai_chatqna.py that take precedence over the existing HOST_IP + PORT pattern. This is the only Python code change needed for the entire distributed deployment feature.

Problem

ChatQnA constructs service URLs using f"http://{HOST_IP}:{PORT}" at ~30 locations. This pattern cannot express:

  • https:// schemes (required for cross-node TLS)
  • Path prefixes (e.g., /v1/chat/completions on non-standard paths)
  • Full URLs with authentication or custom ports

Implementation

1. Add new environment variables (top of file, near existing env var reads)

# Full URL overrides (take precedence over HOST_IP + PORT)
LLM_SERVER_URL = os.getenv("LLM_SERVER_URL", "")
EMBEDDING_SERVER_URL = os.getenv("EMBEDDING_SERVER_URL", "")
RETRIEVER_SERVICE_URL = os.getenv("RETRIEVER_SERVICE_URL", "")
RERANK_SERVER_URL = os.getenv("RERANK_SERVER_URL", "")
TRANSLATION_SERVICE_URL = os.getenv("TRANSLATION_SERVICE_URL", "")
GUARDRAIL_SERVICE_URL = os.getenv("GUARDRAIL_SERVICE_URL", "")

2. Add URL resolution helper

def _resolve_url(url_env, host_env, port_env):
    if url_env:
        return url_env.rstrip("/")
    return f"http://{host_env}:{port_env}"

3. Resolve all 6 service URLs

LLM_URL = _resolve_url(LLM_SERVER_URL, LLM_SERVER_HOST_IP, LLM_SERVER_PORT)
EMBEDDING_URL = _resolve_url(EMBEDDING_SERVER_URL, EMBEDDING_SERVER_HOST_IP, EMBEDDING_SERVER_PORT)
RETRIEVER_URL = _resolve_url(RETRIEVER_SERVICE_URL, RETRIEVER_SERVICE_HOST_IP, RETRIEVER_SERVICE_PORT)
RERANK_URL = _resolve_url(RERANK_SERVER_URL, RERANK_SERVER_HOST_IP, RERANK_SERVER_PORT)
TRANSLATION_URL = _resolve_url(TRANSLATION_SERVICE_URL, TRANSLATION_SERVICE_HOST_IP, TRANSLATION_SERVICE_PORT)
GUARDRAIL_URL = _resolve_url(GUARDRAIL_SERVICE_URL, GUARDRAIL_SERVICE_HOST_IP, GUARDRAIL_SERVICE_PORT)

4. Update URL construction throughout the file

Replace inline f"http://{HOST}:{PORT}/path" with f"{RESOLVED_URL}/path" at all ~30 locations. The paths (/v1/chat/completions, /v1/retrieval, etc.) stay the same — only the base URL changes.

Files

File Lines Change
genie-ai-overlay/chatqna/genieai_chatqna.py ~30 URL constructions Add env vars, helper, replace inline URL construction

Backward Compatibility

Fully backward compatible. When no _URL env var is set, _resolve_url() falls back to the existing HOST_IP + PORT pattern, producing identical URLs to current behavior.

Acceptance Criteria

  • 6 new _URL env vars added and read at module level
  • _resolve_url() helper implemented
  • All URL constructions use resolved URLs instead of inline HOST:PORT
  • Default behavior unchanged (no _URL set = same URLs as before)
  • Setting LLM_SERVER_URL=http://10.0.0.50:8080 correctly routes LLM requests
  • Ruff lint passes