Skip to content

Documentation & Playbook Review - New Deployer Issues (6 findings)

Context

Thorough review of all deployment documentation and Ansible playbook from a new deployer's perspective — starting from scratch with no prior knowledge of the system. Findings span the Ansible playbook (deploy/ansible/), env template (templates/env.j2), and docs (docs/).

🔴 Critical: Playbook Overwrites Source Compose — README Claims It Doesn't

Commit e827cb870 updated the README to describe separating the resolved compose file, but the playbook code was never changed.

The Ansible README states:

"It generates a resolved docker-compose.resolved.yaml with all variables substituted... the source template is never modified."

The playbook at deploy/ansible/deploy.yml still writes to docker-compose.yaml:

  • Line 612: config > {{ deploy_dir }}/docker-compose.yaml.tmp
  • Line 635: mv {{ deploy_dir }}/docker-compose.yaml.tmp {{ deploy_dir }}/docker-compose.yaml
  • Line 656: docker stack deploy -c {{ deploy_dir }}/docker-compose.yaml {{ stack_name }}
  • Zero references to docker-compose.resolved.yaml in the code

Impact: Running --tags deploy without --tags prepare uses the previously-resolved compose with frozen variable values. Changes to .env (e.g., adding SMTP config) won't take effect because ${VAR} references are already resolved to literal values. This caused the Invalid sender address 'null' Keycloak failure on cloud_deploy — SMTP vars were added to .env but the resolved compose had KC_SMTP_HOST: "" baked in.

Fix: Update deploy.yml to write resolved output to docker-compose.resolved.yaml and deploy from that file, leaving the source docker-compose.yaml untouched from git.

🟡 Moderate: KEYCLOAK_URL Routes Backend Through Public Proxy Chain

The env template (templates/env.j2 line 94) generates:

KEYCLOAK_URL=https://{{ nginx_public_domain | default('localhost') }}/auth

For cloud_deploy this becomes https://10.0.0.101/auth. Every JWT validation by the backend traverses the full proxy chain:

Backend → (HTTPS) → nginx → Kong → Keycloak

Instead of the direct Docker network path:

Backend → (HTTP) → keycloak:8080

Measured impact on cloud_deploy (10.0.0.101):

  • Internal Keycloak OIDC discovery: 9ms
  • External (WireGuard): 1.5s — ~1.1s is Keycloak processing AFTER TLS
  • Every backend API call triggers this round-trip for JWT verification

Fix: Add a KEYCLOAK_INTERNAL_URL variable defaulting to http://keycloak:8080 for Swarm/Compose deployments. Use it for backend-internal communication. Keep KEYCLOAK_URL (public) for the frontend OIDC config only.

🟡 Moderate: Keycloak Fails Silently Without SMTP Configuration

The keycloak-config service enables VERIFY_EMAIL=true in the default realm config. If SMTP is not configured or EMAIL_FROM is empty, Keycloak rejects realm creation with:

Invalid sender address 'null'

A new deployer would:

  1. Create vault with required secrets (arango_password, keycloak_client_secret, etc.)
  2. Run the playbook
  3. See keycloak-config fail with a cryptic 400 error
  4. Have no idea why — no warning in the README or troubleshooting section

Fix (one of):

  • Document that email_host, email_user, email_password, and email_from are required when using the default realm config
  • Default VERIFY_EMAIL to false when email is not configured
  • Add a troubleshooting entry for the Invalid sender address 'null' error

🟡 Moderate: Manual Swarm Deploy Guide Doesn't Resolve Variables

docs/docker-swarm-setup.md Step 8b says:

set -a && source .env && set +a
docker stack deploy -c docker-compose.yaml genieai

But docker stack deploy does NOT perform variable substitution from environment variables. Variables like ${ARANGO_PASSWORD} remain unresolved literals. Services crash.

Fix: Document that manual Swarm deployment requires pre-resolving:

docker compose config > docker-compose.resolved.yaml
docker stack deploy -c docker-compose.resolved.yaml genieai

Or add a clear note pointing to the Ansible playbook as the recommended approach.

🟢 Minor: Self-Signed Cert Decision Matrix Missing from Ansible README

Both docs/docker-compose-setup.md and docs/docker-swarm-setup.md have excellent self-signed cert decision matrices explaining NODE_TLS_REJECT_UNAUTHORIZED, OPEA_SSL_SKIP_VERIFY, and KEYCLOAK_SSL_SKIP_VERIFY. The Ansible README doesn't have this matrix.

The env template does auto-set NODE_TLS_REJECT_UNAUTHORIZED=0 when self_signed_certs: true, which is good. But a deployer with self_signed_certs: true + remote GPU wouldn't know they also need opea_ssl_skip_verify: "1" in their vars.yml.

🟢 Minor: Unused node_tls_reject_unauthorized Var in cloud_deploy vars.yml

The cloud_deploy group_vars/cloud_deploy/vars.yml line 40 has node_tls_reject_unauthorized: "0", but the env template at templates/env.j2 line 331 uses self_signed_certs | bool to set NODE_TLS_REJECT_UNAUTHORIZED. The node_tls_reject_unauthorized var is unused — the template ignores it. Confusing for deployers.

Summary

Priority Issue Fix Location
🔴 Critical Playbook overwrites source compose; README claims it doesn't deploy/ansible/deploy.yml lines 612-656
🟡 Moderate KEYCLOAK_URL routes backend through public proxy chain deploy/ansible/templates/env.j2 line 94
🟡 Moderate Keycloak fails without SMTP — no warning in docs deploy/ansible/README.md + realm config
🟡 Moderate Manual Swarm deploy guide doesn't resolve variables docs/docker-swarm-setup.md Step 8b
🟢 Minor Self-signed cert decision matrix missing from Ansible README deploy/ansible/README.md
🟢 Minor Unused node_tls_reject_unauthorized var group_vars/cloud_deploy/vars.yml line 40