Skip to content

AI-Driven Deployment: Make GENIE.AI deployable by any AI agent

Problem

Over 5 deployment sessions to cloud (10.0.0.101), 29 distinct mistakes were made deploying GENIE.AI. Root cause analysis reveals the documentation and playbook have gaps that allow a deployer (human or AI) to make critical errors despite the answers existing somewhere in the docs. The docs are written for humans who can infer missing context — they are NOT structured for AI agents to follow mechanically.

Goal

"The dumbest of dumb users MUST be able to use ANY AI agent (Claude Code, Codex, Cursor, GitHub Copilot, Windsurf) to deploy GENIE.AI to a new server without any prior knowledge of the project."

This issue tracks ALL changes needed to achieve AI-driven deployment: a new AGENTS.md, a critical playbook fix, documentation gap remediation, and platform compatibility files.

Deliverables

D1: AGENTS.md — Cross-Platform AI Agent Deployment Guide (NEW)

File: AGENTS.md (repo root, ~450 lines)

The centerpiece. A self-contained deployment guide that any AI agent can follow mechanically. Does NOT duplicate CLAUDE.md (which covers coding conventions, testing, architecture).

Structure:

  1. Pre-Deployment Questionnaire — 9 questions the agent asks the user IN ORDER before touching any files:

    • Q1: Server access (IP, SSH user, SSH key path — offer to generate if missing)
    • Q2: Environment name (underscores only, no hyphens)
    • Q3: Topology decision tree → one of 4 topologies:
      • single-node-gpu (server has GPU)
      • single-node-no-gpu (no GPU, no remote GPU)
      • multi-node-remote-gpu (no local GPU, remote GPU endpoint)
      • multi-node-local-gpu (advanced, multi-node with local GPU)
    • Q4: GPU details (which model → env file: T4 → env.t4, RTX 6000 → env.rtx6000)
    • Q5: SSL certificates (self-signed / user-provided / Let's Encrypt)
    • Q6: Domain and ports
    • Q7: Email/SMTP (required for user registration)
    • Q8: Observability (optional, needs extra Grafana secrets)
    • Q9: Git access (deploy key on server?)

  2. Step-by-Step Deployment Procedure — 7 steps with exact commands:

    • Phase 0: Prerequisites check (Ansible installed, SSH connectivity)
    • Step 1: Install Ansible collections
    • Step 2: Create inventory file (full template including ansible_ssh_private_key_file)
    • Step 3: Create vars.yml (topology-specific template)
    • Step 4: Create vault.yml (all 14+ secrets, with openssl rand -base64 24 generate commands)
    • Step 5: Handle SSL certificates (3 paths)
    • Step 6: Execute deployment (ansible-playbook command)
    • Step 7: Verify (3 health checks)

  3. Topology-Specific Configurations — Exact vars.yml and vault.yml content for each of the 4 topologies

  4. Error Recovery — Common failures and exact fix commands

  5. Rollback Procedure — teardown.yml usage with volume/registry cleanup options

  6. Vault Secrets Reference (structured YAML) — Every secret, required/optional, which topology, auto-generate command

  7. Variables Reference (structured YAML) — Every non-secret variable with defaults

Key design decisions:

  • Questionnaire FIRST — prevents mid-deploy stops to ask for passwords

  • Topology as first-class concept — the #1 (closed) historical mistake was confusing deploy_opea with GPU settings

  • Structured YAML blocks — agents can parse programmatically

  • Self-contained — no "see README.md for details" for critical steps

  • Create AGENTS.md at repository root

  • Pre-deployment questionnaire (9 questions)

  • Topology decision tree (4 topologies)

  • Step-by-step procedure with exact commands

  • Verification steps after each phase

  • Error recovery procedures

  • Rollback procedure

  • Structured YAML data blocks for secrets and variables

  • Under 500 lines, no duplication of CLAUDE.md

D2: Fix deploy.yml Critical Bug — Resolved Compose Overwrites Source

File: deploy/ansible/deploy.yml (lines ~612-656)

Bug: The playbook runs docker compose config and writes the result back to docker-compose.yaml, destroying the source template. On tagged re-runs (--tags build,deploy), frozen values from the previous run are baked in. Changing a vault secret and re-deploying does NOT propagate the change because the template is gone.

This single bug caused 5 of the 29 historical mistakes (#13 (closed), #14 (closed), #17 (closed), #19 (closed), #28 (closed)).

The README (lines 591, 598) already describes the INTENDED behavior: write to docker-compose.resolved.yaml.

Required changes:

  • Lines 633-636: diff/mv targets → docker-compose.resolved.yaml

  • Line 649: file permissions → docker-compose.resolved.yaml

  • Line 656: docker stack deploy -c → use docker-compose.resolved.yaml

  • Add cleanup task: remove any stale overwritten docker-compose.yaml from previous deploys (detect by absence of ${VARIABLE} references)

  • Change resolved output filename to docker-compose.resolved.yaml

  • Update stack deploy command to use resolved file

  • Update permissions task

  • Add cleanup for old overwritten files

  • Verify tagged re-runs still work correctly

D3: Fix CLAUDE.md Quick Start Paths

File: CLAUDE.md

  • Line 44: cp inventory.example → cp inventory/inventory.example (missing directory prefix)

  • Line 45: cp group_vars/test.vault.example group_vars/test.vault — this file does NOT exist. Vault files are created via ansible-vault create

  • Replace with correct sequence: mkdir -p group_vars/test → cp group_vars/itu_rtx_test/vars.yml group_vars/test/vars.yml → ansible-vault create group_vars/test/vault.yml

  • Add cross-reference to AGENTS.md for deployment

  • Fix inventory path

  • Remove non-existent vault.example reference

  • Add correct vault creation command

  • Add AGENTS.md cross-reference

D4: Fill Ansible README Documentation Gaps

File: deploy/ansible/README.md

Add these missing sections:

A. SSH Key Setup (new subsection under Prerequisites):

  • Generate key: ssh-keygen -t ed25519 -f ~/.ssh/deploy-key -N ""
  • Copy to server: ssh-copy-id -i ~/.ssh/deploy-key.pub user@server
  • WSL caveat: keys on Windows mounts have 777 permissions, SSH refuses them
  • Test: ssh <user>@<host> "docker --version"
  • Set ansible_ssh_private_key_file in inventory

B. Self-Signed Certificate Decision Matrix (table):

Scenario self_signed_certs certbot_email certbot_replicas SSL files
Testing/internal true (empty) "0" None (auto-generated)
Production own certs false (empty) "0" server.crt + server.key
Production Let's Encrypt true (fallback) admin@example.com "1" None initially

C. DEPLOY_OPEA Warning (prominent callout before GPU section):

WARNING: Setting deploy_opea: "0" disables ALL AI/ML services (ChatQnA, Retriever, Dataprep, Reranker, vLLM, TEI). The chat feature will NOT work. Only use deploy_opea: "0" for frontend-only or API-only deployments. If you have a remote GPU node, keep deploy_opea: "1" and set gpu_node_host — the playbook automatically sets GPU_MODEL_REPLICAS=0 to skip GPU containers while deploying orchestrators locally.

D. Email/SMTP Requirements (after Email Configuration):

  • Email is REQUIRED for user registration and password reset

  • All 6 email fields must be set together

  • If SMTP unavailable: set keycloak_verify_email: "false" in vars.yml

  • Fix vault table: mark email_password as "required for user registration" not "always"

  • Add SSH Key Setup section

  • Add Self-Signed Certificate Decision Matrix

  • Add DEPLOY_OPEA warning box

  • Add Email/SMTP Requirements section

  • Fix email_password vault table entry

D5: Update inventory.example

File: deploy/ansible/inventory/inventory.example

  • Add commented ansible_ssh_private_key_file=~/.ssh/id_ed25519 to [all:vars] section

  • Add SSH access instructions comment block near top

  • Add ansible_ssh_private_key_file parameter

  • Add SSH access instructions

D6: Add nginx_permissions_policy Default

File: deploy/ansible/group_vars/all.yml

Add one line after nginx port defaults:

nginx_permissions_policy: "camera=(), microphone=(), geolocation=()"

The README (line 185) says "Default: camera=(), microphone=()" but the default does NOT exist in all.yml. Only 2 of 5 environments define it. New environments created from scratch hit an undefined variable error.

  • Add default to all.yml

D7: Platform Compatibility Symlinks

Files: .cursorrules → AGENTS.md, .windsurfrules → AGENTS.md

Create symlinks so Cursor and Windsurf users get deployment guidance automatically without manual configuration. Git stores symlinks natively.

  • Create .cursorrules symlink → AGENTS.md
  • Create .windsurfrules symlink → AGENTS.md

Acceptance Criteria

  1. Fresh agent test: A user with zero GENIE.AI knowledge instructs any supported AI agent to "deploy GENIE.AI to server X" and the agent completes it using only AGENTS.md
  2. All 4 topologies can be deployed using AGENTS.md guidance (single-node-gpu, single-node-no-gpu, multi-node-remote-gpu, multi-node-local-gpu)
  3. deploy.yml fix: Source docker-compose.yaml is never overwritten; resolved output goes to docker-compose.resolved.yaml
  4. No undefined variable errors for new environments (nginx_permissions_policy, email variables)
  5. CLAUDE.md paths correct: Quick Start commands all reference files that actually exist
  6. Cross-platform: AGENTS.md works with Claude Code, Codex, and GitHub Copilot; symlinks cover Cursor and Windsurf

Implementation Order

  1. D6 (all.yml default) — one-liner, zero risk
  2. D2 (deploy.yml fix) — critical, must work before testing
  3. D5 (inventory.example) — small, template for AGENTS.md
  4. D4 (README gaps) — informs AGENTS.md content
  5. D1 (AGENTS.md) — main deliverable
  6. D3 (CLAUDE.md fixes) — small corrections
  7. D7 (symlinks) — trivial, last
Background: 28 Documentation Gaps Identified

CRITICAL (2)

  1. Resolved compose file path — README says docker-compose.resolved.yaml, playbook overwrites docker-compose.yaml in place
  2. nginx_permissions_policy has no default in all.yml — README claims it does

HIGH (6)

  1. CLAUDE.md Quick Start has wrong paths (inventory.example missing inventory/ prefix, test.vault.example does not exist)
  2. Email/SMTP documentation inconsistent — Compose guide omits entirely, Ansible marks as "always required" but playbook does not validate
  3. SSH key setup not documented as prerequisite — inventory example does not show ansible_ssh_private_key_file
  4. Self-signed cert decision matrix missing from Ansible README — both Compose and Swarm guides have it
  5. DEPLOY_OPEA vs GPU_MODEL_REPLICAS not clearly distinguished — no warning in Ansible README (docker-swarm-setup.md has one)
  6. email_password listed as "always required" but playbook uses default('') — inconsistent

MEDIUM (8)

9-16. Missing secrets in manual deployment guides: KC_DATAPREP_CLIENT_SECRET, KC_MOBILE_CLIENT_ID, KC_MOBILE_REDIRECT_SCHEME, KEYCLOAK_CLIENT_SECRET, KEYCLOAK_PROXY_CLIENT_SECRET, GENIE_ADMIN_EMAIL, VUE_APP_CSP_CONNECT_SRC, OPEA_HOST not in env.j2

LOW (6)

17-22. Index Quick Start misleading, proxy client secret missing, inventory example lacks GPU group explanation, source .env breaks with special chars, observability vault secrets not validated, smoke tests assume localhost

INFO (2)

23-24. Unreviewed files (deploy-gpu.yml, teardown.yml), redundant node_tls_reject_unauthorized in cloud_deploy/vars.yml

Background: Root Cause Patterns (6 cross-cutting failures)

  1. Not reading memory/docs despite instructions — Memory files had SSH paths, vault passwords, working configs, and known issues. AI did not read them at session start. User had to remind 3+ times per session.

  2. Manual changes overwritten by playbook — At least 4 instances where manual .env or docker-compose.yaml edits were wiped by Ansible template regeneration.

  3. Cascading failures from compose overwrite — The source compose overwrite (D2) caused: empty SMTP vars, NODE_TLS_REJECT_UNAUTHORIZED stuck at 1, stale image tags, COMPOSE_UNCHANGED on re-runs. Each symptom debugged individually instead of tracing to root cause.

  4. Wrong network/performance diagnoses — Three wrong diagnoses: "security group" (was MTU), "TLS+WireGuard" (was missing services + backend Keycloak round-trips), "nginx server_name" (was MTU).

  5. SSH/credential friction — Every cloud session wasted 3-6 turns on SSH failures: passphrase-protected keys, executable vault files, expired PATs, wrong keys.

  6. Lazy pattern-matching — "deploy_opea: 0 means no local GPU" without reading the variable. "It's TLS handshake time" without profiling. Docs had answers but were not structured to prevent misinterpretation.

Related Issues

Edited by David FORDEN