Skip to content

fix(ansible): preserve nvidia runtime when clearing userns-remap + flush before verify

Jérôme Revillard requested to merge worktree-fix-ansible-gpu-runtime into main

Problem

The GPU-node deploy broke the nvidia Docker runtime on every ansible run (Verify NVIDIA runtime failed; docker info showed only runc despite daemon.json having the nvidia config). Two compounding bugs:

  1. roles/docker — Clear Docker daemon userns-remap config wrote a bare {} to /etc/docker/daemon.json, clobbering the nvidia runtime config owned by deploy-gpu.yml. Each run wiped nvidia → restarted Docker → docker info showed runc only. Manual restarts didn't help because the next run wiped it again.
  2. deploy-gpu.yml — Verify NVIDIA runtime ran BEFORE the Restart Docker handler applied the freshly-written daemon.json, so it verified stale state.

Fix

  1. roles/docker/tasks/main.yml — slurp daemon.json and strip ONLY the userns-remap key, preserving everything else (nvidia runtime, DNS, default-runtime). Skips if daemon.json is absent.
  2. deploy-gpu.yml — add meta: flush_handlers before Verify NVIDIA runtime so the Docker restart (applying daemon.json + config.toml) lands before the check.

Validation

Jinja render via the real ansible engine (sample daemon.json with nvidia + userns-remap + dns):

{ "default-runtime": "nvidia", "dns": ["1.1.1.1"], "runtimes": { "nvidia": { "path": "nvidia-container-runtime" } } }

→ userns-remap stripped; default-runtime, runtimes.nvidia, dns preserved. Confirmed working on the el-salvador GPU node (10.0.0.110): nvidia runtime registered, deploy proceeds.

Merge request reports