ai: migrate AWS/LiteLLM/OpenWebUI/OAuth secrets to Woodpecker #4

Merged
AVB merged 1 commits from ai-secrets-migration into main 2026-08-25 23:01:10 -07:00
Owner

What

Migrates 8 secrets currently living in plaintext in ai/ai.env on docker-2 to Woodpecker from_secret: values, matching the existing pattern used for vaultwarden/immich/mealie/etc.

Secrets migrated

  • AWS_ACCESS_KEY_IDai_aws_access_key_id
  • AWS_SECRET_ACCESS_KEYai_aws_secret_access_key
  • LITELLM_MASTER_KEYai_litellm_master_key
  • LITELLM_SALT_KEYai_litellm_salt_key
  • POSTGRES_PASSWORD (+ embedded in DATABASE_URL) → ai_litellm_db_password
  • WEBUI_SECRET_KEYai_webui_secret_key
  • OPEN_WEBUI_DATABASE_URL (full DSN) → ai_open_webui_database_url
  • OAUTH_CLIENT_SECRETai_oauth_client_secret

Explicitly NOT touched (deferred / out of scope)

  • MCPO_API_KEY — shared by mcpo and mcpo-critical in ai.yaml. Per agreed plan: migrate mcpo's usage in a follow-up PR only after this one is verified healthy, since ai_mcpo-critical is off-limits for routine changes. This PR's ai) provisioning case never reads or writes MCPO_API_KEY.
  • AWS_REGION_NAME, OAUTH_CLIENT_ID, WEBUI_URL, and other non-secret config vars stay as plain values in ai.env.

Mechanism

ai.yaml is unchanged — LiteLLM/boto3 and Open WebUI don't support the Docker-secret _FILE convention for these vars, so native Pattern C isn't a drop-in without adding wrapper entrypoints (higher risk, out of scope here). Instead, the provision-secrets step's ai) case now SSHes to the swarm manager and rewrites only the 8 migrated lines in /volume1/docker/compose-files/ai/ai.env on every deploy of the ai stack, using grep -v + printf (not sed, since some values contain $, /, &). Every other line in that file is left byte-for-byte untouched.

⚠️ Required before merge

Add these 8 secrets in Woodpecker UI → homelab/compose-files → Settings → Secrets, using the current values from /volume1/docker/compose-files/ai/ai.env on docker-2 (do not reuse old values after rotation — see note below):

ai_aws_access_key_id
ai_aws_secret_access_key
ai_litellm_master_key
ai_litellm_salt_key
ai_litellm_db_password
ai_webui_secret_key
ai_open_webui_database_url
ai_oauth_client_secret

Verification after merge

  1. Watch the provision-secrets step log for [OK] ai/ai.env secrets updated.
  2. ssh root@192.168.4.32 "grep -c '=' /volume1/docker/compose-files/ai/ai.env" — should be unchanged count, just refreshed values.
  3. Confirm ai_litellm and ai_open-webui services redeploy cleanly: docker stack ps ai --filter desired-state=running.
  4. docker service inspect ai_litellm --format '{{json .Spec.TaskTemplate.ContainerSpec.Env}}' — confirm no blank values.
  5. Confirm mcpo and mcpo-critical show no config diff / no restart (MCPO_API_KEY untouched).
  6. Hit https://ai.bryanmail.net and https://llm.bryanmail.net to confirm OAuth login and LiteLLM/Bedrock calls still work.

Follow-up (separate PR, after this is confirmed healthy)

  • Migrate MCPO_API_KEY to a Woodpecker secret for mcpo only, with explicit confirmation before extending to mcpo-critical.
  • Recommend rotating the AWS access key — it has been sitting in plaintext in a .env file on disk (not in git history, but flagging per usual hygiene).
## What Migrates 8 secrets currently living in plaintext in `ai/ai.env` on docker-2 to Woodpecker `from_secret:` values, matching the existing pattern used for vaultwarden/immich/mealie/etc. ## Secrets migrated - `AWS_ACCESS_KEY_ID` → `ai_aws_access_key_id` - `AWS_SECRET_ACCESS_KEY` → `ai_aws_secret_access_key` - `LITELLM_MASTER_KEY` → `ai_litellm_master_key` - `LITELLM_SALT_KEY` → `ai_litellm_salt_key` - `POSTGRES_PASSWORD` (+ embedded in `DATABASE_URL`) → `ai_litellm_db_password` - `WEBUI_SECRET_KEY` → `ai_webui_secret_key` - `OPEN_WEBUI_DATABASE_URL` (full DSN) → `ai_open_webui_database_url` - `OAUTH_CLIENT_SECRET` → `ai_oauth_client_secret` ## Explicitly NOT touched (deferred / out of scope) - `MCPO_API_KEY` — shared by `mcpo` **and** `mcpo-critical` in `ai.yaml`. Per agreed plan: migrate `mcpo`'s usage in a follow-up PR only after this one is verified healthy, since `ai_mcpo-critical` is off-limits for routine changes. This PR's `ai)` provisioning case never reads or writes `MCPO_API_KEY`. - `AWS_REGION_NAME`, `OAUTH_CLIENT_ID`, `WEBUI_URL`, and other non-secret config vars stay as plain values in `ai.env`. ## Mechanism `ai.yaml` is unchanged — LiteLLM/boto3 and Open WebUI don't support the Docker-secret `_FILE` convention for these vars, so native Pattern C isn't a drop-in without adding wrapper entrypoints (higher risk, out of scope here). Instead, the `provision-secrets` step's `ai)` case now SSHes to the swarm manager and rewrites **only the 8 migrated lines** in `/volume1/docker/compose-files/ai/ai.env` on every deploy of the `ai` stack, using `grep -v` + `printf` (not `sed`, since some values contain `$`, `/`, `&`). Every other line in that file is left byte-for-byte untouched. ## ⚠️ Required before merge Add these 8 secrets in Woodpecker UI → `homelab/compose-files` → Settings → Secrets, using the **current values from `/volume1/docker/compose-files/ai/ai.env` on docker-2** (do not reuse old values after rotation — see note below): ``` ai_aws_access_key_id ai_aws_secret_access_key ai_litellm_master_key ai_litellm_salt_key ai_litellm_db_password ai_webui_secret_key ai_open_webui_database_url ai_oauth_client_secret ``` ## Verification after merge 1. Watch the `provision-secrets` step log for `[OK] ai/ai.env secrets updated`. 2. `ssh root@192.168.4.32 "grep -c '=' /volume1/docker/compose-files/ai/ai.env"` — should be unchanged count, just refreshed values. 3. Confirm `ai_litellm` and `ai_open-webui` services redeploy cleanly: `docker stack ps ai --filter desired-state=running`. 4. `docker service inspect ai_litellm --format '{{json .Spec.TaskTemplate.ContainerSpec.Env}}'` — confirm no blank values. 5. Confirm `mcpo` and `mcpo-critical` show **no config diff / no restart** (MCPO_API_KEY untouched). 6. Hit https://ai.bryanmail.net and https://llm.bryanmail.net to confirm OAuth login and LiteLLM/Bedrock calls still work. ## Follow-up (separate PR, after this is confirmed healthy) - Migrate `MCPO_API_KEY` to a Woodpecker secret for `mcpo` only, with explicit confirmation before extending to `mcpo-critical`. - Recommend rotating the AWS access key — it has been sitting in plaintext in a `.env` file on disk (not in git history, but flagging per usual hygiene).
admin added 1 commit 2026-08-25 22:40:29 -07:00
Adds 8 new from_secret-backed env vars to provision-secrets and rewrites
the `ai)` case to do a targeted update of only those 8 keys in the
remote ai/ai.env via grep -v + printf (no sed, safe for values containing
/, $, &, etc). All other lines in ai.env (MCPO_API_KEY, OAUTH_CLIENT_ID,
WEBUI_URL, etc.) are left completely untouched -- MCPO_API_KEY migration
is deferred to a follow-up per plan, and this change never reads or
writes that value.

New secrets required in Woodpecker (Settings -> Secrets) before merge:
  ai_aws_access_key_id
  ai_aws_secret_access_key
  ai_litellm_master_key
  ai_litellm_salt_key
  ai_litellm_db_password
  ai_webui_secret_key
  ai_open_webui_database_url
  ai_oauth_client_secret
AVB merged commit 7d26f97a1e into main 2026-08-25 23:01:10 -07:00
Sign in to join this conversation.