homelab/compose-files
0
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
5edebd977d4948426cbce6dbfa844daa34827cd6
compose-files/.woodpecker/deploy.yml
T
Bot eee6e543b1 Fix phantom verify failure: exclude secrets/ from stack detection + errexit-proof verify's stack-ps 
Root cause of PR #16's pipeline failure (#398-#401): secrets/ is a
tooling folder (secrets-map.yaml, *.secrets.example), not a stack, but
folder-detection (cut -d/ -f1, excluding only deploy/) computed
ALL_STACKS="ai secrets" the first time a commit touched secrets/. The
deploy step survived only because 'secrets' coincidentally sits in the
bootstrap-tier skip list; verify had no guard — `docker stack ps
secrets` exited non-zero and, under errexit, the OUTPUT=$(...)
assignment killed the step before the designed WARNING path could run.
The ai deploy itself was fully successful (complete 21-key render, all
6 services healthy) — the failure was pure phantom. Hazard first
flagged July 2026 in PR #3 (closed unmerged).

Changes:
- grep -v '^deploy$'  ->  grep -vE '^(deploy|secrets)$' in ALL five
  folder-detection sites (validate, provision-secrets x2, deploy,
  verify)
- verify: `|| true` inside the stack-ps command substitution so a
  genuinely missing stack warns instead of aborting (comment added)
- deploy step: also rsync secrets/ tooling to the host mirror
  unconditionally, same treatment as deploy/
- header: 2026-09-08 incident note

Everything else byte-identical to main. Dollar-escape audit done.
2026-09-07 22:26:43 -07:00

502 lines
26 KiB
YAML
Raw Blame History

when:
- event: push
branch: main
# ─────────────────────────────────────────────────────────────────────────────
# IMPORTANT — Woodpecker variable substitution rules (learned the hard way):
#
# Woodpecker pre-processes the ENTIRE yaml text (comments included!) before
# the shell runs anything:
# - A dollar sign followed by a braced variable name is substituted at
# pipeline-compile time from Woodpecker's own metadata (the CI_* vars).
# Secrets DO NOT exist in that map, so a braced reference to a
# secret-backed env var silently becomes an EMPTY STRING.
# - A double dollar sign is unescaped to a single dollar sign and passed
# through to the shell untouched.
#
# Therefore:
# - Braced, single-dollar form: ONLY for Woodpecker CI_* metadata vars.
# - Double-dollar braced form: for everything that must be resolved by the
# shell at runtime (i.e., every from_secret-backed environment variable).
# - Bare single-dollar VAR (no braces) also passes through to the shell.
# - NEVER write a literal dollar-brace sequence in comments either — the
# substitution engine parses comments too and will fail the pipeline
# with "missing closing brace" on anything it cannot parse.
#
# This was the root cause of a long-running "SWARM_MANAGER_IP secret is
# empty" failure: braced references were blanked at compile time before the
# shell ever saw them.
#
# 2026-08-26 HOTFIX: a full-file rewrite (AI secrets migration PR) dropped
# one $ from every $${VAR} occurrence throughout this file, re-introducing
# exactly the bug described above for EVERY secret-backed var, not just the
# new AI ones. The if-empty guards caught it immediately (SWARM_MANAGER_IP
# came back blank) and aborted before any ssh/scp/rsync ran, so no live
# secret or service was touched — but no CI provisioning/deploy could run
# until this was restored. Lesson: grep for the literal string '$${' and
# diff the count against the previous version before ever committing a
# full-file rewrite of this pipeline.
#
# 2026-08-26 FIX: FOLDER_STACKS/FOLDERS detection (grep -E '^[^/.][^/]*/')
# matches ANY non-dot top-level folder in the changed-files list, including
# deploy/ — the shared tooling folder, not a stack. A PR touching only
# deploy/envparse.py caused stack-deploy.sh to be invoked with "deploy" as
# a stack name, which correctly errored ("No main compose file... in
# .../deploy") since deploy/ has no deploy.yaml. No live service was
# affected (the error occurs before any redeploy), but it produced a
# confusing pipeline failure on an otherwise-correct change. deploy/ is
# already unconditionally rsynced at the top of the deploy step regardless
# of which stacks changed, so it's safe to exclude it from the stack list
# everywhere folders are detected below.
#
# 2026-09-03/07 INCIDENT + REDESIGN: hand-maintained grep -v + printf
# line-surgery cases repeatedly drifted (missing '=', duplicated keys,
# mismatched env var names rendering EMPTY secrets) and broke live
# services. Root cause: no authoritative key list and shell heredocs
# hostile to hand-editing. Stacks migrate one at a time to a data-driven
# model: secrets/secrets-map.yaml (data only) + per-stack .env.template
# (authoritative FULL file) + deploy/provision-stack.py (whole-file
# render, hard failure naming any missing value). Migrated stacks call
# the script; unmigrated stacks keep legacy case entries until their own
# PR. See the "Secrets & Deployment Architecture — Global Direction" note.
#
# 2026-09-08 FIX: secrets/ is a tooling/docs folder (secrets-map.yaml +
# *.secrets.example), not a stack — but folder-detection treated it as one
# the first time a commit touched it (PR #16). deploy survived only because
# 'secrets' sits in the bootstrap-tier skip list; verify had no guard and
# died on `docker stack ps secrets` failing under errexit (assignment from
# a failing command substitution aborts the step). Fixed by excluding
# secrets/ alongside deploy/ in ALL folder-detection sites, and by
# tolerating a failing stack-ps in verify (|| true) so a genuinely missing
# stack produces the designed WARNING instead of killing the step. This
# hazard was first flagged in July (PR #3, closed unmerged).
# ─────────────────────────────────────────────────────────────────────────────
steps:
validate:
image: alpine:latest
commands:
- apk add --no-cache docker-cli docker-cli-compose
- |
# Collect changed stacks — both flat files and folder-based
# NOTE: Woodpecker 3.16 exposes changed files as CI_PIPELINE_FILES, a JSON
# array string. CI_* metadata vars are correctly substituted at compile time.
CHANGED_FILES=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' '\n')
# Flat: any root-level *.yaml
FLAT=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.ya?ml$' || true)
# Folder: any file under a subfolder (e.g. immich/immich.yml).
# Exclude dotfolders (.woodpecker, .git, .gitea, etc.) and the
# non-stack tooling folders deploy/ and secrets/ (see notes above).
FOLDERS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -vE '^(deploy|secrets)$' | sort -u || true)
[ -z "$FLAT" ] && [ -z "$FOLDERS" ] && echo "No stacks changed" && exit 0
for f in $FLAT; do
[ -f "$f" ] || continue
docker compose -f "$f" config --no-interpolate -q \
&& echo " OK $f" || { echo " FAIL $f"; exit 1; }
done
for d in $FOLDERS; do
[ -d "$d" ] || continue
# Build -f flags for all compose files in folder, main first
MAIN=""
EXTRAS=""
for ext in yml yaml; do
[ -f "$d/$d.$ext" ] && MAIN="$d/$d.$ext" && break
done
[ -z "$MAIN" ] && echo " SKIP $d (no main compose file)" && continue
for cf in $(find "$d" -maxdepth 1 \( -name '*.yml' -o -name '*.yaml' \) | sort); do
[ "$cf" = "$MAIN" ] && continue
EXTRAS="$EXTRAS -f $cf"
done
docker compose -f "$MAIN" $EXTRAS config --no-interpolate -q \
&& echo " OK $d/" || { echo " FAIL $d/"; exit 1; }
done
provision-secrets:
image: alpine:latest
environment:
SSH_KEY:
from_secret: ssh_key
SWARM_MANAGER_IP:
from_secret: swarm_manager_ip
PRINT3D_DB_PASSWORD:
from_secret: 3dprint_db_password
GAMMA_AUTH_TOKEN:
from_secret: gamma_auth_token
GIT_DB_PASSWORD:
from_secret: git_db_password
GIT_RUNNER_TOKEN:
from_secret: git_runner_token
GIT_MCP_ACCESS_TOKEN:
from_secret: git_mcp_access_token
HOMEASSISTANT_FRIGATE_RTSP_PASSWORD:
from_secret: homeassistant_frigate_rtsp_password
HOMEASSISTANT_IMMICH_API_KEY:
from_secret: homeassistant_immich_api_key
IMMICH_DB_PASSWORD:
from_secret: immich_db_password
IMMICH_KIOSK_BASICAUTH:
from_secret: immich_kiosk_basicauth
MEALIE_DB_PASSWORD:
from_secret: mealie_db_password
MEALIE_LDAP_QUERY_PASSWORD:
from_secret: mealie_ldap_query_password
MESHCENTRAL_BACKUP_PASSWORD:
from_secret: meshcentral_backup_password
N8N_DB_PASSWORD:
from_secret: n8n_db_password
N8N_ENCRYPTION_KEY:
from_secret: n8n_encryption_key
POSTGRESQL_PASSWORD:
from_secret: postgresql_password
POSTGRESQL_PGADMIN_PASSWORD:
from_secret: postgresql_pgadmin_password
POSTGRESQL_REPLICATION_PASSWORD:
from_secret: postgresql_replication_password
POSTGRESQL_PATRONI_PASSWORD:
from_secret: postgresql_patroni_password
PRODUCTIVITY_PAPERLESS_SECRET_KEY:
from_secret: productivity_paperless_secret_key
PRODUCTIVITY_DB_PASSWORD:
from_secret: productivity_db_password
PRODUCTIVITY_OIDC_PROVIDERS:
from_secret: productivity_oidc_providers
VAULTWARDEN_ADMIN_TOKEN:
from_secret: vaultwarden_admin_token
VAULTWARDEN_DATABASE_URL:
from_secret: vaultwarden_database_url
ENTERTAINMENT_DISCORD_TOKEN:
from_secret: entertainment_discord_token
ENTERTAINMENT_DISCORD_CLIENT_SECRET:
from_secret: entertainment_discord_client_secret
ENTERTAINMENT_SECRET_KEY_BASE:
from_secret: entertainment_secret_key_base
ENTERTAINMENT_BASIC_AUTH_PASSWORD:
from_secret: entertainment_basic_auth_password
ENTERTAINMENT_SPARKY_DB_PASSWORD:
from_secret: entertainment_sparky_db_password
ENTERTAINMENT_SPARKY_APP_DB_PASSWORD:
from_secret: entertainment_sparky_app_db_password
ENTERTAINMENT_SPARKY_ENCRYPTION_KEY:
from_secret: entertainment_sparky_encryption_key
ENTERTAINMENT_BETTER_AUTH_SECRET:
from_secret: entertainment_better_auth_secret
# ── ai stack (manifest-driven — secrets/secrets-map.yaml +
# ai/ai.env.template + deploy/provision-stack.py). Env var names
# below match the template placeholders EXACTLY; this block is the
# only per-secret touchpoint left in this file for migrated stacks
# (Woodpecker v3 requires explicit from_secret declarations). ──
AI_AWS_ACCESS_KEY_ID:
from_secret: ai_aws_access_key_id
AI_AWS_SECRET_ACCESS_KEY:
from_secret: ai_aws_secret_access_key
AI_LITELLM_MASTER_KEY:
from_secret: ai_litellm_master_key
AI_LITELLM_SALT_KEY:
from_secret: ai_litellm_salt_key
AI_LITELLM_DATABASE_URL:
from_secret: ai_litellm_database_url
AI_LITELLM_POSTGRES_PASSWORD:
from_secret: ai_litellm_postgres_password
AI_OPEN_WEBUI_SECRET_KEY:
from_secret: ai_webui_secret_key
AI_OPEN_WEBUI_DATABASE_URL:
from_secret: ai_open_webui_database_url
AI_OPEN_WEBUI_OAUTH_CLIENT_SECRET:
from_secret: ai_oauth_client_secret
AI_MCPO_API_KEY:
from_secret: ai_mcpo_api_key
FLOWAGENT_AZURE_CLIENT_ID:
from_secret: flowagent_azure_client_id
FLOWAGENT_AZURE_TENANT_ID:
from_secret: flowagent_azure_tenant_id
FLOWAGENT_AZURE_CLIENT_SECRET:
from_secret: flowagent_azure_client_secret
commands:
- apk add --no-cache openssh-client python3 py3-yaml
- mkdir -p ~/.ssh
- echo "$SSH_KEY" | base64 -d > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- |
if [ -z "$${SWARM_MANAGER_IP}" ]; then
echo "ERROR: SWARM_MANAGER_IP secret is empty. Check Woodpecker repo secrets."
exit 1
fi
- ssh-keyscan -H $${SWARM_MANAGER_IP} >> ~/.ssh/known_hosts 2>/dev/null || true
- |
CHANGED_FILES=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' '\n')
FLAT_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.yaml$' | sed 's/\.yaml$//' || true)
FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -vE '^(deploy|secrets)$' | sort -u || true)
ALL_STACKS=$(printf '%s\n%s' "$FLAT_STACKS" "$FOLDER_STACKS" | grep -v '^$' | sort -u)
[ -z "$ALL_STACKS" ] && echo "No stacks changed, skipping" && exit 0
- scp -o StrictHostKeyChecking=no deploy/create-secrets.sh root@$${SWARM_MANAGER_IP}:/tmp/cs.sh
- |
CHANGED_FILES=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' '\n')
FLAT_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.yaml$' | sed 's/\.yaml$//' || true)
FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -vE '^(deploy|secrets)$' | sort -u || true)
ALL_STACKS=$(printf '%s\n%s' "$FLAT_STACKS" "$FOLDER_STACKS" | grep -v '^$' | sort -u)
for STACK in $ALL_STACKS; do
echo "Provisioning: $STACK"
case "$STACK" in
maintenance|media|unifi|guacamole|security|auth|traefik|meshcentral|ddm)
echo " No Docker secrets for $STACK — secrets in host .env";;
immich)
ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "source /tmp/cs.sh
create_or_update_secret 'immich_db_password' '$${IMMICH_DB_PASSWORD}'
create_or_update_secret 'immich_kiosk_basicauth' '$${IMMICH_KIOSK_BASICAUTH}'";;
woodpecker)
echo " Manual only — skipping";;
3dprint)
ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "source /tmp/cs.sh
create_or_update_secret '3dprint_db_password' '$${PRINT3D_DB_PASSWORD}'";;
gamma)
ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "source /tmp/cs.sh
create_or_update_secret 'gamma_auth_token' '$${GAMMA_AUTH_TOKEN}'";;
git)
# PATTERN B, DELIBERATE (see decision notes) — git hosts the source
# of truth for every other stack's compose files, so it must be
# restorable from a flat git.yaml + git.env backup alone, with zero
# dependency on a running Swarm's Docker secret store. Native Docker
# secrets (Pattern C) can't satisfy that: they only exist inside an
# already-running Swarm, which is exactly the circular dependency
# this stack can't have. Mirrors the retired ai) case's grep -v +
# printf rewrite-in-place approach, never sed (values may contain
# slash, dollar sign, ampersand).
#
# TEST PHASE: target is git.env.pipelinetest, NOT the real git.env.
# The real file is never opened for writing by this step. First run
# seeds the test file from the real git.env (carries over all
# non-secret lines untouched); every push after that only refreshes
# the 3 secret lines below. Also strips the legacy GITEA_ACCESS_TOKEN
# key name so the test file converges on the git.env.example-
# documented GITEA_MCP_ACCESS_TOKEN key. Cutover to the real file —
# and pointing git.yaml/stack-deploy at it — is a deliberate,
# separate follow-up after manually diffing this render.
# (Candidate for the secrets-map.yaml/provision-stack.py migration
# in its own PR; kept legacy for now.)
ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "FILE=/volume1/docker/compose-files/git.env.pipelinetest
TMP=\$FILE.tmp.\$\$
[ -f \$FILE ] || cp /volume1/docker/compose-files/git.env \$FILE
grep -vE '^(GITEA__database__PASSWD|GITEA_RUNNER_REGISTRATION_TOKEN|GITEA_MCP_ACCESS_TOKEN|GITEA_ACCESS_TOKEN)=' \$FILE > \$TMP 2>/dev/null || touch \$TMP
{ cat \$TMP
printf 'GITEA__database__PASSWD=%s\n' '$${GIT_DB_PASSWORD}'
printf 'GITEA_RUNNER_REGISTRATION_TOKEN=%s\n' '$${GIT_RUNNER_TOKEN}'
printf 'GITEA_MCP_ACCESS_TOKEN=%s\n' '$${GIT_MCP_ACCESS_TOKEN}'
} > \$FILE
rm -f \$TMP
echo ' [OK] git.env.pipelinetest updated - real git.env untouched'";;
homeassistant)
ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "source /tmp/cs.sh
create_or_update_secret 'homeassistant_frigate_rtsp_password' '$${HOMEASSISTANT_FRIGATE_RTSP_PASSWORD}'
create_or_update_secret 'homeassistant_immich_api_key' '$${HOMEASSISTANT_IMMICH_API_KEY}'";;
mealie)
ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "source /tmp/cs.sh
create_or_update_secret 'mealie_db_password' '$${MEALIE_DB_PASSWORD}'
create_or_update_secret 'mealie_ldap_query_password' '$${MEALIE_LDAP_QUERY_PASSWORD}'";;
n8n)
ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "source /tmp/cs.sh
create_or_update_secret 'n8n_db_password' '$${N8N_DB_PASSWORD}'
create_or_update_secret 'n8n_encryption_key' '$${N8N_ENCRYPTION_KEY}'";;
postgresql)
# NOTE: bootstrap-tier stack (manual deploy only via stack-deploy.sh).
# Secrets are still auto-provisioned here so they exist on the host
# before the manual `stack-deploy.sh postgresql` run picks them up.
ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "source /tmp/cs.sh
create_or_update_secret 'postgresql_password' '$${POSTGRESQL_PASSWORD}'
create_or_update_secret 'postgresql_pgadmin_password' '$${POSTGRESQL_PGADMIN_PASSWORD}'
create_or_update_secret 'postgresql_replication_password' '$${POSTGRESQL_REPLICATION_PASSWORD}'
create_or_update_secret 'postgresql_patroni_password' '$${POSTGRESQL_PATRONI_PASSWORD}'";;
productivity)
ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "source /tmp/cs.sh
create_or_update_secret 'productivity_paperless_secret_key' '$${PRODUCTIVITY_PAPERLESS_SECRET_KEY}'
create_or_update_secret 'productivity_db_password' '$${PRODUCTIVITY_DB_PASSWORD}'
create_or_update_secret 'productivity_oidc_providers' '$${PRODUCTIVITY_OIDC_PROVIDERS}'";;
vaultwarden)
# NOTE: vaultwarden_database_url cannot be rotated in-place — Swarm refuses
# to remove a secret referenced by a running service's spec. We provision
# under a versioned name instead; vaultwarden.yaml maps it back to the same
# in-container filename via target. The old secret is removed manually once
# the compose file cutover is confirmed healthy.
ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "source /tmp/cs.sh
create_or_update_secret 'vaultwarden_admin_token' '$${VAULTWARDEN_ADMIN_TOKEN}'
create_or_update_secret 'vaultwarden_database_url_v2' '$${VAULTWARDEN_DATABASE_URL}'";;
ai)
# MIGRATED (2026-09-07) to manifest-driven provisioning after the
# line-surgery approach repeatedly drifted (dropped keys 2026-09-03;
# duplicate AI_LITELLM_POSTGRES_PASSWORD + stray legacy keys +
# WEB_UI/WEBUI env-name mismatch rendering an EMPTY OAuth client
# secret, found 2026-09-07). All logic lives in
# deploy/provision-stack.py; the authoritative key list lives in
# ai/ai.env.template; the mapping lives in secrets/secrets-map.yaml.
# This case is intentionally one line.
python3 deploy/provision-stack.py ai;;
entertainment)
ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} "source /tmp/cs.sh
create_or_update_secret 'entertainment_discord_token' '$${ENTERTAINMENT_DISCORD_TOKEN}'
create_or_update_secret 'entertainment_discord_client_secret' '$${ENTERTAINMENT_DISCORD_CLIENT_SECRET}'
create_or_update_secret 'entertainment_secret_key_base' '$${ENTERTAINMENT_SECRET_KEY_BASE}'
create_or_update_secret 'entertainment_basic_auth_password' '$${ENTERTAINMENT_BASIC_AUTH_PASSWORD}'
create_or_update_secret 'entertainment_sparky_db_password' '$${ENTERTAINMENT_SPARKY_DB_PASSWORD}'
create_or_update_secret 'entertainment_sparky_app_db_password' '$${ENTERTAINMENT_SPARKY_APP_DB_PASSWORD}'
create_or_update_secret 'entertainment_sparky_encryption_key' '$${ENTERTAINMENT_SPARKY_ENCRYPTION_KEY}'
create_or_update_secret 'entertainment_better_auth_secret' '$${ENTERTAINMENT_BETTER_AUTH_SECRET}'";;
*)
echo " No secrets case for $STACK";;
esac
done
deploy:
image: alpine:latest
environment:
SSH_KEY:
from_secret: ssh_key
SWARM_MANAGER_IP:
from_secret: swarm_manager_ip
commands:
- apk add --no-cache openssh-client rsync
- mkdir -p ~/.ssh
- echo "$SSH_KEY" | base64 -d > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- |
if [ -z "$${SWARM_MANAGER_IP}" ]; then
echo "ERROR: SWARM_MANAGER_IP secret is empty. Check Woodpecker repo secrets."
exit 1
fi
- ssh-keyscan -H $${SWARM_MANAGER_IP} >> ~/.ssh/known_hosts 2>/dev/null || true
- |
CHANGED_FILES=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' '\n')
FLAT_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.yaml$' | sed 's/\.yaml$//' || true)
FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -vE '^(deploy|secrets)$' | sort -u || true)
ALL_STACKS=$(printf '%s\n%s' "$FLAT_STACKS" "$FOLDER_STACKS" | grep -v '^$' | sort -u)
[ -z "$ALL_STACKS" ] && echo "No stacks changed" && exit 0
# Always sync deploy/ scripts first
rsync -av -e "ssh -o StrictHostKeyChecking=no -i ~/.ssh/id_rsa" \
deploy/ root@$${SWARM_MANAGER_IP}:/volume1/docker/compose-files/deploy/
# Sync secrets/ tooling (manifest + examples) alongside deploy/ —
# provision-stack.py reads secrets/secrets-map.yaml from the CI
# checkout, but the host mirror should stay complete for emergency
# manual provisioning runs.
rsync -av -e "ssh -o StrictHostKeyChecking=no -i ~/.ssh/id_rsa" \
secrets/ root@$${SWARM_MANAGER_IP}:/volume1/docker/compose-files/secrets/
for STACK in $ALL_STACKS; do
echo "--- Deploying: $STACK ---"
# Sync files to host first (always, even for bootstrap stacks)
if [ -d "$STACK" ]; then
rsync -av -e "ssh -o StrictHostKeyChecking=no -i ~/.ssh/id_rsa" \
"$STACK/" root@$${SWARM_MANAGER_IP}:/volume1/docker/compose-files/$STACK/
elif [ -f "$STACK.yaml" ]; then
rsync -av -e "ssh -o StrictHostKeyChecking=no -i ~/.ssh/id_rsa" \
"$STACK.yaml" root@$${SWARM_MANAGER_IP}:/volume1/docker/compose-files/
fi
# Bootstrap-tier guard: file synced to host, deploy is MANUAL
case "$STACK" in
traefik|woodpecker|postgresql|secrets|git)
echo " [BOOTSTRAP] $STACK: file synced. Deploy is MANUAL."
echo " Run: bash /volume1/docker/compose-files/deploy/stack-deploy.sh $STACK"
continue ;;
esac
# Tier 2: auto-deploy
ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} \
"bash /volume1/docker/compose-files/deploy/stack-deploy.sh $STACK" \
&& echo " OK $STACK" || { echo " FAIL $STACK"; exit 1; }
done
verify:
image: alpine:latest
environment:
SSH_KEY:
from_secret: ssh_key
SWARM_MANAGER_IP:
from_secret: swarm_manager_ip
commands:
- apk add --no-cache openssh-client
- mkdir -p ~/.ssh
- echo "$SSH_KEY" | base64 -d > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- |
if [ -z "$${SWARM_MANAGER_IP}" ]; then
echo "ERROR: SWARM_MANAGER_IP secret is empty. Check Woodpecker repo secrets."
exit 1
fi
- ssh-keyscan -H $${SWARM_MANAGER_IP} >> ~/.ssh/known_hosts 2>/dev/null || true
- |
CHANGED_FILES=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' '\n')
FLAT_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.yaml$' | sed 's/\.yaml$//' || true)
FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -vE '^(deploy|secrets)$' | sort -u || true)
ALL_STACKS=$(printf '%s\n%s' "$FLAT_STACKS" "$FOLDER_STACKS" | grep -v '^$' | sort -u)
[ -z "$ALL_STACKS" ] && exit 0
# NOTE: `docker stack deploy` briefly tears down and recreates tasks in
# Swarm's internal bookkeeping, so `docker stack ps` can transiently
# return nothing right after deploy even when the service is healthy.
# A single `sleep 5` + one-shot check produced false-alarm-looking
# "nothing found in stack" output on ordinary deploys (e.g. vaultwarden,
# 2026-08-25). Retry with backoff instead of a single fixed sleep, and
# only warn (don't fail the pipeline) if tasks never show up.
# 2026-09-08: `|| true` inside the command substitution is REQUIRED —
# this step runs under errexit, and an assignment from a failing
# command substitution (e.g. `docker stack ps` on a stack that doesn't
# exist) kills the whole step before the WARNING path can run.
ATTEMPTS=6
DELAY=5
for STACK in $ALL_STACKS; do
echo "--- $STACK ---"
i=1
while [ "$i" -le "$ATTEMPTS" ]; do
OUTPUT=$(ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} \
"docker stack ps $STACK --filter desired-state=running \
--format ' {{.Name}} {{.CurrentState}}'" 2>/dev/null || true)
if [ -n "$OUTPUT" ]; then
echo "$OUTPUT"
break
fi
if [ "$i" -eq "$ATTEMPTS" ]; then
echo " WARNING: no running tasks found for $STACK after $((ATTEMPTS * DELAY))s."
echo " This may be transient Swarm settle time, or a real problem — check manually:"
echo " ssh root@$${SWARM_MANAGER_IP} 'docker stack ps $STACK --no-trunc'"
else
sleep "$DELAY"
fi
i=$((i + 1))
done
done
notify-success:
image: alpine:latest
environment:
TEAMS_WEBHOOK:
from_secret: teams_webhook
commands:
- apk add --no-cache curl
- |
CHANGED=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' ' ')
PAYLOAD=$(printf '{"title":"Swarm Deploy OK","text":"Stacks: %s Commit: %s","themeColor":"00aa00"}' "$CHANGED" "$CI_COMMIT_MESSAGE")
curl -sf -X POST -H 'Content-Type: application/json' -d "$PAYLOAD" "$TEAMS_WEBHOOK" || true
when:
- status: success
notify-failure:
image: alpine:latest
environment:
TEAMS_WEBHOOK:
from_secret: teams_webhook
WOODPECKER_HOST:
from_secret: woodpecker_host
commands:
- apk add --no-cache curl
- |
PAYLOAD=$(printf '{"title":"Swarm Deploy FAILED","text":"Commit: %s See: https://%s","themeColor":"ff0000"}' "$CI_COMMIT_MESSAGE" "$WOODPECKER_HOST")
curl -sf -X POST -H 'Content-Type: application/json' -d "$PAYLOAD" "$TEAMS_WEBHOOK" || true
when:
- status: failure
Reference in New Issue View Git Blame Copy Permalink