Author SHA1 Message Date
Bot cb10c2e22d provision-secrets: rebuild ai) migration on current main (fixes live env-name drift incl. WEB_UI/WEBUI OAuth secret mismatch)
Rebase of the provisioning-v2 branch's deploy.yml onto current main.
Reconstructed from main line-by-line, then two scoped changes:

1. ai) case -> `python3 deploy/provision-stack.py ai` (one line). Retires
   main's current heredoc, which has active drift found during this
   rebase: AI_LITELLM_POSTGRES_PASSWORD printed TWICE (first copy sourced
   from the retired AI_LITELLM_DB_PASSWORD var), a stray legacy
   AI_WEBUI_SECRET_KEY key current ai.yaml never reads, and printf
   referencing AI_OPEN_WEBUI_OAUTH_CLIENT_SECRET while the env block
   declares AI_OPEN_WEB_UI_OAUTH_CLIENT_SECRET (WEB_UI) — undefined shell
   var at runtime, so rendered ai.env currently carries an EMPTY OAuth
   client secret.

2. env block ai section: normalized to the exact template placeholder
   names (AI_OPEN_WEBUI_OAUTH_CLIENT_SECRET <- ai_oauth_client_secret,
   AI_OPEN_WEBUI_SECRET_KEY <- ai_webui_secret_key), added AI_MCPO_API_KEY
   (NEW Woodpecker secret required pre-merge), pruned now-template-literal
   vars (AI_AWS_REGION_NAME, AI_LITELLM_MODIFY_PARAMS/_DATABASE_MIGRATIONS,
   AI_LITELLM_DB_PASSWORD, the seven ai_open_web_ui_* config secrets).

apk line gains python3 py3-yaml. All other steps and cases byte-match
current main. Dollar-escape audit done per header lesson.
2026-09-06 22:52:38 -07:00
Bot 24cc4e6655 ai.env.template: rebase onto main's AI_<SERVICE>_* naming (2026-09-06 renames)
Key names now match exactly what current ai/ai.yaml references:
AI_OPEN_WEBUI_* (was WEBUI_*/OAUTH_*), AI_LITELLM_* (was LITELLM_*/
DATABASE_URL/POSTGRES_PASSWORD), AI_AWS_REGION_NAME (was AWS_REGION_NAME).
MCPO_API_KEY stays unprefixed (ai.yaml references it unprefixed for both
mcpo and mcpo-critical).

Placeholder names match the CI env vars declared in deploy.yml's
provision-secrets block, including the normalized
AI_OPEN_WEBUI_OAUTH_CLIENT_SECRET (fixing main's WEB_UI/WEBUI typo that
currently renders an EMPTY OAuth client secret into ai.env).
2026-09-06 22:46:16 -07:00
Bot 88b3e46f77 provision-secrets: replace broken ai) line-surgery with one-line call to provision-stack.py
Part 4/4 of the provisioning redesign (matched with secrets-map.yaml,
ai/ai.env.template, deploy/provision-stack.py in this same PR).

Changes, all scoped to the provision-secrets step:
- ai) case: the 54-line grep-v+printf heredoc (which had accumulated a
  missing '=' on the DATABASE_URL printf, duplicate POSTGRES_PASSWORD and
  WEBUI_SECRET_KEY printfs, and a commented-out line from manual edits —
  the 2026-09-03 outage) is replaced by:
      python3 deploy/provision-stack.py ai
  No secret value structure (connection-string shapes etc.) remains in
  this file for the ai stack.
- environment block: pruned the AI_* declarations the template no longer
  needs (region/oauth-config/litellm-boolean values are now literals in
  ai/ai.env.template): AI_LITELLM_DB_PASSWORD, AI_LITELLM_DATABASE_
  MIGRATIONS, AI_LITELLM_MODIFY_PARAMS, AI_AWS_REGION_NAME, and the
  seven AI_OPEN_WEB_UI_* config entries. Added AI_MCPO_API_KEY
  (from_secret: ai_mcpo_api_key — NEW Woodpecker secret, must exist
  before merge). Kept the 9 real ai secrets + 3 flowagent_* entries.
- apk line gains python3 py3-yaml for the provisioner.
- Header comment: documented the 2026-09-03 incident + migration model.

Everything else — validate, all other stacks' case entries, deploy,
verify, notify steps — is byte-for-byte identical to main.
Dollar-escape audit done per the header's own lesson: all remaining
double-dollar-brace references belong to unmigrated legacy cases and are
unchanged; the new ai) case contains no dollar sequences at all.
2026-09-03 22:49:03 -07:00
Bot a91974caa1 Add deploy/provision-stack.py — generic manifest-driven provisioner
Part 3/4 of the provisioning redesign. One script for every migrated
stack; no per-stack shell code.

- Renders the stack's full env file from its .env.template (whole file,
  every run — no line surgery), fails hard listing NAMES of any
  missing/empty vars, fails hard on unresolved placeholders.
- Ships the rendered file over ssh stdin with write-temp + chmod 600 +
  atomic mv. Secret values never appear on any command line (an
  improvement over create-secrets.sh, which passes values as remote
  shell arguments).
- Creates/rotates Docker Swarm secrets from the manifest's
  docker_secrets map, values via ssh stdin, same sha256-checksum-label
  skip-if-unchanged convention as create-secrets.sh.
- Never prints a secret value — names and counts only.
- Stacks absent from the manifest exit 0 (legacy case-entries keep
  handling them), enabling one-stack-at-a-time migration.
2026-09-03 22:43:48 -07:00
Bot 8ae2ddbe97 Add ai/ai.env.template — full-file authoritative env template for the ai stack
Part 2/4 of the provisioning redesign. Key properties:

- Complete key list for ai.env in one reviewable place. Rendered whole
  every run by provision-stack.py — the "grep -v + printf line surgery"
  that dropped MCPO_API_KEY/AWS_REGION_NAME/OAUTH_* keys (2026-09-03
  incident) is gone for this stack.
- Non-secret config (region, OAuth endpoints/IDs/scopes, WEBUI_URL,
  LiteLLM booleans) as literals, values taken from the verified-working
  host ai.env. Secrets as placeholders resolved from Woodpecker secrets;
  renderer fails hard on any missing/empty placeholder.
- LiteLLM DATABASE_URL comes from ai_litellm_database_url as a complete
  opaque value — no connection-string structure in git (addresses the
  plaintext-structure concern in deploy.yml).
- Drops legacy plain AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY keys —
  ai.yaml only references the AI_-prefixed names.
- NEW Woodpecker secret required before merging: ai_mcpo_api_key
  (value = current MCPO_API_KEY from host ai.env).
2026-09-03 22:40:11 -07:00
Bot 0869cd319d Add secrets/secrets-map.yaml — data-only provisioning manifest (ai stack first)
Part 1/4 of the provisioning architecture redesign after the 2026-09-03
ai.env incident (line-surgery provisioning dropped keys; broken services;
manual host-side recovery forked ai.yaml/ai.env).

Design: plain-data manifest consumed by deploy/provision-stack.py.
No code, no shell, no secret values, no value structure (e.g. no
connection-string shapes) live in this file or in deploy.yml anymore.
2026-09-03 22:40:10 -07:00
3 changed files with 97 additions and 310 deletions
+8 -30
View File
@@ -59,17 +59,6 @@ when:
# render, hard failure naming any missing value). Migrated stacks call # render, hard failure naming any missing value). Migrated stacks call
# the script; unmigrated stacks keep legacy case entries until their own # the script; unmigrated stacks keep legacy case entries until their own
# PR. See the "Secrets & Deployment Architecture — Global Direction" note. # PR. See the "Secrets & Deployment Architecture — Global Direction" note.
#
# 2026-09-08 FIX: secrets/ is a tooling/docs folder (secrets-map.yaml +
# *.secrets.example), not a stack — but folder-detection treated it as one
# the first time a commit touched it (PR #16). deploy survived only because
# 'secrets' sits in the bootstrap-tier skip list; verify had no guard and
# died on `docker stack ps secrets` failing under errexit (assignment from
# a failing command substitution aborts the step). Fixed by excluding
# secrets/ alongside deploy/ in ALL folder-detection sites, and by
# tolerating a failing stack-ps in verify (|| true) so a genuinely missing
# stack produces the designed WARNING instead of killing the step. This
# hazard was first flagged in July (PR #3, closed unmerged).
# ───────────────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────────────
steps: steps:
@@ -87,9 +76,9 @@ steps:
FLAT=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.ya?ml$' || true) FLAT=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.ya?ml$' || true)
# Folder: any file under a subfolder (e.g. immich/immich.yml). # Folder: any file under a subfolder (e.g. immich/immich.yml).
# Exclude dotfolders (.woodpecker, .git, .gitea, etc.) and the # Exclude dotfolders (.woodpecker, .git, .gitea, etc.) and deploy/
# non-stack tooling folders deploy/ and secrets/ (see notes above). # (shared tooling, not a stack — see note above).
FOLDERS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -vE '^(deploy|secrets)$' | sort -u || true) FOLDERS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -v '^deploy$' | sort -u || true)
[ -z "$FLAT" ] && [ -z "$FOLDERS" ] && echo "No stacks changed" && exit 0 [ -z "$FLAT" ] && [ -z "$FOLDERS" ] && echo "No stacks changed" && exit 0
@@ -230,14 +219,14 @@ steps:
- | - |
CHANGED_FILES=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' '\n') CHANGED_FILES=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' '\n')
FLAT_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.yaml$' | sed 's/\.yaml$//' || true) FLAT_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.yaml$' | sed 's/\.yaml$//' || true)
FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -vE '^(deploy|secrets)$' | sort -u || true) FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -v '^deploy$' | sort -u || true)
ALL_STACKS=$(printf '%s\n%s' "$FLAT_STACKS" "$FOLDER_STACKS" | grep -v '^$' | sort -u) ALL_STACKS=$(printf '%s\n%s' "$FLAT_STACKS" "$FOLDER_STACKS" | grep -v '^$' | sort -u)
[ -z "$ALL_STACKS" ] && echo "No stacks changed, skipping" && exit 0 [ -z "$ALL_STACKS" ] && echo "No stacks changed, skipping" && exit 0
- scp -o StrictHostKeyChecking=no deploy/create-secrets.sh root@$${SWARM_MANAGER_IP}:/tmp/cs.sh - scp -o StrictHostKeyChecking=no deploy/create-secrets.sh root@$${SWARM_MANAGER_IP}:/tmp/cs.sh
- | - |
CHANGED_FILES=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' '\n') CHANGED_FILES=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' '\n')
FLAT_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.yaml$' | sed 's/\.yaml$//' || true) FLAT_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.yaml$' | sed 's/\.yaml$//' || true)
FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -vE '^(deploy|secrets)$' | sort -u || true) FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -v '^deploy$' | sort -u || true)
ALL_STACKS=$(printf '%s\n%s' "$FLAT_STACKS" "$FOLDER_STACKS" | grep -v '^$' | sort -u) ALL_STACKS=$(printf '%s\n%s' "$FLAT_STACKS" "$FOLDER_STACKS" | grep -v '^$' | sort -u)
for STACK in $ALL_STACKS; do for STACK in $ALL_STACKS; do
@@ -371,7 +360,7 @@ steps:
- | - |
CHANGED_FILES=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' '\n') CHANGED_FILES=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' '\n')
FLAT_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.yaml$' | sed 's/\.yaml$//' || true) FLAT_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.yaml$' | sed 's/\.yaml$//' || true)
FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -vE '^(deploy|secrets)$' | sort -u || true) FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -v '^deploy$' | sort -u || true)
ALL_STACKS=$(printf '%s\n%s' "$FLAT_STACKS" "$FOLDER_STACKS" | grep -v '^$' | sort -u) ALL_STACKS=$(printf '%s\n%s' "$FLAT_STACKS" "$FOLDER_STACKS" | grep -v '^$' | sort -u)
[ -z "$ALL_STACKS" ] && echo "No stacks changed" && exit 0 [ -z "$ALL_STACKS" ] && echo "No stacks changed" && exit 0
@@ -379,13 +368,6 @@ steps:
rsync -av -e "ssh -o StrictHostKeyChecking=no -i ~/.ssh/id_rsa" \ rsync -av -e "ssh -o StrictHostKeyChecking=no -i ~/.ssh/id_rsa" \
deploy/ root@$${SWARM_MANAGER_IP}:/volume1/docker/compose-files/deploy/ deploy/ root@$${SWARM_MANAGER_IP}:/volume1/docker/compose-files/deploy/
# Sync secrets/ tooling (manifest + examples) alongside deploy/ —
# provision-stack.py reads secrets/secrets-map.yaml from the CI
# checkout, but the host mirror should stay complete for emergency
# manual provisioning runs.
rsync -av -e "ssh -o StrictHostKeyChecking=no -i ~/.ssh/id_rsa" \
secrets/ root@$${SWARM_MANAGER_IP}:/volume1/docker/compose-files/secrets/
for STACK in $ALL_STACKS; do for STACK in $ALL_STACKS; do
echo "--- Deploying: $STACK ---" echo "--- Deploying: $STACK ---"
# Sync files to host first (always, even for bootstrap stacks) # Sync files to host first (always, even for bootstrap stacks)
@@ -432,7 +414,7 @@ steps:
- | - |
CHANGED_FILES=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' '\n') CHANGED_FILES=$(echo "${CI_PIPELINE_FILES}" | tr -d '[]"' | tr ',' '\n')
FLAT_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.yaml$' | sed 's/\.yaml$//' || true) FLAT_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/]+\.yaml$' | sed 's/\.yaml$//' || true)
FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -vE '^(deploy|secrets)$' | sort -u || true) FOLDER_STACKS=$(echo "$CHANGED_FILES" | grep -E '^[^/.][^/]*/' | cut -d/ -f1 | grep -v '^deploy$' | sort -u || true)
ALL_STACKS=$(printf '%s\n%s' "$FLAT_STACKS" "$FOLDER_STACKS" | grep -v '^$' | sort -u) ALL_STACKS=$(printf '%s\n%s' "$FLAT_STACKS" "$FOLDER_STACKS" | grep -v '^$' | sort -u)
[ -z "$ALL_STACKS" ] && exit 0 [ -z "$ALL_STACKS" ] && exit 0
@@ -443,10 +425,6 @@ steps:
# "nothing found in stack" output on ordinary deploys (e.g. vaultwarden, # "nothing found in stack" output on ordinary deploys (e.g. vaultwarden,
# 2026-08-25). Retry with backoff instead of a single fixed sleep, and # 2026-08-25). Retry with backoff instead of a single fixed sleep, and
# only warn (don't fail the pipeline) if tasks never show up. # only warn (don't fail the pipeline) if tasks never show up.
# 2026-09-08: `|| true` inside the command substitution is REQUIRED —
# this step runs under errexit, and an assignment from a failing
# command substitution (e.g. `docker stack ps` on a stack that doesn't
# exist) kills the whole step before the WARNING path can run.
ATTEMPTS=6 ATTEMPTS=6
DELAY=5 DELAY=5
for STACK in $ALL_STACKS; do for STACK in $ALL_STACKS; do
@@ -455,7 +433,7 @@ steps:
while [ "$i" -le "$ATTEMPTS" ]; do while [ "$i" -le "$ATTEMPTS" ]; do
OUTPUT=$(ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} \ OUTPUT=$(ssh -o StrictHostKeyChecking=no root@$${SWARM_MANAGER_IP} \
"docker stack ps $STACK --filter desired-state=running \ "docker stack ps $STACK --filter desired-state=running \
--format ' {{.Name}} {{.CurrentState}}'" 2>/dev/null || true) --format ' {{.Name}} {{.CurrentState}}'" 2>/dev/null)
if [ -n "$OUTPUT" ]; then if [ -n "$OUTPUT" ]; then
echo "$OUTPUT" echo "$OUTPUT"
break break
+19 -20
View File
@@ -16,20 +16,20 @@ services:
start_period: 60s start_period: 60s
environment: environment:
- OLLAMA_BASE_URL=http://ollama-intel-arc:11434 - OLLAMA_BASE_URL=http://ollama-intel-arc:11434
- WEBUI_SECRET_KEY=${AI_OPEN_WEBUI_SECRET_KEY} - WEBUI_SECRET_KEY=${WEBUI_SECRET_KEY}
- WEBUI_DB_HOST=postgresql - WEBUI_DB_HOST=postgresql
- DATABASE_URL=${AI_OPEN_WEBUI_DATABASE_URL} - DATABASE_URL=${OPEN_WEBUI_DATABASE_URL}
- ENABLE_TOOL_SERVER_CALLING=true - ENABLE_TOOL_SERVER_CALLING=true
- TOOL_SERVER_CALLING=true - TOOL_SERVER_CALLING=true
- WEBUI_URL=${AI_OPEN_WEBUI_URL} - WEBUI_URL=${WEBUI_URL}
- ENABLE_OAUTH_SIGNUP=${AI_OPEN_WEBUI_ENABLE_OAUTH_SIGNUP} - ENABLE_OAUTH_SIGNUP=${ENABLE_OAUTH_SIGNUP}
- OAUTH_MERGE_ACCOUNTS_BY_EMAIL=${AI_OPEN_WEBUI_OAUTH_MERGE_ACCOUNTS_BY_EMAIL} - OAUTH_MERGE_ACCOUNTS_BY_EMAIL=${OAUTH_MERGE_ACCOUNTS_BY_EMAIL}
- OAUTH_PROVIDER_NAME=${AI_OPEN_WEBUI_OAUTH_PROVIDER_NAME} - OAUTH_PROVIDER_NAME=${OAUTH_PROVIDER_NAME}
- OPENID_PROVIDER_URL=${AI_OPEN_WEBUI_OPENID_PROVIDER_URL} - OPENID_PROVIDER_URL=${OPENID_PROVIDER_URL}
- OAUTH_CLIENT_ID=${AI_OPEN_WEBUI_OAUTH_CLIENT_ID} - OAUTH_CLIENT_ID=${OAUTH_CLIENT_ID}
- OAUTH_CLIENT_SECRET=${AI_OPEN_WEBUI_OAUTH_CLIENT_SECRET} - OAUTH_CLIENT_SECRET=${OAUTH_CLIENT_SECRET}
- OAUTH_SCOPES=${AI_OPEN_WEBUI_OAUTH_SCOPES} - OAUTH_SCOPES=${OAUTH_SCOPES}
- OPENID_REDIRECT_URI=${AI_OPEN_WEBUI_OPENID_REDIRECT_URI} - OPENID_REDIRECT_URI=${OPENID_REDIRECT_URI}
networks: networks:
- traefik_backend - traefik_backend
- postgresql_db-backend - postgresql_db-backend
@@ -62,13 +62,13 @@ services:
environment: environment:
- AWS_ACCESS_KEY_ID=${AI_AWS_ACCESS_KEY_ID} - AWS_ACCESS_KEY_ID=${AI_AWS_ACCESS_KEY_ID}
- AWS_SECRET_ACCESS_KEY=${AI_AWS_SECRET_ACCESS_KEY} - AWS_SECRET_ACCESS_KEY=${AI_AWS_SECRET_ACCESS_KEY}
- AWS_REGION_NAME=${AI_AWS_REGION_NAME} - AWS_REGION_NAME=${AWS_REGION_NAME}
- LITELLM_MASTER_KEY=${AI_LITELLM_MASTER_KEY} - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
- LITELLM_SALT_KEY=${AI_LITELLM_SALT_KEY} - LITELLM_SALT_KEY=${LITELLM_SALT_KEY}
- DATABASE_URL=${AI_LITELLM_DATABASE_URL} - DATABASE_URL=${DATABASE_URL}
- POSTGRES_PASSWORD=${AI_LITELLM_POSTGRES_PASSWORD} - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
- LITELLM_MODIFY_PARAMS=${AI_LITELLM_MODIFY_PARAMS} - LITELLM_MODIFY_PARAMS=${LITELLM_MODIFY_PARAMS}
- LITELLM_DATABASE_MIGRATIONS=${AI_LITELLM_DATABASE_MIGRATIONS} - LITELLM_DATABASE_MIGRATIONS=${LITELLM_DATABASE_MIGRATIONS}
networks: networks:
- traefik_backend - traefik_backend
- postgresql_db-backend - postgresql_db-backend
@@ -127,7 +127,7 @@ services:
labels: labels:
- traefik.enable=false - traefik.enable=false
mcpo: mcpo:
image: git.bryanmail.net/homelab/flowagent-mcpo:c5b22618404a image: git.bryanmail.net/homelab/flowagent-mcpo:916164714429
command: command:
- --config - --config
- /app/config/config.json - /app/config/config.json
@@ -141,7 +141,6 @@ services:
- /volume1/docker/mcpo/data:/mcpo_data - /volume1/docker/mcpo/data:/mcpo_data
- /volume1/docker/cronicle/ssh_keys:/app/ssh_keys:ro - /volume1/docker/cronicle/ssh_keys:/app/ssh_keys:ro
- /volume1/docker/mcpo/uv-cache:/app/uv-cache - /volume1/docker/mcpo/uv-cache:/app/uv-cache
- /volume1/docker/mcpo/flowagent-auth:/app/flowagent-auth
secrets: secrets:
- source: flowagent_azure_client_id - source: flowagent_azure_client_id
target: flowagent_azure_client_id target: flowagent_azure_client_id
+68 -258
View File
@@ -5,20 +5,10 @@
# Behavior: # Behavior:
# - Clean + up to date -> pass silently # - Clean + up to date -> pass silently
# - Clean + behind (ff-only) -> auto `git pull --ff-only`, then pass # - Clean + behind (ff-only) -> auto `git pull --ff-only`, then pass
# - Clean + ahead only -> interactive: offer to push; non-interactive: BLOCK # - Ahead only (unpushed) -> interactive: offer to push; non-interactive: BLOCK
# (unpushed) # - Dirty tracked changes -> offer to commit + push right now
# - Unresolved merge conflict -> REFUSE immediately. Never auto-commits over # (auto in non-interactive/CI runs, after a
# markers present conflict markers. Prints remediation options.
# - Dirty + local in sync -> offer to commit + push right now
# with origin (auto in non-interactive/CI runs, after a
# secret-pattern scan of the staged diff) # secret-pattern scan of the staged diff)
# - Dirty + local STALE/ -> NEVER commit on top of a stale base. Stash
# diverged vs origin the dirty changes first, resync main with
# origin using the same behind/ahead/diverged
# rules as the clean-tree case, then reapply
# the stash and re-run. On any failure the
# stash is preserved and remediation options
# (with exact commands) are printed.
# - Diverged (local AND -> REFUSE. Never auto-resolves. Prints the # - Diverged (local AND -> REFUSE. Never auto-resolves. Prints the
# remote both moved) backup/stash/reset recovery steps and exits. # remote both moved) backup/stash/reset recovery steps and exits.
# #
@@ -46,246 +36,12 @@ git status --porcelain | grep -q . && DIRTY=1
SECRET_PATTERN='(-----BEGIN [A-Z]+ PRIVATE KEY-----|AKIA[0-9A-Z]{16}|xox[baprs]-[0-9a-zA-Z-]+|password[[:space:]]*[:=][[:space:]]*[^$ ]|api[_-]?key[[:space:]]*[:=][[:space:]]*[^$ ])' SECRET_PATTERN='(-----BEGIN [A-Z]+ PRIVATE KEY-----|AKIA[0-9A-Z]{16}|xox[baprs]-[0-9a-zA-Z-]+|password[[:space:]]*[:=][[:space:]]*[^$ ]|api[_-]?key[[:space:]]*[:=][[:space:]]*[^$ ])'
# resync_with_origin <local_sha> <remote_sha> <base_sha>
#
# Handles the behind/ahead/diverged cases against a CLEAN working tree.
# Shared by both the "tree was already clean" path and the new
# "dirty tree turned out to be stale, so we stashed first" path, so the
# two paths can never drift out of sync with each other.
#
# Returns 0 if it's now safe to deploy, 1 if it could not safely resolve
# (guidance already printed to stdout in that case).
resync_with_origin() {
local local_sha="$1" remote_sha="$2" base_sha="$3"
# ---- Case: fully in sync ----
if [ "$local_sha" = "$remote_sha" ]; then
echo "==> In sync with origin/main ($local_sha). OK to deploy."
return 0
fi
# ---- Case: behind only (fast-forwardable) ----
if [ "$local_sha" = "$base_sha" ]; then
echo "!! Local main is behind origin/main."
if [ "$INTERACTIVE" -eq 1 ]; then
read -rp "Fast-forward pull now? [y/N] " ans
else
ans="y"
echo "(non-interactive session — auto fast-forwarding)"
fi
if [[ "$ans" =~ ^[Yy]$ ]]; then
git pull --ff-only origin main
echo "==> Fast-forwarded to $(git rev-parse --short main). OK to deploy."
return 0
else
echo "Aborting - pull manually, then retry:"
echo " cd $DIR && git pull --ff-only origin main"
return 1
fi
fi
# ---- Case: ahead only (local commits not yet pushed) ----
if [ "$remote_sha" = "$base_sha" ]; then
echo "!! Local main is AHEAD of origin/main (unpushed commits):"
git log --oneline "origin/main..main"
echo
if [ "$INTERACTIVE" -eq 1 ]; then
read -rp "Push local commits to origin/main now? [y/N] " ans
else
ans="n"
echo "(non-interactive session — will NOT auto-push ahead commits; needs human review)"
fi
if [[ "$ans" =~ ^[Yy]$ ]]; then
git push origin main
echo "==> Pushed. OK to deploy."
return 0
else
echo "Aborting. Review with:"
echo " cd $DIR && git log origin/main..main"
echo "Then push manually when ready:"
echo " git push origin main"
return 1
fi
fi
# ---- Case: true divergence (both ahead and behind) — NEVER auto-fix ----
echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
echo "!! DIVERGED: local main and origin/main have both moved independently."
echo "!!"
echo "!! Local-only commits:"
git log --oneline "$base_sha..main" | sed 's/^/!! /'
echo "!!"
echo "!! Remote-only commits:"
git log --oneline "$base_sha..origin/main" | sed 's/^/!! /'
echo "!!"
echo "!! This requires a human decision - git-guard will NOT auto-resolve this."
echo "!!"
echo "!! Before stashing anything new, check whether a git-guard safety stash"
echo "!! ALREADY exists from this same run (avoids confusing duplicate stashes):"
echo "!! git stash list"
echo "!!"
echo "!! Recommended recovery:"
echo "!! 1. tar backup: tar czf /volume1/docker/compose-files-backup-\$(date +%Y%m%d-%H%M%S).tar.gz -C /volume1/docker compose-files"
echo "!! 2. name the branch: git branch backup/pre-reset-\$(date +%Y%m%d)"
echo "!! 3. stash any NEW uncommitted state only if 'git stash list' above"
echo "!! didn't already show one for this run:"
echo "!! git stash push -u -m 'pre-reset-snapshot'"
echo "!! 4. reset to origin: git reset --hard origin/main"
echo "!! 5. selectively restore needed files from the stash/backup branch:"
echo "!! git stash list"
echo "!! git stash show -p stash@{N}"
echo "!! git stash apply stash@{N} # 'apply' keeps the stash as a backup; use 'pop' to also drop it"
echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
return 1
}
# ---- Case: unresolved merge conflict already present ----
# Can happen if a PRIOR git-guard run's `git stash pop` conflicted and the
# resulting conflict markers were never resolved before the next deploy
# attempt. Must be checked BEFORE the dirty-tree commit flow below, because
# an unmerged path shows up as "dirty" too, and `git add -A` would silently
# stage the literal <<<<<<< / ======= / >>>>>>> markers into a real commit.
if git ls-files -u | grep -q .; then
echo "ERROR: unresolved merge conflict markers present in the working tree."
echo "Refusing to auto-commit over a conflict — this would push literal"
echo "<<<<<<< / ======= / >>>>>>> markers to origin/main."
echo
echo "Conflicted paths:"
git diff --name-only --diff-filter=U | sed 's/^/ /'
echo
echo "Remediation options:"
echo " A) Resolve the conflict by hand, then commit and push:"
echo " cd $DIR"
echo " git status # see conflicted paths"
echo " git diff # inspect the conflict markers"
echo " \$EDITOR <conflicted-file> # remove markers, keep correct content"
echo " git add <conflicted-file>"
echo " git commit -m 'resolve git-guard stash-pop conflict'"
echo " git push origin main"
echo " B) Discard the conflicted merge attempt entirely and start clean from"
echo " origin/main, then decide separately whether to re-apply anything"
echo " from a prior safety stash:"
echo " cd $DIR"
echo " git checkout -- ."
echo " git reset --hard origin/main"
echo " git stash list # look for a git-guard-safety-stash-* entry"
echo " git stash show -p stash@{N} # inspect before deciding"
echo " C) Once resolved (via A or B) and no longer needed, clean up the stash:"
echo " git stash drop stash@{N}"
exit 1
fi
# ---- Case: dirty tracked changes ---- # ---- Case: dirty tracked changes ----
if [ "$DIRTY" -eq 1 ]; then if [ "$DIRTY" -eq 1 ]; then
echo "!! WORKING TREE DIRTY — uncommitted changes detected:" echo "!! WORKING TREE DIRTY — uncommitted changes detected:"
git status --short git status --short
echo echo
# If local is ALSO stale/diverged from origin, committing right now would
# create a doomed commit on top of a base that's about to be rejected on
# push (this is exactly what caused a real incident: a stray on-disk edit
# sat on a checkout that was 4 commits behind, git-guard auto-committed
# anyway, then the push bounced). Stash first, resync safely using the
# same rules as the clean-tree path, then reapply.
if [ "$LOCAL" != "$REMOTE" ]; then
echo "!! Local main is ALSO stale/diverged from origin/main."
echo " Refusing to commit on top of a stale base — stashing the dirty"
echo " changes safely first, then resyncing with origin."
echo
STASH_MSG="git-guard-safety-stash-$(date -u +%Y%m%dT%H%M%SZ)"
if ! git stash push -u -m "$STASH_MSG"; then
echo "ERROR: 'git stash push' itself failed (disk full, permissions, or"
echo " some other git error). Your changes are still on disk,"
echo " uncommitted — nothing has been lost, but git-guard cannot"
echo " proceed safely until this is resolved."
echo
echo "Remediation options:"
echo " A) Check disk space and permissions, then retry the deploy:"
echo " df -h $DIR"
echo " ls -la $DIR"
echo " B) Identify and manually move aside whatever is blocking the stash,"
echo " then retry:"
echo " cd $DIR"
echo " git status --short # find the offending path(s)"
echo " mv <path> <path>.bak-\$(date +%s)"
echo " C) Inspect the raw git error above for specifics before proceeding."
exit 1
fi
echo "==> Stashed as: $STASH_MSG"
if resync_with_origin "$LOCAL" "$REMOTE" "$BASE"; then
echo "==> Resync succeeded. Reapplying stashed changes..."
if git stash pop; then
echo "==> Stash reapplied cleanly. Re-checking sync state..."
exec bash "$0" "$@"
else
echo "ERROR: 'git stash pop' did not complete successfully."
echo
if git ls-files -u | grep -q .; then
echo "This is a MERGE CONFLICT — your stashed changes were partially"
echo "applied and conflict markers (<<<<<<< / ======= / >>>>>>>) are now"
echo "in the working tree. The stash itself is still preserved as a backup."
echo
echo "Remediation options:"
echo " A) Resolve the conflict by hand, then commit and push:"
echo " cd $DIR"
echo " git status # see conflicted paths"
echo " git diff # inspect the markers"
echo " \$EDITOR <conflicted-file> # remove markers, keep correct content"
echo " git add <conflicted-file>"
echo " git commit -m 'resolve git-guard stash-pop conflict'"
echo " git push origin main"
echo " git stash list # confirm which entry is: $STASH_MSG"
echo " git stash drop stash@{N} # once confirmed no longer needed"
echo " B) Abandon this merge attempt and fall back to a clean, resynced"
echo " tree, then re-apply the change manually with full visibility:"
echo " cd $DIR"
echo " git checkout -- ."
echo " git reset --hard origin/main # now matches origin, no conflict"
echo " git stash list # find: $STASH_MSG"
echo " git stash show -p stash@{N} # review the content"
echo " git stash apply stash@{N} # 'apply' keeps the backup; use 'pop' to also drop it"
else
echo "This looks like an UNTRACKED-FILE COLLISION, not a merge conflict"
echo "(a file added upstream shares a path with an untracked file in your"
echo "stash). No conflict markers were written; the stash was NOT applied"
echo "and remains fully intact."
echo
echo "Remediation options:"
echo " A) Move the colliding upstream file aside, pop, then reconcile:"
echo " cd $DIR"
echo " git status --short # identify the colliding path"
echo " mv <path> <path>.upstream-\$(date +%s)"
echo " git stash pop"
echo " diff <path> <path>.upstream-* # reconcile manually, then remove the .upstream-* backup"
echo " B) Inspect the stash without applying, and hand-merge the needed"
echo " pieces instead:"
echo " git stash list # find N"
echo " git stash show -p stash@{N}"
fi
echo
echo "Your stash reference for this run: $STASH_MSG"
exit 1
fi
else
echo "ERROR: could not safely resync with origin/main."
echo "Your uncommitted changes are preserved in the stash: $STASH_MSG"
echo
echo "Remediation options:"
echo " A) Follow the manual recovery steps printed above (from the"
echo " behind/ahead/diverged case), THEN reapply your change:"
echo " cd $DIR"
echo " git stash list # find: $STASH_MSG"
echo " git stash apply stash@{N} # or 'pop' to also drop it once resynced"
echo " B) If the stashed change is no longer needed (e.g. it's already"
echo " represented in a since-merged PR), verify then drop it:"
echo " git stash show -p stash@{N}"
echo " git stash drop stash@{N}"
exit 1
fi
fi
if [ "$INTERACTIVE" -eq 1 ]; then if [ "$INTERACTIVE" -eq 1 ]; then
read -rp "Commit and push these changes to origin/main now? [y/N] " ans read -rp "Commit and push these changes to origin/main now? [y/N] " ans
else else
@@ -298,8 +54,7 @@ if [ "$DIRTY" -eq 1 ]; then
if git diff --cached | grep -Eiq "$SECRET_PATTERN"; then if git diff --cached | grep -Eiq "$SECRET_PATTERN"; then
echo "ERROR: possible secret detected in staged changes. Refusing to auto-commit." echo "ERROR: possible secret detected in staged changes. Refusing to auto-commit."
echo "Review manually:" echo "Review manually: git diff --cached"
echo " cd $DIR && git diff --cached"
git reset git reset
exit 1 exit 1
fi fi
@@ -308,25 +63,80 @@ if [ "$DIRTY" -eq 1 ]; then
if git push origin main; then if git push origin main; then
echo "==> Pushed. Re-checking sync state..." echo "==> Pushed. Re-checking sync state..."
exec bash "$0" "$@" exec "$0" "$@"
else else
echo "ERROR: push failed (likely diverged from origin). Aborting deploy." echo "ERROR: push failed (likely diverged from origin). Aborting deploy."
echo "Run:" echo "Run: cd $DIR && git status"
echo " cd $DIR && git status"
exit 1 exit 1
fi fi
else else
echo "Aborting deploy - commit or stash changes manually, then retry:" echo "Aborting deploy - commit or stash changes manually, then retry."
echo " cd $DIR"
echo " git add -A && git commit -m 'your message' && git push origin main"
echo " # or: git stash push -u -m 'manual-stash'"
exit 1 exit 1
fi fi
fi fi
# ---- Clean tree: resync with origin using the shared logic above ---- # ---- Case: fully in sync ----
if resync_with_origin "$LOCAL" "$REMOTE" "$BASE"; then if [ "$LOCAL" = "$REMOTE" ]; then
echo "==> In sync with origin/main ($LOCAL). OK to deploy."
exit 0
fi
# ---- Case: behind only (fast-forwardable) ----
if [ "$LOCAL" = "$BASE" ]; then
echo "!! Local main is behind origin/main."
if [ "$INTERACTIVE" -eq 1 ]; then
read -rp "Fast-forward pull now? [y/N] " ans
else
ans="y"
echo "(non-interactive session — auto fast-forwarding)"
fi
if [[ "$ans" =~ ^[Yy]$ ]]; then
git pull --ff-only origin main
echo "==> Fast-forwarded to $(git rev-parse --short main). OK to deploy."
exit 0 exit 0
else else
echo "Aborting deploy - pull manually, then retry."
exit 1 exit 1
fi fi
fi
# ---- Case: ahead only (local commits not yet pushed) ----
if [ "$REMOTE" = "$BASE" ]; then
echo "!! Local main is AHEAD of origin/main (unpushed commits):"
git log --oneline "origin/main..main"
echo
if [ "$INTERACTIVE" -eq 1 ]; then
read -rp "Push local commits to origin/main now? [y/N] " ans
else
ans="n"
echo "(non-interactive session — will NOT auto-push ahead commits; needs human review)"
fi
if [[ "$ans" =~ ^[Yy]$ ]]; then
git push origin main
echo "==> Pushed. OK to deploy."
exit 0
else
echo "Aborting deploy. Review with: git log origin/main..main"
exit 1
fi
fi
# ---- Case: true divergence (both ahead and behind) — NEVER auto-fix ----
echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
echo "!! DIVERGED: local main and origin/main have both moved independently."
echo "!!"
echo "!! Local-only commits:"
git log --oneline "$BASE..main" | sed 's/^/!! /'
echo "!!"
echo "!! Remote-only commits:"
git log --oneline "$BASE..origin/main" | sed 's/^/!! /'
echo "!!"
echo "!! This requires a human decision - git-guard will NOT auto-resolve this."
echo "!! Recommended recovery:"
echo "!! 1. tar backup: tar czf /volume1/docker/compose-files-backup-\$(date +%Y%m%d-%H%M%S).tar.gz -C /volume1/docker compose-files"
echo "!! 2. name the branch: git branch backup/pre-reset-\$(date +%Y%m%d)"
echo "!! 3. stash all state: git stash push -u -m 'pre-reset-snapshot'"
echo "!! 4. reset to origin: git reset --hard origin/main"
echo "!! 5. selectively restore needed files from the stash/backup branch"
echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
exit 1