During the traefik/keepalived incident (2026-09-11/12), we hit a real bootstrapping circular dependency: Traefik down → the VIP/hostname git.bryanmail.net unreachable → git-guard.sh's git fetch origin fails → stack-deploy.sh traefik refuses to run → can't redeploy Traefik to fix Traefik. The manual workaround (temporarily pointing git remote origin at Gitea's direct port-3000 ingress on a node IP) worked, but required hand-editing git config under pressure during an active outage.
What this PR adds
Two opt-in flags on stack-deploy.sh, passed straight through to git-guard.sh:
-e / --emergency
Before the normal fetch, tries candidate Gitea endpoints in order, each over plain http://<host>:3000 (Gitea's direct ingress port, verified to work unauthenticated via git ls-remote — bypasses Traefik/VIP entirely since that's exactly what's broken in this scenario):
https://git.bryanmail.net (the normal path — tried first in case only this check's assumption about the outage is wrong)
Switches origin to the first reachable one, then runs the normal sync logic (behind/ahead/diverged handling is completely unchanged) against it. Prints a reminder to restore the real origin URL once the normal path is healthy again; never persists the swap anywhere (not committed, not written to any tracked file).
If none of the 5 candidates respond, exits 1 with guidance pointing at -f/--force as the next option, since that means Gitea itself is down, not just routing.
-f / --force
Skips git-guard.sh's sync check entirely — no fetch, no comparison, no commit/push offer. Deploys whatever is on disk right now. Genuine last resort for when Gitea itself is unreachable (not just routing) — prints a loud warning banner every time it's used.
Usage
bash deploy/stack-deploy.sh traefik -e # try alternate hosts first
bash deploy/stack-deploy.sh -e traefik # flag order doesn't matter
bash deploy/stack-deploy.sh traefik -f # skip the check entirely (last resort)
What's deliberately unchanged
Normal (no-flag) behavior of both scripts is byte-for-byte identical to before.
-e never skips the actual safety logic (behind/ahead/diverged/dirty-tree handling) — it only changes which remote URL that logic runs against.
Neither flag touches the render/mount-guard/deploy steps in stack-deploy.sh.
Verified git ls-remote succeeds unauthenticated over plain HTTP against the port-3000 endpoint before writing this, so there's no credential-smuggling concern with the candidate URLs.
Testing suggestion (safe, non-destructive)
cd /volume1/docker/compose-files
bash deploy/git-guard.sh -e # should report "Normal origin URL ... is reachable — no swap needed" when Traefik is healthy
To actually exercise the failover path without a real outage, temporarily point DNS/hosts at an unreachable IP for git.bryanmail.net and re-run, or just trust the git ls-remote verification already done manually against each of the 5 URLs during this incident.
## Problem
During the traefik/keepalived incident (2026-09-11/12), we hit a real bootstrapping circular dependency: Traefik down → the VIP/hostname `git.bryanmail.net` unreachable → `git-guard.sh`'s `git fetch origin` fails → `stack-deploy.sh traefik` refuses to run → can't redeploy Traefik to fix Traefik. The manual workaround (temporarily pointing `git remote origin` at Gitea's direct port-3000 ingress on a node IP) worked, but required hand-editing git config under pressure during an active outage.
## What this PR adds
Two opt-in flags on `stack-deploy.sh`, passed straight through to `git-guard.sh`:
### `-e` / `--emergency`
Before the normal fetch, tries candidate Gitea endpoints in order, each over plain `http://<host>:3000` (Gitea's direct ingress port, verified to work unauthenticated via `git ls-remote` — bypasses Traefik/VIP entirely since that's exactly what's broken in this scenario):
1. `https://git.bryanmail.net` (the normal path — tried first in case only *this* check's assumption about the outage is wrong)
2. `http://192.168.4.30:3000` (VIP)
3. `http://192.168.4.31:3000`, `.32`, `.33` (each node directly)
Switches `origin` to the first reachable one, then runs the **normal** sync logic (behind/ahead/diverged handling is completely unchanged) against it. Prints a reminder to restore the real origin URL once the normal path is healthy again; never persists the swap anywhere (not committed, not written to any tracked file).
If none of the 5 candidates respond, exits 1 with guidance pointing at `-f`/`--force` as the next option, since that means Gitea itself is down, not just routing.
### `-f` / `--force`
Skips `git-guard.sh`'s sync check entirely — no fetch, no comparison, no commit/push offer. Deploys whatever is on disk right now. Genuine last resort for when Gitea itself is unreachable (not just routing) — prints a loud warning banner every time it's used.
## Usage
```bash
bash deploy/stack-deploy.sh traefik -e # try alternate hosts first
bash deploy/stack-deploy.sh -e traefik # flag order doesn't matter
bash deploy/stack-deploy.sh traefik -f # skip the check entirely (last resort)
```
## What's deliberately unchanged
- Normal (no-flag) behavior of both scripts is byte-for-byte identical to before.
- `-e` never skips the actual safety logic (behind/ahead/diverged/dirty-tree handling) — it only changes which remote URL that logic runs against.
- Neither flag touches the render/mount-guard/deploy steps in `stack-deploy.sh`.
- Verified `git ls-remote` succeeds unauthenticated over plain HTTP against the port-3000 endpoint before writing this, so there's no credential-smuggling concern with the candidate URLs.
## Testing suggestion (safe, non-destructive)
```bash
cd /volume1/docker/compose-files
bash deploy/git-guard.sh -e # should report "Normal origin URL ... is reachable — no swap needed" when Traefik is healthy
```
To actually exercise the failover path without a real outage, temporarily point DNS/hosts at an unreachable IP for `git.bryanmail.net` and re-run, or just trust the `git ls-remote` verification already done manually against each of the 5 URLs during this incident.
Adds two opt-in flags to break the circular dependency discovered during
the 2026-09-12 traefik/keepalived incident: Traefik down -> VIP/hostname
unreachable -> git-guard can't fetch -> stack-deploy.sh traefik blocked ->
can't redeploy traefik to fix Traefik.
-e/--emergency: probes git.bryanmail.net, then the VIP (192.168.4.30), then
each node's direct IP (.31/.32/.33) on Gitea's direct ingress port 3000
(bypasses Traefik/VIP entirely), switches origin to the first reachable one,
then runs the NORMAL sync logic against it (behind/ahead/diverged handling
unchanged — this only changes which host is used, never skips the safety
checks). Prints a reminder to restore the real origin URL afterward; never
persists the swap anywhere.
-f/--force: skips the sync check entirely, for the genuine last-resort case
where Gitea itself (not just routing) is unreachable. Loud warning banner.
Verified git ls-remote succeeds unauthenticated over plain http against
each candidate before writing this, so no credential-smuggling concern.
Flags may appear before or after the stack name (stack-deploy.sh traefik -e
and stack-deploy.sh -e traefik both work) — the stack name is whatever
argument isn't a recognized flag. No change to the render/mount-guard/deploy
logic; this only changes how the git-guard.sh pre-flight is invoked. See
deploy/git-guard.sh header for what each flag actually does.
AVB
merged commit 9895249e88 into main2026-09-12 00:07:58 -07:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Problem
During the traefik/keepalived incident (2026-09-11/12), we hit a real bootstrapping circular dependency: Traefik down → the VIP/hostname
git.bryanmail.netunreachable →git-guard.sh'sgit fetch originfails →stack-deploy.sh traefikrefuses to run → can't redeploy Traefik to fix Traefik. The manual workaround (temporarily pointinggit remote originat Gitea's direct port-3000 ingress on a node IP) worked, but required hand-editing git config under pressure during an active outage.What this PR adds
Two opt-in flags on
stack-deploy.sh, passed straight through togit-guard.sh:-e/--emergencyBefore the normal fetch, tries candidate Gitea endpoints in order, each over plain
http://<host>:3000(Gitea's direct ingress port, verified to work unauthenticated viagit ls-remote— bypasses Traefik/VIP entirely since that's exactly what's broken in this scenario):https://git.bryanmail.net(the normal path — tried first in case only this check's assumption about the outage is wrong)http://192.168.4.30:3000(VIP)http://192.168.4.31:3000,.32,.33(each node directly)Switches
originto the first reachable one, then runs the normal sync logic (behind/ahead/diverged handling is completely unchanged) against it. Prints a reminder to restore the real origin URL once the normal path is healthy again; never persists the swap anywhere (not committed, not written to any tracked file).If none of the 5 candidates respond, exits 1 with guidance pointing at
-f/--forceas the next option, since that means Gitea itself is down, not just routing.-f/--forceSkips
git-guard.sh's sync check entirely — no fetch, no comparison, no commit/push offer. Deploys whatever is on disk right now. Genuine last resort for when Gitea itself is unreachable (not just routing) — prints a loud warning banner every time it's used.Usage
What's deliberately unchanged
-enever skips the actual safety logic (behind/ahead/diverged/dirty-tree handling) — it only changes which remote URL that logic runs against.stack-deploy.sh.git ls-remotesucceeds unauthenticated over plain HTTP against the port-3000 endpoint before writing this, so there's no credential-smuggling concern with the candidate URLs.Testing suggestion (safe, non-destructive)
To actually exercise the failover path without a real outage, temporarily point DNS/hosts at an unreachable IP for
git.bryanmail.netand re-run, or just trust thegit ls-remoteverification already done manually against each of the 5 URLs during this incident.