Real incident: a run had both patroni-0 and patroni-1 stuck forever on
"waiting for standby_leader to bootstrap", never even attempting to race
for the role. Root cause was leftover etcd/patroni data on disk from a
prior interrupted run (operator stopped it short) — the postgresqlha
stack itself was gone, but etcd-1/2/3-data still had persisted raft state
including a real /service/postgres-ha/initialize key and old replication
slot records. Fresh Patroni nodes booting against that non-fresh etcd
correctly concluded the cluster already existed and deferred forever
waiting for a leader that could never appear, since nobody actually held
the lock. rollback.sh's own data-dir wipe only fires when it detects the
HA stack IS currently present (Case D/E) — if the stack was already gone
by the time cleanup ran, its Case A path never touches the data dirs,
leaving exactly this trap.
Added a "Pre-Phase-0" check that runs before preflight.sh's ~6+ minute
pg_dumpall: detects a still-present postgresqlha stack OR non-empty
etcd-*/patroni-*-data left over from a prior run, and — since this is
destructive and the operator explicitly wants this to be a deliberate
choice, not silent automatic cleanup — prompts interactively before
tearing down/wiping. Non-interactive sessions (no tty) hard-fail with a
clear message rather than guessing; AUTO_CLEANUP=yes in the environment
skips the prompt for deliberate unattended re-runs. Legacy production
data is never touched by any of this.
Root cause of a real run's failure: Phase 2 passed (patroni-0 reached
standby_leader, patroni-1's basebackup completed, role flipped to
"replica"), but Phase 3's canary write then failed to propagate within
5s moments later. patroni_lag() queried the REPLICA's own /patroni
endpoint for lag data — but that field only exists on the LEADER side
(derived from pg_stat_replication); a replica's own /patroni response
never has it. So $LAG_INFO was always empty, and Phase 2's gate
`[ -z "$LAG_INFO" ] || ...` short-circuited permanently true — the lag
check never actually ran. Phase 2 degraded to "did role flip to replica
3x in a row", which can be true before the replica has genuinely caught
up on WAL backlog from its own basebackup.
Fixed by replacing patroni_lag() with cluster_member_lag_state(), which
queries the LEADER's /cluster endpoint (real pg_stat_replication-backed
data, same shape verified in the original dry run) and extracts the
specific replica's state/lag fields from its member object. Phase 2 now
requires literal state=streaming AND (lag=0 or absent), not just "field
was empty because we asked the wrong node." Phase 4's informational lag
log line updated to match.
Mirrors the fix in postgresql-ha-staging.yaml: added
primary_slot_name: standby_leader_slot to both patroni-0 and patroni-1's
standby_cluster blocks so this file's specs stay byte-identical to
staging's, per this file's own "do not recreate the live promoted
primary/replica" design requirement. See postgresql-ha-staging.yaml
header and the ADR-0001 note for full incident detail.
Root cause of a real cutover run's failure: the standby_leader's basebackup
from legacy completed, but its Postgres process then got permanently stuck
in "starting" because legacy had no replication slot reserving WAL — normal
WAL recycling (checkpoint_timeout=300s) deleted the segment needed to
resume streaming during the 8-11 min basebackup window. This also explained
why the cascade replica's own basebackup (which targets the standby_leader)
failed with "database system is starting up" — one root cause, not two.
Fix: created a physical replication slot (standby_leader_slot) on live
production legacy, and added primary_slot_name: standby_leader_slot under
bootstrap.dcs.standby_cluster in both patroni-0 and patroni-1's
SPILO_CONFIGURATION so Patroni pins the slot automatically. Also bumped
wal_keep_size to 4GB on legacy as defense-in-depth.
Full incident detail documented in this file's header for future reference.
Phase 2's cascade-replica bootstrap does its own full basebackup FROM the
new standby_leader (an extra hop beyond Phase 1's legacy-direct copy), so
per operator request its window is extended further than Phase 1's —
from 1200s (20 min, matched to observed 8-11 min legacy-direct timing) to
2100s (35 min), giving more margin for the additional hop. Phase 1's
window is intentionally left unchanged at 1200s since it already has
comfortable headroom against the timing we've actually observed for that
specific bootstrap path.
Two real-run bugs found and fixed:
1. Phase 1 hardcoded patroni-1 as the expected standby_leader. The
bootstrap-race winner is actually nondeterministic (Patroni/etcd lock
race) — the original prod attempt had patroni-0 win it instead, which
the dry run never exercised. Fixed by polling BOTH patroni-0 and
patroni-1 each iteration and capturing whichever wins into
$LEADER_HOST, with the other becoming $REPLICA_HOST. Every later phase
(2,3,4,5,6,8,10) now references $LEADER_HOST/$REPLICA_HOST instead of
hardcoded hostnames.
2. Phase 1's wait window (240s) and Phase 2's (300s) were both far shorter
than the observed real basebackup duration for a ~42GB cluster
(8-11 minutes per prior dry-run/live polling). The role only flips to
standby_leader/replica AFTER the full copy completes, so both timeouts
could fire — and did — while a legitimate basebackup was still
in-progress, triggering a false-negative rollback. Both windows
extended to 1200s (20 min), with per-poll data-dir size logging
(timeout-guarded du -sh) so progress is observable instead of a silent
binary wait.
Increases the delay before Swarm restarts a failed uptime-kuma task
from the default 5s to 30s. Applied live via `docker service update
--restart-delay 30s` on 2026-07-26; this persists it so it survives
the next stack deploy.
Note: this does not pin the task to a specific node or prevent
cross-node reschedule races against the CephFS-backed SQLite data
dir - it only slows the restart-after-failure loop. See prior
incident notes for the DB corruption root cause discussion.
Root cause of Woodpecker push pipelines silently no-op'ing since
2026-07-18: Woodpecker's config resolution is
.woodpecker/*.{yaml,yml} -> .woodpecker.yaml -> .woodpecker.yml
Once .woodpecker/renovate.yml existed, the root .woodpecker.yml was
entirely ignored. Every push webhook hit renovate.yml's `when: event:
cron` filter, matched zero steps, and Woodpecker logged
"ignoring hook: 'when' filters filtered out all steps" instead of
running deploy.
Moving the push pipeline into .woodpecker/deploy.yml restores it as
an independent workflow alongside renovate.yml -- each file's own
`when:` filter now applies correctly (push -> deploy.yml, cron ->
renovate.yml).
Adds powerautomate entry with Azure app registration credentials.
Auth dir (/volume1/docker/mcpo/powerautomate-auth/) and config stub
already exist on host from prior session.
NOTE: powerautomate-mcp package not yet on npm - npx will fail until
published. Tracked here for GitOps pipeline; will update command when
package is available or we switch to build-from-source approach.
- Pin ms365 to 0.129.0 with --discovery flag (fixes MCPO schema parse error)
- Add powerautomate entry (PA_MCP_CLIENT_ID placeholder - needs Azure app reg)
- Update authentik token (was expired)
- teams-mcp-auth volume now source of truth for ms365/teams auth cache
- powerautomate-auth volume wired for future PA auth cache
NOTE: config.json was previously managed manually on disk only.
This commit brings it under GitOps control.
reverse-proxy publishes host-mode ports (80/443/1883/5201/5431) — start-first
can never bind while the old task holds the ports, deadlocking every rolling
update. keepalived master/backup run VRRP on the host network — concurrent
old+new speakers with the same router-id risk VIP flapping. Both now use
stop-first (global mode + parallelism=1 + VIP failover keeps ingress up).
Stateless helpers (whoami, speedtest, iperf3, certs-dumper) keep start-first.
Step 1 of Tier 0/1 pipeline inclusion (prep for removing traefik/woodpecker
from the bootstrap manual-deploy guard).
Image ghcr.io/thebluffman85/crowdsec-web-ui:latest never existed (typo in org name).
Correct upstream project is TheDuffman85/crowdsec-web-ui on GitHub. Fixed to
ghcr.io/theduffman85/crowdsec-web-ui:latest.
Deployed and verified live on docker-2 prior to this commit:
- security_crowdsec-web-ui now pulls successfully and is Running
- Host(`secure.bryanmail.net`) rule intact, all CROWDSEC_* env vars correct
- security_crowdsec unaffected, still running
Docker secrets (postgresql_password, postgresql_pgadmin_password) already handled
POSTGRES_PASSWORD_FILE/PGADMIN_DEFAULT_PASSWORD_FILE. This removes the two remaining
`env_file: - postgresql.env` lines (postgresql + pgadmin services) which were redundant
and left the stack in a half-migrated hybrid state. Non-secret vars (POSTGRES_USER,
PGADMIN_DEFAULT_EMAIL, DATABASUS_HOST, PGADMIN_HOST) continue to be supplied via
stack-deploy.sh's envsubst mechanism from postgresql.env at deploy time — no functional
change to those values.
Deployed and verified live on docker-2 prior to this commit:
- postgresql_postgresql: pg_isready OK, all 21 databases intact, psql auth via Docker secret confirmed
- postgresql_pgadmin: Host(`dba.bryanmail.net`) rule + PGADMIN_DEFAULT_EMAIL correct
- postgresql_databasus: running
- All dependent stacks (mealie, vaultwarden, auth, media, homeassistant, guacamole, security) remained healthy throughout, zero reconnect issues
admin