355 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
admin	7a06b040ad	postgresql-ha-staging.yaml: add primary_slot_name to fix real bootstrap failure ... Root cause of a real cutover run's failure: the standby_leader's basebackup from legacy completed, but its Postgres process then got permanently stuck in "starting" because legacy had no replication slot reserving WAL — normal WAL recycling (checkpoint_timeout=300s) deleted the segment needed to resume streaming during the 8-11 min basebackup window. This also explained why the cascade replica's own basebackup (which targets the standby_leader) failed with "database system is starting up" — one root cause, not two. Fix: created a physical replication slot (standby_leader_slot) on live production legacy, and added primary_slot_name: standby_leader_slot under bootstrap.dcs.standby_cluster in both patroni-0 and patroni-1's SPILO_CONFIGURATION so Patroni pins the slot automatically. Also bumped wal_keep_size to 4GB on legacy as defense-in-depth. Full incident detail documented in this file's header for future reference.	2026-08-04 20:04:01 -07:00
admin	e57f4b3592	cutover.sh: extend Phase 2 cascade-replica wait window to 35 min ... Phase 2's cascade-replica bootstrap does its own full basebackup FROM the new standby_leader (an extra hop beyond Phase 1's legacy-direct copy), so per operator request its window is extended further than Phase 1's — from 1200s (20 min, matched to observed 8-11 min legacy-direct timing) to 2100s (35 min), giving more margin for the additional hop. Phase 1's window is intentionally left unchanged at 1200s since it already has comfortable headroom against the timing we've actually observed for that specific bootstrap path.	2026-08-04 00:48:32 -07:00
admin	29cd73eabc	cutover.sh: dynamic standby_leader detection + extended bootstrap wait windows ... Two real-run bugs found and fixed: 1. Phase 1 hardcoded patroni-1 as the expected standby_leader. The bootstrap-race winner is actually nondeterministic (Patroni/etcd lock race) — the original prod attempt had patroni-0 win it instead, which the dry run never exercised. Fixed by polling BOTH patroni-0 and patroni-1 each iteration and capturing whichever wins into $LEADER_HOST, with the other becoming $REPLICA_HOST. Every later phase (2,3,4,5,6,8,10) now references $LEADER_HOST/$REPLICA_HOST instead of hardcoded hostnames. 2. Phase 1's wait window (240s) and Phase 2's (300s) were both far shorter than the observed real basebackup duration for a ~42GB cluster (8-11 minutes per prior dry-run/live polling). The role only flips to standby_leader/replica AFTER the full copy completes, so both timeouts could fire — and did — while a legitimate basebackup was still in-progress, triggering a false-negative rollback. Both windows extended to 1200s (20 min), with per-poll data-dir size logging (timeout-guarded du -sh) so progress is observable instead of a silent binary wait.	2026-08-04 00:40:20 -07:00
admin	9c1f31a70e	ADR-0001 rollback.sh: accept any 2xx in consumer health checks (204 from Woodpecker was a false-positive failure)	2026-08-02 22:15:15 -07:00
admin	325daca86b	ADR-0001 cutover.sh: fix Phase 1 node-label check (broken Go template on hyphenated label) and accept any 2xx in consumer health checks	2026-08-02 22:14:17 -07:00
admin	066cbaf97f	ADR-0001 Phase 3: add rollback.sh (standalone, idempotent, grace-window guard against post-cutover data loss)	2026-08-02 17:38:50 -07:00
admin	1203630bb5	ADR-0001 Phase 3: add cutover.sh (11-phase scripted cutover with auto-rollback)	2026-08-02 17:32:42 -07:00
admin	a4f5d54b82	ADR-0001 Phase 3: add postgresql-ha-final.yaml (Stage 3 - HAProxy alias handoff, no external port/Traefik yet)	2026-08-02 17:28:26 -07:00
admin	01768d14f5	ADR-0001 Phase 3: add preflight.sh (disk/backup hard gates, catalog+consumer enumeration)	2026-08-02 16:45:44 -07:00
admin	f49b576806	ADR-0001 Phase 3: add cutover RUNBOOK (dependency map, phased procedure, rollback)	2026-08-02 16:35:52 -07:00
admin	8d30f19c8c	postgresql.yaml: update stale comment referencing old CLONE_* rationale to reference standby_cluster design (no functional change - this file never used CLONE_* itself)	2026-07-30 22:03:44 -07:00
admin	4dcb3561ed	postgresql-ha-staging.yaml: switch from CLONE_WITH_BASEBACKUP to Patroni standby_cluster (continuous streaming, validated in pgha-test dry run) - closes pre-cutover write gap	2026-07-30 21:59:44 -07:00
admin	dc6120ffd4	pgha-dryrun.yaml: trigger secret re-provisioning after regenerating postgresql_replication_password (excludes &<>\" per pystache HTML-escaping bug found this session)	2026-07-30 21:44:49 -07:00
admin	8afe0049c4	pgha-dryrun.yaml: switch from CLONE_WITH_BASEBACKUP to Patroni standby_cluster (continuous streaming) to close the pre-cutover write gap	2026-07-30 21:24:28 -07:00
admin	499402cd0d	postgresql-ha-staging.yaml: port dry-run fixes (bugs 1,2,4,5) - $$(...) escaping (incl. CLONE_PASSWORD), ETCD3_HOSTS, post_init_wrapper.sh SUPERUSER role fix	2026-07-30 15:56:43 -07:00
admin	7e8e3388f4	postgresql.yaml: port dry-run fixes (bugs 1,2,4,5) - $$(...) escaping, ETCD3_HOSTS, post_init_wrapper.sh SUPERUSER role fix	2026-07-30 15:55:26 -07:00
admin	b7f0d9dbfb	pgha dry-run: fix bug 5 - grant SUPERUSER to placeholder postgres role	2026-07-29 09:32:09 -07:00
admin	e8e506dc4d	add: post_init_wrapper.sh to work around Spilo's hardcoded 'postgres' role name assumption in post_init.sh. Does not fork/modify Spilo's script — creates missing role then execs the original unchanged.	2026-07-29 07:21:34 -07:00
admin	7529e6cb36	fix: override bootstrap.post_init via SPILO_CONFIGURATION to create missing 'postgres' role before Spilo's real post_init.sh runs. Spilo hardcodes ALTER VIEW...OWNER TO postgres with no way to parameterize, which fails since our superuser is PGadmin not postgres.	2026-07-29 06:35:24 -07:00
admin	b86784fe3a	fix: legacy container needs pg_hba.conf replication rule for pg_basebackup — added initdb.d hook script. Disposable test only, uses 'trust' since this container is not auth-representative.	2026-07-28 21:27:13 -07:00
admin	1dcb5ec467	fix: use ETCD3_HOSTS not ETCD_HOSTS — confirmed against spilo source that etcd/etcd3 are distinct DCS backends (v2 vs v3 API). Our etcd 3.5.9 containers have v2 API disabled, causing 404s with the old var name.	2026-07-28 21:10:13 -07:00
admin	83480aa23b	fix: escape \$(...) as \$\$(...) in patroni command blocks — Compose interpolation was choking on \$( before the shell ever saw it (invalid interpolation format error)	2026-07-28 21:03:56 -07:00
admin	20e1210441	postgresql: add disposable dry-run test stack for cutover validation (pgha-test, own network + data dirs, zero prod impact)	2026-07-28 11:12:19 -07:00
admin	f4f0749969	postgresql: add cutover staging compose (ADR-0001 Phase 3). Lives in cutover/ subdir — deliberately excluded from stack-deploy.sh folder merge. Deployed only by cutover script as separate postgresqlha stack.	2026-07-28 11:07:03 -07:00
admin	87ae54fb3f	postgresql: add HAProxy config for Patroni-aware TCP routing (ADR-0001 Phase 2)	2026-07-28 08:53:55 -07:00
admin	c8d45e9e61	postgresql: add folder-based Patroni+etcd+HAProxy HA stack draft (ADR-0001 Phase 2). NOT deployed — bootstrap tier, manual deploy only. Coexists with flat postgresql.yaml until cutover.	2026-07-28 08:53:44 -07:00
AVB	129badada4	Add secrets (and add postgresql) back so they are seen as utility folders and do deploy	2026-07-28 08:36:56 -07:00
AVB	e1f569b3ec	Add secrets folder to utility folders list so it isn't auto-deployed as a stack	2026-07-28 08:25:35 -07:00
AVB	b5767ec7d9	revert 8719c398fd ... revert Update .woodpecker/deploy.yml	2026-07-28 08:24:29 -07:00
AVB	8719c398fd	Update .woodpecker/deploy.yml	2026-07-28 08:22:57 -07:00
admin	9a5b028e07	postgresql: document replication + patroni secrets (ADR-0001 Phase 1)	2026-07-28 07:20:52 -07:00
admin	0f26d31a02	postgresql: add replication + patroni Woodpecker secrets to provision-secrets step (ADR-0001 Phase 1)	2026-07-28 07:20:35 -07:00
AVB	efd90dac3e	Update ai/ai.yaml	2026-07-27 16:14:54 -07:00
admin	6789d2530a	vaultwarden: retry #8 -- trigger after fixing pipeline compile error (dollar-brace literal in comments)	2026-07-27 13:36:17 -07:00
admin	e0a6b7d233	deploy.yml: fix pipeline compile error 'missing closing brace' -- the documentation comment itself contained a literal dollar-brace sequence with invalid contents (dollar-brace CI_...), which Woodpecker's substitution engine parses even inside comments. Rewrote header to describe the rules in words without any literal dollar-brace sequences. Also removed backtick-quoted dollar-brace remnants from the vaultwarden case comment.	2026-07-27 13:35:13 -07:00
admin	9b3bba636c	vaultwarden: retry #7 -- trigger after fixing Woodpecker compile-time ${VAR} substitution root cause	2026-07-27 12:52:59 -07:00
admin	6757b0dfe1	deploy.yml: THE ACTUAL ROOT CAUSE FIX. Woodpecker substitutes ${VAR} at pipeline-compile time from its CI metadata map, where secrets do not exist -- so every braced reference to a secret-backed env var (SWARM_MANAGER_IP, all provisioning values) was silently blanked before the shell ever ran. Proven by debug output showing eval "val=\$$v" arriving as val=\$v ($$ unescaped to $) and lengths matching var NAMES. Fix: escape all runtime secret refs as $${VAR}; keep ${CI_PIPELINE_FILES} braced (compile-time is correct for CI metadata). Removes flawed debug step; adds header documenting the substitution rules.	2026-07-27 12:50:21 -07:00
admin	7a81bc73bd	deploy.yml: add temporary debug-secrets-all step declaring the identical 31-secret environment block as provision-secrets, printing byte length of every var (never values). Runs as its own step so it completes even when provision-secrets fails, telling us in one run whether all secrets are empty, only some, or only swarm_manager_ip.	2026-07-27 12:39:53 -07:00
admin	1a1068a154	vaultwarden: retry #6 -- re-trigger after fixing invalid 3DPRINT_DB_PASSWORD env var name	2026-07-27 12:13:05 -07:00
admin	332708ad41	deploy.yml: rename env var alias 3DPRINT_DB_PASSWORD -> PRINT3D_DB_PASSWORD in provision-secrets. Shell/env var identifiers cannot start with a digit (POSIX); this invalid key was likely poisoning the entire environment block for this step, explaining why SWARM_MANAGER_IP read empty specifically in provision-secrets/deploy/verify (which all declare 20-30 secrets) while working fine in the isolated 2-secret debug step. The underlying Woodpecker secret name (3dprint_db_password) is unchanged -- only the YAML alias and its $-reference were renamed.	2026-07-27 12:12:39 -07:00
admin	cfd735f14d	vaultwarden: retry #5 -- real end-to-end test now that repo is public and pipeline bugs are fixed	2026-07-27 11:57:30 -07:00
admin	01333893f1	deploy.yml: remove temporary debug-secrets step. Secret injection confirmed working (SSH_KEY=516 bytes, SWARM_MANAGER_IP=12 bytes) after repo was switched back to public. Root cause of the entire multi-day secret-provisioning saga: (1) CI_COMMIT_CHANGED_FILES renamed to CI_PIPELINE_FILES in Woodpecker 3.16, (2) --no-interpolate flag was misplaced, (3) dotfolders matched as bogus stack names, (4) private-repo flip broke Woodpecker's secret injection entirely. All four now fixed.	2026-07-27 11:57:02 -07:00
admin	d89cfbf055	deploy.yml: add temporary debug-secrets step (prints byte length of SSH_KEY and SWARM_MANAGER_IP only, never values) to determine whether secret injection is failing repo-wide or just for swarm_manager_ip. Repo has been switched back to public.	2026-07-27 11:54:54 -07:00
admin	6fd3018a8c	vaultwarden: retry #4 -- re-trigger after fixing --no-interpolate flag position and dotfolder false-match bugs	2026-07-27 11:03:58 -07:00
admin	c511b2b607	deploy.yml: fix two bugs surfaced now that CI_PIPELINE_FILES parsing actually works: (1) --no-interpolate was placed before the config subcommand instead of after, causing 'unknown flag' on any real folder-stack validate; (2) folder-detection regex matched dotfolders like .woodpecker as bogus stack names, causing SSH/SCP attempts with empty SWARM_MANAGER_IP. Also added explicit empty-var guards for SWARM_MANAGER_IP in provision-secrets/deploy/verify so future secret misconfig fails fast with a clear message instead of a cryptic ssh error.	2026-07-27 11:03:35 -07:00
admin	a9297397c7	vaultwarden: retry #3 -- re-trigger secret provisioning now that CI_PIPELINE_FILES parsing is fixed	2026-07-27 10:52:01 -07:00
admin	206f6ff103	deploy.yml: fix root cause - CI_COMMIT_CHANGED_FILES does not exist in Woodpecker 3.16; the correct variable is CI_PIPELINE_FILES (JSON array format, e.g. [\"a/b.yaml\",\"c.yaml\"]). Updated all 5 usages (validate, provision-secrets x2, deploy, verify, notify-success) to read CI_PIPELINE_FILES and strip [ ] \" before splitting on comma. Removes temporary debug-env step.	2026-07-27 10:51:37 -07:00
admin	2b6286a1c9	deploy.yml: add temporary debug-env step to diagnose CI_COMMIT_CHANGED_FILES not populating despite Woodpecker UI showing correct changed files. No logic changes to existing steps -- remove once root cause is found.	2026-07-27 10:31:10 -07:00
admin	9447b277a1	vaultwarden: remap DATABASE_URL secret to versioned name (vaultwarden_database_url_v2), same in-container target path. Fixes rotation being silently blocked by Swarm (cannot rm a secret referenced by a running service).	2026-07-27 08:50:43 -07:00
admin	6ff6260b2f	deploy.yml: fix vaultwarden secret provisioning (was bypassing shared helper, silently failing on rm-while-in-use). Provision under versioned name vaultwarden_database_url_v2; compose-side remap happens in a follow-up commit.	2026-07-27 08:50:16 -07:00