100 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
admin	325daca86b	ADR-0001 cutover.sh: fix Phase 1 node-label check (broken Go template on hyphenated label) and accept any 2xx in consumer health checks	2026-08-02 22:14:17 -07:00
admin	066cbaf97f	ADR-0001 Phase 3: add rollback.sh (standalone, idempotent, grace-window guard against post-cutover data loss)	2026-08-02 17:38:50 -07:00
admin	1203630bb5	ADR-0001 Phase 3: add cutover.sh (11-phase scripted cutover with auto-rollback)	2026-08-02 17:32:42 -07:00
admin	a4f5d54b82	ADR-0001 Phase 3: add postgresql-ha-final.yaml (Stage 3 - HAProxy alias handoff, no external port/Traefik yet)	2026-08-02 17:28:26 -07:00
admin	01768d14f5	ADR-0001 Phase 3: add preflight.sh (disk/backup hard gates, catalog+consumer enumeration)	2026-08-02 16:45:44 -07:00
admin	f49b576806	ADR-0001 Phase 3: add cutover RUNBOOK (dependency map, phased procedure, rollback)	2026-08-02 16:35:52 -07:00
admin	8d30f19c8c	postgresql.yaml: update stale comment referencing old CLONE_* rationale to reference standby_cluster design (no functional change - this file never used CLONE_* itself)	2026-07-30 22:03:44 -07:00
admin	4dcb3561ed	postgresql-ha-staging.yaml: switch from CLONE_WITH_BASEBACKUP to Patroni standby_cluster (continuous streaming, validated in pgha-test dry run) - closes pre-cutover write gap	2026-07-30 21:59:44 -07:00
admin	dc6120ffd4	pgha-dryrun.yaml: trigger secret re-provisioning after regenerating postgresql_replication_password (excludes &<>\" per pystache HTML-escaping bug found this session)	2026-07-30 21:44:49 -07:00
admin	8afe0049c4	pgha-dryrun.yaml: switch from CLONE_WITH_BASEBACKUP to Patroni standby_cluster (continuous streaming) to close the pre-cutover write gap	2026-07-30 21:24:28 -07:00
admin	499402cd0d	postgresql-ha-staging.yaml: port dry-run fixes (bugs 1,2,4,5) - $$(...) escaping (incl. CLONE_PASSWORD), ETCD3_HOSTS, post_init_wrapper.sh SUPERUSER role fix	2026-07-30 15:56:43 -07:00
admin	7e8e3388f4	postgresql.yaml: port dry-run fixes (bugs 1,2,4,5) - $$(...) escaping, ETCD3_HOSTS, post_init_wrapper.sh SUPERUSER role fix	2026-07-30 15:55:26 -07:00
admin	b7f0d9dbfb	pgha dry-run: fix bug 5 - grant SUPERUSER to placeholder postgres role	2026-07-29 09:32:09 -07:00
admin	e8e506dc4d	add: post_init_wrapper.sh to work around Spilo's hardcoded 'postgres' role name assumption in post_init.sh. Does not fork/modify Spilo's script — creates missing role then execs the original unchanged.	2026-07-29 07:21:34 -07:00
admin	7529e6cb36	fix: override bootstrap.post_init via SPILO_CONFIGURATION to create missing 'postgres' role before Spilo's real post_init.sh runs. Spilo hardcodes ALTER VIEW...OWNER TO postgres with no way to parameterize, which fails since our superuser is PGadmin not postgres.	2026-07-29 06:35:24 -07:00
admin	b86784fe3a	fix: legacy container needs pg_hba.conf replication rule for pg_basebackup — added initdb.d hook script. Disposable test only, uses 'trust' since this container is not auth-representative.	2026-07-28 21:27:13 -07:00
admin	1dcb5ec467	fix: use ETCD3_HOSTS not ETCD_HOSTS — confirmed against spilo source that etcd/etcd3 are distinct DCS backends (v2 vs v3 API). Our etcd 3.5.9 containers have v2 API disabled, causing 404s with the old var name.	2026-07-28 21:10:13 -07:00
admin	83480aa23b	fix: escape \$(...) as \$\$(...) in patroni command blocks — Compose interpolation was choking on \$( before the shell ever saw it (invalid interpolation format error)	2026-07-28 21:03:56 -07:00
admin	20e1210441	postgresql: add disposable dry-run test stack for cutover validation (pgha-test, own network + data dirs, zero prod impact)	2026-07-28 11:12:19 -07:00
admin	f4f0749969	postgresql: add cutover staging compose (ADR-0001 Phase 3). Lives in cutover/ subdir — deliberately excluded from stack-deploy.sh folder merge. Deployed only by cutover script as separate postgresqlha stack.	2026-07-28 11:07:03 -07:00
admin	87ae54fb3f	postgresql: add HAProxy config for Patroni-aware TCP routing (ADR-0001 Phase 2)	2026-07-28 08:53:55 -07:00
admin	c8d45e9e61	postgresql: add folder-based Patroni+etcd+HAProxy HA stack draft (ADR-0001 Phase 2). NOT deployed — bootstrap tier, manual deploy only. Coexists with flat postgresql.yaml until cutover.	2026-07-28 08:53:44 -07:00
admin	9a5b028e07	postgresql: document replication + patroni secrets (ADR-0001 Phase 1)	2026-07-28 07:20:52 -07:00
admin	0f26d31a02	postgresql: add replication + patroni Woodpecker secrets to provision-secrets step (ADR-0001 Phase 1)	2026-07-28 07:20:35 -07:00
admin	6789d2530a	vaultwarden: retry #8 -- trigger after fixing pipeline compile error (dollar-brace literal in comments)	2026-07-27 13:36:17 -07:00
admin	e0a6b7d233	deploy.yml: fix pipeline compile error 'missing closing brace' -- the documentation comment itself contained a literal dollar-brace sequence with invalid contents (dollar-brace CI_...), which Woodpecker's substitution engine parses even inside comments. Rewrote header to describe the rules in words without any literal dollar-brace sequences. Also removed backtick-quoted dollar-brace remnants from the vaultwarden case comment.	2026-07-27 13:35:13 -07:00
admin	9b3bba636c	vaultwarden: retry #7 -- trigger after fixing Woodpecker compile-time ${VAR} substitution root cause	2026-07-27 12:52:59 -07:00
admin	6757b0dfe1	deploy.yml: THE ACTUAL ROOT CAUSE FIX. Woodpecker substitutes ${VAR} at pipeline-compile time from its CI metadata map, where secrets do not exist -- so every braced reference to a secret-backed env var (SWARM_MANAGER_IP, all provisioning values) was silently blanked before the shell ever ran. Proven by debug output showing eval "val=\$$v" arriving as val=\$v ($$ unescaped to $) and lengths matching var NAMES. Fix: escape all runtime secret refs as $${VAR}; keep ${CI_PIPELINE_FILES} braced (compile-time is correct for CI metadata). Removes flawed debug step; adds header documenting the substitution rules.	2026-07-27 12:50:21 -07:00
admin	7a81bc73bd	deploy.yml: add temporary debug-secrets-all step declaring the identical 31-secret environment block as provision-secrets, printing byte length of every var (never values). Runs as its own step so it completes even when provision-secrets fails, telling us in one run whether all secrets are empty, only some, or only swarm_manager_ip.	2026-07-27 12:39:53 -07:00
admin	1a1068a154	vaultwarden: retry #6 -- re-trigger after fixing invalid 3DPRINT_DB_PASSWORD env var name	2026-07-27 12:13:05 -07:00
admin	332708ad41	deploy.yml: rename env var alias 3DPRINT_DB_PASSWORD -> PRINT3D_DB_PASSWORD in provision-secrets. Shell/env var identifiers cannot start with a digit (POSIX); this invalid key was likely poisoning the entire environment block for this step, explaining why SWARM_MANAGER_IP read empty specifically in provision-secrets/deploy/verify (which all declare 20-30 secrets) while working fine in the isolated 2-secret debug step. The underlying Woodpecker secret name (3dprint_db_password) is unchanged -- only the YAML alias and its $-reference were renamed.	2026-07-27 12:12:39 -07:00
admin	cfd735f14d	vaultwarden: retry #5 -- real end-to-end test now that repo is public and pipeline bugs are fixed	2026-07-27 11:57:30 -07:00
admin	01333893f1	deploy.yml: remove temporary debug-secrets step. Secret injection confirmed working (SSH_KEY=516 bytes, SWARM_MANAGER_IP=12 bytes) after repo was switched back to public. Root cause of the entire multi-day secret-provisioning saga: (1) CI_COMMIT_CHANGED_FILES renamed to CI_PIPELINE_FILES in Woodpecker 3.16, (2) --no-interpolate flag was misplaced, (3) dotfolders matched as bogus stack names, (4) private-repo flip broke Woodpecker's secret injection entirely. All four now fixed.	2026-07-27 11:57:02 -07:00
admin	d89cfbf055	deploy.yml: add temporary debug-secrets step (prints byte length of SSH_KEY and SWARM_MANAGER_IP only, never values) to determine whether secret injection is failing repo-wide or just for swarm_manager_ip. Repo has been switched back to public.	2026-07-27 11:54:54 -07:00
admin	6fd3018a8c	vaultwarden: retry #4 -- re-trigger after fixing --no-interpolate flag position and dotfolder false-match bugs	2026-07-27 11:03:58 -07:00
admin	c511b2b607	deploy.yml: fix two bugs surfaced now that CI_PIPELINE_FILES parsing actually works: (1) --no-interpolate was placed before the config subcommand instead of after, causing 'unknown flag' on any real folder-stack validate; (2) folder-detection regex matched dotfolders like .woodpecker as bogus stack names, causing SSH/SCP attempts with empty SWARM_MANAGER_IP. Also added explicit empty-var guards for SWARM_MANAGER_IP in provision-secrets/deploy/verify so future secret misconfig fails fast with a clear message instead of a cryptic ssh error.	2026-07-27 11:03:35 -07:00
admin	a9297397c7	vaultwarden: retry #3 -- re-trigger secret provisioning now that CI_PIPELINE_FILES parsing is fixed	2026-07-27 10:52:01 -07:00
admin	206f6ff103	deploy.yml: fix root cause - CI_COMMIT_CHANGED_FILES does not exist in Woodpecker 3.16; the correct variable is CI_PIPELINE_FILES (JSON array format, e.g. [\"a/b.yaml\",\"c.yaml\"]). Updated all 5 usages (validate, provision-secrets x2, deploy, verify, notify-success) to read CI_PIPELINE_FILES and strip [ ] \" before splitting on comma. Removes temporary debug-env step.	2026-07-27 10:51:37 -07:00
admin	2b6286a1c9	deploy.yml: add temporary debug-env step to diagnose CI_COMMIT_CHANGED_FILES not populating despite Woodpecker UI showing correct changed files. No logic changes to existing steps -- remove once root cause is found.	2026-07-27 10:31:10 -07:00
admin	9447b277a1	vaultwarden: remap DATABASE_URL secret to versioned name (vaultwarden_database_url_v2), same in-container target path. Fixes rotation being silently blocked by Swarm (cannot rm a secret referenced by a running service).	2026-07-27 08:50:43 -07:00
admin	6ff6260b2f	deploy.yml: fix vaultwarden secret provisioning (was bypassing shared helper, silently failing on rm-while-in-use). Provision under versioned name vaultwarden_database_url_v2; compose-side remap happens in a follow-up commit.	2026-07-27 08:50:16 -07:00
admin	71cb6a7e81	vaultwarden: retry secret re-provision trigger (previous run had empty changed-files)	2026-07-26 22:27:58 -07:00
admin	4a3eb44a62	vaultwarden: trigger secret re-provision (db -> postgresql alias)	2026-07-26 22:10:46 -07:00
admin	61b1a645bb	uptime-kuma: set restart_policy.delay to 30s ... Increases the delay before Swarm restarts a failed uptime-kuma task from the default 5s to 30s. Applied live via `docker service update --restart-delay 30s` on 2026-07-26; this persists it so it survives the next stack deploy. Note: this does not pin the task to a specific node or prevent cross-node reschedule races against the CephFS-backed SQLite data dir - it only slows the restart-after-failure loop. See prior incident notes for the DB corruption root cause discussion.	2026-07-26 17:34:35 -07:00
admin	2b0ef411d1	chore(ai): sync ai.yaml with host (manual litellm v1.93.0 upgrade comment)	2026-07-25 22:20:28 -07:00
admin	cc9880e09c	chore(traefik): remove flat traefik.env, superseded by traefik/traefik.env	2026-07-25 22:20:27 -07:00
admin	1494224194	chore(traefik): remove flat traefik.yaml, superseded by traefik/traefik.yaml	2026-07-25 22:20:27 -07:00
admin	d16203b873	chore(auth): remove flat auth.yaml, superseded by auth/auth.yaml	2026-07-25 22:20:26 -07:00
admin	c2069e3e0d	feat(traefik): migrate to folder-based stack layout	2026-07-25 22:19:26 -07:00
admin	6caa7f0a33	feat(traefik): migrate to folder-based stack layout	2026-07-25 22:19:25 -07:00
admin	7ccff906fd	feat(auth): migrate to folder-based stack layout	2026-07-25 22:19:24 -07:00
admin	68f89ef139	chore(ci): remove root .woodpecker.yml, superseded by .woodpecker/deploy.yml ... Woodpecker ignores this file entirely once .woodpecker/ folder exists (config resolution: .woodpecker/ > .woodpecker.yaml > .woodpecker.yml). Content preserved verbatim in .woodpecker/deploy.yml.	2026-07-25 22:00:57 -07:00
admin	270489b207	fix(ci): move .woodpecker.yml -> .woodpecker/deploy.yml ... Root cause of Woodpecker push pipelines silently no-op'ing since 2026-07-18: Woodpecker's config resolution is .woodpecker/*.{yaml,yml} -> .woodpecker.yaml -> .woodpecker.yml Once .woodpecker/renovate.yml existed, the root .woodpecker.yml was entirely ignored. Every push webhook hit renovate.yml's `when: event: cron` filter, matched zero steps, and Woodpecker logged "ignoring hook: 'when' filters filtered out all steps" instead of running deploy. Moving the push pipeline into .woodpecker/deploy.yml restores it as an independent workflow alongside renovate.yml -- each file's own `when:` filter now applies correctly (push -> deploy.yml, cron -> renovate.yml).	2026-07-25 22:00:49 -07:00
admin	cc5adc3312	feat(mcpo): sync config.json to Gitea as source of truth ... Adds powerautomate entry with Azure app registration credentials. Auth dir (/volume1/docker/mcpo/powerautomate-auth/) and config stub already exist on host from prior session. NOTE: powerautomate-mcp package not yet on npm - npx will fail until published. Tracked here for GitOps pipeline; will update command when package is available or we switch to build-from-source approach.	2026-07-25 15:10:07 -07:00
admin	963a52d665	litellm: upgrade v1.92.1 -> v1.93.0 to fix Bedrock parallel tool call truncation bug	2026-07-24 22:16:39 -07:00
admin	2a1feee03e	feat(mcpo): sync config.json from host (real PA_MCP_CLIENT_ID + TENANT_ID)	2026-07-24 22:06:24 -07:00
admin	8fde5266ea	chore(ai): remove flat ai.yaml, superseded by ai/ai.yaml	2026-07-24 22:02:22 -07:00
admin	92abfe295f	feat(ai): migrate to folder-based stack layout	2026-07-24 22:01:45 -07:00
admin	3da6a89919	feat(woodpecker): add PA_MCP_CLIENT_ID secret provisioning for ai stack	2026-07-19 14:23:22 -07:00
admin	0030d92ca8	chore(mcpo): sync config.json from live disk - adds powerautomate entry	2026-07-19 14:20:27 -07:00
admin	0f7f00404d	feat(mcpo): add config.json to source control with powerautomate, ms365 fixes ... - Pin ms365 to 0.129.0 with --discovery flag (fixes MCPO schema parse error) - Add powerautomate entry (PA_MCP_CLIENT_ID placeholder - needs Azure app reg) - Update authentik token (was expired) - teams-mcp-auth volume now source of truth for ms365/teams auth cache - powerautomate-auth volume wired for future PA auth cache NOTE: config.json was previously managed manually on disk only. This commit brings it under GitOps control.	2026-07-19 14:16:40 -07:00
admin	815d953cd7	chore(renovate): remove unused TEAMS_WEBHOOK env - notifications via Gitea webhook	2026-07-19 07:00:37 -07:00
admin	310c68376a	chore(renovate): revert postUpgradeTasks - notifications via Gitea webhook instead	2026-07-19 07:00:17 -07:00
admin	43f3fdce94	chore(renovate): add postUpgradeTasks Teams notification via Power Automate webhook	2026-07-19 06:59:49 -07:00
admin	d9a6d217bc	chore(renovate): pass TEAMS_WEBHOOK secret into renovate container	2026-07-19 06:59:28 -07:00
admin	088d884a88	chore(renovate): add Teams webhook notification via Power Automate	2026-07-19 06:59:02 -07:00
admin	96783bce8c	chore(renovate): fix config warnings - remove global-only options, disable woodpecker manager	2026-07-18 22:48:22 -07:00
admin	8185fb5842	feat: add woodpecker cron pipeline for renovate (v3 syntax)	2026-07-18 22:05:19 -07:00
admin	fe6c95f54a	feat: add renovate.json configuration	2026-07-18 22:05:01 -07:00
admin	aea8657092	feat: add renovate env example	2026-07-18 22:05:01 -07:00
admin	ac87ea2520	feat: add renovate stack (image update automation)	2026-07-18 22:05:00 -07:00
admin	5a5718a2d3	traefik: fix update order for host-mode/VRRP services (start-first -> stop-first) ... reverse-proxy publishes host-mode ports (80/443/1883/5201/5431) — start-first can never bind while the old task holds the ports, deadlocking every rolling update. keepalived master/backup run VRRP on the host network — concurrent old+new speakers with the same router-id risk VIP flapping. Both now use stop-first (global mode + parallelism=1 + VIP failover keeps ingress up). Stateless helpers (whoami, speedtest, iperf3, certs-dumper) keep start-first. Step 1 of Tier 0/1 pipeline inclusion (prep for removing traefik/woodpecker from the bootstrap manual-deploy guard).	2026-07-16 23:04:00 -07:00
admin	5a24c162cc	fix(pipeline): sync bootstrap stack files before guard, not after	2026-07-16 22:29:36 -07:00
admin	4100daa199	Add bootstrap-tier guard to deploy step (traefik/woodpecker/postgresql skip auto-deploy)	2026-07-16 21:44:53 -07:00
admin	26177f5706	Add update_config/rollback_config start-first rolling deploy; fix depends_on indent bug	2026-07-16 21:44:47 -07:00
admin	beb53ce9ff	Add update_config/rollback_config rolling deploy to all traefik services (start-first, parallelism=1)	2026-07-16 21:44:41 -07:00
admin	6221f5d213	chore: bump litellm-database v1.90.3 -> v1.92.0 ... Latest stable release. Includes Bedrock fixes since 1.90.x (anthropic-beta header forwarding, converse stream handling, STS region decoupling).	2026-07-15 22:00:16 -07:00
admin	bc1f4f288a	fix(security): correct crowdsec-web-ui image typo — thebluffman85 -> theduffman85 ... Image ghcr.io/thebluffman85/crowdsec-web-ui:latest never existed (typo in org name). Correct upstream project is TheDuffman85/crowdsec-web-ui on GitHub. Fixed to ghcr.io/theduffman85/crowdsec-web-ui:latest. Deployed and verified live on docker-2 prior to this commit: - security_crowdsec-web-ui now pulls successfully and is Running - Host(`secure.bryanmail.net`) rule intact, all CROWDSEC_* env vars correct - security_crowdsec unaffected, still running	2026-07-10 15:43:01 -07:00
admin	1876fedfff	feat(postgresql): complete Pattern C migration — remove leftover env_file directives ... Docker secrets (postgresql_password, postgresql_pgadmin_password) already handled POSTGRES_PASSWORD_FILE/PGADMIN_DEFAULT_PASSWORD_FILE. This removes the two remaining `env_file: - postgresql.env` lines (postgresql + pgadmin services) which were redundant and left the stack in a half-migrated hybrid state. Non-secret vars (POSTGRES_USER, PGADMIN_DEFAULT_EMAIL, DATABASUS_HOST, PGADMIN_HOST) continue to be supplied via stack-deploy.sh's envsubst mechanism from postgresql.env at deploy time — no functional change to those values. Deployed and verified live on docker-2 prior to this commit: - postgresql_postgresql: pg_isready OK, all 21 databases intact, psql auth via Docker secret confirmed - postgresql_pgadmin: Host(`dba.bryanmail.net`) rule + PGADMIN_DEFAULT_EMAIL correct - postgresql_databasus: running - All dependent stacks (mealie, vaultwarden, auth, media, homeassistant, guacamole, security) remained healthy throughout, zero reconnect issues	2026-07-10 15:36:15 -07:00
admin	fc1c808041	chore(guacamole): remove DOMAIN - provided by global.env	2026-07-10 15:31:30 -07:00
admin	6f571ae877	chore(mealie): remove DOMAIN - provided by global.env	2026-07-10 15:31:29 -07:00
admin	ee7034c4e7	chore(security): remove DOMAIN - provided by global.env	2026-07-10 15:31:29 -07:00
admin	e41149f0ab	chore(vaultwarden): remove DOMAIN+ALT_DOMAIN - provided by global.env	2026-07-10 15:31:28 -07:00
admin	30dc3d0f22	chore(homeassistant): remove DOMAIN - provided by global.env	2026-07-10 15:31:28 -07:00
admin	6587c62d80	chore(maintenance): remove DOMAIN - provided by global.env	2026-07-10 15:31:27 -07:00
admin	2ac4a73ae7	chore(traefik): remove DOMAIN - provided by global.env	2026-07-10 15:31:27 -07:00
admin	09e97b1572	chore(media): remove DOMAIN - provided by global.env	2026-07-10 15:31:26 -07:00
admin	8884e9c0ff	fix(global.env): ALT_DOMAIN=whooshh.net	2026-07-10 15:31:26 -07:00
admin	2b6e937b57	feat(stack-deploy): load global.env as base, stack env overrides on top	2026-07-08 21:29:15 -07:00
admin	b664e9d41d	feat(envparse): add merge mode for global+stack env layering	2026-07-08 21:29:13 -07:00
admin	78fe8092ad	feat(deploy): add global.env for shared vars (DOMAIN, DOMAIN_NAME, ALT_DOMAIN)	2026-07-08 21:29:12 -07:00
admin	9f7dcc3970	fix(media): add DOMAIN to media.env — was missing, causing blank Traefik hostnames	2026-07-08 21:25:34 -07:00
admin	2b4332896e	fix(ci): verify step fails pipeline on zero running tasks; increase settle time	2026-07-08 21:23:00 -07:00
admin	33f63924fc	feat(postgresql): migrate to Pattern C — Docker secrets for POSTGRES_PASSWORD and PGADMIN_DEFAULT_PASSWORD (_FILE convention)	2026-07-06 23:37:16 -07:00
admin	01469a7138	ci: trigger pipeline for ai stack (litellm-database:main-latest)	2026-07-05 21:50:20 -07:00
admin	2b547de27a	Update README.md	2026-07-05 21:42:27 -07:00
admin	76815f93e4	fix(ai): litellm image litellm-database:main-latest to match running stack	2026-07-05 21:41:49 -07:00
admin	c55eae3f24	ci: update .woodpecker.yml ai provision to Pattern B no-op	2026-07-04 22:35:54 -07:00
admin	db27239e68	ci: add ai.yaml (Pattern B - remove env_file, inline env vars)	2026-07-04 22:35:54 -07:00
admin	eff099eddf	immich: restore yaml default to :-release, version controlled via immich.env	2026-07-02 07:32:38 -07:00