Operations

Recurring operational procedures for ByteBite infrastructure. Risky operations include pre-check, backup, action, post-validation, and rollback guidance.

Dokploy upgrades

Pre-check

• Note current Dokploy version (dokploy/dokploy:v0.30.5 as of 2026-09-08)
• Export Traefik container inspect: docker inspect dokploy-traefik > /root/traefik-pre-upgrade-inspect.json
• Confirm backup snapshots exist in /root/

Action

• Upgrade Dokploy via Dokploy's own update mechanism or host package manager (follow Dokploy documentation)

Post-validation

• Verify ByteBite staging application containers are healthy
• Critical: Run the Traefik post-upgrade checklist below
• Run Validation smoke tests

Rollback

• Restore Dokploy to previous version if application deploy breaks
• Restore Traefik from backup if ingress breaks

Traefik upgrades / recreates

Dokploy may recreate dokploy-traefik during platform upgrades. Custom DNS-01 wiring is not part of standard Dokploy and may be lost.

Traefik post-upgrade checklist

After any Dokploy upgrade or Traefik container recreation:

1. Inspect container mounts:

   docker inspect dokploy-traefik --format '{{json .Mounts}}' | python3 -m json.tool

   Verify /etc/dokploy/traefik/secrets → /run/secrets (read-only)

2. Inspect environment:

   docker inspect dokploy-traefik --format '{{range .Config.Env}}{{println .}}{{end}}' | grep CF_DNS

   Expected: CF_DNS_API_TOKEN_FILE=/run/secrets/cloudflare-dns-api-token

3. Verify static config contains letsencrypt-cloudflare resolver in /etc/dokploy/traefik/traefik.yml

4. Verify ACME files exist:

   • /etc/dokploy/traefik/dynamic/acme.json
   • /etc/dokploy/traefik/dynamic/acme-cloudflare.json

5. Test wildcard TLS:

   TENANT_HOST="ops-check-$(date +%s).staging.getbytebite.co"
   echo | openssl s_client -connect "${TENANT_HOST}:443" -servername "${TENANT_HOST}" 2>/dev/null | openssl x509 -noout -subject

6. Test wildcard API:

   curl -s "https://${TENANT_HOST}/api/health"

7. Verify explicit hosts still work (api, docs, dashboard, luigi, mario)

8. If any check fails, restore from /root/bytebite-traefik-post-wildcard-20260908-175333/ — see Disaster recovery

Ubuntu updates (APP VM)

Pre-check

• Confirm staging is healthy
• Schedule maintenance window if kernel update requires reboot

Action

sudo apt update && sudo apt upgrade -y

Reboot if required: sudo reboot

Post-validation

• SSH access restored
• Docker daemon running
• Dokploy UI accessible
• Full Validation suite

PostgreSQL upgrades

Managed by Dokploy. Follow Dokploy documentation for PostgreSQL version upgrades.

Pre-check

• No off-host backup exists yet — accept data loss risk or establish backup first
• Note current version (postgres:18)

Post-validation

• curl -s https://api.staging.getbytebite.co/health/db
• Application smoke tests

Docker cleanup / disk monitoring

Periodic checks

docker system df
df -h /

Cleanup (destructive — removes unused images/containers)

docker system prune -f

Do not prune volumes without confirming they are unused. PostgreSQL data volumes must not be pruned.

TLS checks

Run monthly or after infrastructure changes. See Validation TLS section.

Cloudflare token rotation

1. Generate new token (same scopes: Zone Read + DNS Edit on getbytebite.co)
2. Write to /etc/dokploy/traefik/secrets/cloudflare-dns-api-token (mode 600)
3. Restart dokploy-traefik
4. Verify wildcard TLS
5. Revoke old token in Cloudflare

Secret rotation (application)

1. Update value in Dokploy environment
2. Redeploy or restart affected service
3. Verify application health

Backup verification

Periodically confirm Traefik backup snapshots exist and are readable:

ls -la /root/bytebite-traefik-*

Establish PostgreSQL backup verification when off-host backups are implemented.

Dependency / runtime upgrades

Application runtime versions are pinned in Dockerfiles (Node 24, pnpm 12.3.4). Upgrade by:

1. Update Dockerfile ARG NODE_VERSION or pnpm version
2. Test locally with compose.deploy.yml
3. Deploy to staging
4. Run validation suite

Smoke tests

The canonical smoke test reference is Validation. Run after every infrastructure change.

Related

• Disaster recovery — failure runbooks
• Backups — snapshot locations
• Traefik — configuration details
• Known gaps — unresolved operational items