Disaster Recovery

Practical recovery procedures for known failure modes. Where a complete recovery path does not exist, the gap is labeled explicitly.

Traefik broken after config change

Symptoms: TLS errors, routing failures, or Traefik container crash after editing host config.

Diagnosis (read-only)

docker logs dokploy-traefik --tail 100
docker inspect dokploy-traefik --format '{{json .Mounts}}' | python3 -m json.tool

Recovery (destructive — restores prior config)

1. Stop Traefik: docker stop dokploy-traefik
2. Restore files from the most recent backup:
   • Pre-wildcard: /root/bytebite-traefik-backup-20260908-172700/
   • Post-wildcard: /root/bytebite-traefik-post-wildcard-20260908-175333/
3. Copy traefik.yml, dynamic files, and ACME JSON back to /etc/dokploy/traefik/
4. Start Traefik: docker start dokploy-traefik
5. Run Validation smoke tests

Traefik recreated by Dokploy upgrade

Symptoms: Wildcard TLS fails; CF_DNS_API_TOKEN_FILE missing; letsencrypt-cloudflare resolver absent.

Diagnosis (read-only)

docker inspect dokploy-traefik --format '{{range .Config.Env}}{{println .}}{{end}}' | grep CF_DNS
docker inspect dokploy-traefik --format '{{json .Mounts}}' | python3 -m json.tool

Check for:

• CF_DNS_API_TOKEN_FILE=/run/secrets/cloudflare-dns-api-token
• Mount: /etc/dokploy/traefik/secrets → /run/secrets
• traefik.yml contains letsencrypt-cloudflare resolver

Recovery

Host files (traefik.yml, acme-cloudflare.json, token file) likely survived on disk. Re-apply container customization:

1. Verify host files exist and token file is mode 600
2. Recreate or reconfigure dokploy-traefik with secret mount and env var (via Dokploy or manual docker run matching prior inspect)
3. Use post-wildcard backup inspect output as reference
4. Run full Validation including wildcard TLS

See Operations upgrade checklist.

Wildcard TLS broken

Symptoms: Explicit hosts work; arbitrary *.staging.getbytebite.co hosts show TLS errors.

Diagnosis

TENANT_HOST="dr-test-$(date +%s).staging.getbytebite.co"
echo | openssl s_client -connect "${TENANT_HOST}:443" -servername "${TENANT_HOST}" 2>&1 | head -20
docker logs dokploy-traefik 2>&1 | grep -i acme | tail -20

Check acme-cloudflare.json exists and Cloudflare token is valid.

Recovery

1. Verify Cloudflare token file and mount (see Traefik recreate runbook above)
2. If acme-cloudflare.json is corrupt, restore from post-wildcard backup
3. Restart Traefik and wait for ACME retry or trigger re-issuance
4. Validate wildcard TLS

Cloudflare token invalid or rotated

Symptoms: ACME DNS-01 challenge failures in Traefik logs.

Recovery

1. Generate new token in Cloudflare (Zone Read + DNS Edit on getbytebite.co)
2. Write to /etc/dokploy/traefik/secrets/cloudflare-dns-api-token (mode 600)
3. Restart dokploy-traefik
4. Verify wildcard TLS

Never document or commit the token value.

APP VM lost

Status: GAP — no complete recovery path exists.

Required to rebuild (high level):

1. Provision new VM with 88.99.0.118 (or update DNS if IP changes)
2. Install Dokploy, recreate ByteBite staging project
3. Restore Traefik config from backup snapshots (if accessible)
4. Restore PostgreSQL from off-host backup (no backup exists yet)
5. Redeploy application from Git
6. Run migrations and staging seeds on fresh DB
7. Full validation

Without off-host PostgreSQL backups, database data is unrecoverable.

PostgreSQL failure

Symptoms: /health/db returns degraded; API tenant routes return 503.

Diagnosis (read-only)

# From APP VM, if Docker access is available
docker ps --filter name=postgres

Check Dokploy PostgreSQL service status in Dokploy UI.

Recovery

MAIL VM lost

Impact: Mail DNS points to 88.99.0.97 but mail was not fully configured. Recovery requires reprovisioning BYTEBITE-MAIL-01 and reconfiguring Stalwart (planned).

DNS mistake

Symptoms: Hostname resolves to wrong IP or does not resolve.

Diagnosis

dig +short edge.getbytebite.co
dig +short api.staging.getbytebite.co

Recovery

1. Correct record in Cloudflare dashboard (DNS-only, not proxied)
2. Wait for TTL propagation
3. Re-run DNS validation from Validation

Failed ByteBite application deployment

Symptoms: 502/503 from Traefik; containers unhealthy in Dokploy.

Diagnosis (read-only)

Check Dokploy deployment logs and container health:

docker ps --filter name=bytebite-staging
docker logs <api-container-id> --tail 50

Recovery

1. Identify failing service from logs
2. If migration needed: node db/dist/migrate.js in api container
3. If bad commit: redeploy previous Git commit via Dokploy
4. Validate

Related

• Backups — snapshot locations
• Operations — preventive maintenance
• Known gaps — unresolved recovery paths
• Troubleshooting — symptom index