My web application went down: incident response plan
Published on 2026-11-14 by frfrand
Operational guide for containing, diagnosing, recovering, and communicating an application outage with a small team.
# My web application went down: incident response plan
When an application stops responding, the order of decisions matters. Repeatedly restarting processes can erase evidence, worsen a data problem, or hide its cause. An effective response combines rapid diagnosis, containment, recovery, and communication.
1. Confirm the scope
Reproduce the failure from different networks. Ask: does it affect all routes or just one? Frontend, API, database, DNS, TLS certificate, or external provider? Did the outage start after a deployment? Record time in UTC, symptoms, recent changes, and responsible persons.
Initial indicative checks:
curl -I https://ejemplo.com
dig ejemplo.com
systemctl --failed
df -h
free -h Run commands only on authorized infrastructure. An HTTP 200 response from the home page does not prove that login, payments, and API work.
2. Classify and contain
Define severity by impact: data loss, total service failure, partial degradation, or visual defect. In case of intrusion indicators, preserve logs and evidence before cleaning or restarting; isolate compromised credentials and consider specialized advice. If it is a faulty deployment, avoid continuing to deploy without resolving the cause.
3. Diagnose by layers
| Layer | Indicator | Safe action | | --- | --- | --- | | DNS/TLS | Domain does not resolve or expired certificate | Check resolution and certificate | | Reverse proxy | 502/504 | Check upstream and times | | Application | 500 or crashes | Inspect logs and changes | | Database | Exhausted connections or slow queries | Measure pool, locks and availability | | System | Disk/CPU/RAM saturated | Measure consumption and responsible process | | Third parties | External API with failures | Activate degradation or limited retries |
Consult journalctl -u servicio --since "30 minutes ago" if you use systemd, respecting log privacy. Check Nginx and metrics before blindly increasing resources.
4. Restore with a reversible plan
If the latest release caused the failure, consider application rollback only when schema and data allow it. Do not revert a database with new writes without analyzing loss. If recovery from backup is needed, verify a reliable restore point and agree on possible data loss with the person in charge.
5. Communicate without inventing deadlines
Issue a brief message with affected functions, ongoing actions, temporary alternative, and time of the next update. Do not attribute cause without evidence. Keep a record of decisions and responsible persons. For clients, distinguish response time from resolution time.
6. Postmortem and prevention
After recovering, identify the technical cause and organizational factors. Include timeline, detection, impact, mitigation action, and preventive tasks with owner. Add business health checks, alerts, tested backups, and reversible deployments.
Related: DNS diagnosis and cron vs systemd timers.
Frequently asked questions
Is restarting always the first step? No; it depends on the risk of losing evidence and data. What does 502 mean? The proxy did not receive a valid response from upstream. When to inform the client? Early, with confirmed facts and agreed updates.