You found out from a customer
Something broke, nobody knew until someone complained, and the honest answer is that it could happen again tomorrow.
Does this sound familiar?
Almost every outage we have investigated began weeks earlier as something small that nothing was watching.
- The first report came from a customer, not a monitor
- It took a while to work out which part had actually failed
- Nobody is certain the same thing will not happen again
- The fix is done, but the reason it went unnoticed has not been addressed
What we do, in order
Roughly the first week. The sequence matters more than the speed.
We establish what really happened
Not who to blame — what the chain was. An expired certificate, a full disk and a failed backup have different causes and different fixes, and the visible symptom rarely names the right one.
We check whether it is still true
The same class of problem is usually present elsewhere. If one certificate lapsed, we check every certificate. If one backup failed, we test every restore.
We put watching in place
Expiry, renewals, disk, and availability, all checked continuously with alerts that reach a person. The point is not dashboards — it is that somebody is told.
We take over the routine
The work that prevents recurrence is unglamorous and constant. That is the part we do.
Why the fix is not the fix
Repairing what broke restores service without changing the thing that let it go unnoticed for weeks. The next failure will be a different component and the same story.
Common questions
Why did our website go down without any warning?
Because nothing was watching. Most outages begin weeks earlier as something small — a certificate approaching expiry, a disk filling, a backup silently failing. Without a check that alerts a person, the first report comes from a customer.
My SSL certificate expired even though auto-renewal was on. How?
Automatic renewal attempts can fail quietly. Certbot retries twice a day, logs the failure and exits, and will do that for thirty days without notifying anyone. The usual causes are DNS having moved, a changed web-server config, or a duplicate certificate where the wrong one is being served.
How quickly should we find out about an outage?
Within minutes, from a monitor rather than a customer. Equally important is being warned before failure — an expiry date approaching is a problem you can fix calmly; an expired certificate is an incident.
We fixed what broke. Is that enough?
It restores service, but it does not change the reason nobody noticed for weeks. The next failure will be a different component and the same story unless something is watching.