Incident

an event that puts a system's data or service at risk

What it is

An incident is a problem in a running system that has become serious enough for people to drop their planned work and deal with it now: a service that has stopped answering, records that have been altered, a machine that has run out of memory, an account someone else is using. Security teams say “security incident”, operations teams say “production incident” or simply “outage”, and most organisations sort their incidents into severity levels.

NIST’s incident-response guidance draws the line in three steps, and they hold well outside security. An event is any observable occurrence involving computing assets: a login attempt, a software update, an app answering a request. An adverse event is one with a negative consequence, whatever caused it, and the guide puts natural disasters and power failures beside attacks. An incident is an adverse event that someone has judged to meet the organisation’s criteria, and the guide’s verb for that judgement is declared. The definition it relies on, taken from US federal law, covers an occurrence that actually or imminently puts information or a system at risk of losing its confidentiality, integrity or availability, or that breaks security policy.

Security and operations use the word with different centres of gravity. In security work an incident usually implies an adversary, a policy breach or data that may have left the building. In operations it means any unplanned interruption or serious degradation of a service, with no attacker in the story. The two overlap more than the vocabulary suggests: a denial-of-service attack and an ordinary traffic spike can look alike on a graph at first, and a disk that fills up can lose data as thoroughly as an intruder.

The neighbouring words are worth keeping apart. A defect is a flaw in the product; it can sit unnoticed for years and becomes part of an incident only when it starts doing harm. An alert is a message saying that something may be wrong; plenty of alerts never become incidents, and plenty of incidents are first reported by a customer. Service-management practice tracks the cause behind one or more incidents separately, as a problem.

Why it matters

The decision hiding inside the word is a threshold. At what point does a slow page, an odd login or a growing queue stop being one engineer’s ticket and become the team’s first priority? Set the bar too high and a spreading breach gets handled as routine work by someone who cannot see it growing. Set it too low and people are pulled out of their work so often that declaring stops meaning anything. NIST’s answer is to agree the criteria before they are needed: incidents are declared when adverse events meet criteria the organisation has already defined, with known false positives taken into account. Written criteria with examples are far cheaper than an argument at two in the morning.

A declaration changes who does what. One person leads and coordinates rather than debugging, others are released from their normal work, one channel carries the facts and a timestamped record starts. Without that, the same people do the same work in parallel, and the first clear statement about the problem is the one a customer writes.

For some companies there is also a legal clock. The EU’s NIS2 directive, Directive (EU) 2022/2555, requires the organisations it covers to report a significant incident to the national authority: an early warning within 24 hours of becoming aware of it, an incident notification within 72 hours, and a final report within one month of that notification. Its definition reaches beyond attacks to any event compromising the availability, authenticity, integrity or confidentiality of data or of the services a network and information system offers, so a severe outage can count. A team that starts asking “is this an incident?” on the second day has already spent the first window.

The running cost is people’s time, and it is larger than it used to be. The 2025 revision of NIST’s guide notes that incidents now happen frequently and that recovering from them often takes weeks or months, where its 2012 predecessor assumed they were rare and over within a day or two. Budgeting for incidents means budgeting for the follow-up work as well as for the night itself.

What it looks like in practice

The best-measured incident on this site happened on a macOS desktop application. The application filled memory at roughly 41 MB a second until the operating system killed it; the recorded incident reached 111 GB of compressed pages on a 36 GB machine, and one short session left a log file of 610,996 lines. The response found three causes that multiplied each other: an event stream consumed with no bound, subscribers that each received every event, and logging whose volume grew with the fault it was recording. Each path got a limit, a regression test now asserts the memory ceiling under a high event rate, and the same session writes 1,411 lines. The price is named too: the logging budget discards information, so the record of the next fast failure will be deliberately incomplete.

Other incidents are the repeating kind. Eight WireGuard and IPsec tunnels between Google Cloud and Linux systems were fragile and slow to diagnose, and a dead tunnel could cut communication between environments. The work standardised their configuration, monitored their health and went after the recurring routing and key-exchange problems at the root rather than restarting the tunnels, and the recurring connectivity incidents stopped. When the same thing happens for the third time, the cause is the incident.

And some incidents begin with somebody trying to help. On a platform with no admin tool, approving a claim, hiding a review or clearing a cache falls to an engineer with a database console, which the project page calls slow, unlogged and one typo away from an incident. The operator back-office built there, around 40 admin routes made from 128 components, puts that work behind the same authorization and the same audit trail as the rest of the product, at the ongoing cost of a large internal surface to keep tested and accessible.

Where it goes wrong

Five people work on it and nobody leads it. The chat channel fills with theories, two engineers roll back different things, and customers hear nothing for an hour, because nobody declared and so nobody was given the job of coordinating. Let anyone on the team declare, and make the first act of a declaration naming one lead who keeps the picture and the record.

The same incident returns every few weeks. The response ends when the service is back, and a restart counts as the fix. Treat restoring service and removing the cause as two tasks: close the first on the night, and give the second an owner and a date before anyone goes back to bed.

The tooling makes the incident bigger. Logging that writes a line per event produces a log as large as the fault, and alert rules send hundreds of notifications for one cause. Give logs and alerts a budget in advance, collapse repeats, and test the reporting path under the load of the failure it reports.

The repair causes the next incident. A hurried change typed by hand against production fixes one symptom, starts another problem, and leaves a hole in the timeline the review needs later. Keep a short list of pre-approved recovery actions that leave a trail, and record every change made during an incident.

The lessons arrive months late, or arrive as blame. A review held after weeks of recovery remembers little, and one that asks who made the mistake teaches people to hide the next one. NIST’s revised guide says lessons learned should often be shared as soon as they are identified, and its recovery guidance ends with an after-action report on the incident, the actions taken and what was learned. Write down what made the mistake easy, and change that.

Incident

What it is

Why it matters

What it looks like in practice

Where it goes wrong

Further reading