{"canonical_url":"https://engineer.company/glossary/incident/","content_markdown":"# Incident\n\n## What it is\n\nAn incident is a problem in a running system that has become serious enough for people to drop\ntheir planned work and deal with it now: a service that has stopped answering, records that have\nbeen altered, a machine that has run out of memory, an account someone else is using. Security\nteams say “security incident”, operations teams say “production incident” or simply “outage”, and\nmost organisations sort their incidents into severity levels.\n\nNIST's incident-response guidance draws the line in three steps, and they hold well outside\nsecurity. An event is any observable occurrence involving computing assets: a login attempt, a\nsoftware update, an app answering a request. An adverse event is one with a negative consequence,\nwhatever caused it, and the guide puts natural disasters and power failures beside attacks. An\nincident is an adverse event that someone has judged to meet the organisation's criteria, and the\nguide's verb for that judgement is _declared_. The definition it relies on, taken from US federal\nlaw, covers an occurrence that actually or imminently puts information or a system at risk of losing\nits confidentiality, integrity or availability, or that breaks security policy.\n\nSecurity and operations use the word with different centres of gravity. In security work an\nincident usually implies an adversary, a policy breach or data that may have left the building. In\noperations it means any unplanned interruption or serious degradation of a service, with no\nattacker in the story. The two overlap more than the vocabulary suggests: a denial-of-service\nattack and an ordinary traffic spike can look alike on a graph at first, and a disk that fills up\ncan lose data as thoroughly as an intruder.\n\nThe neighbouring words are worth keeping apart. A defect is a flaw in the product; it can sit\nunnoticed for years and becomes part of an incident only when it starts doing harm. An alert is a\nmessage saying that something may be wrong; plenty of alerts never become incidents, and plenty of\nincidents are first reported by a customer. Service-management practice tracks the cause behind\none or more incidents separately, as a problem.\n\n## Why it matters\n\nThe decision hiding inside the word is a threshold. At what point does a slow page, an odd login\nor a growing queue stop being one engineer's ticket and become the team's first priority? Set the\nbar too high and a spreading breach gets handled as routine work by someone who cannot see it\ngrowing. Set it too low and people are pulled out of their work so often that declaring stops\nmeaning anything. NIST's answer is to agree the criteria before\nthey are needed: incidents are declared when adverse events meet criteria the organisation has\nalready defined, with known false positives taken into account. Written criteria with examples\nare far cheaper than an argument at two in the morning.\n\nA declaration changes who does what. One person leads and coordinates rather than debugging,\nothers are released from their normal work, one channel carries the facts and a timestamped\nrecord starts. Without that, the same people do the same work in parallel, and the first clear\nstatement about the problem is the one a customer writes.\n\nFor some companies there is also a legal clock. The EU's NIS2 directive, Directive (EU) 2022/2555,\nrequires the organisations it covers to report a significant incident to the national authority:\nan early warning within 24 hours of becoming aware of it, an incident notification within 72\nhours, and a final report within one month of that notification. Its definition reaches beyond\nattacks to any event compromising the availability, authenticity, integrity or confidentiality of\ndata or of the services a network and information system offers, so a severe outage can count. A\nteam that starts asking “is this an incident?” on the second day has already spent the first\nwindow.\n\nThe running cost is people's time, and it is larger than it used to be. The 2025 revision of\nNIST's guide notes that incidents now happen frequently and that recovering from them often takes\nweeks or months, where its 2012 predecessor assumed they were rare and over within a day or two.\nBudgeting for incidents means budgeting for the follow-up work as well as for the night itself.\n\n## What it looks like in practice\n\nOne incident on this site concerns a macOS desktop application. Its\n[diagnostic reports recorded 111 GB of compressed pages on a 36 GB machine](/portfolio/an-app-that-ate-41-mb-a-second-stopped/)\non 2026-09-01; a separate footprint sample recorded growth of about 41 MB per second during\na connection retry loop emitting core events. The response found log events retained in an\nunbounded main stream and copied to subscribers that did not use them. Buffers were bounded,\nsubscriptions filtered by event type and logging given a rate budget. A full test-run\ncomparison recorded 610,996 core log lines falling to 1,411. Those measurements concern the\ntest run, not production traffic or post-fix memory. Architecture checks and application\nbudget tests defend the bounded design. Its overflow policy may drop repetitive diagnostics\nand refresh affected state; separate shutdown timeouts needed their own correction.\n\nOther incidents are the repeating kind. Eight WireGuard and IPsec tunnels between Google Cloud and\nLinux systems were fragile and slow to diagnose, and a dead tunnel could cut communication between\nenvironments. The work standardised their configuration, monitored their health and went after\nthe recurring routing and key-exchange problems at the root rather than restarting the tunnels,\nand\n[the recurring connectivity incidents stopped](/portfolio/managed-8-wireguard-and-ipsec-vpn-connections/).\nWhen the same thing happens for the third time, the cause is the incident.\n\nAnd some incidents begin with somebody trying to help. On a platform with no admin tool,\napproving a claim, hiding a review or clearing a cache falls to an engineer with a database\nconsole, which the project page calls slow, unlogged and one typo away from an incident. The\noperator back-office built there,\n[around 40 admin routes made from 128 components](/portfolio/built-the-operator-back-office-for-the-platform/),\nputs that work behind the same authorization and the same audit trail as the rest of the product,\nat the ongoing cost of a large internal surface to keep tested and accessible.\n\n## Where it goes wrong\n\n**Five people work on it and nobody leads it.** The chat channel fills with theories, two\nengineers roll back different things, and customers hear nothing for an hour, because nobody\ndeclared and so nobody was given the job of coordinating. Let anyone on the team declare, and make\nthe first act of a declaration naming one lead who keeps the picture and the record.\n\n**The same incident returns every few weeks.** The response ends when the service is back, and a\nrestart counts as the fix. Treat restoring service and removing the cause as two tasks: close the\nfirst on the night, and give the second an owner and a date before anyone goes back to bed.\n\n**The tooling makes the incident bigger.** Logging that writes a line per event produces a log as\nlarge as the fault, and alert rules send hundreds of notifications for one cause. Give logs and\nalerts a budget in advance, collapse repeats, and test the reporting path under the load of the\nfailure it reports.\n\n**The repair causes the next incident.** A hurried change typed by hand against production fixes\none symptom, starts another problem, and leaves a hole in the timeline the review needs later.\nKeep a short list of pre-approved recovery actions that leave a trail, and record every change\nmade during an incident.\n\n**The lessons arrive months late, or arrive as blame.** A review held after weeks of recovery\nremembers little, and one that asks who made the mistake teaches people to hide the next one.\nNIST's revised guide says lessons learned should often be shared as soon as they are identified,\nand its recovery guidance ends with an after-action report on the incident, the actions taken and\nwhat was learned. Write down what made the mistake easy, and change that.\n\n## Further reading\n\n- [Incident](https://csrc.nist.gov/glossary/term/incident) — the definitions NIST publications use,\n  each shown with the law or standard it comes from.\n- [SP 800-61 Rev. 3](https://csrc.nist.gov/pubs/sp/800/61/r3/final) — NIST's incident response\n  recommendations of 2025, built around the CSF 2.0 Functions instead of a single response cycle.\n- [Cybersecurity Framework](https://www.nist.gov/cyberframework) — the framework whose Govern,\n  Identify, Protect, Detect, Respond and Recover Functions the incident guide maps onto.\n- [RFC 2350: Expectations for Computer Security Incident Response](https://www.rfc-editor.org/rfc/rfc2350)\n  — the IETF best current practice for what an incident response team should publish about itself.\n\n\u003chttps://engineer.company/glossary/incident/\u003e\n","date_modified":"2026-11-08T02:22:23Z","description":"Where an incident starts, who should declare one, the reporting clock it can start under NIS2, and the habits that turn one bad night into a pattern.","formats":{"gemtext":"https://engineer.company/glossary/incident/index.gmi","gopher":"https://engineer.company/glossary/incident/gophermap","html":"https://engineer.company/glossary/incident/","json":"https://engineer.company/glossary/incident/index.json","markdown":"https://engineer.company/glossary/incident/index.md","pdf":"https://engineer.company/glossary/incident/index.pdf"},"kind":"page","language":"en","title":"What is an incident?","url":"https://engineer.company/glossary/incident/","version":1}