{"canonical_url":"https://engineer.company/glossary/incident-response/","content_markdown":"# Incident response\n\n## What it is\n\nIncident response is the coordinated work of handling a disruption or a serious threat to a system.\nAn incident is the event a team decides needs attention. The response is what the team does about\nit, from recognizing and declaring the incident to coordinating action, restoring a usable service\nand handing the remaining work to named people. It may involve engineering, operations, customer\nsupport and people with authority over legal, security or business decisions.\n\nNIST's incident response recommendations for cybersecurity describe a cycle of detection, response\nand recovery supported by preparation and ongoing improvement. The publication is about cybersecurity.\nAn engineering team can use related coordination practices for a broken release or a failed\ndependency, with the right people and safeguards for its own situation. The response to a suspected\ndisclosure may require preserving evidence and protecting affected people even when the application\nis still serving traffic. In an ordinary outage the urgent choice may be how to restore that traffic\nwithout adding new failures.\n\nGoogle's account of incident management describes explicit coordination roles. An incident commander\nkeeps a view of the whole response, an operations lead coordinates changes to the service, and a\ncommunications lead keeps stakeholders informed. One person can cover several roles early on. As\nthe work grows, naming who owns each function helps people avoid duplicating effort or giving\ncompeting instructions. The roles describe responsibilities a team can allocate to suit its size.\n\n## Why it matters\n\nThe early minutes of a problem bring information and decisions at different speeds. A check can\nsay that payment failures rose within a few seconds, while the team may need several minutes to\nlearn which customers or payment methods are affected. Support may see a pattern before an engineer\ndoes. A shared timeline can record both observations with their sources and the confidence the team\nhas in them. Other responders can then investigate without treating the first explanation as fact.\n\nA response also gives the team control of changes made under pressure. A rollback and an emergency\nconfiguration change can each look sensible on their own and be harmful if they happen together.\nThe person coordinating operations needs to know what others are about to change, when the result can\nbe observed and what the reversal would be. Clear authority at that moment leaves room for a\nspecialist to propose a solution without leaving the outcome to whoever reacts first.\n\nRecovery and diagnosis run on different clocks. Redirecting traffic away from a failing instance\nmay restore the service while the underlying fault remains unknown. Marking recovery ends the period\nof interruption using an agreed observation, such as successful orders reaching the expected volume\nand stalled orders being reconciled. The later postmortem can examine why the failure was possible,\nhow well the response worked and what to change. Its conclusions are stronger when decision history\nand test evidence have been preserved during the response.\n\n## What it looks like in practice\n\nImagine that a booking service starts taking payment while sometimes failing to issue a ticket.\nSupport sees reports first and engineering confirms that requests have started failing. The first\nresponder declares an incident and records when the symptoms were first seen. The team agrees on\na short description of the known effect and appoints someone to coordinate the service changes.\nSupport has a point of contact and a time for the next update, even if the cause remains unknown.\n\nOperations inspects a small sample of affected bookings and finds a recent change that might explain\nthe timing. The coordinator asks the team to confirm that returning to the previous version leaves\nalready captured payments visible. One person carries out that rollback and a second watches the\nrelevant booking and payment indicators. Support preserves the identifiers for users who need follow-up.\nCustomer communication reports the effect and advises what to do with an unconfirmed booking. The\nteam can revise the message as it learns more.\n\nThe rollback is judged from complete transactions, including whether a payment and a ticket match,\nand from the progress of any backlog. The team records when the route became usable again and who\nwill reconcile unsettled bookings. It writes down the decisions and observations needed for follow-up.\nThe response closes when the current risk can be handed back to ordinary operations. The investigation\ninto the cause can continue with an owner and a date.\n\nEngineer's [case on a macOS application's memory growth](/portfolio/an-app-that-ate-41-mb-a-second-stopped/)\nrecords separate observations of memory growth and the subsequent work to bound retained events. It\nillustrates why it is useful to preserve what each measurement actually covers. The booking service\nabove is a hypothetical example of coordination and customer follow-up.\n\n## Where it goes wrong\n\n**The team waits for certainty before declaring.** A probable impact can be investigated with explicit\nownership and a scheduled update. A clear threshold helps responders recognize when coordination is\nworth starting. The team can stand down if later evidence shows the situation is less serious.\n\n**Everyone changes the system at once.** Keep a short shared record of actions, including proposed\nchanges and who will execute them. Observe the result of a change before introducing another where\npossible. When independent actions can really proceed together, make their boundaries clear.\n\n**Customers hear guesses.** Report what users are likely to experience, any step they can safely take\nand when to expect another update. Explain uncertainty directly. An explanation of the cause can wait\nuntil the team has evidence that supports it.\n\n**The symptoms end and unfinished work disappears.** Make the handoff explicit. A recovered service\ncan still have a backlog, users who have not been contacted or a provisional safeguard that needs an\nowner. Put those tasks where the team will track them after the incident channel falls quiet.\n\n## Further reading\n\n- [NIST SP 800-61 Rev. 3](https://csrc.nist.gov/pubs/sp/800/61/r3/final) — incident response and cybersecurity risk management, including preparation and improvement.\n- [Managing incidents](https://sre.google/sre-book/managing-incidents/) — Google's description of coordination roles, communications and an incident record.\n- [Incident response](https://sre.google/workbook/incident-response/) — incident-management experience and practical examples from Google's SRE workbook.\n\n\u003chttps://engineer.company/glossary/incident-response/\u003e\n","date_modified":"2026-11-19T19:18:49Z","description":"How teams coordinate the first minutes of an incident, restore the service and hand reliable evidence to the people who follow up.","formats":{"gemtext":"https://engineer.company/glossary/incident-response/index.gmi","gopher":"https://engineer.company/glossary/incident-response/gophermap","html":"https://engineer.company/glossary/incident-response/","json":"https://engineer.company/glossary/incident-response/index.json","markdown":"https://engineer.company/glossary/incident-response/index.md","pdf":"https://engineer.company/glossary/incident-response/index.pdf"},"kind":"page","language":"en","title":"What is incident response?","url":"https://engineer.company/glossary/incident-response/","version":1}