{"canonical_url":"https://engineer.company/glossary/postmortem/","content_markdown":"# Postmortem\n\n## What it is\n\nA postmortem is the written account a team produces after an incident: what\nhappened, what it cost, what was done to stop it, why it happened, and what will\nchange so that it does not happen the same way again. The word is also written\npost-mortem, with a hyphen; the meaning is the same. Google's Site Reliability\nEngineering book defines it in those terms, as an account that connects an\nincident and its consequences to the response, the causes and the work needed to\nprevent a repeat.\n\nThe book gives the exercise three aims: to document the incident, to\nunderstand every contributing root cause rather than the first one found, and\nto put preventive actions in place that actually work. The plural matters.\nIncidents in systems of any size rarely have a single cause, and a postmortem\nthat stops at the first plausible explanation tends to fix the trigger and\nleave the conditions that let the trigger do damage.\n\nWhat separates the practice from an ordinary report is that it is blameless. A\nblameless postmortem looks for contributing causes without accusing any person\nor team of bad behaviour, and it assumes that the people involved meant well and\nmade decisions using the information available to them then. The SRE book traces\nthe idea to healthcare and aviation, two fields where a mistake can be fatal and\nwhere every error is treated as a chance to make the system stronger. Its\nreasoning is practical rather than kind: a person cannot be patched, but a\nsystem or a process can.\n\nA postmortem is not written for every hiccup, since it costs time. The book\nlists the usual triggers, and teams adjust the thresholds to suit them:\n\n- user-visible downtime or degradation beyond an agreed limit;\n- data loss of any kind;\n- an on-call engineer having to intervene, by rolling back a release or\n  rerouting traffic, for instance;\n- a resolution time above an agreed limit;\n- a failure of monitoring, which usually means a human found the problem\n  first.\n\nThe criteria are agreed before anything breaks, so that nobody has to argue\nduring an outage about whether this one counts, and any stakeholder may ask\nfor a postmortem when an event falls outside them.\n\n## Why it matters\n\nAn incident that is fixed but not written up is likely to happen again. The\nSRE Workbook works through a postmortem that was published four months after\nthe incident it described; in the meantime the same incident recurred, and\nthe team had to reconstruct what it had learned the first time. A record\nwritten promptly, and kept where people can find it, turns one team's bad\nafternoon into knowledge the whole organisation can use.\n\nThe record also forces the follow-up to exist. Ben Treynor Sloss, quoted in the\nWorkbook, puts it bluntly: users gain nothing from a postmortem unless the team\nacts on its findings. Google therefore requires every postmortem that follows a\nuser-affecting outage to have at least one tracking ticket at one of the two\nhighest priorities. The postmortem is where the argument for that work is made,\nwith evidence, and where its progress can be checked later.\n\nBlamelessness is what makes the evidence trustworthy. Where people are shamed\nfor mistakes, the SRE book observes, they stop bringing problems to light, and\nthe Workbook adds that they may hide exactly the facts needed to prevent a\nrecurrence. A blameless culture gets the full timeline, including the command\nsomeone ran by mistake, because nobody is punished for telling it. The book\ngoes further: a team or a person who writes many postmortems should not be\ntreated as a problem, since the alternative is incidents nobody writes about.\n\nTaken together, postmortems become a data set. Once each one carries the same\nmetadata, an organisation can follow trends such as time to detect, time to\nresolve and how far an incident spread, and see whether reliability work is\npaying off. The same records serve as training: Google reenacts old postmortems as\nrole-play exercises for new engineers, and runs reading clubs and a newsletter\nfeaturing a postmortem of the month.\n\n## What it looks like in practice\n\nThe SRE book publishes a complete example, Appendix D, written for a fictional\nsearch service that went down for 66 minutes under a sudden surge of traffic.\nIts sections are a good default: status, summary, impact, root causes,\ntrigger, resolution, detection, action items, lessons learned (what went well,\nwhat went wrong, where the team got lucky) and a timeline in UTC. The timeline\nmarks four moments separately: when the outage began, when the incident was\ndeclared, when the outage ended, and when the incident was formally closed.\nThey are different moments, and the gaps between them are where the lessons\nusually sit.\n\nHere is a postmortem in that shape for an invented incident, cut down to what\nfits on a screen:\n\n```text\nPostmortem: checkout failures after a timeout change\nStatus:      final, reviewed\nOwner:       on-call engineer, with the incident commander\nSummary:     checkout failed for about a third of requests for 31 min\nImpact:      about 4,100 failed checkouts, no data lost; the 28-day\n             error budget for checkout spent in one evening\nRoot causes: payment timeout cut from 10 s to 1 s in a change that\n             was reviewed as \"config only\"; nothing exercised it\n             under peak load before release\nTrigger:     the evening peak slowed the payment provider past 1 s\nDetection:   error-rate alert paged on-call 6 min after the first error\nResolution:  change rolled back; failures stopped 4 min later\n\nAction items\n  Type      Priority  Owner          Done when\n  prevent   P1        platform lead  config changes run the load test\n  detect    P1        on-call lead   alert fires on payment timeouts\n  mitigate  P2        checkout lead  rollback of config takes one step\n  process   P2        incident lead  config changes appear in deploy log\n\nLessons\n  Went well:   rollback worked first time\n  Went wrong:  a timeout change was not treated as a code change\n  Got lucky:   the peak was short; a sale day would have tripled it\n\nTimeline (UTC)\n  18:02  change merged and deployed\n  18:40  traffic peak; first checkout failures        outage begins\n  18:46  error-rate alert pages on-call\n  18:52  incident declared, commander named           incident begins\n  19:07  timeout change identified, rollback started\n  19:11  failures stop                                outage ends\n  19:41  thirty minutes at normal error rate          incident ends\n```\n\nNotice what is absent: no names in the causes, and no action item that asks\nanyone to be more careful. Each action has one owner, a priority and an end\nstate that someone else can verify. Most of the raw material does not have to\nbe written from memory either. The SRE book's chapter on managing incidents\nhas the incident commander keep a living document during the incident and\nasks for it to be kept for the postmortem, and the Workbook describes tooling\nthat fills in the roles, the timeline and the affected services automatically.\n\nOn this site, the\n[macOS event-buffer incident record](/portfolio/an-app-that-ate-41-mb-a-second-stopped/),\nfrom September 2026, has much of a postmortem's anatomy. It distinguishes diagnostic reports\nof 111 GB of compressed pages on a 36 GB machine from a separate footprint sample showing\ngrowth of about 41 MB per second. It identifies an unbounded main event stream and log events\ncopied to subscribers that did not use them, then bounds buffers, filters subscriptions by\nevent type and gives logging a rate budget. A full test-run comparison records 610,996 core\nlog lines falling to 1,411. Architecture checks and budget tests defend the design, while\nthe account states what the evidence does not establish: a measured post-fix memory figure\nor a production benchmark. Separate shutdown timeouts needed their own fix, and the budget\ndeliberately allows repetitive diagnostic events to be dropped. The account of\n[three locks on the door, none of them locked](/portfolio/three-locks-on-the-door-none-of-them-locked/)\ndescribes a review rather than an outage, but it ends where a good postmortem\nshould: the three fixes were small, and what came out of the work was two\nrules that now govern the repository, so that the whole class of failure is\ncaught rather than the three instances.\n\n## Where it goes wrong\n\n**The document names a culprit.** A postmortem that says one person was solely\nresponsible, or that describes their \"careless ignorance\", teaches everyone\nelse to keep quiet next time. Write about what happened and what the person\nknew at the time, not about who they are, and keep the language factual;\nthe Workbook flags dramatic phrasing and exclamation marks as a distraction\nfrom the facts.\n\n**The action items ask humans to try harder.** \"Be more careful\" and \"improve\nthe runbook\" cannot be closed, and changing behaviour is less reliable than\nchanging the system. Give every item a type (prevent, mitigate, detect), one\nowner, a priority and a verifiable end state, such as an alert that fires\nabove a stated threshold, and open a ticket for each one in the tracker the\nteam already watches.\n\n**The root cause stops at the trigger.** \"A config change caused the outage\"\nnames the trigger. The root causes are why the change could do that much\nharm: why the review missed it, why no test caught it, why the alert came\nlate. Keep the two headings separate, as the SRE book's example does.\n\n**The impact is a shrug.** If the only number in the document is the duration,\nnobody can tell whether the fix worked. Estimate the users affected, the\nrequests lost and the error budget spent; the Workbook's advice is that a\nwell-informed estimate beats having no data at all.\n\n**Four people own it, so nobody does.** The postmortem needs one owner who is\nthe single point of contact for writing it and seeing the follow-up through,\nwith others helping. The same goes for every action item.\n\n**It is written late, or read by nobody.** Memories fade within days, and a\ndocument circulated months later arrives after the next occurrence. The\nWorkbook's good example went out within a week of closing the incident, to a\nwide audience. Review it, too: \"An unreviewed postmortem might as well never\nhave existed,\" as the SRE book has it. Hold regular review sessions, then keep\nthe result in a searchable archive.\n\n**The actions never close.** When the same incident keeps coming back, check\nwhether the action items are taking too long and whether feature work keeps\nwinning over the reliability fixes. Track closure, not authorship: Google\nholds FixIt weeks twice a year and rewards the engineers who close the most\npostmortem action items.\n\n## Further reading\n\n- [Postmortem Culture: Learning from Failure](https://sre.google/sre-book/postmortem-culture/)\n  — chapter 15 of Google's Site Reliability Engineering book: triggers,\n  blamelessness, review and the culture around it.\n- [Postmortem Culture: Learning from Failure](https://sre.google/workbook/postmortem-culture/)\n  — the SRE Workbook's chapter, with a bad and a good postmortem of the same\n  incident and the reasons each is what it is.\n- [Example Postmortem](https://sre.google/sre-book/example-postmortem/) — a\n  complete postmortem for a fictional outage, with action items, lessons and a\n  timeline.\n- [Managing Incidents](https://sre.google/sre-book/managing-incidents/) —\n  chapter 14 of the SRE book: the incident commander, the live incident\n  document and the handoffs that the postmortem later draws on.\n\n\u003chttps://engineer.company/glossary/postmortem/\u003e\n","date_modified":"2026-11-08T02:22:23Z","description":"What a postmortem records after an incident, why it is blameless, what a useful one contains, and the habits that turn it into paperwork.","formats":{"gemtext":"https://engineer.company/glossary/postmortem/index.gmi","gopher":"https://engineer.company/glossary/postmortem/gophermap","html":"https://engineer.company/glossary/postmortem/","json":"https://engineer.company/glossary/postmortem/index.json","markdown":"https://engineer.company/glossary/postmortem/index.md","pdf":"https://engineer.company/glossary/postmortem/index.pdf"},"kind":"page","language":"en","title":"What is a postmortem?","url":"https://engineer.company/glossary/postmortem/","version":1}