{"canonical_url":"https://engineer.company/glossary/regression-testing/","content_markdown":"# Regression testing\n\n## What it is\n\nRegression testing checks whether software still performs previously correct behavior after a\nchange. A new export setting can work exactly as specified and still alter the rows downloaded by\nexisting users. Testing only the new setting answers whether the requested capability works. A\nregression test exercises the old path and asks whether a behavior people rely on still holds.\n\nThe NASA Software Engineering Handbook discusses regression testing as a way to confirm that\nchanges have not introduced new defects. Its guidance includes re-executing existing tests that\ntrace to modified code or requirements and choosing a scope that covers interacting modules. The\nunderlying idea applies whether the change affects source code, a dependency, configuration, data\nor the environment. The same code can produce a different result when an input or browser\nchanges.\n\nVerification of a defect fix and regression testing work together. First reproduce the problem\nand confirm the new result after the fix. Then test nearby behavior that must remain correct. The\ncase for the original fault can remain in the suite so it warns if the defect returns. The\nexpected result should describe the behavior the team promises. When requirements deliberately\nchange, update those expectations and document which existing callers have a different path.\n\n## Why it matters\n\nSoftware changes rarely stay inside the visible edit. Turning an internal list into a set can\nremove duplicates, which may solve one problem and reorder a report. A date parser change can\naccept the new format and drop support for empty values handled elsewhere. An automated\nregression check catches particular effects while the team can still connect them to a specific\nchange. This matters especially for behavior used by another integration that the author rarely\nruns locally.\n\nFor a small change, start with the nearest shared rule and one or two realistic different\ncallers. Ask how the change could disturb default behavior, boundaries, error handling or already\nstored data. For higher-impact changes, consider effects across the product, such as preserving\nkey performance and accessibility behavior as well as values. Select checks whose evidence is\ngood enough for the consequence of a miss. A stable unit test can cover a conversion precisely. A\nbrowser check might be needed to see the actual visual or keyboard behavior using the same path\nas a reader.\n\nRecorded examples help maintainers understand why a rule exists. A test called “empty date stays\nempty in an export of older cases” says more than “fixture three passes”. Show the important\ninputs and the relevant result. The next person can decide whether a new requirement\nintentionally changes the judgment or whether a failure signals something previously used has\nbroken.\n\n## What it looks like in practice\n\nImagine a CSV export with a new “include archived accounts” option. The default export must keep\nexcluding archived accounts. Active accounts remain sorted by account reference, and a missing\nlast-contact date stays empty. The new option must add archived accounts without duplicating\nactive rows. Give the fixture two active accounts in reverse order and one archived account.\nLeave the date blank for one active row. Exercise the old and new export paths through the actual\nexporter.\n\nOn the default path, compare the account references and their order, and confirm the empty date\ncell remains empty. With the option on, confirm the same active accounts plus the archived\nrecord, each exactly once. Include a name containing a comma, because an exporter that builds\nrows by concatenation can still return the correct row count while producing the wrong columns.\nParse the generated CSV with the kind of reader a caller uses. The fixture should be small enough\nto read and representative enough to expose the effect.\n\nNow check whether the tests reject a real regression. Temporarily make the default include\narchived accounts, or remove quoting for the comma. The relevant old-path checks should fail for\ndistinct, readable reasons. Restore the implementation and run again. That exercise checks that a\nmisleading assertion such as “the output file exists” did not quietly replace comparison of the\nrecords. Testing another known case with no archives helps preserve the valid empty selection\npath.\n\nOur [note on Safari underlines and font subsets](/notes/safari-underline-font-subset/) offers a\ndifferent example of evidence boundaries. It explains that a CSS change appeared to repair\nChromium's rendering while Safari still drew ink across letter descenders. A regression suite for\nthe affected font behavior would need observations in the engines and cases it claims to protect.\nThe note documents the diagnosis and browser evidence. The proposed suite is an example of how to\ncarry the lesson forward.\n\n## Where it goes wrong\n\nOne trap is accepting test count as coverage. Hundreds of simple assertions can miss the only\ncaller that reads historical data. Choose cases from the behavior inventory. Start with what real\nusers do, which inputs exist and which interactions the change disturbs. When an issue escaped\nbefore, protect the workflow that suffered and check that the assertion fails when the faulty\nbehavior returns.\n\nAnother trap is comparing too much. Full snapshots can change when file order, random identifiers\nor time vary. If the export promises the order of records, assert that order. If it promises only\na set of records, compare the set while separately detecting duplicates. Test the quality the\nuser depends on. Review any intentionally updated expectation as carefully as a code change and\nretain a reason for it.\n\nFinally, a flaky check makes repeated failures hard to interpret. Replace timing guesses with\ncontrolled waits for observable conditions, use known test data and ensure one example does not\nleave state for the next. A test still needs a time bound and a readable failure. Keep broader\nchecks for the points where genuine interactions matter, and keep quick focused checks close to\ndevelopment. The combination gives the team evidence about different parts of the same change.\n\n## Further reading\n\n- [NASA Software Engineering Handbook on software regression\n  testing](https://swehb.nasa.gov/spaces/SWEHBVC/pages/50889419/SWE-191+-+Software+Regression+Testing)\n- [pytest documentation on parametrizing\n  examples](https://docs.pytest.org/en/stable/how-to/parametrize.html)\n- [Software Engineering at Google on effective unit\n  tests](https://abseil.io/resources/swe-book/html/ch12.html)\n\n\u003chttps://engineer.company/glossary/regression-testing/\u003e\n","date_modified":"2026-11-19T19:18:49Z","description":"How regression testing checks that an improvement preserves existing behavior, with a worked export example and practical test selection.","formats":{"gemtext":"https://engineer.company/glossary/regression-testing/index.gmi","gopher":"https://engineer.company/glossary/regression-testing/gophermap","html":"https://engineer.company/glossary/regression-testing/","json":"https://engineer.company/glossary/regression-testing/index.json","markdown":"https://engineer.company/glossary/regression-testing/index.md","pdf":"https://engineer.company/glossary/regression-testing/index.pdf"},"kind":"page","language":"en","title":"What is regression testing?","url":"https://engineer.company/glossary/regression-testing/","version":1}