{"canonical_url":"https://engineer.company/glossary/regular-expression/","content_markdown":"# Regular expression\n\n## What it is\n\nA regular expression is a short pattern that describes a set of strings, so that\na program can pick out the ones that belong to it. The POSIX standard, published\nby The Open Group and the IEEE, opens its chapter on the subject by describing\nregular expressions as the tool for picking particular strings out of a set of\ncharacter strings. In everyday speech, and in many function and library names,\nthe term shrinks to regex, or to regexp, the spelling of JavaScript's RegExp\nobject and of Go's regexp package.\n\nMost characters in a pattern stand for themselves: an `a` in the pattern matches\nan `a` in the text. A few are special. A period matches any character, `*`\nrepeats what comes before it zero or more times, a bracket expression such as\n`[0-9]` matches one character from a set, `^` and `$` tie the pattern to the\nstart and the end, and an interval such as `{4}` asks for an exact number of\nrepetitions. So `^[0-9]{4}$` describes every string made of exactly four digits,\nwith no other characters before or after. The standard's own example of an\ninterval is `c{3}` applied to `abababccccccd`: it matches the seventh to ninth\ncharacters, the first three of the six c's.\n\nThere is more than one dialect. POSIX itself defines two: basic regular\nexpressions (BRE), which most of its utilities use, and extended regular\nexpressions (ERE), which some use instead. The C functions `regcomp()` and\n`regexec()` offer both. The difference is not cosmetic. In an ERE, `a+` means\none or more `a`. In a BRE, `+` is an ordinary character, and the standard leaves\nit to each implementation whether `\\+` means one or more or a literal plus sign.\n\nThe other large family descends from Perl. The ECMAScript specification states\nthat the form and functionality of JavaScript's regular expressions are modelled\non the regular expression facility of Perl 5. Go's regexp package says it\naccepts broadly the same syntax as Perl, Python and other languages, and\nPython's `re` module is a third member. These engines share most of their\nnotation, but not every detail. The shorthand classes are a case in point: `\\d`\n(a digit) and `\\w` (a word character) look the same everywhere, but what they\ncover depends on the engine and its flags, as the practice section below shows.\n\n## Why it matters\n\nRegular expressions sit inside a great deal of software that does not look like\ntext processing: validation of form fields, filters that pull values out of log\nlines, routing rules in web servers, search and replace in text editors. Each\npattern is short, which makes it easy to read past what it actually accepts. Two\nquestions decide the result, and the answer to both depends on the engine: which\nmatch is chosen when several are possible, and how long it may take to find one.\n\nPOSIX chooses the leftmost-longest match: of all the matches that start earliest\nin the text, the longest wins. The standard's examples are the BRE `bb*`, which\nmatches the second to fourth characters of `abbbc`, and the ERE\n`(wee|week)(knights|night)`, which matches all ten characters of `weeknights`.\nThe Perl family works differently. It tries the alternatives in the order they\nare written and takes the first one that leads to a match, which Go's\ndocumentation calls leftmost-first. The same pattern on the same text can\ntherefore give two answers:\n\n```text\n$ echo ab | grep -oE 'a|ab'\nab\n\u003e\u003e\u003e re.search(\"a|ab\", \"ab\").group()\n'a'\n\u003e \"ab\".match(/a|ab/)[0]\n'a'\n```\n\nGo offers both behaviours. Its `Compile` function uses leftmost-first semantics,\nand `CompilePOSIX` restricts the syntax to POSIX ERE and switches to\nleftmost-longest. Even then the package keeps its own rule for choosing\nsubmatches, and its documentation explains why: the POSIX rule, it says, would\ncost too much to compute and is not even clearly defined.\n\nThe second question is time. Python's `re`, like most engines in the Perl\nfamily, backtracks. When part of a pattern cannot go on, the engine steps back\nand tries another way of matching what came before. Python's documentation shows\nthe idea with `a*a` against `aaaa`: `a*` first takes all four letters, then\ngives one back so that the final `a` can match. For most patterns the cost is\ninvisible. For some, the number of ways to try grows exponentially with the\nlength of the input, and a string of a few dozen characters can keep a processor\nbusy for minutes. MITRE catalogues this as CWE-1333, Inefficient Regular\nExpression Complexity, and lists ReDoS (regular expression denial of service)\nand catastrophic backtracking as other names for it. Go's regexp package was\ndesigned around the opposite promise. Its documentation states that matching is\n\"guaranteed to run in time linear in the size of the input\", and adds that most\nopen-source implementations make no such guarantee.\n\n## What it looks like in practice\n\nTake a sign-up form for customers in Denmark and Ukraine, and a first-name field\nthat should accept letters only. The obvious pattern is `[A-Za-z]+`, and in\nPython 3.15 it behaves like this:\n\n```text\n\u003e\u003e\u003e import re\n\u003e\u003e\u003e re.fullmatch(r\"[A-Za-z]+\", \"Mia\")\n\u003cre.Match object; span=(0, 3), match='Mia'\u003e\n\u003e\u003e\u003e re.fullmatch(r\"[A-Za-z]+\", \"Søren\")\n\u003e\u003e\u003e re.fullmatch(r\"[A-Za-z]+\", \"Олена\")\n\u003e\u003e\u003e re.fullmatch(r\"\\w+\", \"Søren_42\")\n\u003cre.Match object; span=(0, 8), match='Søren_42'\u003e\n\u003e\u003e\u003e re.fullmatch(r\"[^\\W\\d_]+\", \"Олена\")\n\u003cre.Match object; span=(0, 5), match='Олена'\u003e\n\u003e\u003e\u003e re.fullmatch(r\"[^\\W\\d_]+\", \"Søren_42\")\n\u003e\u003e\u003e\n```\n\nA call that returns `None` prints no line at all in the interactive shell, so\nthe ASCII class turns away Søren and Олена (Olena), two entirely ordinary names\nin Copenhagen and Kyiv. Switching to `\\w` lets them in, but in Python `\\w`\ncovers every character for which `str.isalnum()` is true, plus the underscore,\nso it also lets in `Søren_42`. The class `[^\\W\\d_]` excludes non-word\ncharacters, decimal digits and the underscore. It also permits some non-decimal\nnumeric characters, so it is not a strict letters-only test. For names of more\nthan one word `[^\\W\\d_]+( [^\\W\\d_]+)*` accepts `Anne Sofie Østergaard` and\n`Олена Коваль` alike. It still turns away a hyphenated `Anne-Marie`, which is a\nreason to keep a name field as permissive as the business allows. POSIX tools\nhave the same problem in another form: the standard leaves the meaning of a\nrange such as `A-Z` unspecified outside the POSIX locale, and offers the class\n`[[:alpha:]]`, whose members are defined by the current locale.\n\nGetting this right matters well beyond one form. One of this site's services,\n[internationalization and localization](/services/i18n-localization/), describes\nits aim as software that reads naturally in every market a client sells in, down\nto the way names sort. A name field that refuses Søren falls short of that aim\nlong before any sorting happens.\n\nDigits raise the opposite question. A Danish postcode is four digits, so `\\d{4}`\nlooks like a safe check for `2200`. In Python, though, `\\d` in a text pattern\nmatches any Unicode decimal digit, not just 0 to 9, and accepts the same\npostcode written in Arabic-Indic digits:\n\n```text\n\u003e\u003e\u003e s = \"\\u0662\\u0662\\u0660\\u0660\"\n\u003e\u003e\u003e print(s, int(s))\n٢٢٠٠ 2200\n\u003e\u003e\u003e re.fullmatch(r\"\\d{4}\", s)\n\u003cre.Match object; span=(0, 4), match='٢٢٠٠'\u003e\n\u003e\u003e\u003e re.fullmatch(r\"\\d{4}\", s, re.ASCII)\n\u003e\u003e\u003e re.fullmatch(r\"[0-9]{4}\", s)\n\u003e\u003e\u003e\n```\n\nJavaScript takes the other side. The ECMAScript specification defines `\\d` as\nexactly the ten characters 0 to 9, so `/^\\d{4}$/.test(s)` returns `false` in\nNode.js for the same string, while `/^\\p{Nd}{4}$/u` returns `true`. A form\nchecked in the browser with one rule and on the server with the other will\ndisagree about the same input. Python's `int()` reads the string as 2200, but a\ndatabase column, a lookup table or another service may not. Decide which digits\nare meant: `[0-9]`, or the `re.ASCII` flag, when only ASCII digits will do, and\n`\\p{Nd}` with the `u` flag in JavaScript when any decimal digit is welcome.\n\n## Where it goes wrong\n\n**A pattern that doubles its running time with every letter.** A name pattern\nmeant to allow words separated by single spaces is easily written as\n`([^\\W\\d_]+ ?)+`. On a valid name it is fast. On a run of letters followed by a\ncharacter it cannot accept, the engine tries every way of dividing the letters\ninto words before it gives up, and the number of ways doubles with each letter.\nOn one machine with Python 3.15, the fastest of five runs at each length gave\nthese times for `fullmatch` on a string of letters followed by `!`:\n\n```text\nletters   seconds\n     18     0.017\n     20     0.067\n     22     0.27\n     24     1.0\n     26     4.1\n```\n\nEach letter doubles the time, so two more letters multiply it by four.\nExtrapolating from that factor, thirty letters would take about a minute and\nthirty-two more than four minutes. This is the case CWE-1333 describes: the\nbacktracking attempts grow exponentially with the input, the input does not have\nto match, and the input can be long enough. The rewrite `[^\\W\\d_]+( [^\\W\\d_]+)*`\ndescribes the same names without a nested quantifier, rejects the 24-letter\nstring in about a microsecond and a million letters followed by `!` in about\nthirty milliseconds. Remove nested quantifiers, cap the length of the input\nbefore matching it, and run patterns that see untrusted input on an engine with\na linear-time guarantee, such as Go's.\n\n**Searching where the whole value must match.** In Python, `re.search` accepts a\nmatch anywhere in the string, and `re.match` anchors only at the start; Python\n3.15 gives it the clearer name `prefixmatch` and soft-deprecates the old one,\nnoting that other languages use match for what Python has always called search.\nSo `re.prefixmatch(r\"\\d{4}\", \"22000\")` succeeds on the first four digits, and\n`re.search(r\"\\d{4}\", \"DK-2200\")` finds the postcode inside the string. Even\n`re.search(r\"^\\d{4}$\", \"2200\\n\")` succeeds, because `$` also matches just before\na newline at the end. `re.fullmatch`, available since Python 3.4, refuses all\nthree. Use `fullmatch` whenever a pattern describes the whole value, or anchor\nwith `\\A` and `\\Z`, which Python 3.14 also spells `\\z`.\n\n**An allowlist with a loose dot.** A check that only lets redirects go to the\ncompany's own site might be written as `^https://www.example.com`. It accepts\n`https://www.example.com.evil.example/`, because the pattern stops before the\nhost name does, and each unescaped dot matches any character, so a host that\ndiffers from the real one by a single character in place of a dot passes too.\nEscaping the dots, as `re.escape(\"www.example.com\")` does, and ending the\npattern with `/` closes both holes. Better still, parse the URL and compare its\nhost name with the expected one as a whole string.\n\n**A pattern copied between engines.** A pattern tested with `grep -E` and pasted\ninto Python can pick a different match, as the `a|ab` example shows, and one\nthat works in one BRE tool with `\\+` can mean a literal plus sign in another.\nThe `\\d` that accepts Arabic-Indic digits in Python rejects them in JavaScript.\nTest every pattern in the engine that will run it, with the flags it will run\nwith.\n\n**A global flag that remembers.** In JavaScript, a regular expression with the\n`g` or `y` flag stores the position where its last match ended in `lastIndex`\nand starts the next search there; after a failed attempt it resets `lastIndex`\nto 0. A pattern kept in a constant and reused for validation then answers\ndifferently on alternate calls:\n\n```text\n\u003e const re = /^\\d{4}$/g;\nundefined\n\u003e re.test(\"2200\")\ntrue\n\u003e re.test(\"2200\")\nfalse\n\u003e re.test(\"2200\")\ntrue\n```\n\nLeave the `g` flag off patterns used with `test()`, or set `lastIndex` to 0\nbefore every call.\n\n**Text that looks the same but is not.** The name André can be stored with a\nsingle precomposed é or with a plain e followed by a combining accent. Both look\nidentical on screen, but POSIX states that matching follows the bit pattern used\nto encode a character, not its graphic representation, and Python's\n`re.search(\"André\", \"Andre\\u0301\")` returns `None`. After\n`unicodedata.normalize(\"NFC\", ...)` on the text it finds the name. Normalise the\npattern and the text to the same form before matching, with\n`unicodedata.normalize` in Python or `String.prototype.normalize` in JavaScript.\n\n## Further reading\n\n- [The Open Group Base Specifications Issue 8: Regular Expressions](https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap09.html)\n  — the POSIX definition: BRE and ERE, bracket expressions, character classes\n  and locales, and the leftmost-longest rule.\n- [regexp package (Go)](https://pkg.go.dev/regexp)\n  — the linear-time guarantee, leftmost-first against leftmost-longest matching,\n  and why CompilePOSIX departs from POSIX on submatches.\n- [re — Regular expression operations (Python)](https://docs.python.org/3/library/re.html)\n  — the syntax, Unicode and ASCII classes, backtracking and possessive\n  quantifiers, and search, prefixmatch and fullmatch.\n- [ECMAScript: Text Processing (living draft, ECMAScript 2027)](https://tc39.es/ecma262/multipage/text-processing.html)\n  — JavaScript's regular expressions: the Perl 5 heritage, the definition of\n  `\\d`, Unicode property escapes and how lastIndex is used.\n- [CWE-1333: Inefficient Regular Expression Complexity](https://cwe.mitre.org/data/definitions/1333.html)\n  — catastrophic backtracking and ReDoS: the conditions that make a pattern\n  exploitable, observed examples and mitigations.\n\n\u003chttps://engineer.company/glossary/regular-expression/\u003e\n","date_modified":"2026-11-09T21:50:25Z","description":"How a regular expression picks out strings, why engines read one pattern differently, and how backtracking, loose anchors and ASCII classes break validation.","formats":{"gemtext":"https://engineer.company/glossary/regular-expression/index.gmi","gopher":"https://engineer.company/glossary/regular-expression/gophermap","html":"https://engineer.company/glossary/regular-expression/","json":"https://engineer.company/glossary/regular-expression/index.json","markdown":"https://engineer.company/glossary/regular-expression/index.md","pdf":"https://engineer.company/glossary/regular-expression/index.pdf"},"kind":"page","language":"en","title":"What is a regular expression?","url":"https://engineer.company/glossary/regular-expression/","version":1}