Regular Expressions, Without the Mysticism

I have a theory about why regular expressions feel hard, and it is that almost every regex tutorial leads with the formal grammar - here is what a metacharacter is, here is what a quantifier is, here is what an anchor is - instead of leading with the actual mental model of what a regex engine is doing. The mental model is straightforward once you have it. A regex is a description of a pattern, and the engine slides that pattern across your input string from left to right, asking at each position: does the pattern match starting here? If yes, record the match. If no, slide one character to the right and try again.

That is it. The rest of regex - all the strange punctuation, all the lookaheads and lookbehinds and capture groups - is just notation for describing patterns more compactly. Once you have the sliding-window model in your head, everything else clicks into place.

This post is the explanation I wish someone had given me when I was learning regex, plus a horror story about catastrophic backtracking that prompted me to add Web Worker isolation to Toolium's regex tester.

The five tokens that do 90 percent of the work

The full regex spec has dozens of features. In practice, almost every regex I write uses five of them:

• Character classes. [abc] matches one character that is either a, b, or c. [a-z] matches one lowercase letter. [^abc] (the caret inside the brackets) matches any character except a, b, or c. Character classes are how you say "one of these things, please."
• Quantifiers. * means zero or more of the previous thing. + means one or more. ? means zero or one (often used to make a thing optional). {2,5} means between 2 and 5 of the previous thing. Quantifiers are how you say "however many of these are here."
• Anchors. ^ means "at the start of the string" (or line, with the m flag). $ means "at the end." Anchors are how you say "this pattern only matches if it is at the edge."
• Groups. (abc) captures whatever matches "abc" into a numbered group you can reference later. (?:abc) groups without capturing, which is sometimes useful for performance or readability. Named groups ((?<name>...)) make complex patterns much easier to maintain.
• Alternation. cat|dog matches either "cat" or "dog." Alternation is how you say "any one of these."

Beyond these, the things you reach for most often are the shorthand character classes: \d for digits, \w for word characters (letters, digits, underscores), \s for whitespace. Each has an uppercase inverse: \D, \W, \S respectively.

The "match an email" pattern, and why every one you see is wrong

Every regex tutorial includes an "email validation" example. Almost every one of them is wrong, because the actual specification for valid email addresses (RFC 5322) is famously baroque. The full grammar fills several pages and includes things like quoted local parts, comments, and IP-address-literal domains.

For 95 percent of real-world use cases, this is plenty:

^[\w.+-]+@[\w.-]+\.[a-zA-Z]{2,}$

What this says, reading left to right: the string must start with one or more word characters (or dots, pluses, or hyphens), followed by an @, followed by one or more word characters or dots or hyphens, followed by a dot, followed by at least two letters, and the string must end there.

The "good enough" pattern misses some legal email addresses (anything with an IP-literal domain like user@[192.168.0.1], anything with quoted-local-part syntax like "weird email"@example.com) and accepts some illegal ones (it does not check that the TLD is real). For UI validation in a form, this is fine. For anything that actually has to match the RFC, use a library; do not write your own regex.

The catastrophic backtracking story

Here is the bug that prompted me to add a Web Worker timeout to the Regex Tester.

A user reported that the tester was freezing their browser. They had typed the pattern (a+)+b and pasted a 20-character string of aaaaaaaaaaaaaaaaaaaa as the test input. The browser tab became unresponsive for about 30 seconds, then the OS killed the tab.

The pattern (a+)+b looks innocuous. It says "one or more groups of one or more a's, followed by a b." Against the input "aaaaaaaaaaaaaaaaaaaa" (twenty a's with no trailing b), the regex engine has to verify that no match exists. To do that, it tries every possible way to partition the a's into groups of one-or-more, and there are an enormous number of those partitions. The number of partitions grows exponentially with the input length, so 20 a's produces over a million combinations to check. 25 a's produces over 33 million. Your browser sits there and burns CPU.

This is "catastrophic backtracking," and it is a well-known regex footgun. The fix in the engine is to either use a different regex engine (RE2, which Google uses, runs in guaranteed linear time but doesn't support some features) or to add timeouts. I went with timeouts: the tester now runs the regex inside a Web Worker, and if the worker takes more than one second to return a result, the main page kills it and shows a warning instead of freezing the tab.

If your regex causes the warning, the regex is almost certainly bad and would also be slow in production. Common patterns to avoid: nested quantifiers like (a+)+, alternations where the branches overlap like (a|aa)+, and lookaheads inside loops.

Flags, briefly

The five flags you actually care about:

• g (global) - find all matches, not just the first. Without this, most regex APIs return only the first match.
• i (insensitive) - case-insensitive matching. [A-Z] and [a-z] become equivalent.
• m (multiline) - ^ and $ match at line boundaries, not just the start and end of the whole input. Useful when your input has multiple lines.
• s (dotAll) - the . character matches newlines, which by default it does not. Useful for matching across line boundaries.
• u (unicode) - properly handle multi-byte characters. Important for non-Latin text and emojis. With this flag on, the behavior of some metacharacters changes slightly.

A few patterns I actually keep around

The ones I find myself copy-pasting from old projects:

• Strip leading/trailing whitespace: ^\s+|\s+$ (replace with empty string)
• Multiple spaces to single space: \s+ (replace with " ")
• Find words inside quotes: "([^"]*)" - the capture group has the unquoted contents
• Hex color codes: #[0-9a-fA-F]{6}\b
• ISO date: \d{4}-\d{2}-\d{2} (doesn't check that the month and day are valid, but catches the shape)
• URL (rough): https?://[\w.-]+(?:/[\w./?=&-]*)? - misses some legal URLs but is good enough for log scanning
• IPv4 address: \b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b - doesn't check that each octet is 0-255

Notice that almost all of these are deliberately loose. Writing a regex that exactly matches the spec for something like an IPv4 address gets ugly fast. For 95% of use cases, the loose version is fine. For the other 5%, use a library or a parser, not a regex.

What regex is bad at

A short list of things people try to use regex for and shouldn't:

• Parsing HTML. There is a famous Stack Overflow answer that just shouts about this. HTML is not regular, so a regex that works on the simple cases will fail on the realistic ones. Use an HTML parser.
• Parsing JSON. Same reason. JSON has nested structure, regex cannot count nesting levels reliably.
• Validating natural language. Email address validation is the canonical example. The full spec is too complex; either use a "good enough" regex or use a library.
• Anything that needs balancing. Matching balanced parentheses, balanced HTML tags, balanced quotes with escapes - these need a real parser.

For everything else, regex is genuinely a superpower. Five minutes of pattern-tinkering can replace half an hour of writing a parser by hand. The Toolium Regex Tester is a good place to do that tinkering - real-time match highlighting, capture group display, the Web Worker timeout so a bad pattern doesn't freeze your browser, all running locally without uploading your test input anywhere.