{"canonical_url":"https://engineer.company/glossary/full-text-search/","content_markdown":"# Full-text search\n\n## What it is\n\nFull-text search finds the documents that contain the words of a query, in\nwhatever form those words take, and can put the best matches first. It is also\nwritten full text search, without the hyphen. A document here is whatever the\napplication treats as one unit of searching: an email, an article, a support\nticket, or the title and body of one row put together.\n\nWhat makes it different from a pattern match is the work done before anything is\ncompared. The text is first broken into tokens: words, numbers, email addresses\nand other classes. Each token is then normalised into a lexeme, a form chosen so\nthat variants of the same word come out alike. Upper case is folded to lower\ncase, a stemmer strips endings such as the English plural, and stop words, the\nwords so common that they carry no signal, are dropped. The result is stored\ntogether with the position of each lexeme, which later lets a ranking function\nprefer documents where the query words stand close together. The query goes\nthrough the same process, so a search for \"payments\" and a document that says\n\"payment\" meet in the middle.\n\nPostgreSQL builds this into the database. Its `tsvector` type holds a document\nin the normalised form, `tsquery` holds a query, and the operator `@@` answers\nwhether one matches the other. A text search configuration decides how\nnormalisation happens: which parser splits the text and which dictionaries turn\ntokens into lexemes. The standard distribution ships configurations for many\nlanguages, among them `english`, `danish`, `norwegian` and `swedish`, and the\npsql command `\\dF` lists the ones installed. SQLite offers the same idea through\nits FTS5 extension, where a virtual table holds the index and the `MATCH`\noperator queries it.\n\n## Why it matters\n\nThe PostgreSQL documentation explains what the older operators lack. `LIKE`,\n`ILIKE` and regular expressions have no linguistic support, so a search for\n\"satisfy\" misses a document that says \"satisfies\" unless someone spells out\nevery derived form. They have no ranking, so a thousand matches come back in no\nuseful order. And they cannot use an ordinary index for this kind of search, so\nevery query reads every document. A pattern such as `%cat%` also matches\n\"category\" and \"concatenate\", which is noise rather than recall.\n\nPeople type words into a search box and expect to find the documents about those\nwords. On a contract archive, a help desk or a product catalogue, full-text\nsearch is the difference between a search box that people trust and one they\nstop using after the third miss. The ranking matters as much as the matching:\nsomeone looking for \"late payment\" wants the reminder letter at the top, not the\nhundred invoices that mention payment once.\n\nHaving it inside the database you already run has a practical side. The search\nindex lives in the same transaction as the data, so a row is searchable the\nmoment it is committed, and a deleted row disappears from results at the same\nmoment. Filters on dates, customers or permissions are ordinary `WHERE` clauses\nnext to the text condition. A separate search server can do more, but it is a\nsecond system to fill and to keep in step with the first, and every gap between\nthem shows up as a result that is missing or one that should be gone.\n\nFull-text search also has a limit worth knowing from the start: it matches\nwords, not meaning. A query for \"car\" does not find \"automobile\" unless a\nsynonym dictionary maps one to the other. Vector search, which compares\nnumerical representations of meaning, covers that case, and the two are often\ncombined.\n\n## What it looks like in practice\n\nTake a table of documents with a title, a body and the place each one came from.\nA stored generated column keeps the searchable form up to date on every insert\nand update, and a GIN index makes it fast to search:\n\n```sql\nCREATE TABLE documents (\n    id         bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n    title      text,\n    body       text,\n    source     text NOT NULL,\n    search_vec tsvector GENERATED ALWAYS AS (\n                   setweight(to_tsvector('english', coalesce(title, '')), 'A')\n                || setweight(to_tsvector('english', coalesce(body, '')), 'D')\n               ) STORED\n);\n\nCREATE INDEX documents_search_idx ON documents USING GIN (search_vec);\n```\n\nThree details carry the weight. The configuration is named, `'english'`, rather\nthan left to the session's `default_text_search_config`; the two-argument form\nof `to_tsvector` is the one PostgreSQL can store and index. `coalesce` turns a\nmissing title or body into an empty string, because a single NULL would\notherwise make the whole document NULL. And `setweight` labels each lexeme with\nwhere it came from: A for the title, D for the body. The ranking functions use\nthe weights {0.1, 0.2, 0.4, 1.0} for D, C, B and A unless told otherwise, so a\nquery word in the title counts ten times as much as the same word in the body.\n\nGIN is the index type the documentation prefers for text search: an inverted\nindex with one entry per lexeme and a compressed list of the rows that contain\nit. GiST also works, but it is lossy and has to check the table row to rule out\nfalse matches.\n\nThe query takes the words a user typed, as they typed them:\n\n```sql\nSELECT id, title, source,\n       ts_rank_cd(search_vec, query) AS rank\nFROM   documents,\n       websearch_to_tsquery('english', 'unpaid invoice -draft') AS query\nWHERE  search_vec @@ query\nORDER  BY rank DESC\nLIMIT  10;\n```\n\n`websearch_to_tsquery` reads input the way a web search box does. Plain words\nmust all appear, in any form the English stemmer folds together, text in double\nquotes must appear as a phrase, the word \"or\" offers an alternative, and a\nleading minus excludes a word, so this query leaves out every document that\nmentions a draft. The function never raises a syntax error, which is why it is\nthe safe choice for raw input. `ts_rank_cd` ranks by cover density, a method\nfrom a 1999 paper by Clarke, Cormack and Tudhope that also rewards query words\nstanding close together. Its sibling `ts_rank` counts how often the words\nappear. Both accept a normalisation flag: 1, for instance, divides the rank by\none plus the logarithm of the document's length, so a long document does not win\nonly because it is long.\n\nThe same need runs through the company's\n[work on company data for AI agents](/ai-data/). The page describes old PDFs,\nscans and mail archives turned into searchable text with their source kept\nbeside them, so they can be found rather than only stored. The `source` column\nabove has that job: a match is worth more when the reader can open the original.\n\nSQLite reaches the same result with less ceremony. A virtual table declared with\n`CREATE VIRTUAL TABLE notes USING fts5(title, body)` is queried with\n`WHERE notes MATCH 'invoice NOT draft'`, and `ORDER BY bm25(notes)` puts the\nbest matches first: FTS5 returns better matches as numerically lower values, so\nthe default ascending order is the right one. The default tokenizer is\n`unicode61`; the `porter` tokenizer adds stemming, which its documentation says\nis designed for English only.\n\n## Where it goes wrong\n\n**An index the query never touches.** An expression index built on\n`to_tsvector('english', body)` serves only queries that repeat exactly that\nexpression. A query written as `to_tsvector(body)`, relying on the default\nconfiguration, reads the whole table. Store the vector in a generated column, as\nabove, and query the column; then the query cannot drift from the index.\n\n**One NULL that hides a document.** `to_tsvector(NULL)` returns NULL, and\nconcatenating a NULL title with a body gives NULL, so a row without a title\nsilently drops out of every search. Wrap every field in `coalesce`.\n\n**Raw input in `to_tsquery`.** `to_tsquery` expects its own operator syntax and\nis less forgiving of its input than the other conversion functions, so a user\nwho types an unbalanced parenthesis or a stray ampersand gets an error page.\nPass user input through `websearch_to_tsquery`, or `plainto_tsquery` when\noperators are not wanted at all.\n\n**The wrong language.** Danish text indexed with the English configuration is\nstemmed by English rules and filtered by English stop words, so half the\nvariants never meet. Pick the configuration for the language of the text. A\ncollection in several languages can store the configuration name in a column of\nits own and index `to_tsvector(config_name, body)`, as long as the queries name\nthe same column.\n\n**Ranking every match.** Ranking reads the `tsvector` of each matching row, and\nthe documentation warns that this can be I/O bound and slow. A one-word query\nthat matches half the table makes the database rank half the table before\n`LIMIT` keeps ten. Narrow the candidates with ordinary conditions first, such as\na date range or a customer, and keep the vector stored so it is not computed\nagain for every row.\n\n**A rank shown as a percentage.** The ranking functions use no information about\nthe collection as a whole, so a rank cannot be turned into a fair \"87%\nrelevant\". The normalisation flag 32 maps every rank to a value between 0 and 1,\nbut the documentation calls that a cosmetic change. Use the rank to order\nresults and nothing else.\n\n**Highlights that carry markup.** `ts_headline` returns an excerpt with the\nquery words marked, which is just what a results page wants. PostgreSQL warns\nthat its output is \"not guaranteed to be safe for direct inclusion in web\npages\". A document that holds HTML can carry a script into the page. Strip the\nmarkup from the input or run the output through an HTML sanitiser. The function\nalso works on the original text, not the vector, so call it only for the rows on\nthe page, after `LIMIT`, not for every match.\n\n**Documents too large for one vector.** A `tsvector` must stay under 1 MB, and\nit keeps no more than 256 positions per lexeme. A whole book or a year of\nminutes in one row runs into both. Split long documents into sections or pages,\nwhich also gives each result a more precise place to point to.\n\n**Fragments and part numbers.** Full-text search compares whole lexemes, so\n\"inv\" finds nothing about invoices and a part number such as `AB-1042-X` may be\nsplit by the parser into pieces. Keep an ordinary index on the columns people\nsearch by exact value, or, in SQLite, use the `trigram` tokenizer, which treats\nevery run of three characters as a token and so supports substring matching.\n\n**An SQLite index that drifts from its table.** An FTS5 table declared with\n`content=` reads its text from another table, and the documentation leaves it to\nthe user to keep the two consistent. When they drift, queries return rows that\nseem to contradict each other. Create the insert, update and delete triggers the\ndocumentation shows, in the same migration as the virtual table.\n\n## Further reading\n\n- [Introduction to full text search](https://www.postgresql.org/docs/current/textsearch-intro.html)\n  — PostgreSQL's definition of documents, tokens, lexemes and configurations,\n  and what `LIKE` and regular expressions lack.\n- [Tables and Indexes](https://www.postgresql.org/docs/current/textsearch-tables.html)\n  — searching a table, expression indexes against a stored generated column, and\n  why the configuration must be named.\n- [Controlling Text Search](https://www.postgresql.org/docs/current/textsearch-controls.html)\n  — weights, `websearch_to_tsquery`, `ts_rank` and `ts_rank_cd`, normalisation\n  flags and the `ts_headline` warning.\n- [Preferred Index Types for Text Search](https://www.postgresql.org/docs/current/textsearch-indexes.html)\n  — GIN and GiST compared, including why GiST needs to recheck rows.\n- [Limitations](https://www.postgresql.org/docs/current/textsearch-limitations.html)\n  — the size limits on lexemes, vectors, positions and queries.\n- [SQLite FTS5 Extension](https://www.sqlite.org/fts5.html) — virtual tables,\n  the `MATCH` operator, the built-in tokenizers, `bm25()` and external content\n  tables.\n\n\u003chttps://engineer.company/glossary/full-text-search/\u003e\n","date_modified":"2026-10-29T11:17:41Z","description":"What full-text search is: documents reduced to lexemes, GIN indexes and relevance ranking in PostgreSQL, SQLite FTS5, and where search breaks.","formats":{"gemtext":"https://engineer.company/glossary/full-text-search/index.gmi","gopher":"https://engineer.company/glossary/full-text-search/gophermap","html":"https://engineer.company/glossary/full-text-search/","json":"https://engineer.company/glossary/full-text-search/index.json","markdown":"https://engineer.company/glossary/full-text-search/index.md","pdf":"https://engineer.company/glossary/full-text-search/index.pdf"},"kind":"page","language":"en","title":"What is full-text search?","url":"https://engineer.company/glossary/full-text-search/","version":1}