{"canonical_url":"https://engineer.company/glossary/database/","content_markdown":"# Database\n\n## What it is\n\nA database is a body of data kept in a fixed structure, with rules about what\nmay go into it, so that software can find, change and check its contents; the\nsame word, often shortened to DB, is also used loosely for the program that does\nthe keeping, whose exact name is a database management system or DBMS. NIST's\nglossary prints two definitions, and the broader one describes a repository of\ninformation or data \"which may or may not be a traditional relational database\nsystem\".\n\nThe word does three jobs, and a conversation goes in circles when two people\nmean different ones. It names the data: the customers, orders and sensor\nreadings an organisation keeps. It names the engine that keeps them: PostgreSQL,\nSQL Server, SQLite. And inside a server it names one container among several.\nPostgreSQL's tutorial sets out the levels: rows with named, typed columns make\nup tables, tables are grouped into databases, and the databases that one server\ninstance manages together form a database cluster. SQL Server draws the same\nline between an instance and the databases it hosts. \"The database is slow\" can\nmean one table, one container or one whole machine.\n\nMost engines in daily use are relational: they store data as relations, the\nmathematical term for tables, and take questions in SQL. The PostgreSQL tutorial\nnames other arrangements. The files and directories of a Unix-like system form a\nhierarchical database, and object-oriented databases came later. Redis calls\nitself a data structure server: in place of tables it offers strings, hashes,\nlists, sets and other native types, put to work for caching, queuing and event\nprocessing.\n\nEngines also differ in where they run. A client/server engine is a long-running\nprocess that many programs reach over a network. An embedded engine is a library\ninside one program, and SQLite keeps a whole database, tables, indexes, triggers\nand views included, in one ordinary file; the project says it competes with\n`fopen()` rather than with server databases. The difference shows in how a\nprogram connects:\n\n```sh\n# client/server: connect to a service that is already running\npsql \"host=db.internal dbname=shop user=app\"\n\n# embedded: the program opens a file and the engine runs inside it\nsqlite3 shop.db\n```\n\nA third family is built for analysis. BigQuery stores each column separately,\nwhere many traditional systems keep a row's fields together on disk. Row storage\nsuits looking up one record; column storage suits summing one field across\nmillions of rows, because the query reads that column and leaves the rest alone.\n\n## Why it matters\n\nFiles hold data too. A database adds promises about that data for the moment\nmany programs use it at once.\n\nThe first promise is a consistent picture under concurrent use. PostgreSQL gives\neach statement a snapshot of the data as it stood at a moment, a design called\nmultiversion concurrency control, or MVCC, so a read never waits for a write and\na write never waits for a read. SQLite allows any number of readers at once and\none writer at any instant, with the other writers waiting their turn. The two\nsuit different numbers of writers, and that number does much of the choosing.\n\nThe second promise is that data keeps its shape. A schema states which columns a\ntable has and what type each holds; keys and constraints state which rows may\nexist. When the database enforces those rules, every program that writes to it\nobeys them. When the rules live only in application code, the first program that\nskips them leaves rows behind that every later query has to defend against.\n\nThe third is a shared language, with accents. SQL is the standard ISO/IEC 9075,\nlast revised as SQL:2023, yet PostgreSQL's conformance appendix notes that no\ncurrent system claims full conformance to its Core. A schema and its queries\nbelong to their engine, and moving them to another is a migration with its own\ntesting.\n\nSQLite's own documentation turns the choice of engine into a short checklist:\ndata across a network from the code that issues the SQL, many processes writing\nat the same instant, or content heading for the terabyte range each point to\nclient/server, and otherwise SQLite is the simpler answer. Reporting at scale\nadds one more question: whether heavy reads belong on a separate analytical\ncopy, filled by ETL, so that a long report never competes with the queries\ncustomers wait on.\n\n## What it looks like in practice\n\nAt the large end of the scale, a database engineer role at a rapidly growing\ntech company\n[created, managed and optimised 100+ databases across PostgreSQL, Microsoft SQL Server and Google BigQuery](/portfolio/ran-100-postgresql-ms-sql-bigquery-warehouses/),\nmost of them holding GIS data and time-series data such as sensor readings, logs\nand real-time metrics. The work produced normalised and denormalised schemas for\nPostgreSQL and SQL Server, with spatial indexing (PostGIS on the PostgreSQL\nside) and time-series partitioning, while BigQuery carried large time-series\ndata in time-partitioned, clustered tables. Once those databases had indexing,\nmaterialised views and caching, GIS and time-series queries were answered in\n40–60% less time, and automated monitoring reduced downtime by 50%.\n\nIn a smaller system, at one documented stage of development, a single SQLite file [held the facts behind a CV, a reference sheet, a portfolio and a cover letter](/portfolio/built-a-database-driven-cv-generator/) in three languages. That database had twenty-three tables, and one command built them in nineteen ordered steps, any of which could safely run again. Six output formats were rendered from the database by 41 Python modules totalling 10,580 lines. A document aimed at one kind of reader was \"a query rather than a copy\": a profile selected which achievements appeared and in what order. The trade-off was that a document could then come from that database and nowhere else. These implementation counts describe that recorded stage, not the current repository or its language coverage.\n\nBetween those sizes sits the question every busy database meets, and\n[a note on optimising before sharding](/notes/optimise-before-you-shard/)\nanswers it with an order of work: look, tighten, take the reads off the primary,\npartition the tables around the hot spot, and shard only when one machine is\nfull of good work. Looking means asking PostgreSQL itself: `pg_stat_statements`\nfor the statements that cost most in total, then `EXPLAIN (ANALYZE, BUFFERS)`\nfor what the costliest one does.\n\n## Where it goes wrong\n\nMost database trouble starts in a decision made early, when the data was small\nand the decision looked free.\n\n**Two copies of one fact drift apart.** The CV generator above exists because\nfacts kept as four documents get corrected four times, until two of them\ndisagree and nothing says which is current. The same happens to a table exported\nto a spreadsheet and edited there, or a second service with its own list of\ncustomers. Give each fact one store that owns it, and make every other copy say\nwhere it came from and when. The note's cache follows this rule strictly: it may\ntell a buyer an item is fully reserved, and every yes still comes from the\ndatabase.\n\n**More machines arrive before anyone reads the plans.** Splitting data across\nservers spreads the slow statements over more hardware without making any of\nthem cheaper, and it puts a lasting bill on everything else: as the note lists\nit, every migration, backup, restore and report runs once per shard, and a join\nacross shards becomes application code. Read `pg_stat_statements` and the plans\nfirst, and keep what they show as the baseline.\n\n**A read lands on a copy that is behind.** A streaming replica takes reads off\nthe primary and runs a little behind it. A user who has just written something\nand then reads it back from the replica may see the old value. Read a user's own\nrecent writes from the primary.\n\n**Reports and customers share one engine.** A report that scans two years of\nhistory holds memory and disk that the shop's own queries need, and the people\nbuying feel it. Move heavy analysis to a warehouse copy, and keep the working\ndatabase for the many small reads and writes it serves.\n\n## Further reading\n\n- [NIST glossary: database](https://csrc.nist.gov/glossary/term/database) — two\n  short definitions from NIST publications, one of which allows for databases\n  that are not relational.\n- [PostgreSQL tutorial: Concepts](https://www.postgresql.org/docs/current/tutorial-concepts.html)\n  — tables, rows, columns, databases and the database cluster, in one page.\n- [PostgreSQL: Introduction to MVCC](https://www.postgresql.org/docs/current/mvcc-intro.html)\n  — how snapshots let readers and writers work without blocking each other.\n- [PostgreSQL: SQL Conformance](https://www.postgresql.org/docs/current/features.html)\n  — the parts and versions of ISO/IEC 9075 and what Core conformance means.\n- [Appropriate Uses For SQLite](https://www.sqlite.org/whentouse.html) — where\n  an embedded engine fits, where a client/server one fits better, and a\n  checklist for deciding.\n- [Overview of BigQuery storage](https://cloud.google.com/bigquery/docs/storage_overview)\n  — row-oriented and column-oriented storage, and why analysis favours the\n  second.\n\n\u003chttps://engineer.company/glossary/database/\u003e\n","date_modified":"2026-11-08T02:22:23Z","description":"What the word database names, how server, embedded and analytical engines differ, how to pick one, and the habits that make a database hard to live with.","formats":{"gemtext":"https://engineer.company/glossary/database/index.gmi","gopher":"https://engineer.company/glossary/database/gophermap","html":"https://engineer.company/glossary/database/","json":"https://engineer.company/glossary/database/index.json","markdown":"https://engineer.company/glossary/database/index.md","pdf":"https://engineer.company/glossary/database/index.pdf"},"kind":"page","language":"en","title":"What is a database?","url":"https://engineer.company/glossary/database/","version":1}