{"canonical_url":"https://engineer.company/glossary/storage/","content_markdown":"# Data storage\n\n## What it is\n\nData storage, usually shortened to storage, is the layer of a system where data\nlives between uses, on media that hold it through a reboot and a power cut. The\nNIST glossary gives the word two senses, both taken from SP 800-88r2: the\nretrievable retention of data, and the hardware that data may be entered onto\nand retrieved from.\n\nEvery kind rests on a physical medium: a spinning hard disk, a solid-state\ndrive, an SD card, a tape. Software sees one of three layers above it:\n\n- **Block storage** presents a device as a run of fixed-size blocks addressed by\n  number. A disk partition, a RAID array and a cloud virtual disk all look like\n  this to the kernel, and a filesystem or a database engine decides what the\n  blocks mean.\n- **File storage** lays a filesystem over those blocks: names, directories,\n  sizes, owners and permissions, and a file can be changed a few bytes at a\n  time. Shared over NFS or SMB, one tree serves many machines.\n- **Object storage** keeps whole objects, each a run of bytes plus metadata,\n  under a name inside a container, and is reached through an HTTP API rather\n  than a mount. Google Cloud Storage stores data as objects in containers called\n  buckets, every bucket belongs to a project, and each object holds a file of\n  any format as data that cannot be changed once written.\n\nThe three part ways over what a change costs. A block device rewrites one block\nin place. A filesystem edits part of a file and moves it with a rename. An\nobject store replaces the whole object: Google's objects page rules out appends\nand truncation, and a replacement is atomic, so readers get the old version\nuntil the new upload completes. Depending on the tool, a rename can mean a copy\nunder the new name and a delete of the original; Cloud Storage also supports\natomic object renames within a bucket. In a bucket with a flat namespace,\nfolders are a picture the tools draw from slashes in object names; the string\n`folder1/` is part of the name, and the bucket stores no folder at all.\n\nThe same file at each layer looks like this:\n\n```sh\n# block: the kernel's view of a disk as numbered blocks\nlsblk /dev/sda\n\n# file: a filesystem mounted over those blocks, with names and directories\ndf -h /srv\n\n# object: a whole object sent to a bucket over an API\ngcloud storage cp readings.csv.gz gs://example-bucket/2024/01/\n```\n\n## Why it matters\n\nThe medium decides how data is lost. A hard disk, an SSD and an SD card each\nfail in their own way, and a recovery method that suits one is the wrong tool\nfor the next. Redundancy inside one box, such as a mirrored pair or a RAID\narray, keeps data readable after a single disk dies, but it writes every\ndeletion to every disk as faithfully as it writes the data.\n\nDurability and availability are separate promises. Durability is the chance that\nstored data survives; availability is the chance it can be read at a given\nmoment. Cloud Storage states 99.999999999% annual durability for every storage\nclass, while the typical monthly availability it gives runs from 99.9% to above\n99.99%, depending on class and location. A storage class is metadata on every\nobject that sets its availability and pricing model, and new objects inherit\ntheir bucket's default. The colder classes trade a lower price for keeping data\nagainst slightly lower availability, a minimum storage duration of 30, 90 or 365\ndays and a charge for reading the data back; Google aims them at data read once\na month or less, at most once a quarter, and less than once a year.\n\nLocation is part of the data. A copy on the same disk, the same machine or the\nsame cloud account shares that thing's fate, and a copy elsewhere does not.\nLocation also sets speed and cost: Google notes that keeping data in the same\nregion as the compute that uses it maximises performance for data-intensive work\nand can reduce network charges.\n\nHow software reaches storage shapes the rest. Restic, a backup program, defines\nits repository as nothing more than a directory holding a set of subdirectories\nand files, and that same directory can sit on a local disk, on a server reached\nover SFTP, or in an object store such as Google Cloud Storage or Amazon S3. The\nbytes stay the same while the latency, the failure modes and the person who\ncontrols the account all change.\n\n## What it looks like in practice\n\nFrom the bottom of the stack up, the founder's earlier work touched the physical\nmedium that fails and the object store that moves whole files. The company's\nown backup implementation sets out automation for a copy at another provider as\nprotection against losing a host; activating that storage and verifying a live\nrestore are separate steps.\n\nThe medium:\n[data recovery ran across pretty much every common storage type](/portfolio/recovered-data-from-sd-cards-hdds-ssds-and-raid/),\nfrom SD cards and spinning HDDs to SSDs, RAID arrays, external drives and Mac\nsystems. People brought in storage that had failed or been damaged, holding\nphotos, business files, the only copy of something that mattered. Each medium\nfails and gives up its data differently: an SSD is a different problem from a\nplatter drive, a RAID array is a different problem again, and a Mac filesystem\nhas quirks of its own. Knowing the right approach for the specific medium\nmattered as much as any single technique, and clients got critical data back\nfrom devices they had already written off as gone.\n\nThe object store: in\n[six ETL/ELT pipelines feeding Google BigQuery](/portfolio/designed-and-ran-6-etl-elt-pipelines/),\na Google Cloud Storage bucket stood between the source and the warehouse.\nCurated datasets and tables were exported to intermediate formats, compressed\nwith gzip and transferred to the bucket, and from there the bq CLI and Python\nscripts loaded them into BigQuery. Whole compressed files went in and whole\nfiles came out, the pattern object storage is built for.\n\nThe copy elsewhere is part of the company's\n[backup and restore-test implementation](/portfolio/backups-that-have-actually-been-restored/).\nAn Ansible role installs restic scripts and systemd units for a database dump where configured,\nan encrypted, deduplicated snapshot to storage at another provider over SFTP, retention pruning\nand an integrity check. A separate monthly workflow restores a known file to a scratch location\nand compares its bytes, testing the read-back path rather than just the write. Backup and\nrestore have separate success signals, and the timers require a configured repository.\nThe implementation record dated 2026-10-27 says those timers were disabled without a repository\nand records no completed live restore. It documents the automation and what must be activated;\nit does not prove that the company can already lose the host and recover its data.\n\n## Where it goes wrong\n\nMost storage trouble comes from a copy, a check or a cost that looked fine until\nthe day it was needed.\n\n**A copy shares a fate with the original.** A snapshot in the same account, a\nbackup on the same disk, a replica in the same building: each survives the small\nfailures and goes down with the large one. List the single events that could\ntake out both, and move one copy until the list is short.\n\n**Redundancy gets counted as a backup.** A mirror copies every write to every\ndisk, deletions and corruption included. It answers a dead disk; a separate copy\nwith its own history answers a mistake.\n\n**An object store is treated as a filesystem.** Code written for files expects\ncheap renames and directory listings. Some tools implement object renames as a\ncopy plus a delete, even though Cloud Storage also supports atomic renames\nwithin a bucket. Google notes that deeply nested, directory-like structures in\na flat namespace lack the listing performance of a native filesystem. Lay out\nobject names for the way they will be read, and keep work that edits files in\nplace on file or block storage.\n\n**The check reads the index and skips the data.** By default, restic's `check`\ncommand verifies the repository's structure and leaves the pack files that hold\nthe backed-up data unread, because reading them means downloading every one.\n`--read-data` reads them all, and `--read-data-subset=1/5` through `5/5` covers\nthe repository in five runs. A check that only lists files proves that names\nexist, and little about the bytes behind them.\n\n**A cold class holds hot data.** A file read daily from a class priced for\nmonthly reads pays a charge on every read, so match the class to the read\npattern the data actually has.\n\n## Further reading\n\n- [NIST glossary: storage](https://csrc.nist.gov/glossary/term/storage) — the\n  two senses of the word as NIST SP 800-88r2 defines them.\n- [About Cloud Storage objects](https://cloud.google.com/storage/docs/objects) —\n  object data and metadata, naming rules, immutability and the flat namespace\n  behind simulated folders.\n- [Cloud Storage: Storage classes](https://cloud.google.com/storage/docs/storage-classes)\n  — durability, availability, minimum storage durations and retrieval fees for\n  each class.\n- [restic: Preparing a new repository](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html)\n  — what a repository is, and the local, SFTP and object-store backends it can\n  live on.\n- [restic: Working with repositories](https://restic.readthedocs.io/en/stable/045_working_with_repos.html)\n  — checking a repository's structure and reading back the data it holds.\n- [Cloud Storage: Move objects](https://docs.cloud.google.com/storage/docs/json_api/v1/objects/move)\n  — the atomic rename operation within one bucket, distinct from tools that\n  simulate a move by copying and deleting.\n\n\u003chttps://engineer.company/glossary/storage/\u003e\n","date_modified":"2026-11-09T21:50:25Z","description":"How block, file and object storage differ, what durability and storage classes promise, and why where a copy sits matters as much as the copy.","formats":{"gemtext":"https://engineer.company/glossary/storage/index.gmi","gopher":"https://engineer.company/glossary/storage/gophermap","html":"https://engineer.company/glossary/storage/","json":"https://engineer.company/glossary/storage/index.json","markdown":"https://engineer.company/glossary/storage/index.md","pdf":"https://engineer.company/glossary/storage/index.pdf"},"kind":"page","language":"en","title":"What is data storage?","url":"https://engineer.company/glossary/storage/","version":1}