Data storage

keeping data, or the media that hold it, for later reading

What it is

Data storage, usually shortened to storage, is the layer of a system where data lives between uses, on media that hold it through a reboot and a power cut. The NIST glossary gives the word two senses, both taken from SP 800-88r2: the retrievable retention of data, and the hardware that data may be entered onto and retrieved from.

Every kind rests on a physical medium: a spinning hard disk, a solid-state drive, an SD card, a tape. Software sees one of three layers above it:

  • Block storage presents a device as a run of fixed-size blocks addressed by number. A disk partition, a RAID array and a cloud virtual disk all look like this to the kernel, and a filesystem or a database engine decides what the blocks mean.
  • File storage lays a filesystem over those blocks: names, directories, sizes, owners and permissions, and a file can be changed a few bytes at a time. Shared over NFS or SMB, one tree serves many machines.
  • Object storage keeps whole objects, each a run of bytes plus metadata, under a name inside a container, and is reached through an HTTP API rather than a mount. Google Cloud Storage stores data as objects in containers called buckets, every bucket belongs to a project, and each object holds a file of any format as data that cannot be changed once written.

The three part ways over what a change costs. A block device rewrites one block in place. A filesystem edits part of a file and moves it with a rename. An object store replaces the whole object: Google’s objects page rules out appends and truncation, and a replacement is atomic, so readers get the old version until the new upload completes. A rename becomes a copy under the new name and a delete of the original. In a bucket with a flat namespace, folders are a picture the tools draw from slashes in object names; the string folder1/ is part of the name, and the bucket stores no folder at all.

The same file at each layer looks like this:

# block: the kernel's view of a disk as numbered blocks
lsblk /dev/sda

# file: a filesystem mounted over those blocks, with names and directories
df -h /srv

# object: a whole object sent to a bucket over an API
gcloud storage cp readings.csv.gz gs://example-bucket/2024/01/

Why it matters

The medium decides how data is lost. A hard disk, an SSD and an SD card each fail in their own way, and a recovery method that suits one is the wrong tool for the next. Redundancy inside one box, such as a mirrored pair or a RAID array, keeps data readable after a single disk dies, but it writes every deletion to every disk as faithfully as it writes the data.

Durability and availability are separate promises. Durability is the chance that stored data survives; availability is the chance it can be read at a given moment. Cloud Storage states 99.999999999% annual durability for every storage class, while the typical monthly availability it gives runs from 99.9% to above 99.99%, depending on class and location. A storage class is metadata on every object that sets its availability and pricing model, and new objects inherit their bucket’s default. The colder classes trade a lower price for keeping data against slightly lower availability, a minimum storage duration of 30, 90 or 365 days and a charge for reading the data back; Google aims them at data read once a month or less, at most once a quarter, and less than once a year.

Location is part of the data. A copy on the same disk, the same machine or the same cloud account shares that thing’s fate, and a copy elsewhere does not. Location also sets speed and cost: Google notes that keeping data in the same region as the compute that uses it maximises performance for data-intensive work and can reduce network charges.

How software reaches storage shapes the rest. Restic, a backup program, defines its repository as nothing more than a directory holding a set of subdirectories and files, and that same directory can sit on a local disk, on a server reached over SFTP, or in an object store such as Google Cloud Storage or Amazon S3. The bytes stay the same while the latency, the failure modes and the person who controls the account all change.

What it looks like in practice

From the bottom of the stack up, the work has touched the physical medium that fails, the object store that moves whole files, and the copy kept at another provider so that losing a host is survivable.

The medium: data recovery ran across pretty much every common storage type, from SD cards and spinning HDDs to SSDs, RAID arrays, external drives and Mac systems. People brought in storage that had failed or been damaged, holding photos, business files, the only copy of something that mattered. Each medium fails and gives up its data differently: an SSD is a different problem from a platter drive, a RAID array is a different problem again, and a Mac filesystem has quirks of its own. Knowing the right approach for the specific medium mattered as much as any single technique, and clients got critical data back from devices they had already written off as gone.

The object store: in six ETL/ELT pipelines feeding Google BigQuery, a Google Cloud Storage bucket stood between the source and the warehouse. Curated datasets and tables were exported to intermediate formats, compressed with gzip and transferred to the bucket, and from there the bq CLI and Python scripts loaded them into BigQuery. Whole compressed files went in and whole files came out, the pattern object storage is built for.

The copy elsewhere: restic now ships encrypted backups over SFTP to a second provider. Before that, everything the company holds sat on one cloud instance whose provider snapshot was the whole recovery position: a snapshot that lived in the same account as the machine it protected, nobody at the company had encrypted it, and nobody had ever restored from one. A systemd timer now runs a database dump where one exists, an encrypted, deduplicated snapshot to the other provider, a retention prune and an integrity check. Once a month one known file is restored from the other provider’s storage and compared, which proves the remote copy can be read back and not merely written. With the copy at a second provider, the company can lose the host and still get its data back.

Where it goes wrong

Most storage trouble comes from a copy, a check or a cost that looked fine until the day it was needed.

A copy shares a fate with the original. A snapshot in the same account, a backup on the same disk, a replica in the same building: each survives the small failures and goes down with the large one. List the single events that could take out both, and move one copy until the list is short.

Redundancy gets counted as a backup. A mirror copies every write to every disk, deletions and corruption included. It answers a dead disk; a separate copy with its own history answers a mistake.

An object store is treated as a filesystem. Code written for files expects cheap renames and directory listings. In a flat-namespace bucket a rename is a copy plus a delete, and Google notes that deeply nested, directory-like structures lack the listing performance of a native filesystem. Lay out object names for the way they will be read, and keep work that edits files in place on file or block storage.

The check reads the index and skips the data. By default, restic’s check command verifies the repository’s structure and leaves the pack files that hold the backed-up data unread, because reading them means downloading every one. --read-data reads them all, and --read-data-subset=1/5 through 5/5 covers the repository in five runs. A check that only lists files proves that names exist, and little about the bytes behind them.

A cold class holds hot data. A file read daily from a class priced for monthly reads pays a charge on every read, so match the class to the read pattern the data actually has.

Further reading

Glossary