{"canonical_url":"https://engineer.company/glossary/raid/","content_markdown":"# RAID\n\n## What it is\n\nRAID runs several disks as a single block device, so that the group can be\nfaster, larger or harder to kill than any one of its members. The name comes\nfrom a 1987 technical report from the University of California, Berkeley, by\nDavid Patterson, Garth Gibson and Randy Katz: _A Case for Redundant Arrays of\nInexpensive Disks (RAID)_. Linux spells the levels in lower case and without a\nspace, raid1 or raid5, which is how its md driver reports them.\n\nThe report starts from a mismatch. Processors and memory were getting faster,\nand the abstract warns that the gain is wasted unless input and output keep up.\nIt sets the Single Large Expensive Disk, or SLED, whose capacity had grown\nquickly while its performance had improved only modestly, against arrays built\nfrom the cheap disks made for personal computers, which it expects to beat the\nbig disk roughly tenfold on speed, reliability, power draw and room to grow. The\ncomparison is concrete: an IBM 3380 mainframe disk, a Fujitsu Super Eagle for\nminicomputers, and the 100 MB Conner CP 3100 for personal computers.\n\nThen comes the catch, in a section the authors title as bad news. If disks fail\nindependently and at a constant rate, which the paper notes is what disk makers\nassume when they quote a mean time to failure (MTTF), an array's MTTF is that of\none disk divided by the number of disks. A hundred CP 3100s, each rated at\n30,000 hours, give 300 hours, under two weeks; a thousand give 30 hours. The\nconclusion is blunt: “Without fault tolerance, large arrays of inexpensive disks\nare too unreliable to be useful.”\n\nThe answer is redundancy. The disks are split into reliability groups, each with\nextra check disks that hold redundant information. When a disk fails, it is\nreplaced and its contents are rebuilt onto the new disk from that information;\nthe time this takes is the mean time to repair, and hot standby spares, switched\nin electronically when a disk dies, shorten it. The report then lays out five\norganisations and calls each one a level. Level 1 mirrors every disk. Level 2\ncorrects errors with a Hamming code, as memory systems did. Level 3 needs only a\nsingle check disk per group. Level 4 lets small reads and writes go to separate\ndisks at the same time. Level 5 spreads data and check information across every\ndisk, so that no single check disk holds up the writes.\n\nPractice has added to the list. mdadm, the tool that manages Linux software\nRAID, accepts linear, raid0 (or stripe), raid1, raid4, raid5, raid6 and raid10,\namong others. Striping without redundancy, now called level 0, is not in the\n1987 taxonomy, which begins with mirrored disks. The report left open whether an\narray is built in hardware or in software and called that a question of cost and\nbenefit alone. Linux md is the software kind; a controller card that shows the\noperating system one large disk is the hardware kind.\n\n## Why it matters\n\nA disk wears out, and on a server with one disk its failure stops the machine\nand leaves the data to whatever backup exists. The report opens its section on\nreliability with exactly that: disks are unreliable enough that systems managers\nback up everything every night. With redundancy, a failed disk becomes a repair\njob instead of an outage. The array carries on from the members that remain, in\na state Linux calls degraded, and returns to full strength once a replacement\nhas been added and rebuilt.\n\nThat protection lasts only as long as the remaining disks do. The report's\narithmetic counts data as lost when a second disk in the same group fails before\nthe first one has been repaired, so the length of the repair window matters as\nmuch as the number of check disks. With hot spares and short repairs the figures\nbecome impressive: the report computes that a level 5 array of 1,000 disks, in\ngroups of ten with 5 percent standby spares, would still have an MTTF of 90\nyears. Elsewhere it warns that an earthquake or a power surge can make\nthe disks in one array fail together, and its formulas assume they do not.\n\nChoosing a level is a trade between capacity, speed and cost. The report's own\nconclusion is that mirrored disks suit transaction processing that uses no more\nthan half of the storage, and level 5 looks best when more of the capacity is\nneeded, or for supercomputer work, or for a mix of that and transaction\nprocessing.\n\nWhen the redundancy runs out, what is left is recovery. The portfolio entry on\n[recovering data from SD cards, hard disks, SSDs and RAID](/portfolio/recovered-data-from-sd-cards-hdds-ssds-and-raid/)\ntreats a failed RAID array as a problem of its own, set apart from a dead SSD or\na failed platter drive, with the right approach depending on the medium.\n\n## What it looks like in practice\n\nTake a server with one partition on each of two disks, `/dev/sda1` and\n`/dev/sdb1`, to be mirrored for a data volume. With Linux md, three commands\nbuild the array, put a file system on it and record it in the configuration:\n\n```sh\nmdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1\nmke2fs -t ext4 -L data /dev/md0\nmdadm --detail --scan \u003e\u003e /etc/mdadm/mdadm.conf\n```\n\nThe first creates `/dev/md0`, a RAID1 array of the two partitions, and the\nkernel starts a resync, recalculating the redundancy, as it does after any\ncreation. The second formats the array exactly as it would a single disk; the\nfile system sees one device. The third appends a line describing the array to\nmdadm's configuration file, which lives at `/etc/mdadm.conf` or\n`/etc/mdadm/mdadm.conf`, and the manual asks for that output to be reviewed\nbefore use, since it may carry detail nobody wants there.\n\nThree files show how the array is doing:\n\n```sh\ncat /proc/mdstat\ncat /sys/block/md0/md/degraded\ncat /sys/block/md0/md/sync_action\n```\n\n`/proc/mdstat` lists every active md device with its state, and mdadm reads it\nto find arrays and to follow reconstruction. Each array also has a directory\nunder `/sys/block/`, and in it `degraded` holds the number of members missing: 0\nfor a whole array, 1 with one disk gone. `sync_action` reads `idle` when nothing\nis happening, `resync` after creation or an unclean shutdown, `recover` while a\nhot spare takes over the contents of a lost member, and `check` or `repair`\nduring a full verification. Writing `check` to it starts that verification,\nwhich reads every block of the array:\n\n```sh\necho check \u003e /sys/block/md0/md/sync_action\ncat /sys/block/md0/md/mismatch_cnt\n```\n\nAfterwards `mismatch_cnt` gives the number of sectors that were rewritten, or\nthat would have been. The kernel documentation notes that most levels work in\npages rather than sectors, so the count can be larger than the number of real\nerrors.\n\nSomeone also has to hear about a failure. `mdadm --monitor` polls the arrays and\nturns what it notices into events, which it can mail to an address or hand to a\nprogram. On a system run by systemd, the manual names `mdmonitor.service` as the\nmain route, and `MAILADDR` in `mdadm.conf` as the setting to fill in for mail.\nOnly five events send mail: `Fail`, `FailSpare`, `DegradedArray`,\n`SparesMissing` and `TestMessage`. The last exists to prove the path works:\n\n```text\nMAILADDR ops@example.com\n```\n\n```sh\nmdadm --monitor --scan --oneshot --test\nmdadm --detail --test /dev/md0\n```\n\nThe first sends a test message for every array it finds and then exits. The\nsecond suits a script: with `--test`, its exit status is 0 for a healthy array,\n1 when at least one device has failed, 2 when so many have failed that the array\nis unusable, and 4 when mdadm hit an error while gathering information about\nthe array.\n\nThe array covers one event, a disk dying, and nothing else. The portfolio entry on\n[backup and restore-test automation](/portfolio/backups-that-have-actually-been-restored/)\ndescribes a separate recovery design: encrypted, deduplicated restic snapshots to storage at\nanother provider over SFTP, systemd timers and a monthly workflow that restores a known file\nto a scratch location and compares its bytes. That is a test of restoration in the design,\nnot evidence that it has run. The implementation record dated 2026-10-27 says the timers were\ndisabled without a configured repository and records no completed live restore.\n\n## Where it goes wrong\n\nAn array is built for one event, a member disk dying, and most of the ways it\nloses data sit just beside that event.\n\n**The array is taken for a backup.** A mirror writes every change to both disks\nat once, and that includes a file deleted by mistake, a table dropped by a bad\nmigration or a volume encrypted by ransomware. Every member then holds the same\ndamage. The fix is a separate backup kept off the host, with restores tested on\na schedule, so the array guards against hardware and the backup against\neverything else.\n\n**A degraded array goes unnoticed.** A degraded array keeps serving, and nothing\nabout it looks different until the next disk fails. mdadm sends no mail unless\nit has an address or a program: with `--scan` and neither configured, the manual\nsays it monitors nothing. `DegradedArray` also covers only arrays that are\nalready degraded when mdadm first sees them; a disk that dies later produces a\n`Fail` event. The fix is to set `MAILADDR`, make sure `mdmonitor.service` is\nrunning, and send a `--test` message to prove that mail arrives.\n\n**The second disk fails during the rebuild.** A rebuild reconstructs the missing\ndisk from all the others, and the report's model loses data exactly when another\ndisk in the group fails within that repair time. A failed disk left in place for\na week stretches the window to a week. The fix is to shorten it: keep a hot\nspare in the array, which mdadm can set up at creation with `--spare-devices`,\nreplace failed disks promptly, and run a scheduled `check`, so that a disk with\nproblems is found while the array can still correct them.\n\n**The disks are not independent.** The MTTF arithmetic assumes that disks fail\non their own, and the report itself names an earthquake and a power surge as\ncases where they would not. Two disks in one case share a power supply, a\ncontroller, a room and a building. The fix is to treat the calculated figure as\nan upper bound and to cover shared causes with something outside the box, such\nas a copy at another site.\n\n**A RAID 5 or 6 array is dirty and degraded at once.** If the machine stops\nuncleanly while a member is missing, the parity cannot be trusted and some\nblocks cannot be rebuilt, so the kernel documentation warns of undetectable\ncorruption, and md refuses to start the array. Forcing it with\n`mdadm --assemble --force`, or for a root file system with the\n`start_dirty_degraded` module parameter, accepts that risk. The fix is to choose\na consistency policy in advance: md can log writes to a journal device for\nlevels 4, 5 and 6, and for level 5 it offers a partial parity log, which the\ndocumentation says closes the write hole. When forcing is the only way in, treat\nthe data as suspect and check it against a backup.\n\n**RAID 0 is counted as protection.** md lists raid0 and linear arrays as having\nno redundancy information, and mdadm's monitor reports them with `Wrong-Level`,\nbecause they support none of the failure, spare and resync operations it\nwatches. There is nothing to rebuild a missing member from. The fix is to use\nlevel 0 only for data that can be thrown away or recreated, and never to count\nit as RAID in the sense of the 1987 report.\n\n**A hardware controller hides the disks.** Behind a controller card, the\noperating system sees one logical device, and asking that device about drive\nhealth reaches the controller, not the drives. smartctl, for example, needs\n`-d megaraid,N` to address disk N behind a MegaRAID controller, which its manual\nsays also covers Dell PERC cards. The fix is to monitor each physical disk\nthrough the controller's own interface, so that a failing member is noticed\nwhile the array is still whole.\n\n## Further reading\n\n- [A Case for Redundant Arrays of Inexpensive Disks (RAID)](https://www2.eecs.berkeley.edu/Pubs/TechRpts/1987/5853.html)\n  — the Berkeley technical report's page, with its abstract and the PDF.\n- [UCB/CSD-87-391, the full report (PDF)](https://www2.eecs.berkeley.edu/Pubs/TechRpts/1987/CSD-87-391.pdf)\n  — the MTTF arithmetic, reliability groups and the five levels.\n- [RAID arrays, Linux kernel documentation](https://docs.kernel.org/admin-guide/md.html)\n  — md's files under `/sys`, dirty degraded arrays and consistency policies.\n- [mdadm(8)](https://man7.org/linux/man-pages/man8/mdadm.8.html) — creating\n  arrays, monitor events, mail and the `--detail --test` exit codes.\n- [smartctl(8), smartmontools](https://github.com/smartmontools/smartmontools/blob/main/src/smartctl.8.in)\n  — reaching disks behind RAID controllers with the `-d` option.\n\n\u003chttps://engineer.company/glossary/raid/\u003e\n","date_modified":"2026-11-08T02:22:23Z","description":"What RAID is, where the five levels of the 1987 Berkeley paper came from, how Linux builds and watches an md array, and where arrays lose data.","formats":{"gemtext":"https://engineer.company/glossary/raid/index.gmi","gopher":"https://engineer.company/glossary/raid/gophermap","html":"https://engineer.company/glossary/raid/","json":"https://engineer.company/glossary/raid/index.json","markdown":"https://engineer.company/glossary/raid/index.md","pdf":"https://engineer.company/glossary/raid/index.pdf"},"kind":"page","language":"en","title":"What is RAID?","url":"https://engineer.company/glossary/raid/","version":1}