The Mathematics of RAID6 Revisited: RAID Storage for a Martian Future
Welcome to StreamScale: Building the Cosmic Ray Umbrella for Mars and Earth.
raid for mars
A Brief History of RAID Storage Technology
Patterson, Gibson, and Katz introduced RAID in 1988, defining levels 1–5 with RAID5 using single parity for one failure. Their focus was Earth-based systems, not accounting for cosmic rays or silent corruption, which are critical on Mars (10x bit flips, 230 mSv/year).
In 1997, James S. Plank published “A Tutorial on Reed-Solomon Coding for Fault-Tolerance in RAID-like Systems,” aiming to explain how Reed-Solomon codes could be applied to RAID for multiple failure tolerance. Plank suggested using a Vandermonde matrix to encode data into check symbols, computing syndromes as weighted sums of data symbols alone (e.g., for check symbols P and Q, using rows of the Vandermonde matrix. However, Plank misunderstood the role of the Vandermonde matrix—it’s not for encoding but for defining the parity-check equations (H matrix). In a correct Reed-Solomon code, a codeword c = (d₁, d₂, ..., dₖ, c₁, c₂, ..., cₘ) (data + check symbols) must satisfy Hc = 0, meaning syndromes evaluate to zero at the polynomial roots (α⁰, α¹, ..., αᵐ⁻¹). Plank’s method produced check symbols from data alone, so the syndromes of the full codeword didn’t evaluate to zero, making error location impossible.
Relevance to Mars: Plank’s flawed approach couldn’t reliably locate unknown errors, a fatal flaw on Mars where cosmic ray-induced bit flips (10x rate) and silent corruption require precise error detection and correction for life-critical data (e.g., oxygen levels).
In 2003, Plank and Ying Ding published a correction, “Note: Correction to the 1997 Tutorial on Reed-Solomon Coding,” addressing errors in the original paper (e.g., incorrect matrix operations in GF(2^w)). However, even with the correction, Plank’s approach still used an inverted and normalized Vandermonde matrix for encoding and computed check symbols from data alone, failing to produce codewords that evaluate to zero at the polynomial roots. The syndromes of the full codeword (data + check) were non-zero, so the code remained ineffective for unknown error location.
Relevance to Mars: The corrected paper still fails to address a critical flaw—without zero syndromes, unknown error positions remain undetectable, risking catastrophic data loss on Mars, where cosmic rays and DRAM errors exert far greater stress than on Earth. For instance, in 2013, the Mars Curiosity rover suffered its most significant malfunction to date, just seven months into its mission, likely due to high-energy solar and cosmic ray strikes, as noted by project manager Richard Cook in a National Geographic report: “On other space missions, similar problems were caused by high-energy solar and cosmic ray strikes. He said that’s probably what happened this time.” This underscores the urgent need for robust error correction, like proper Reed-Solomon codewords with zero syndromes, to ensure data integrity for Mars missions facing such harsh conditions.
One Misstep Begets Another
Peter Anvin’s “The Mathematics of RAID-6” (2004–2011) built on Plank’s work, citing Plank's 1997 paper as a reference. Anvin adopted the same flawed assumption that the Vandermonde matrix is used for encoding, defining P as the parity of data symbols only (P = d₁ ⊕ d₂ ⊕ ... ⊕ dₖ) and Q as a weighted sum of data (Q = 1·d₁ ⊕ 2·d₂ ⊕ ... ⊕ k·dₖ). Anvin also misunderstood syndromes, thinking they were computed over data alone, not the entire codeword. This ordering (P then Q) meant P didn’t include Q, so the codeword didn’t satisfy the parity-check equations (Hc ≠ 0). The syndromes didn’t evaluate to zero, making unknown error correction impossible.
- Greg Tucker’s GitHub issues (#10 (https://github.com/intel/isa-l/issues/10) on ISA-L highlight the practical fallout of Plank and Anvin’s misunderstandings. ISA-L followed the same flawed approach (Vandermonde for encoding, syndromes over data only), leading to:
- Issue #10: “corrupted fragment on decode.” Tucker noted that the Vandermonde matrix fails for m - k errors (“unrecoverable”), a direct result of improper Reed Solomon encoding. He tried to explain this as “normal and usual,” ("If you increase m and k for this first case you can find an unrecoverable code where errors <= m - k but this is an expected result if you are using Vandermonde matrix. This is fundamental to the math and not particular to ISA-L EC"). But actually, it is a fundamental flaw in his approach to the problem, not an "expected result" or "fundamental to the math" if you use Vandermonde to decode properly encoded Reed Solomon codewords.
- Issues #13, #26, #40 and #46 all spring from the same flawed approach: Mixed data/check symbols fail to decode because syndromes don’t include check symbols.
Relevance to Mars: Tucker’s claim that customer-identified errors in his work were “fundamental to the math” is incorrect. His proposed fix—limiting the size of the Vandermonde matrix for encoding—might have reduced the errors customers encountered, but it failed to address the core issue: large codewords didn't scale and couldn’t “self-report” error locations and values, a capability standard Reed-Solomon (RS) codewords have provided for decades. For instance, in 1977, RS codes were integral to the Voyager Program, ensuring robust error correction for data transmission across vast distances—a necessity for space missions. Yet, modern large-scale storage systems like Hadoop, Ceph, and Swift still rely on weaker, non-scalable “erasure codes” that lack RS’s superior error detection and correction. On Mars, where reliable data storage is critical for rovers and habitats facing harsh conditions, it’s imperative that modern storage systems adopt proper RS codewords to ensure data integrity and mission success.
ECC Upgrade for Mars and Earth
.jpg/:/cr=t:0%25,l:0%25,w:100%25,h:100%25/rs=w:370,cg:true)
.jpg/:/cr=t:0%25,l:0%25,w:100%25,h:100%25/rs=w:370,cg:true)
.jpg/:/cr=t:0%25,l:0%25,w:100%25,h:100%25/rs=w:370,cg:true)
.jpg/:/cr=t:0%25,l:0%25,w:100%25,h:100%25/rs=w:370,cg:true)
.jpg/:/cr=t:0%25,l:0%25,w:100%25,h:100%25/rs=w:370,cg:true)