---
title: "ECC memory errors: rates, causes, and consequences in datacenter and AI systems"
slug: ecc-memory-errors
question: What does the evidence show about the rate, causes, and system-level consequences of ECC-protected memory errors in modern computing systems, and how effective are the mitigations?
status: published
depth: deep
created: 2026-08-08
updated: 2026-08-08
summary: "Error-correcting-code (ECC) memory protects datacenter and AI systems from bit flips that would otherwise corrupt results, but the evidence shows the protection is partial and the threat is changing. Field studies spanning 1979-2026 find real DRAM error rates orders of magnitude above lab estimates (more than 8% of DIMMs affected per year at Google), show that most errors are hard, repeatable faults rather than cosmic-ray soft errors, and document that ECC catches most but not all of them - single-bit-correction codes can miscorrect double-bit errors, on-die ECC hides raw error patterns from operators, and GPU/HBM studies reveal error rates varying by three orders of magnitude across otherwise identical clusters. The largest caveat: the load-bearing field studies are from a handful of large operators (Google, Meta, LANL, BSC, Alibaba), and AI-specific evidence is young, with several key GPU/HBM results still preprints."
disciplines: ["computer systems", "computer architecture", "reliability engineering"]
tags: ["ECC memory", "DRAM errors", "HBM reliability", "GPU memory errors", "silent data corruption", "row hammer", "memory scrubbing", "soft errors"]
source_count: 84
year_range: [1979, 2026]
confidence: moderate
search:
  databases: ["OpenAlex", "Crossref", "arXiv API", "Semantic Scholar (best-effort)", "Unpaywall"]
  queries:
    - "DRAM errors in the wild"
    - "memory errors in the wild"
    - "uncorrectable memory errors"
    - "DRAM failures in the field"
    - "GPU soft errors"
    - "GPU memory errors"
    - "HBM memory errors"
    - "silent data corruption"
    - "memory scrubbing"
    - "on-die ECC"
    - "row hammer"
    - "cosmic rays memory errors"
    - "alpha particle soft errors"
    - "architectural vulnerability factor"
    - "ECC memory reliability"
    - "memory errors in modern systems"
    - "A Realistic Evaluation of Memory Hardware Errors and Software System Susceptibility"
    - "Cycles Cells and Platters An Empirical Analysis of Hardware Failures on a Million Consumer PCs"
    - "A Study of DRAM Failures in the Field"
    - "A Large Scale Study of Soft-Errors on GPUs in the Field"
    - "Silent Data Corruptions at Scale"
    - "RAIDR Retention-Aware Intelligent DRAM Refresh"
    - "Flipping Bits in Memory Without Accessing Them"
    - "Lessons Learned from the Analysis of System Failures at Petascale The Case of Blue Waters"
    - "Virtualized and Flexible ECC for Main Memory"
    - "A Systematic Methodology to Compute the Architectural Vulnerability Factors for a High-Performance Microprocessor"
    - "GPU DRAM errors"
    - "chipkill"
  last_run: 2026-08-08
---


## Summary

Error-correcting-code (ECC) memory is the main line of defense against bit flips in DRAM and HBM, and the literature shows both that it is indispensable and that it is incomplete. Large-scale field studies consistently measure DRAM error rates orders of magnitude above laboratory estimates — Google found more than 8% of DIMMs affected per year [8] — and the dominant mechanism is not cosmic rays but hard, repeatable device faults [13]. ECC catches the overwhelming majority of these errors (Blue Waters logged 99.997% coverage over 1.5 million errors [14]), but single-error-correction codes miscorrect double-bit errors at alarming rates [47], ECC does not cover CPU/GPU datapaths where silent data corruption originates [63], and on-die ECC now hides raw error patterns from operators and researchers [46]. The GPU/HBM era adds a new front: error rates vary by over three orders of magnitude across otherwise similar A100 clusters [29], and fault-injection studies of LLM training show silent degradation that loss-NaN checks cannot see [31]. Confidence is moderate: the field-study core is replicated across operators and systems, but most AI-era results are young preprints, and several foundational papers could only be read at abstract level.

## Why this question

Memory errors used to be a server-reliability footnote — an ECC DIMM costing a few percent more, a background scrub, a page retired. That changed twice. First, the field studies of 2009-2015 showed that real error rates were hundreds of times higher than the radiation-physics community had estimated, and that hard faults — not cosmic rays — dominated; any operator sizing redundancy or checkpoint frequency off the old numbers was wrong by orders of magnitude. Second, AI training and inference turned memory into the workload: HBM is the largest and least-repaired memory subsystem ever deployed at scale, a single corrupted weight can silently poison a multi-million-dollar training run, and vendors are now actively proposing to weaken or remove on-die ECC to cut HBM cost [32].

For an infrastructure architect, the practical questions are: how often do ECC errors actually happen, what do they do to applications, what does ECC actually cover, and where should the next layer of defense live — stronger codes, prediction, or application-level fault tolerance? This review assembles what 47 years of measurements, simulations, and architecture proposals collectively establish.

## Scope and methods

**Question.** What does the evidence show about the rate, causes, and system-level consequences of ECC-protected memory errors in modern computing systems, and how effective are the mitigations?

**Inclusion criteria.** Field studies measuring errors in production systems (datacenters, HPC clusters, consumer fleets, GPU farms) of any era; radiation-physics and mechanism papers for DRAM/SRAM soft and hard errors; ECC architecture and coding proposals for main memory, on-die ECC, and HBM; row-hammer and retention studies; silent-data-corruption (SDC) studies tied to hardware faults; error-prediction and mitigation systems; scrubbing theory. Peer-reviewed work plus reputable preprints (AI-era results are frequently preprint-only). Years 1979-2026.

**Exclusion criteria.** ECC in the cryptographic sense (elliptic-curve cryptography); error correction for NAND flash and storage devices; FPGA configuration-memory scrubbing; cache-level ECC microarchitecture (L1/L2); algorithmic SDC detection for numerical HPC (software-level only, no hardware error data); device-level DRAM process papers without field or system implications.

**Search and screening.** 16 concept queries plus 12 targeted title searches across OpenAlex, Crossref, the arXiv API, and Semantic Scholar (best-effort; it 429'd most of the session), followed by citation-graph snowballing (references and citing-works of the 2009 and 2015 landmark studies). 3,112 raw records were merged and deduplicated (DOI, then title+first-author+year) to 3,087; a venue-plus-title screen produced 329 candidates; manual curation against the criteria selected 88; four were dropped at retrieval because no accessible full text or abstract existed (a USENIX paper with no DOI, two 2026 conference papers with no retrievable abstract, and an SSRN/IOP-walled extended abstract). Final included set: **84 sources**. All DOIs were verified to resolve; every load-bearing source was retraction-checked via Crossref (none retracted). Access is honest: 23 sources were read in full text, 61 are abstract-only (classic ACM/IEEE papers are paywalled); claims resting on abstracts are hedged accordingly.

## The landscape

The literature has three distinct eras, and knowing which era a claim comes from matters more than in most fields. **1979-2005** is the physics era: the discovery that alpha particles from packaging contaminate DRAM [2], that cosmic-ray neutrons and muons flip bits at ground level [3], the first ground-level neutron flux measurements [4], and the scaling analyses warning that soft-error rates would rise [5]. **2009-2016** is the field-study era: Google [9], Microsoft's consumer-PC study [10], the LANL/Cielo studies [13], Blue Waters [14], and the first GPU studies [26][27]. **2017-2026** is the AI era: HBM-focused GPU studies [28], silent-data-corruption measurement and detection [64][67], LLM-training fault injection [70][71], and proposals to redesign HBM ECC around inference workloads [33].

Two shape observations. First, the field-study core is produced by a small number of large operators — Google, Microsoft, LANL/OLCF, BSC (Barcelona), Alibaba, Meta — and one research group (Schroeder and collaborators at Carnegie Mellon/Toronto) appears in or anchors most of the CPU-era landmarks; the AI-era GPU studies come from national-lab and university consortia plus Meta. The measurements are therefore concentrated in a handful of fleets, and generalization across operators is an assumption, not a result. Second, the methods split cleanly: field studies measure what ECC reports (correctable/uncorrectable errors), while fault injection and radiation testing measure what ECC would miss — the two literatures rarely cite each other's numbers, which is precisely where the disagreements in this review come from.

## How memory errors arise: mechanisms

The mechanism taxonomy is stable across 47 years. **Alpha particles** from radioactive contaminants in packaging and solder were the first identified cause of soft errors in DRAM [2], and the mechanism persists in modern parts through package materials [5]. **Cosmic-ray neutrons and muons** were identified within months [1], with the first conclusive ground-level demonstration in commercial DRAM coming from AT&T's study of altitude dependence — error rates rose sharply with elevation, and fell to near zero 200 m underground [3]. The neutron flux itself was later measured precisely: a ground-level spectrum spanning over twelve decades of energy [4]. Scaling made the picture more complicated rather than uniformly worse: circuit SER in modern processes has fallen in some structures — 14 nm tri-gate technology improved radiation-induced SER by up to roughly 23× over its 32 nm predecessor [6] — while the sheer number of bits and the shrinking charge per cell keep the aggregate exposure high [5].

The second major class is **hard errors**: permanent or intermittent faults in specific cells, rows, or devices. Heavy ions were shown experimentally to induce hard errors in one-transistor DRAM cells as early as 1994 [7], but the field studies made the hard/soft split quantitative and inverted the popular belief: at Google, the overwhelming majority of logged errors were hard — repeatable on the same addresses — not transient radiation events [12]. This distinction is not academic: hard errors respond to address remapping and page retirement, while soft errors respond to scrubbing and stronger codes.

Two further mechanisms sit outside the radiation/defect framing. **Retention failures** — cells that lose charge faster than the refresh interval — are data- and temperature-dependent, are distributed with heavy tails, and motivated the RAIDR proposal to refresh based on measured retention time (74.6% refresh reduction, 16.1% DRAM power savings in simulation) [22]. The same research line showed that variable retention time (VRT) cells are common enough that in-field detection and mitigation can cut testing cost [21], and that data-dependent failure patterns are spatially correlated, exploitable by techniques like PARBOR [23]. **Row hammer** — charge leakage from repeatedly activated wordlines — is a disturbance mechanism, not a decay mechanism, and is treated separately below because it is largely invisible to ECC.

## Measured error rates in production: CPU and DRAM

The quantitative anchor of the entire field is the Google study: roughly 2.5 years of error logs across a large fleet of commodity servers, six hardware platforms, and millions of DIMM-days. It reported FIT rates of 25,000-70,000 errors per billion device-hours per Mbit — orders of magnitude above the 200-5,000 FIT/Mbit range the lab literature had suggested — with more than 8% of DIMMs affected per year, about a third of machines experiencing at least one memory error per year, and no evidence that newer DIMM generations were more reliable [9]. The follow-up work at the same scale (roughly 300 terabyte-years of additional data) established that most errors are hard, that error-prone DIMMs tend to stay error-prone, and that sacrificing a small amount of DRAM via page retirement can mask most errors at negligible cost [12].

Subsequent studies replicated the headline numbers in different populations. On Cielo (11 months, ~100k DIMMs), 32% of DIMMs had errors and uncorrected errors were dominated by permanent faults [11]; the same group's two leadership-class systems study concluded that counting errors rather than faults systematically misleads reliability analysis, and that DRAM needs stronger resilience while SRAM needs less than commonly assumed [13]. Blue Waters logged 1.5+ million errors over 17 months at an average of ~250 errors/hour, with chipkill/ECC/parity catching all but 28 multi-bit events [14]. Samsung's replacement-network data (~40,000 DIMMs) found 24% of DIMMs with errors, with uncorrectable-error-only DIMMs outnumbering correctable-only ones, and a 6.7× FIT gap between 2,400 Mbps and 1,066 Mbps parts [17]. Alibaba's cloud study (>3 million modules, 8 months) found that correctable errors precede most failures and that most failing modules show a short error burst right before failure — the property that makes prediction feasible [16]. BSC's MareNostrum 3 analysis (3,056 nodes, >25,000 DIMMs, 2,000 billion MB-hours) counted 4.5 million corrected and just 71 uncorrected errors, putting the mean time between uncorrected errors around 10 days compared with ~14 seconds implied by the corrected-error rate — a five-order-of-magnitude gap that is exactly what ECC is buying [19].

Not all evidence comes from servers. Microsoft's study of a million consumer PCs found that hardware failures are strongly recurrent — a machine that crashed once had up to two orders of magnitude higher risk of crashing again — and that memory errors were among the more common hardware faults [10]. In Linux systems, fault injection showed that many soft errors are never activated (the overwritten data is dead), so software recovery can avoid many reboots [15]. And the spatial structure matters operationally: in a large HPC cluster, nodes with high error counts cluster in space, suggesting shared manufacturing or environmental causes [18].

## GPU and HBM errors in the field

The GPU era began with two findings that set expectations. Folding@home's volunteer fleet (50,000+ hosts) showed that roughly two-thirds of GPUs exhibited pattern-sensitive soft errors while lab controls showed none — an early warning that GPU memory had idiosyncratic failure modes [27]. On Titan, GPU-related failure events occurred on average about once every two days across 18,688 GPUs — far below the >2 failures/day a naive vendor-MTBF estimate would predict — with off-the-bus errors, ECC page retirement, and double-bit errors dominating [26]. Neutron-beam testing in the same study found double-bit error rates of ~6% (Fermi) to ~1% (Kepler) of events, with no triple-bit corruptions [26]. The Titan dataset also produced the first large-scale characterization of GPU soft-error classes by type [25], and machine-learning prediction on the same fleet reached F1 0.81 (recall 0.87) for single-bit-error prediction [43].

The Ampere-generation studies changed the resolution. A 2025 study of 10,693 NVIDIA A100/A40 GPUs across three national-lab supercomputers (Delta, Polaris, Perlmutter) — 67.77 million GPU device-hours — found single-bit error rates from 0.53 to 2.83 per GPU per day and double-bit rates from 0.0082 to 0.069, with error rates varying by over three orders of magnitude across clusters that are nominally the same hardware, and only 0.24%-5.06% of GPUs ever experiencing errors [29]. The same study found error arrivals bursty rather than Poisson — relevant to checkpointing — and no strong correlation with temperature or utilization. Radiation testing of HBM2 GPU DRAM showed that the right ECC can cut SDC risk by up to five orders of magnitude and uncorrectable errors 7.87× versus SEC-DED [28], while fault-injection studies on GPU architecture show that GPU SDC corruptions are dominated by multi-bit, structured patterns — NaN/Inf account for only ~1% of outcomes and single-bit flips for under 40% of events, in contrast to CPU studies reporting 72-98% single-bit flips [30]. That structural difference matters for both detection (NaN-checks are nearly useless on GPUs) and modeling (single-bit-based fault models understate GPU corruption).

For AI workloads specifically, fault-injection evidence is accumulating fast. LLM inference on A100s shows abnormal outputs in roughly 15-30% of runs at a single injected fault, rising above 75% at eight faults, with larger models more resilient [34]. LLM pre-training fault injection (7,664 runs) found crash rates up to ~60% for faults in forward outputs and backward input gradients, silent perplexity deviations up to ~600% in FP16 (and ~400% in BF16) that loss-NaN checks do not catch, and much lower impact in FP8 [31]. A complementary study found backward-pass exponent flips spiking gradient norms to ~10^19, with detection-plus-recompute restoring baseline loss at roughly 1% training overhead — and cites industry incidents (Gemini-class models encountering SDC-related disruptions roughly every one to two weeks; Meta and ByteDance reporting similar events) as motivation [70]. SDC impact on training correlates with training stability and loss-landscape region, not just fault rate [71], and distributed training frameworks differ measurably in resilience [73].

## What ECC covers — and what it misses

ECC's coverage is the single most consequential fact for system design, and the literature is unusually precise about its limits.

**It catches most memory errors.** SEC-DED (single-error-correct, double-error-detect) is the baseline; chipkill/SDDC corrects full device failures. On Blue Waters, 99.997% of 1.5+ million errors were handled [14]. On Cielo, chipkill cut the node failure rate from uncorrected DRAM errors by 42× versus SEC-DED [11]. The MareNostrum ratio (14 seconds of corrected-error "MTBF" vs 10 days of uncorrected) is the same statement in different units [19].

**It miscorrects.** When a double-bit error lands in a SEC-DED codeword, the code can "correct" it to the wrong value — a silent data corruption. On-die SEC was found to miscorrect double-bit errors more than 45% of the time, and controller-side miscorrection rates above 55% were reported; the COMET scheme eliminates both and corrects 99.9997% of tested patterns [47]. This is the concrete mechanism by which ECC itself becomes an SDC source.

**It does not cover the datapath.** Meta's fleet-scale study detected hundreds of CPUs with reproducible, non-transient silent data corruption across hundreds of thousands of machines monitored for 18+ months — rates orders of magnitude above soft-error FIT simulations, which treat SDC as roughly a one-in-a-million event [63]. ECC reduces SRAM error rates by ~1000× but leaves logic, queues, and datapaths unprotected [63]. Meta's follow-up describes running a large library of silent-error test scenarios across the fleet to find these CPUs [64]. The phenomenon has been documented since at least 2008 [65], and recent work argues the industry still does not know how SDCs manifest in silicon well enough to screen them cheaply [66][72]. Architecturally, the vulnerability of unprotected state is quantified by AVF analysis: a fault in a structure only matters if it reaches program output, and AVF/PVF methodologies exist precisely to size where protection pays [81][82].

**On-die ECC hides the raw error surface.** LPDDR4/DDR5/HBM on-die ECC corrects errors before the memory controller sees them, which means operators and researchers observe post-correction rates. The BEER work reverse-engineered the on-die ECC functions of 80 real LPDDR4 chips and showed that pre-correction error rates can be reconstructed — but only with significant effort, and the functions are trade secrets [46]. Every GPU field study above measures what ECC reports; the raw HBM error rate is systematically under-observed.

**What a fault does to an application varies wildly.** Injected hard-but-correctable faults slowed SPEC CPU2006 by up to ~2.5× in the worst cases [74]. Cooperative OS/application recovery can keep systems alive through uncorrectable errors by containing damage [75], and in-memory key-value stores can be surgically pruned of objects damaged by uncorrectable ECC errors rather than crashing [76].

## ECC architecture: from SEC-DED to domain-specific HBM codes

The architecture literature attacks two problems: making ECC stronger without proportional cost, and making ECC cheaper. The classic proposals are well replicated in simulation: virtualized ECC achieves near-chipkill protection with ~1% performance overhead on ECC DIMMs and up to 27% DRAM power savings [44]; LOT-ECC delivers chipkill-level protection on commodity DRAM with up to 44.8% lower power and 46.9% lower latency [45]; ARCC and multidimensional-parity chipkill variants cut code overhead and power (36% average power reduction; up to 41% at 1% extra code) [48][49]; configurable ECC trades protection per access for energy and FIT reductions (17-21% energy, 20× FIT) [57]; Stealth ECC and Twin ECC reduce system failure probability by ~48% and ~50-65% respectively with sub-1% performance overhead and no storage overhead [51]. Scrubbing theory quantifies the other half of the bargain: periodic scrub sweeps keep soft-error accumulation below the correction threshold, with ordering strategies roughly doubling effective MTTF [83][84].

The HBM era shifts the design point. HBM's on-die ECC (SEC-DED at the die level) is being extended in several directions: error-pattern-aligned codes that beat SEC-DED on real HBM multi-bit patterns without extra redundancy [53], prediction-based two-tiered ECC for stuck-word-line errors [52], DBB-ECC correcting burst plus random double-bit errors for HBM3 [54], ROSE combining 100% double-bit correction with metadata ECC for SDC detection [55], and CXL-attached memory ECC delivering >10^9 reliability gain with bandwidth overhead cut from 63.5% to 3.4% [56].

The most provocative thread is explicitly AI-motivated: because LLM inference tolerates a modest bit error rate in weights (up to ~10^-3 raw BER), two proposals argue that HBM's fixed on-die ECC is the wrong abstraction. One keeps on-die ECC but adds a controller-managed long Reed-Solomon outer code, extending tolerable device error rates by about three orders of magnitude while keeping ~79% of on-die-ECC throughput [32]; the other removes on-die ECC entirely, using large-codeword Reed-Solomon plus CRC, retaining >78% of throughput and ≥97% PIQA / ≥94% MMLU accuracy at 10^-3 raw BER — while noting that exponent-only corruption is the catastrophic case (PIQA down to 61.7%, MMLU to 38.1%) and must be protected [33]. Both are simulations with synthetic error injection, and both make the same underlying claim: HBM reliability can be traded against cost per byte for inference workloads. That is a genuinely new design space, and the field has not yet measured whether the raw error rates of shipping HBM3/HBM4 actually sit in the range these proposals assume.

## Row hammer and retention: ECC-blind disturbance failures

Row hammer is the case where ECC is structurally the wrong tool: the attacker (or the workload) deliberately induces errors, and no error-correcting code can distinguish a hammered bit from a radiation event. The original study found disturbance errors in 110 of 129 commodity modules (836 of 972 chips) from three manufacturers, with all 2012-2013 modules vulnerable, as few as ~139K wordline activations causing an error, and up to 1 in 1,700 cells susceptible [58]. The phenomenon matured into a security industry: Google Project Zero's 2015 user-to-kernel privilege escalation, remote and VM-based attacks, and the retrospective literature documenting that >80% of tested modules were vulnerable and the activation threshold fell to ~10K [59][60]. Software mitigations are defeatable [61]; real production systems have logged row-hammer-induced errors that no standards body was addressing at the time [24]; and the mechanism family continues to expand — RowPress, a retention-adjacent disturbance mode in sub-20 nm DRAM, was characterized via TCAD simulation in 2024 [62]. ECC's role here is containment (a hammered row produces correctable errors until it doesn't), not prevention; the literature's mitigation work is architectural (refresh, shuffling, counters), which is outside this review's scope.

## Predicting, detecting, and recovering at the system level

Because uncorrectable errors are rare but catastrophic, a substantial literature tries to predict them. The consistent finding is that correctable-error telemetry is predictive. LLNL's physics-informed models on Cielo/Hopper data (1.1+ million correctable errors) reached F1 up to 0.89±0.01, with spatial locality stronger than temporal [37]. BSC's cost-aware prediction on MareNostrum 3 reduced lost compute time up to 57% (~21,000 node-hours/year saved) and established that only 67 of 333 uncorrected errors had reliability impact — most are harmless if handled quickly [35]; an adaptive reinforcement-learning mitigation then cut lost compute time 54% versus no mitigation [36]. Workload-aware models showed error rates varying 8× across workloads and 188× across chips, with prediction error ~10% versus 2.9× for workload-unaware baselines [38]; the same group's ML-driven governor relaxed refresh/voltage guardbands to cut DRAM power 24% with failure probability bounded below ~10^-45 [39]. Alibaba found that CE storms drive 41% of DRAM-caused node unavailability, and that hybrid rule-plus-ML prediction beats pure ML by >40% on unavailability reduction [40]; a 2026 study on >130,000 DDR4 DIMMs reached F1 of 49.9-57.6% for failure prediction [41]. CARE showed that near-chipkill resilience can be delivered from the memory controller with ~58KB of state and no capacity penalty [42].

For SDC specifically, detection is the hard open problem. Phoebe's microarchitectural fault-injection methodology (with >1000× speedup over prior injection) is one systematic approach [67]; Druto bounds SDC vulnerability in GPU applications with estimates up to 74× higher than prior methods [80]; others argue mitigation must be tuned to availability-reliability trade-offs approaching information-theoretic bounds [68]. Cloud providers report that CPU-oriented SDC detection does not transfer to GPU-based AI datacenters [69]. GPU program resilience varies dynamically within executions, complicating static analysis [79], and the efficacy of ECC itself on GPUs depends on transistor layout and fault type [77][78].

## Where the evidence disagrees

**Are DRAM errors mostly hard or soft?** The Google studies found the majority of errors hard and repeatable [12], and the Cielo studies found permanent faults dominating uncorrected failures [13]. The radiation-physics literature — and much industry folklore — centers cosmic rays [3]. The disagreement dissolves on method: field studies count *correctable errors logged by ECC*, which are dominated by a small number of sick devices (a few DIMMs generate most events), while radiation experiments measure *per-bit cross sections* under accelerated beams. Both are true; the reconciliation is that radiation dominates at the level of individual bits across a fleet (soft errors are everywhere at low rate), while hard faults dominate at the level of DIMMs and machines (a few devices produce most of the load). The recent BSC cosmic-ray study pushes further: at two Barcelona systems, neutron counts showed no influence on DRAM error rates at all [20] — a claim that should be read against its setting (sea level, ~100 m altitude), since the classic altitude-dependence result [3] is not in dispute.

**Is SDC rare or systemic?** Fault-injection-based FIT analysis treats SDC as roughly one-in-a-million events [63], while field screening found hundreds of reproducible SDC CPUs in one fleet [64]. The resolution is that the two literatures measure different things: injection measures random faults in a model; the field detects latent manufacturing defects (e.g., aging or marginal cells) that are non-random and concentrated in specific units. The disagreement is therefore about what fraction of real hardware carries such defects — and that number is still not public at scale.

**How much does ECC cost in performance?** Simulation-era proposals report sub-1% overheads [50][51], while the HBM proposals report 20-25% throughput deltas between on-die ECC and their alternatives [33]. These are different measurement objects (controller energy vs inference tokens/s), and neither camp has measured on real silicon at fleet scale.

## Gaps and open questions

- **Raw HBM error rates are unmeasured in the public literature.** Every field study sees post-ECC numbers [46]; the pre-correction rates that the ECC-reduction proposals assume [33] are inferred, not measured. The BEER methodology [46] is the only public path, and it predates HBM3.
- **No large-scale HBM3/HBM4 field study exists.** The Ampere study is HBM2E [29]; vendor data is not public. The entire "trade ECC for cost" design space hangs on this.
- **SDC defect prevalence is unknown.** Meta published detection counts but not rates [63]; no operator has published SDC per-million-unit rates. This single number would settle how much application-level fault tolerance AI systems need.
- **LLM tolerance thresholds are model- and format-dependent** [34][71]; the field lacks a fault-tolerance spec that hardware vendors could design to.
- **The 8% DIMMs/year figure is 17 years old** [8]; nothing of comparable scale has been published since — Alibaba [16] and BSC [19] are the closest, and neither covers modern DDR5.

## Confidence and limitations

The core claims — error rates far above lab estimates, hard-error dominance, ECC's high but incomplete coverage, chipkill's 1-2 order-of-magnitude gains, row-hammer's universality — are replicated across operators, systems, and eras, and carry high confidence. Claims specific to AI systems (HBM rates, LLM training impact, ECC-reduction proposals) rest on young preprints and simulations, and carry moderate-to-low confidence; they are labeled as such throughout.

This review's own limitations: 61 of 84 sources were read at abstract level (the classic ACM/IEEE field studies are paywalled), so fine-grained numbers from those are hedged; the search was English-only; the date cutoff is August 2026; gray literature (vendor whitepapers, errata) was deliberately excluded, which means the operational picture is missing vendor-side data; and the field-study population is concentrated in a handful of operators' fleets. One source (zivanovic2019dram) was dropped at DOI stage for lacking a DOI; two 2026 conference papers and one IOP-walled extended abstract were dropped as unretrievable. Where a finding rests on a single source, it is stated as such.

## Evidence table

| key | design | sample | measure | finding | limitations | confidence | access | note |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| schroeder2009dram | field study | Google production server fleet, 6 hardware platforms, ~2.5 years (Jan 2006-Jun 2008), many millions of DIMM days, DDR1/DDR2/FBDIMM DIMMs at 1/2/4 GB from multiple vendors | correctable (CE) and uncorrectable (UE) DRAM error rates: FIT per Mbit, % of DIMMs affected per year, CE/UE per machine per year | reports FIT rates of 25,000 to 70,000 errors per billion device hours per Mbit and more than 8% of DIMMs affected by errors per year, with about a third of machines experiencing at least one memory error per year and an average of over 22,000 correctable errors per year. | measurement infrastructure cannot reliably distinguish hard from soft errors; temperature values reported only normalized because absolute levels are sensitive; errors logged every 10 minutes may miss some events | high | full-text | Foundational large-scale DRAM field study establishing real-world error rates orders of magnitude above lab-based estimates and dominance of hard errors |
| zhu2025understandin | field study | 10,693 NVIDIA Ampere GPUs (A100/A40) on three supercomputers (Delta, Polaris, Perlmutter), 67.77 million GPU device-hours of ECC-reported error logs | single-bit (SBE) and double-bit (DBE) ECC error rates per GPU per day, mean-time-between-errors (MTBE), interarrival time distributions | reports observed GPU error rates varying by over three orders of magnitude across the three clusters, with SBE rates of 0.53 (Polaris) and 2.83 (Perlmutter) per GPU per day, DBE rates of 0.069 and 0.0082 respectively, and only 0.24%-5.06% of GPUs experiencing errors. | A100 HBM2 only (no HBM3 or newer GPUs); Perlmutter logs at 1-hour granularity force aggregation; no strong correlation found with temperature/power/utilization but causal analysis not possible | high | full-text | Large-scale cross-supercomputer A100 GPU HBM reliability characterization; shows cluster-to-cluster rate variability and bursty error patterns relevant to checkpointing |
| dixit2021silent | field study | Facebook datacenter fleet of hundreds of thousands of servers; silent-error test scenarios run across hundreds of thousands of machines, monitored for longer than 18 months | number of CPUs detected with silicon-defect silent data corruption (SDC); comparison of observed CPU SDC rates vs soft-error FIT simulations | reports hundreds of CPUs detected with reproducible, non-transient silent data corruption across the fleet, and observes CPU SDCs at rates orders of magnitude higher than soft-error based FIT simulations (which treat SDCs as ~one in a million occurrences); ECC reduces SRAM error rates by 1000x but does not cover all datapaths. | no quantitative rate (e.g., SDCs per million CPUs) or detection coverage statistics reported; performance-vs-detection tradeoff costs deferred to a follow-up publication; one detailed case study presented | moderate | full-text | Industry evidence that SDC is a systemic, repeatable silicon-defect problem beyond DRAM ECC coverage, motivating system-level fault tolerance |
| tiwari2015understandin | field study | All 18,688 GPUs of the Titan supercomputer (OLCF) plus a LANL GPU cluster; neutron-beam experiments at LANSCE and ISIS on Kepler K20 and Fermi C2050 GPUs | GPU failure event frequency, single/double-bit ECC errors, per-bit and per-structure neutron cross sections | reports GPU-related failure events on Titan occur about once every two days on average (much lower than the >2 failures/day expected from vendor MTBF), with off-the-bus, ECC page retirement and double-bit errors dominant; neutron tests found DBE rates of ~6% (close to 1% for Fermi) with no triple-bit corruptions, and Kepler showing better per-bit reliability than Fermi. | ECC-protected structures only in field data (unprotected logic errors not logged); neutron flux much higher than sea level; thermal neutron component at ISIS; vendor resilience details business-sensitive | high | full-text | First large-scale GPU field study plus radiation testing; shows SECDED ECC suffices for most radiation-induced GPU memory bit corruptions |
| kim2014flipping | fault injection | 129 commodity DRAM modules (972 DRAM chips) from three major manufacturers, including FPGA-based testing platform and real Intel/AMD systems | presence and rate of DRAM disturbance (row hammer) errors; minimum activations to induce an error; fraction of disturbable cells | reports disturbance errors induced in 110 of 129 modules (836 of 972 chips) from three manufacturers, with all modules manufactured in 2012-2013 vulnerable, as few as 139K wordline activations causing an error, and up to one in every 1.7K cells susceptible. | results are for sampled modules and may not generalize to all DRAM; mechanisms hypothesized (coupling, bridges, hot-carrier injection) not definitively confirmed at device level | high | full-text | The original RowHammer discovery paper; demonstrates a circuit-level disturbance mechanism that defeats memory isolation without ECC visibility |
| mutlu2019rowhammer | survey | Retrospective synthesis of the RowHammer literature; reproduces Kim et al. 2014 data on 129 DRAM modules (2008-2014 manufacture dates) from three manufacturers | RowHammer error rates (errors per 1e9 cells) vs manufacturing date; survey of attacks and mitigations | reports that 110 of 129 tested modules exhibited RowHammer errors with all 2012-2013 modules vulnerable (error rates up to ~1e6 errors per 1e9 cells), that Google Project Zero demonstrated user-level kernel privilege escalation in 2015, and that many follow-up attacks enable remote/virtual-machine/mobile takeover. | retrospective/secondary source for the underlying measurements; primary quantitative data from the original 2014 paper rather than new experiments | moderate | full-text | Widely-cited survey establishing RowHammer as the canonical circuit-level failure mechanism with system security consequences |
| patel2020bit | fault injection | 80 real LPDDR4 DRAM chips with on-die ECC from three major manufacturers; simulation of 115,300 single-error-correction Hamming codes with word lengths 4-247 bits | ability to recover the full on-die ECC parity-check matrix (BEER); bit-exact pre-correction error recovery (BEEP); SAT solver runtime/memory | reports BEER successfully identified the on-die ECC functions of 80 real LPDDR4 chips (different manufacturers use different functions) and correctly recovered functions for 115,300 simulated Hamming codes, with median SAT runtime of 57.1 hours and 6.3 GiB memory for representative 128-bit codes. | no ground truth available because on-die ECC functions are trade secrets; uncovered ECC functions cannot be published for confidentiality; experiments limited to LPDDR4 chips with on-die ECC | moderate | full-text | Shows on-die ECC obfuscates raw DRAM error patterns and provides a method to reconstruct bit-exact pre-correction error rates, complicating third-party ECC error studies |
| boixaderas2024dram | field study | MareNostrum 3 production supercomputer (3,056 nodes, >25,000 DDR3-1600 DIMMs, Chipkill ECC) with 2,000 billion MB-hours and 4.5 million corrected plus 71 uncorrected errors; Mont-Blanc prototype (LPDDR3-1600, no ECC) with 135 million MB-hours and 25 million errors | correlation between cosmic-ray neutron counts and corrected/transient/uncorrected DRAM error rates, using quantitative analysis, statistical tests and machine learning | reports no indications that cosmic rays have any influence on DRAM errors in either cluster, despite 4.5M corrected and 71 uncorrected errors over 2000 billion MB-hours on MareNostrum 3 and 25M errors over 135 million MB-hours on Mont-Blanc. | both systems located in Barcelona at 100 meters above sea level (low altitude); conclusions may not generalize to high-altitude clusters; MCA registers sampled every 100 ms so sub-interval error details are limited | high | full-text | Challenges the widely assumed cosmic-ray explanation for DRAM error rates with large-scale production data and a repeatable methodology |
| tung2026anatomy | fault injection | Gate-level single stuck-at fault injection on a synthesized production-class data-center GPU model (reduced two-SM configuration) consuming over 3 million simulator hours across 63 CUDA micro-benchmarks | SDC outcome distribution (nullified vs bit-flipped vs NaN/INF), bit-flip count distribution, warp-aligned spatial periodicity of corruption addresses | reports NaN/+INF/-INF account for only 1.01% of SDC outcomes, single-bit flips constitute less than 40% of bit-flip events (contrasting with CPU reports of 72-98% single-bit flips), and corruption addresses exhibit warp-aligned periodicity. | simulation only (no silicon data); stuck-at fault model approximates real defects; reduced 2-SM configuration and micro-benchmarks may not capture full GPU behavior | moderate | full-text | Shows GPU SDC corruption is dominated by multi-bit, structured, non-special-value patterns, undermining single-bit and NaN-based fault models |
| tyagi2026llm | simulation | 7,664 LLM pre-training runs using RTL-level GPU fault characterization coupled with a stochastic injection engine in Megatron-LM, across FP16, BF16 and FP8 regimes (small models: GPT-2 Small/Medium) | perplexity deviation, crash/divergence rate, downstream task accuracy (CBT, Winograd) under permanent/intermittent faults vs fault type, rate, phase, and numeric format | reports that while LLMs resist low-frequency faults, impact is highly non-uniform: forward-output and backward-input-gradient faults yield the highest crash rates (~60%), non-diverged FP16 runs show PPL deviations up to ~600% (BF16 up to ~400%, FP8 under 1%), and loss-NaN checking converts many crashes into spike-and-recover but leaves silent degradation. | small-scale models only; conclusions depend on fidelity of hardware fault models; single distributed training configuration (sequence parallelism) may not capture other parallelism strategies | moderate | full-text | First hardware-grounded characterization of LLM pre-training SDC resilience; shows critical-datapath and precision-format-dependent catastrophic divergence and NaN-detector blind spots |
| xie2025making | architecture proposal | REACH controller-managed two-level Reed-Solomon ECC for HBM evaluated on three LLMs (LLaMA-3.1-8B, Voxtral-Mini-3B, Qwen3-4B) at 8K context; ASAP7 7nm controller implementation at 3.56 TB/s | qualified tokens/s vs raw bit error rate (BER), throughput relative to on-die ECC, ECC area and power, uncorrectable error probability | reports REACH keeps about 79% of on-die ECC throughput at BER=0 (110.1 vs 139 tokens/s for LLaMA-3.1-8B) and stays qualified up to raw BER of 1e-3, extending tolerable device error rates by about three orders of magnitude while keeping tokens/s nearly flat, with a controller of 15.2 mm2 and 17.5 W that reduces ECC area 11.6x and power ~60% vs a naive long-RS baseline. | design evaluated by simulation with synthetic BER fault injection, not silicon or real field error traces; inference workloads only; depends on typical LLM access patterns (low random-access ratios) | moderate | full-text | Proposes moving HBM reliability from fixed on-die ECC to controller-managed long codes, trading higher raw HBM BER for lower $/GB in AI inference |
| xie2025breaking | architecture proposal | Domain-specific ECC framework (large-codeword Reed-Solomon + fine-grained CRC, no on-die ECC) evaluated on LLM inference workloads (LLaMA-3.1-8B, Voxtral-Mini-3B, Qwen3-4B) on PIQA and MMLU | inference throughput (tokens/s) and accuracy (PIQA, MMLU) vs raw HBM bit error rate up to 1e-3, relative to error-free HBM | reports the system retains over 78% of throughput and at least 97% PIQA and 94% MMLU accuracy relative to error-free HBM even at raw HBM bit error rates up to 1e-3, and models that increasing codeword size from 32B to 2KB raises tolerable raw BER by more than five orders of magnitude. | simulation-based evaluation with synthetic BER, not measured field error rates; inference-only scope; exponent-only corruption at 1e-3 can drop PIQA to 61.7% and MMLU to 38.1%, so tunable/importance-based protection is required | moderate | full-text | Companion HBM cost-reduction proposal eliminating on-die ECC entirely; quantifies accuracy/throughput headroom under relaxed raw BER for AI inference |
| zivanovic2019dram | field study | MareNostrum 3 supercomputer: 3056 servers, more than 25,000 DIMMs from all three major manufacturers in three DRAM technologies; 2000 billion MB-hours; observation Oct 2014-Nov 2016 (25 months); 4.5 million corrected and 71 uncorrected DRAM errors | Corrected vs uncorrected DRAM error counts, errors per MB-hour, MTBF, percentage of DIMMs affected per manufacturer/technology, statistical significance (independence tests) | Over 2000 billion MB-hours on MareNostrum 3, 4.5 million corrected but only 71 uncorrected DRAM errors were detected, yielding an uncorrected-error MTBF of about 10 days versus a corrected-error-based MTBF of 14 seconds (five orders of magnitude difference), with average error rates volatile even after a year of logging. | Authors: very small number of uncorrected errors (71) makes their rates unstable; corrected errors are a misleading reliability indicator; pre-failure DIMM retirement may bias data; single system. | high | full-text | Anchor field study: quantifies corrected vs uncorrected DRAM error incidence and MTBF, and warns that standard error-rate metrics are volatile without statistical rigor. |
| boixaderas2020cost | ML model | MareNostrum 3: 3056 nodes, more than 25,000 DDR3-1600 DIMMs (6694/5207/13419 per manufacturer A/B/C), SDDC (chipkill) ECC; 2 years of production logs (Oct 2014-Nov 2016), 2000+ billion MB-hours; 4.5 million corrected and 333 uncorrected errors, of which only 67 had reliability impact | Lost compute time (node-hours) saved by predicting and mitigating uncorrected DRAM errors; precision/recall shown insufficient vs cost-benefit analysis | A random forest trained on two years of MareNostrum 3 logs predicts uncorrected DRAM errors well enough to reduce lost compute time by up to 57%, a net saving of 21,000 node-hours per year, and only 67 of 333 detected uncorrected errors (one fifth) actually impacted reliability. | Authors: single system; only 67 impactful UEs for training/evaluation; bias from pre-failure DIMM retirement; standard metrics (precision, recall, F1) do not correlate with saved compute time. | high | full-text | Shows uncorrected-error prediction is practically actionable and that cost-benefit, not precision/recall, should evaluate predictors. |
| boixaderas2024reinforcemen | ML model | MareNostrum 3 error and job logs: 3056 nodes, more than 25,000 DDR3-1600 DIMMs (6694/5207/13419 per manufacturer), SDDC ECC; more than two years of production (Oct 2014-Nov 2016); 4.5 million corrected and 333 uncorrected errors; time-series cross-validation from untrained model | Lost compute time (node-hours) under RL-triggered mitigation vs no mitigation and vs optimal Oracle; classical ML metrics plus cost-benefit analysis | The first adaptive reinforcement-learning mitigation scheme reduces lost compute time by 54% compared with no mitigation (a saving of more than 40,000 node-hours over two years) and is just 6% below the optimal Oracle method on MareNostrum production logs. | Authors: evaluation on historical logs with assumed mitigation costs; only the first UE in a burst impacts reliability; single-system dataset; applicability to other systems argued but not field-tested. | high | full-text | First adaptive RL-based uncorrected-DRAM-error mitigation; quantifies achievable savings relative to an Oracle bound. |
| nie2018machine | ML model | Titan supercomputer: 18,688 NVIDIA K20X GPUs; six-month trace data (Feb-Jun 2015), more than 60 million node hours; single-bit errors (SBEs) collected via nvidia-smi at batch-job granularity | GPU single-bit-error prediction quality: F1 score, precision, recall of machine learning models (LR, GBDT, SVM, NN) using spatial/temporal features | On six months of Titan production traces (18,688 GPUs, >60 million node hours), the GBDT-based predictor achieved an F1 score of 0.81 with recall 0.87 and precision 0.76 for GPU SBE occurrence, significantly outperforming other models. | Authors: SBE snapshots lack per-event timestamps (job-level granularity); imbalanced data; double-bit errors excluded as statistically unsuitable; single GPU platform (K20X); ECC overhead cited up to 10% on GPUs motivates prediction. | moderate | full-text | Large-scale GPU ECC soft-error prediction in production HPC; motivates dynamic ECC on/off via error prediction. |
| mukhanov2019workload | ML model | 72 server-grade DRAM chips on a real 64-bit ARM server; 249 program-inherent features (memory access rate, cache misses, reuse time, data entropy) from compute-intensive, caching and analytics benchmarks; DRAM tested under scaled refresh period and lowered supply voltage and varied temperature | Single- and multi-bit DRAM error rate prediction accuracy (average prediction error %) and workload-induced error rate variation (x-fold) | On 72 server-grade DRAM chips, single- and multi-bit error rates varied across workloads by up to 8x (and up to 188x across chips), and a KNN-based workload-aware model predicted per-module error rates within 10.2% average error versus a 2.9x estimation error for a conventional workload-unaware model. | Authors: controlled stress characterization under relaxed refresh/voltage, not nominal conditions or production logs; single server platform; error behavior induced by deliberately scaled parameters. | moderate | full-text | Quantifies workload dependence of DRAM error rates (8x) - key evidence that program behavior materially affects DRAM reliability. |
| mukhanov2020revealing | ML model | Real server with DDR3 memories (temperatures below 50 C); workload-aware ML DRAM error model from program features; temperature-driven OS governor setting module-specific marginal refresh period under lowered supply voltage | DRAM power reduction from relaxed refresh/voltage guardbands while minimizing failure probability; crash probability bound | The ML-based temperature-driven OS governor reduces DRAM power by 24% on average without triggering any errors, with a mechanism bounding the probability of a system crash at 1 - 4x10^-45, achieving near-maximum power savings for DDR3 below 50 C. | Authors: single server/DDR3 platform; requires a DRAM characterization campaign; power savings demonstrated below 50 C; error-free operation shown for the evaluated workloads only. | moderate | full-text | Shows ML workload-aware models can safely relax DRAM guardbands (24% power saving) - reliability/power tradeoff evidence. |
| baseman2018physics | ML model | Field data from two DOE supercomputers: Cielo (8,944 nodes, 286 TB DDR3, over 700 thousand correctable DRAM error events over 4.5 years, chipkill) and Hopper (6,384 nodes, 212 TB DDR3, over 400 thousand correctable error events in first 22 months); six ML models incl. physics-informed Markov random field | F1 score, false positive and false negative rates for predicting whether a DRAM location will experience an error, including cross-machine train/test transfer | Statistical ML models predict the likelihood of a DRAM location erroring with F1 up to 0.89 +/- 0.01 on Cielo/Hopper field data, and a physics-informed Markov random field outperforms purely statistical models (near-zero false positives) and transfers across machines, showing spatial locality matters more than temporal locality. | Authors: correctable errors only; transferability shown between two DOE systems; F1 on a rare-event prediction task; conference paper with results largely in figures (text reports the 0.89 +/- 0.01 headline). | moderate | full-text | Physics-informed ML for DRAM error location prediction; evidence that spatial structure dominates DRAM error prediction and models transfer across systems. |
| mutlu2023retrospectiv | survey | Retrospective of the ISCA 2014 RowHammer study: commodity DRAM modules from the three major vendors (more than 80% vulnerable); follow-up ISCA 2020 work analyzed 1580 DRAM chips of three types from at least two generations | Prevalence of RowHammer vulnerability (% modules affected), number of row activations needed to induce bitflips, influence on industry defenses | The ISCA 2014 study experimentally demonstrated that more than 80% of all tested DRAM modules from the three major vendors were vulnerable to RowHammer bitflips, and without mitigation one can now induce bitflips with orders of magnitude fewer activations (~10K) in cutting-edge chips. | Authors: retrospective, not a new study; original numbers come from 2014-era chips and cited follow-ups; no new field measurements. | moderate | full-text | Documents RowHammer prevalence (>80% of modules) - disturbance-induced bitflips that bypass ECC assumptions in commodity DRAM. |
| bridges2012cooperative | architecture proposal | Workshop position paper (Euro-Par 2011 workshops): cites prior research that uncorrectable DRAM errors occur in up to 8% of DIMMs per year and that exascale systems are expected to suffer several hard and soft errors per day; proposes app/OS framework implemented with Trilinos fault-tolerant GMRES iterative solver | Convergence of a fault-tolerant iterative linear solver under uncorrectable memory errors; framework viability (initial convergence results, no field measurements) | The paper cites prior research reporting uncorrectable DRAM errors in up to 8% of DIMMs per year and argues exascale systems will suffer several hard and soft errors per day, proposing a cooperative app/OS recovery framework with initial convergence results showing fault-tolerant GMRES can recover (no quantitative error-rate results of its own). | Authors: initial convergence results only, no large-scale evaluation; the 8% DIMMs-per-year figure is cited from prior research, not measured; Linux only notifies of scrubber-detected failures. | low | full-text | Early cooperative application/OS recovery proposal for uncorrectable DRAM errors; supplies the 8% of DIMMs per year uncorrectable-error rate citation. |
| chai2025analysis | fault injection | Instruction-level fault injection (NVBitFI) on NVIDIA A100 (80 GB): GPT2 (124M/774M), Llama3.2 (1.23B/3.21B), Qwen3 (0.59B/1.72B) across six datasets (Lambada, PIQA, HellaSwag, WikiText-2, XSum, GSM8K); 1-8 injected bit-flip faults per run; analysis across instruction type, bit position, task difficulty, operator, layer | Abnormal outcome rates (DUE, SDC, masked), instruction vulnerability factor (IVF), approximate model vulnerability factor (MVF) and its deviation from measured values | LLM inference abnormal-outcome rates grew from roughly 15%, 25% and 30% (single injected fault) to over 75% as injected faults increased from 1 to 8, with larger models more resilient (higher masked-error proportion) and the approximate MVF deviating under 24% from experiments for all models except Qwen3-1.7B (29.3%; GPT family under 7%). | Authors: synthetic faults injected at rates far above realistic field rates; inference only (not training); single GPU platform (A100); instruction-level FI abstracts away microarchitecture. | moderate | full-text | First instruction-level fault injection study of LLM inference; quantifies DUE/SDC behavior and vulnerability factors by instruction, bit position, task and layer. |
| altenbernd2026exploring | fault injection | Controlled single-GPU (NVBit-based) fault injection into LLaMA pretraining: 60M, 350M and 1.3B parameter models on C4 dataset; 1,000-10,000 training steps; 12/6/3 seeds per scale; faults into HMMA matrix-multiply instructions; industry anecdotes cited: Google Gemini SDC disruptions roughly every 1-2 weeks, Meta 6 incidents in a 54-day training run | Evaluation loss, parameter difference, pre-clipping gradient norm, max attention logits under injected faults; detection rate, recompute precision, runtime overhead of detection + recomputation | Exponent-bit flips in backward-pass GEMM kernels spike gradient norms from ~2e+0 to 1e+2-1e+19 (up to infinity) and attention logits to 3e+3-3e+20, costing roughly 30-40 training steps per fault, while detection plus recomputing the last step restored baseline loss (60M: 3.61 +/- 0.037 with faults to 3.50 +/- 0.002, equal to baseline) at ~1% runtime overhead. | Authors: single-GPU setting (distributed effects discussed, not evaluated); fault rates are stress tests, not realistic frequencies; LLaMA-family models only; detection evaluated on backward-pass kernels and bit positions 9-12. | moderate | full-text | Controlled SDC study in LLM training with detection/recompute mitigation; carries industry SDC frequency anecdotes (Gemini ~1-2 weeks; Meta 6 incidents in 54 days). |
| hwang2012cosmic | field study | production systems totaling nearly 300 terabyte-years of main memory, diverse range of systems | DRAM error characteristics: hard vs soft error prevalence, hard error patterns, protection mechanism efficacy | A large fraction of DRAM errors in the field are hard errors, and simple page retirement policies could mask a large number of DRAM errors while sacrificing only a negligible fraction of total DRAM. | Abstract gives no quantitative rates (e.g., error-per-DIMM-per-year); details only in full text | high | abstract-only | Large-scale companion to Schroeder 2009; establishes hard-error dominance and page-retirement mitigation |
| sridharan2015memory | field study | two leadership-class HPC systems (tens of thousands of nodes) | DRAM and SRAM fault/error rates; efficacy of DRAM ECC, DDR address/command parity, SRAM ECC and parity | Counting errors instead of faults leads to incorrect reliability conclusions; DRAM faults will be a major concern needing stronger resilience schemes, while SRAM faults are unlikely to pose a significantly larger threat in future systems. | No quantitative numbers in retrieved abstract text | high | abstract-only | Methodological caution about error-vs-fault counting; DRAM resilience projection for exascale |
| sridharan2012study | field study | 11 months of DRAM errors in a large high-performance computing cluster, nodes with hardware scrubbers | DRAM failure modes, rates, fault types; chipkill vs SEC-DED node failure rate | DRAM failures are dominated by permanent rather than transient faults, large multi-bit (whole row/column) failures occur, and chipkill ECC reduces the node failure rate from uncorrected DRAM errors by 42x compared to SEC-DED ECC. | Single cluster; abstract reports no absolute error rates | high | abstract-only | Canonical SC12 field study; 42x chipkill benefit is a key quantified citation |
| nie2016large | field study | GPU nodes of the Titan supercomputer, large-scale field data | Soft-error rates, characteristics and impact on GPUs in the field | Provides the first large-scale field data analysis quantifying different kinds of soft-errors on Titan's GPU nodes, uncovering previously unknown insights about soft-error characteristics and impact. | Abstract reports no quantitative error rates | high | abstract-only | Key field evidence for GPU/HBM soft errors in HPC |
| nightingale2011cycles | field study | one million consumer PCs, desktops and laptops, 8-month observation window | Hardware failure rates and recurrence for CPU, DRAM and disk subsystems | Hardware-induced failures are recurrent: a machine that crashes once from a hardware fault is up to two orders of magnitude more likely to crash again (e.g., machines with >=30 days CPU time had a 1-in-190 chance of CPU crash, but 1-in-3.3 after a first crash). | Consumer PC telemetry, not datacenter ECC logs; DRAM-specific rates not broken out in abstract | high | abstract-only | Evidence that memory/hardware faults are non-transient and recurrent, motivating proactive replacement |
| dimartino2014lessons | field study | Blue Waters Cray hybrid (CPU/GPU) supercomputer, manual failure reports and event logs over 261 days | Root causes of single-node failures; coverage of memory/processor protection; system-wide outages | Hardware failures were 42% of all failures but only 23% of repair time, because x8/x4 chipkill, ECC and parity handled sustained error rates up to 250 errors/h with 99.997% coverage over >1.5 million errors, with only 28 multiple-bit errors bypassing protection. | Single system (Blue Waters); relies on log quality | high | abstract-only | Quantified evidence that ECC/chipkill absorbs high error rates at petascale |
| messer2004susceptibili | fault injection | commodity PC processors running Linux kernel and a Java virtual machine with sample workloads | Activation of injected memory soft errors and recoverability via software | Many injected soft errors are not activated (mostly due to overwriting), and activated errors that would normally cause reboot need not be fatal if simple software recovery is used. | Small fault-injection study, no field rates; no quantitative results in retrieved text | moderate | abstract-only | Early evidence that software recovery can mitigate soft errors in commodity systems |
| cheng2022depth | field study | eight-month dataset from over three million memory modules in Alibaba production data centers | Correlation between correctable DRAM errors and server failures; ML failure prediction | Correctable DRAM errors of most server failures manifest only shortly before the failure, so failure prediction must run at short intervals; ML-based prediction from DRAM error characterization is feasible, with 14 findings reported. | Abstract gives no prediction accuracy numbers | high | abstract-only | 3M-module scale; links CE error patterns to imminent server failure |
| baeg2019correctable | field study | about 40K DRAM DIMMs over 2.5 years from 23 server types, 3 manufacturers, densities 4-128 GB, speeds 1066-2400 Mbps | Correctable (CE) vs uncorrectable (UE) error counts, FIT by manufacturer and speed | 24% of DIMMs recorded errors (CE-only 28%, UE-only 43%, UE+CE 29% of recorded errors), FIT differed up to 38% across manufacturers, and 2400 Mbps DIMMs had 6.7x the FIT of 1066 Mbps DIMMs. | Replacement-server population; FIT differences partly adjusted for repetitive counts; year/venue metadata missing (s2 record) | high | abstract-only | Quantifies UE vs CE prevalence and speed/manufacturer dependence of DIMM FIT |
| khan2014efficacy | theoretical | conceptual analysis of DRAM retention failures and VRT (Variable Retention Time) | Efficacy of error mitigation techniques for retention failures | Argues that field-time detection and mitigation of retention failures, responsive to VRT cells appearing after assembly, could dramatically reduce testing cost versus manufacture-time testing alone. | Position/analysis paper; no experimental numbers in retrieved text | moderate | abstract-only | Background on VRT retention failures motivating in-field mitigation |
| liu2012raidr | architecture proposal | simulated 8-core system with 32 GB DRAM | DRAM refresh reduction, power, performance, storage overhead | RAIDR achieves 74.6% refresh reduction, 16.1% average DRAM power reduction and 8.6% average system performance improvement at 1.25 KB memory-controller storage overhead. | Simulation-based; assumes retention-time profiling capability | moderate | abstract-only | Shows most DRAM rows retain data far longer than worst-case refresh, enabling refresh skipping |
| khan2016parbor | fault injection | 144 real DRAM chips from three major vendors | Detection of physically neighboring cells; data-dependent failure detection rate | PARBOR detects neighboring cell locations with only 66-90 tests (a 745,654x reduction vs the 49-day naive test) and uncovers 21.9% more failures than a random-pattern test. | Experimental chips, not production field data | moderate | abstract-only | Evidence that data-dependent (pattern-sensitive) failures are detectable and common enough to matter |
| aichinger2015ddr | survey | industry perspective on DDR3 row hammer in cloud servers | Row hammer failure mechanism, mitigations, application testing tool | Row hammer in DDR3 is both a reliability issue and a security risk with no industry standards group addressing it, motivating a unique testing tool to identify applications that can create such failures. | No quantitative data; position paper | low | abstract-only | Industry testimony that row hammer is an unaddressed field reliability/security issue |
| haque2010hard | field study | MemtestG80 run on over 50,000 hosts in the Folding@home distributed network plus controlled lab experiments | Pattern-sensitive memory soft error rate on GPUs | Two-thirds of tested GPUs exhibit a detectable, pattern-sensitive rate of memory soft errors in installed environments, persisting after controlling for overclocking and temperature proxies, though control experiments found no errors. | Volunteer distributed hosts (uncontrolled environment); consumer GPUs without ECC | high | abstract-only | Early large-scale evidence that GPU memory soft errors are widespread in the field |
| sullivan2021characterizi | fault injection | high-energy neutron beam testing of HBM2 memory on a compute-class GPU | Soft error rates and patterns in GPU DRAM; ECC scheme efficacy | Based on beam-test error patterns, novel ECC schemes decrease silent data corruption risk by up to five orders of magnitude relative to SEC-DED and reduce uncorrectable errors by up to 7.87x with no additional redundancy. | Beam testing, not field data; intermittent errors from beam-induced cell damage needed filtering | moderate | abstract-only | Quantifies HBM2 multi-bit error locality and ECC redesign headroom for GPUs |
| yong2026collaborativ | ML model | Alibaba Cloud production DRAM error data; rules + machine learning predictor | DRAM-caused node unavailability (DCNU) prediction, NURR metric | CE storms dominate 41% of DRAM-caused node unavailability, and the proposed rule+ML approach achieves over 40% better NURR than existing methods and runs stably in production. | Abstract gives no dataset size or absolute NURR values | moderate | abstract-only | Shows CE storms, not just UEs, drive node unavailability in hyperscale clouds |
| wang2026predicting | ML model | over 130,000 DDR4 DIMMs from large-scale heterogeneous production clusters over nine months, two Intel x86 architectures, four DRAM vendors | DRAM failure prediction F1-score (CatBoost two-stage model) | The CatBoost-based model achieves F1-scores of 49.9% on Intel x86v5 and 57.6% on Intel x86v6, substantially outperforming existing methods and validating cross-architecture generalization. | Single-vendor (Intel) architectures; abstract lacks precision/recall details | high | abstract-only | Largest recent DIMM-level prediction study; cross-architecture error-pattern differences |
| chen2021care | architecture proposal | simulated memory controller with ~58KB cache-like error-tracking structure | Reliability (vs chipkill), performance and capacity overhead | CARE achieves near-chipkill reliability with ~58KB area overhead in the memory controller, no memory capacity penalty, and negligible performance overhead versus baseline SEC-DED systems. | Simulation-based evaluation; abstract gives no failure-rate numbers | moderate | abstract-only | Cost-effective alternative to chipkill for datacenter DRAM error tolerance |
| yoon2010virtualized | architecture proposal | SPEC CPU 2006 and PARSEC benchmark analysis; ECC DIMM and non-ECC DIMM configurations | Performance overhead, DRAM power, energy-delay product of virtualized two-tier ECC | Virtualized ECC incurs only 1% performance overhead with ECC DIMMs and <10% with non-ECC DIMMs, while enabling DRAM power savings up to 27% and 12% average energy-delay product improvement. | Simulation/analytical evaluation, not field data | moderate | abstract-only | Enables chipkill-class protection even on non-ECC DIMMs at small overhead |
| udipi2012lot | architecture proposal | commodity DRAM systems, memory-controller simulation | Power, latency, fault tolerance of localized multi-tier ECC | LOT-ECC provides strong fault tolerance while activating few chips, reducing power by up to 44.8% and latency by up to 46.9% versus conventional chipkill schemes. | Simulation; abstract gives no reliability quantification | moderate | abstract-only | Shows chipkill-level protection need not activate all chips on every access |
| alam2022comet | architecture proposal | analytical/design study of on-die SEC + in-controller SECDED ECC interaction | Double-bit error miscorrection rate and silent data corruption | On-die SEC miscorrects double-bit errors into triple-bit errors more than 45% of the time, miscorrected in the controller >55% of the time causing SDC, while COMET eliminates all such SDC and corrects almost all (99.9997%) double-bit errors with no added redundancy. | Design/theoretical evaluation; numbers are analytical, not field-measured | moderate | abstract-only | Quantifies a new SDC vulnerability from on-die ECC + controller ECC interaction |
| xunjian2013adaptive | architecture proposal | simulated commercial SCCDCD chipkill-correct memory, page-level adaptation | Memory power vs reliability trade-off of adaptive chipkill strength | ARCC reduces memory power by 36% on average when applied to commercial SCCDCD while keeping storage overhead the same and maintaining similar reliability, citing field studies showing chipkill reduces uncorrectable error rate by 4X-36X vs SECDED. | Simulation-based; relies on cited (not own) field reliability numbers | moderate | abstract-only | Exploits the observation that only a tiny fraction of memory faults during a server's lifetime |
| jian2013high | architecture proposal | PARSEC and SPEC benchmark simulations, conventional double chipkill correct baseline | Power, code overhead, rank size of multidimensional-parity chipkill | The proposed organization halves code overhead versus conventional chipkill for the same rank size, and provides double chipkill correct at half the rank size with up to 41% (32% average) power reduction at only 1% additional code overhead. | Simulation-based; no field error data | moderate | abstract-only | Decouples local-fault and device-fault correction to cut chipkill cost |
| lee2022stealth | architecture proposal | simulated DRAM memory system, x4 chips with bitwise interleaving | System failure probability and performance overhead vs SECDED | Stealth ECC reduces the probability of system failure caused by DRAM errors by 47.9% on average with only 0.9% performance overhead and no storage overhead compared to conventional SECDED. | Simulation-based evaluation | moderate | abstract-only | Data-width-aware adaptive ECC leveraging narrow-width values for multi-bit correction |
| bae2023twin | architecture proposal | simulated 512-bit data ECC DIMM with SECDED baseline | System failure probability reduction and performance overhead | Twin ECC reduces system failure probability by averages of 64.8%, 56.9% and 49.5% when the '1'-to-'0' error portion is 100%, 90% and 80%, with only 0.7% performance overhead and no storage overhead. | Simulation; assumes dominance of '1'-to-'0' error pattern | moderate | abstract-only | Exploits asymmetric DRAM error polarity via data duplication + bitwise OR |
| moon2024novel | architecture proposal | HBM two-tiered ECC architecture (OD-ECC + S-ECC) design with two interleaved Reed-Solomon engines | Error correction capability for subwordline driver (SWD) errors | The proposed prediction-based OD-ECC with data-deinterleaving S-ECC significantly enhances correction capability for SWD errors while preserving correction for other error types and keeping check-bit size unchanged. | No quantitative results in retrieved text | moderate | abstract-only | Targets SWD errors, a primary HBM error cause, without extra check bits |
| kwon2023epa | architecture proposal | HBM2E OD-ECC design informed by soft error experiments on HBM2 | Memory reliability of OD-ECC for multi-bit error patterns | Recent soft error experiments on HBM2 show DRAM frequently experiences multi-bit errors, and EPA ECC's RS-based multi-bit-symbol design provides higher reliability than SEC-DED OD-ECC without increasing redundancy or significant performance degradation. | No quantitative results in retrieved text | moderate | abstract-only | Aligns OD-ECC symbols with observed HBM multi-bit error patterns |
| shin2025dbb | architecture proposal | HBM3 on-die ECC design (SSC RS code baseline) | Correction of burst and random double-bit errors; decoder implementation overhead | DBB-ECC corrects both single-symbol (burst) and random double-bit errors using SSC-RS syndromes without increasing parity bits, needing lower implementation overhead than conventional schemes while significantly enhancing HBM3 reliability. | No quantitative results in retrieved text | moderate | abstract-only | HBM3 on-die ECC must handle both burst and randomly scattered errors |
| shin2025rose | architecture proposal | HBM3 OD-ECC + S-ECC design with 32-bit metadata | Double-bit error correction rate and silent data corruption detection | ROSE achieves 100% correction of double-bit errors via enhanced OD-ECC (single-symbol + double-bit) and strengthens S-ECC with 32-bit metadata for effective SDC detection, without additional redundancy or performance overhead. | Conference short paper; no quantitative results beyond the 100% claim | moderate | abstract-only | Shows combined OD-ECC/S-ECC can be made resource-efficient for HBM3 |
| liu2025cxl | architecture proposal | CXL Memory eXpander Controller (CXL-MXC) with LRC-based Inter-Channel-ECC, experiments vs RAID-enabled CXL switch | DRAM reliability, bandwidth overhead, system performance | CXL-ECC enhances DRAM reliability by more than 10^9 versus state-of-the-art ECC methods, reduces additional bandwidth overhead from 63.5% to 3.4% relative to a RAID-enabled CXL switch, and improves system performance by 12%. | Simulation/experiment-based; CXL reliability field data still lacking | moderate | abstract-only | First ECC design targeting DRAM reliability in the CXL memory expansion architecture |
| chen2018configurable | architecture proposal | HBM system simulation for GPU memory subsystem, fixed 32B/64B ECC baselines | HBM energy, FIT rate, performance overhead of two-tier Config-ECC | Config-ECC reduces HBM energy by 17-21% and reduces the failure-in-time (FIT) rate by 20 times compared to a state-of-the-art fixed 64B ECC scheme, with an insignificant 1.2% performance overhead. | Simulation-based; FIT reduction is design-level, not field-measured | moderate | abstract-only | Flexible HBM ECC for mixed fine/coarse-grained GPU access granularities |
| tatar2018defeating | architecture proposal | DRAM with software mitigations against Rowhammer; no n reported | ability to induce bit flips while evading software defenses | describes a Rowhammer attack ('Surgical Precision Hammer') that defeats software mitigations by inducing bit flips in DRAM rows with surgical precision; no quantitative results in retrieved text | abstract-only; attack is evaluated in a security context, not a field reliability study | low | abstract-only | Documents Rowhammer bit-flip exploitability and why DRAM reliability (ECC/mitigations) is a security concern |
| zhou2024unveiling | simulation | sub-20 nm DRAM cells; 3-D TCAD simulation; single-sided and double-sided RowPress vs RowHammer | leakage mechanism and bit-flip characteristics of RowPress | reports via 3-D TCAD simulation that RowPress-induced leakage in sub-20 nm DRAM is driven mainly by an increased electric field facilitating electron migration, not the e-trap-assisted EM and capacitive crosstalk mechanisms behind RowHammer; no quantitative results in retrieved text | simulation-only, no experimental validation or error-rate numbers in retrieved text | low | abstract-only | Distinguishes RowPress from RowHammer disturbance mechanisms, informing the DRAM disturbance-error taxonomy |
| dixit2023keytone | field study | hundreds of thousands of machines in Meta's fleet; hundreds of CPUs detected with silent error test scenarios | number of CPUs exhibiting silent data corruption in production fleet | reports that running a vast library of silent error test scenarios across hundreds of thousands of Meta fleet machines detected hundreds of CPUs with silent data corruption errors, showing SDCs are a systemic issue across device generations | retrieved 'fulltext' is keynote front matter containing only the abstract; no per-million error rates or device-generation breakdowns reported | high | abstract-only | Meta fleet-scale evidence that SDC-affected CPUs exist in large numbers in production infrastructure |
| constantinescu2008silent | survey | panel discussion; includes results of an extensive SDC study on LANL HPC platforms; no n reported | occurrence and causes of silent data corruption | panel reports real examples of silent data corruption generated at the semiconductor device and virtualization levels, and cites results of an extensive SDC study carried out at Los Alamos National Laboratory on HPC platforms; no quantitative results in retrieved text | panel abstract only; SDC study results are referenced but not detailed | low | abstract-only | Early industry/academic framing of SDC as a real, costly phenomenon including HPC field evidence |
| gizopoulos2025dark | survey | none; commentary on defective-silicon SDCs in computing | severity and frequency of silent data corruptions | argues that silent data corruptions due to defective silicon cause erroneous program results but that nobody knows how severe and frequent the problem is, how much investment is needed, and who should pay; no quantitative results in retrieved text | short editorial abstract; no data | low | abstract-only | Highlights open questions on SDC severity/frequency and mitigation economics |
| chatzopoulos2025phoebe | simulation | systolic array-based AI accelerators; diverse ML models; SRAMs, registers, functional units; no n reported | SDC vulnerability across accelerator design points; fault-injection throughput | presents a microarchitectural modeling methodology with statistical fault injection on SRAMs, registers, and functional units that speeds up injection throughput by more than 1000x without sacrificing accuracy, revealing SDC vulnerability trends across AI accelerator design points | simulation-based; results depend on modeled fault models and design space coverage | moderate | abstract-only | Provides a methodology for SDC vulnerability evaluation of AI accelerators, complementing CPU-focused reliability work |
| saxena2026silent | architecture proposal | datacenter compute at current scales; no n reported | overhead of SDC mitigation vs availability/reliability tradeoff | proposes optimized SDC mitigation strategies that balance availability and reliability to approach Shannon-like bounds on reliable-compute overhead, and advocates algorithm-based error detection (ABED) because ECC leaves residual SDC risks; no quantitative results in retrieved text | abstract-only; overhead figures and strategy evaluations not reported in retrieved text | low | abstract-only | Argues ECC covers transient faults but residual SDC requires algorithm-based error detection in datacenters |
| vallin2026silent | field study | GPU-based public AI datacenters (e.g., Microsoft Azure); no n reported | hardware-induced SDC challenges and screening feasibility in cloud GPUs | examines hardware-induced silent data corruption challenges in GPU-based public AI datacenters, noting that CPU-oriented screening approaches are not as applicable in public cloud environments like Microsoft Azure and highlighting the need for scalable production hardware-defect screening; no quantitative results in retrieved text | abstract-only; no fleet numbers or error rates reported | low | abstract-only | GPU-centric view of SDC screening challenges in hyperscale AI cloud deployments |
| pei2026connecting | measurement | real-world SDC-affected hardware vs healthy hardware; deterministic LLM training workloads; no n reported | relative performance gap caused by SDC; NaN occurrence; effect of gradient bounds | empirically finds that SDC impact on LLM training correlates strongly with training stability and loss landscape regions, with NaN occurring during training of larger models, and that elementwise gradient bounds reduce the relative performance gap caused by SDC but cannot avoid SDC-induced NaN; no quantitative results in retrieved text | abstract-only; magnitude of the performance gap and bound settings not reported | moderate | abstract-only | Empirically connects SDC impact to LLM training characteristics and tests a gradient-bounding mitigation |
| sangani2024possible | theoretical | front-end-of-line device physics; gate-oxide breakdown and random telegraph noise fault signatures; no n reported | properties and identifiable signatures of an 'archetypal' SDC fault | uses physical models of VLSI front-end-of-line defects to define an 'archetypal' SDC fault with characteristic signatures (gate-oxide breakdown, random telegraph noise), proposes a 'prime and test' screening methodology for RTN-related faults, and shows device variability enhances SDC probability; no quantitative results in retrieved text | theoretical analysis; screening methodology not validated with production data in retrieved text | low | abstract-only | Maps SDC to FEOL gate-oxide/RTN defect physics and proposes screening signatures |
| rojas2022exploring | fault injection | two distributed deep learning libraries (Distributed Data Parallel and Horovod); checkpoint-alteration fault injection; no n reported | resilience to bit-flips in internal data structures; fault propagation | reports via checkpoint-alteration fault injection on distributed deep learning training that Horovod is slightly more resilient to SDCs than Distributed Data Parallel, that fault propagation is similar in both, and that the model is more sensitive to SDCs than the optimizer; no quantitative results in retrieved text | abstract-only; no effect sizes or error-rate numbers reported | moderate | abstract-only | Shows SDC sensitivity differences across DL libraries and between model vs optimizer state |
| gottscho2016measuring | fault injection | one real cloud server with faults injected in DRAM; SPEC CPU2006 benchmarks; interactive web-search workload; light and peak traffic loads | single-machine performance degradation from correctable-but-faulty memory | reports that injected DRAM faults with hard-but-correctable errors slow SPEC CPU2006 average execution time by up to 2.5x and degrade interactive web-search query latency by up to 2.3x under light traffic and up to an extreme 3746x under peak load, due to the memory error-reporting stack | single-machine fault injection study; degradation is scenario-dependent (worst-case peak-load figure is extreme) | moderate | abstract-only | Quantifies the performance cost of the error-avalanche scenario in faulty memory, motivating error-reporting stack redesign |
| shimomura2022hardening | architecture proposal | memcached 1.4.39 and Redis 5.0.3 prototypes with injected ECC-uncorrectable memory errors | recovery success and performance vs the all-clean reboot approach | presents a partial-surgery approach that forces in-memory key-value stores to prune objects damaged by ECC-uncorrectable errors and reconstruct internals from undamaged ones, and reports prototypes on memcached 1.4.39 and Redis 5.0.3 that recover from injected errors and significantly outperform the conventional all-clean approach; no quantitative results in retrieved text | prototype-scale evaluation; no quantitative speedup or overhead numbers in retrieved text | moderate | abstract-only | Addresses availability consequences of ECC-uncorrectable errors for in-memory key-value stores |
| lunardi2018efficacy | measurement | GPUs fabricated in CMOS and FinFET technologies under neutron irradiation; no n reported | efficacy of ECC in masking neutron-induced faults; silent data corruption rates | experimentally investigates ECC efficacy in masking neutron-induced faults in modern GPUs, finding that changing transistor technology (FinFET vs CMOS) can be as beneficial as using ECC for reducing silent data corruption rates, and that hidden logic/memory (flip-flops in pipelines and queues) cannot be easily protected by ECC; no quantitative results in retrieved text | abstract-only; error-rate numbers and irradiation fluences not reported | moderate | abstract-only | Shows ECC is not a panacea for GPU radiation faults and that technology choice can match ECC benefits |
| oliveira2015evaluation | measurement | memory structures of modern GPUs; wide set of parallel codes exposed to controlled neutron beams; no n reported | operative error rates of GPUs; pattern dependence; multiple-error occurrences; hardening overheads | evaluates the neutron sensitivity of modern GPU memory structures under controlled neutron beams, highlighting pattern dependence and multiple error occurrences, measures operative error rates for parallel codes, and evaluates ECC, algorithm-based fault tolerance, and duplication-with-comparison hardening via radiation experiments; no quantitative results in retrieved text | abstract-only; measured error rates and overhead figures not reported | moderate | abstract-only | Radiation experiments quantifying GPU memory neutron sensitivity and hardening tradeoffs |
| previlon2020characterizi | fault injection | GPU applications/kernels; fault injection campaigns; no n reported | time-varying soft error vulnerability phases; correlation with execution characteristics | shows that GPU program resilience characteristics change significantly and repetitively during execution, that these vulnerability phases do not align with performance phases but coincide with changes in basic block execution paths, and that exploiting this accelerates fault injection campaigns for reliability assessment by an order of magnitude; no quantitative results in retrieved text | abstract-only; vulnerability magnitude and per-phase values not reported | moderate | abstract-only | Reveals phase behavior of GPU soft-error vulnerability and uses it to speed up fault injection |
| rahman2024druto | fault injection | GPU applications; benchmark test inputs vs Druto-generated inputs; no n reported | upper bound of SDC probability per application input | reports that Druto, a compiler-based input-search technique, derives SDC probability estimates up to 74x higher than existing techniques, and that existing techniques cannot find Druto's inputs even given 5x more search time, showing standard benchmark-input resilience evaluations are over-optimistic | evaluation on benchmark GPU applications; relies on representative-thread ranking property | moderate | abstract-only | Shows input dependence of GPU SDC rates; standard benchmark inputs can underestimate SDC risk by up to 74x |
| mukherjee2003systematic | simulation | instrumented 1A64 (Itanium-family) processor simulator; dynamic sections of the CPU2000 benchmark suite | per-structure architectural vulnerability factor (AVF) | defines architectural vulnerability factor (AVF) as the probability a fault in a structure escapes to program output and computes per-structure AVFs on an instrumented 1A64 processor simulator, finding AVFs of 28% for the instruction queue and 9% for the execution units averaged across the CPU2000 suite | simulation-based on one processor family; AVF excludes raw error-rate (process/circuit) contribution | high | abstract-only | Foundational AVF methodology linking structure error rates to program-visible faults in microprocessors |
| sridharan2009eliminating | theoretical | programs executed on hardware; AVF/PVF analysis; no n reported | program vulnerability factor (PVF); decomposition of AVF into architecture- and microarchitecture-level masking | introduces the program vulnerability factor (PVF) metric that isolates software-dependent (architecture-level) fault masking from hardware-dependent (microarchitecture-level) masking, enabling quantitative statements about a program's soft-error tolerance and explaining workload-driven AVF changes across structures; no quantitative results in retrieved text | abstract-only; PVF values for example programs not reported | moderate | abstract-only | PVF metric separates software from hardware fault masking, informing soft-error-aware software design |
| saleh1990reliability | theoretical | four memory system configurations (no redundancy, SEC-DED only, SEC-DED with exponential or deterministic scrubbing); transient errors at exponentially distributed arrival times | reliability functions and mean time to failure (MTTF) | derives reliability and MTTF equations for SEC-DED memory systems with exponentially distributed and deterministic scrubbing, comparing them with non-redundant and SEC-DED-only systems, and provides easy-to-use MTTF expressions as a major contribution; no quantitative results in retrieved text | analytic modeling only; no empirical validation or specific MTTF numbers in retrieved text | moderate | abstract-only | Foundational analytic treatment of scrubbing-based transient-error recovery in SEC-DED memories |
| reviriego2010optimizing | theoretical | advanced memories (smaller geometries, lower voltages) with significant multibit soft error rates; no n reported | mean time to failure as a function of scrubbing sequence | presents a new scrubbing procedure that orders memory scrubbing to cope with multibit soft errors, claiming that in the presence of multiple cell upsets the mean time to failure could be doubled relative to the traditional scrubbing approach; no quantitative results in retrieved text | theoretical/analytic claim; no experimental validation or absolute MTTF values in retrieved text | moderate | abstract-only | Shows scrubbing order matters when multibit errors dominate, with up to 2x MTTF improvement |
| ziegler1979effect | theoretical | typical memory devices (LSI circuits); sea-level cosmic-ray flux; effects of shielding, altitude, and solar cycle | cosmic-ray-induced error rates in computer memories | develops a method for evaluating cosmic-ray effects on computer memories, estimating that cosmic-ray nucleons and muons cause errors in current memories at a level of marginal significance with a potentially very significant effect in the next generation, and that error rates increase rapidly with altitude; no quantitative results in retrieved text | estimates based on 1970s device technology; error-rate magnitudes not given in retrieved text | high | abstract-only | Foundational analysis establishing cosmic-ray-induced errors in memories and strong altitude dependence |
| may1979alpha | measurement | dynamic RAMs and CCDs; packaging materials containing parts-per-million levels of uranium and thorium; no n reported | alpha-particle-induced soft error mechanism and alpha activity of materials | reports experiments and measurements identifying a new physical soft error mechanism in dynamic RAMs and CCDs: alpha particles emitted by uranium and thorium decay in packaging materials create enough electron-hole pairs near a storage node to cause random single-bit errors, and develops a physical model for the soft error; no quantitative results in retrieved text | abstract-only; measured alpha activities and error rates not reported | high | abstract-only | Discovered the alpha-particle soft-error mechanism in DRAMs from packaging radioactivity (May-Woods effect) |
| ogorman1994effect | measurement | a large number of commercial DRAM chips measured at four US sites: near sea level in Essex Junction VT, 200 m underground in a Kansas salt mine, 1.6 km altitude in Boulder CO, and 3.1 km in Leadville CO | soft error rate (SER) of DRAM chips vs location and altitude | provides conclusive evidence that cosmic rays cause soft errors in commercial DRAM chips at ground level, showing a significant cosmic-ray-attributable SER component even at sea level whose magnitude increases dramatically at higher altitudes (up to 3.1 km); no quantitative results in retrieved text | abstract-only; SER values for each site not reported | high | abstract-only | Conclusive field evidence that cosmic rays drive DRAM soft errors at ground level, with altitude scaling |
| gordon2004measurement | measurement | extended-energy Bonner sphere spectrometer measurements at several US locations; neutron energies covering over twelve decades from meV to GeV | ground-level cosmic-ray-induced neutron flux and energy spectrum | reports new ground-based measurements of the cosmic-ray-induced neutron flux and energy distribution spanning over twelve decades of energy (meV to GeV), provides an analytic expression fitting the spectrum above about 0.4 MeV, and a scaling expression for altitude, geomagnetic, and solar-activity dependence; no quantitative results in retrieved text | abstract-only; flux values and fit parameters not reported | high | abstract-only | Ground neutron flux/spectrum data underpinning soft-error rate estimation models for memories and logic |
| baumann2005soft | survey | modern computer systems; terrestrial radiation mechanisms; technology scaling trends; no n reported | soft-error sensitivity of memory and logic and its dependence on application and technology | comprehensively analyzes soft-error sensitivity in modern systems, showing it to be application dependent, and discusses ground-level radiation mechanisms with the most serious impact on circuit operation along with the effect of technology scaling on soft-error rates in memory and logic; no quantitative results in retrieved text | survey; no original quantitative measurements in retrieved text | moderate | abstract-only | Authoritative survey of terrestrial soft-error mechanisms and scaling trends for SER |
| seifert2015soft | measurement | 14-nm second-generation high-k + metal gate bulk tri-gate devices vs 32-nm planar and first-generation tri-gate; memory cells, sequential elements, and combinational logic; thermal and high-energy neutrons, high-energy protons, alpha particles | radiation-induced soft error rate (upset rates) by device type and radiation source | reports radiation-induced SER improvements up to about 23x for 14-nm second-generation tri-gate technology relative to 32-nm planar devices, with an about 8x reduction of logic upset rates relative to first-generation tri-gate from aggressive fin depopulation and fin parameter scaling | vendor technology-node study; abstract-only (test conditions and fluences not reported) | high | abstract-only | Measured SER improvement trends (~23x) across technology generations, informing scaling of soft-error rates |
| swift1994new | measurement | one-transistor dynamic memory cells and four-transistor static cells under heavy-ion irradiation; no n reported | heavy-ion-induced hard error classes, annealing behavior, and scaling/bias dependence | reports experimental evidence that heavy ions can cause hard errors in one-transistor DRAM cells (previously seen only in four-transistor static cells), identifying a new hard-error class inconsistent with the single-ion dose mechanism that is non-additive between ion hits, resistant to annealing, and accumulates over long missions, with scaling increasing susceptibility and lowered bias mitigating damage; no quantitative results in retrieved text | abstract-only; no error counts or ion parameters reported | moderate | abstract-only | Identifies accumulating heavy-ion hard errors in DRAM cells relevant to long-mission reliability |
| patwari2017understandin | field study | a large production HPC cluster; DRAM error logs; no n reported | spatial distribution of DRAM errors across the cluster; predictability of susceptible regions | analyzes DRAM errors in a large production HPC cluster and finds that nodes with high error counts are grouped in spatial regions over time periods, suggesting these 'susceptible' regions are collectively more vulnerable to errors, and builds a predictor that identifies such regions from neighboring region patterns; no quantitative results in retrieved text | abstract-only; cluster size, error counts, and predictor accuracy not reported | moderate | abstract-only | Shows spatial clustering of DRAM errors across an entire HPC cluster enabling region-level prediction |
| schroeder2011dram | field study | Large fleet of commodity servers in a production cluster, 2.5 years, multiple vendors and DRAM technologies, many millions of DIMMs | correctable and uncorrectable DRAM error rates, DIMM failure rates | journal version of the Google field study: analyzes 2.5 years of memory error measurements in a large fleet, reporting error rates far above lab expectations with more than 8% of DIMMs affected per year. | same dataset and method as the SIGMETRICS 2009 version; abstract does not restate all quantitative results | high | abstract-only | CACM journal version of the foundational Google DRAM field study |

## References

1. Ziegler, J. F. & Lanford, W. A. (1979). *Effect of Cosmic Rays on Computer Memories*. Science. Abstract only. Foundational analysis establishing cosmic-ray-induced errors in memories and strong altitude dependence [doi:10.1126/science.206.4420.776](https://doi.org/10.1126/science.206.4420.776)
2. May, T.C. & Woods, M. H. (1979). *Alpha-particle-induced soft errors in dynamic memories*. IEEE Transactions on Electron Devices. Abstract only. Discovered the alpha-particle soft-error mechanism in DRAMs from packaging radioactivity (May-Woods effect) [doi:10.1109/t-ed.1979.19370](https://doi.org/10.1109/t-ed.1979.19370)
3. O'Gorman, T. (1994). *The effect of cosmic rays on the soft error rate of a DRAM at ground level*. IEEE Transactions on Electron Devices. Abstract only. Conclusive field evidence that cosmic rays drive DRAM soft errors at ground level, with altitude scaling [doi:10.1109/16.278509](https://doi.org/10.1109/16.278509)
4. Gordon, Michael S. et al. (2004). *Measurement of the flux and energy spectrum of cosmic-ray induced neutrons on the ground*. IEEE Transactions on Nuclear Science. Abstract only. Ground neutron flux/spectrum data underpinning soft-error rate estimation models for memories and logic [doi:10.1109/tns.2004.839134](https://doi.org/10.1109/tns.2004.839134)
5. Baumann, R. (2005). *Soft Errors in Advanced Computer Systems*. IEEE Design & Test of Computers. Abstract only. Authoritative survey of terrestrial soft-error mechanisms and scaling trends for SER [doi:10.1109/mdt.2005.69](https://doi.org/10.1109/mdt.2005.69)
6. Seifert, N. et al. (2015). *Soft Error Rate Improvements in 14-nm Technology Featuring Second-Generation 3D Tri-Gate Transistors*. IEEE Transactions on Nuclear Science. Abstract only. Measured SER improvement trends (~23x) across technology generations, informing scaling of soft-error rates [doi:10.1109/tns.2015.2495130](https://doi.org/10.1109/tns.2015.2495130)
7. Swift, G.M. et al. (1994). *A new class of single event hard errors [DRAM cells]*. IEEE Transactions on Nuclear Science. Abstract only. Identifies accumulating heavy-ion hard errors in DRAM cells relevant to long-mission reliability [doi:10.1109/23.340540](https://doi.org/10.1109/23.340540)
8. Schroeder, Bianca et al. (2009). *DRAM errors in the wild*. Proceedings of the eleventh international joint conference on Measurement and modeling of computer systems. Full text read. Foundational large-scale DRAM field study establishing real-world error rates orders of magnitude above lab-based estimates and dominance of hard errors [doi:10.1145/1555349.1555372](https://doi.org/10.1145/1555349.1555372)
9. Schroeder, Bianca et al. (2011). *DRAM errors in the wild*. Communications of the ACM. Abstract only. CACM journal version of the foundational Google DRAM field study; corroborates the 2009 SIGMETRICS numbers [doi:10.1145/1897816.1897844](https://doi.org/10.1145/1897816.1897844)
10. Nightingale, Edmund B. et al. (2011). *Cycles, cells and platters*. arXiv. Abstract only. Evidence that memory/hardware faults are non-transient and recurrent, motivating proactive replacement [doi:10.1145/1966445.1966477](https://doi.org/10.1145/1966445.1966477)
11. Sridharan, Vilas & Liberty, Dean (2012). *A study of DRAM failures in the field*. 2012 International Conference for High Performance Computing, Networking, Storage and Analysis. Abstract only. Canonical SC12 field study; 42x chipkill benefit is a key quantified citation [doi:10.1109/sc.2012.13](https://doi.org/10.1109/sc.2012.13)
12. Hwang, Andy A. et al. (2012). *Cosmic rays don't strike twice*. arXiv. Abstract only. Large-scale companion to Schroeder 2009; establishes hard-error dominance and page-retirement mitigation [doi:10.1145/2150976.2150989](https://doi.org/10.1145/2150976.2150989)
13. Sridharan, Vilas et al. (2015). *Memory Errors in Modern Systems*. ACM SIGARCH Computer Architecture News. Abstract only. Methodological caution about error-vs-fault counting; DRAM resilience projection for exascale [doi:10.1145/2786763.2694348](https://doi.org/10.1145/2786763.2694348)
14. Di Martino, Catello et al. (2014). *Lessons Learned from the Analysis of System Failures at Petascale: The Case of Blue Waters*. 2014 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks. Abstract only. Quantified evidence that ECC/chipkill absorbs high error rates at petascale [doi:10.1109/dsn.2014.62](https://doi.org/10.1109/dsn.2014.62)
15. Messer, Alan et al. (2004). *Susceptibility of commodity systems and software to memory soft errors*. IEEE Transactions on Computers. Abstract only. Early evidence that software recovery can mitigate soft errors in commodity systems [doi:10.1109/tc.2004.119](https://doi.org/10.1109/tc.2004.119)
16. Cheng, Zhinan et al. (2022). *An In-Depth Correlative Study Between DRAM Errors and Server Failures in Production Data Centers*. 2022 41st International Symposium on Reliable Distributed Systems (SRDS). Abstract only. 3M-module scale; links CE error patterns to imminent server failure [doi:10.1109/srds55811.2022.00032](https://doi.org/10.1109/srds55811.2022.00032)
17. Baeg, Sanghyeon et al. (2019). *Correctable and uncorrectable errors using large scale DRAM DIMMs in replacement network servers*. Microelectronics Reliability. Abstract only. Quantifies UE vs CE prevalence and speed/manufacturer dependence of DIMM FIT [doi:10.1016/j.microrel.2019.05.008](https://doi.org/10.1016/j.microrel.2019.05.008)
18. Patwari, Ayush et al. (2017). *Understanding the Spatial Characteristics of DRAM Errors in HPC Clusters*. Proceedings of the 2017 Workshop on Fault-Tolerance for HPC at Extreme Scale. Abstract only. Shows spatial clustering of DRAM errors across an entire HPC cluster enabling region-level prediction [doi:10.1145/3086157.3086164](https://doi.org/10.1145/3086157.3086164)
19. Zivanovic, Darko et al. (2019). *DRAM errors in the field*. Proceedings of the International Symposium on Memory Systems. Full text read. Anchor field study: quantifies corrected vs uncorrected DRAM error incidence and MTBF, and warns that standard error-rate metrics are volatile without statistical rigor. [doi:10.1145/3357526.3357558](https://doi.org/10.1145/3357526.3357558)
20. Boixaderas, Isaac et al. (2024). *DRAM Errors and Cosmic Rays: Space Invaders or Science Fiction?*. 2024 IEEE 36th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD). Full text read. Challenges the widely assumed cosmic-ray explanation for DRAM error rates with large-scale production data and a repeatable methodology [doi:10.1109/sbac-pad63648.2024.00025](https://doi.org/10.1109/sbac-pad63648.2024.00025)
21. Khan, Samira et al. (2014). *The efficacy of error mitigation techniques for DRAM retention failures*. arXiv. Abstract only. Background on VRT retention failures motivating in-field mitigation [doi:10.1145/2591971.2592000](https://doi.org/10.1145/2591971.2592000)
22. Liu, Jamie et al. (2012). *RAIDR*. ACM SIGARCH Computer Architecture News. Abstract only. Shows most DRAM rows retain data far longer than worst-case refresh, enabling refresh skipping [doi:10.1145/2366231.2337161](https://doi.org/10.1145/2366231.2337161)
23. Khan, Samira et al. (2016). *PARBOR: An Efficient System-Level Technique to Detect Data-Dependent Failures in DRAM*. arXiv. Abstract only. Evidence that data-dependent (pattern-sensitive) failures are detectable and common enough to matter [doi:10.1109/dsn.2016.30](https://doi.org/10.1109/dsn.2016.30)
24. Aichinger, Barbara (2015). *DDR memory errors caused by Row Hammer*. 2015 IEEE High Performance Extreme Computing Conference (HPEC). Abstract only. Industry testimony that row hammer is an unaddressed field reliability/security issue [doi:10.1109/hpec.2015.7322462](https://doi.org/10.1109/hpec.2015.7322462)
25. Nie, Bin et al. (2016). *A large-scale study of soft-errors on GPUs in the field*. 2016 IEEE International Symposium on High Performance Computer Architecture (HPCA). Abstract only. First large-scale field study of GPU soft-errors (Titan), grounding GPU reliability research in field data [doi:10.1109/hpca.2016.7446091](https://doi.org/10.1109/hpca.2016.7446091)
26. Tiwari, Devesh et al. (2015). *Understanding GPU errors on large-scale HPC systems and the implications for system design and operation*. arXiv. Full text read. First large-scale GPU field study plus radiation testing; shows SECDED ECC suffices for most radiation-induced GPU memory bit corruptions [doi:10.1109/hpca.2015.7056044](https://doi.org/10.1109/hpca.2015.7056044)
27. Haque, Imran S. & Pande, Vijay S. (2010). *Hard Data on Soft Errors: A Large-Scale Assessment of Real-World Error Rates in GPGPU*. arXiv. Abstract only. Early large-scale evidence that GPU memory soft errors are widespread in the field [doi:10.1109/ccgrid.2010.84](https://doi.org/10.1109/ccgrid.2010.84)
28. Sullivan, Michael B. et al. (2021). *Characterizing and Mitigating Soft Errors in GPU DRAM*. MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture. Abstract only. Quantifies HBM2 multi-bit error locality and ECC redesign headroom for GPUs [doi:10.1145/3466752.3480111](https://doi.org/10.1145/3466752.3480111)
29. Zhu, Zhu et al. (2025). *Understanding the Landscape of Ampere GPU Memory Errors*. arXiv preprint. Full text read. Large-scale cross-supercomputer A100 GPU HBM reliability characterization; shows cluster-to-cluster rate variability and bursty error patterns relevant to checkpointing [doi:10.48550/arxiv.2508.03513](https://doi.org/10.48550/arxiv.2508.03513)
30. Tung, Chung-Hsuan et al. (2026). *The Anatomy of Silent Data Corruption: GPU Error Pattern Study and Modeling Guidance*. arXiv preprint. Full text read. Shows GPU SDC corruption is dominated by multi-bit, structured, non-special-value patterns, undermining single-bit and NaN-based fault models [doi:10.48550/arxiv.2605.04213](https://doi.org/10.48550/arxiv.2605.04213)
31. Tyagi, Abhishek et al. (2026). *LLM-PRISM: Characterizing Silent Data Corruption from Permanent GPU Faults in LLM Training*. arXiv preprint. Full text read. First hardware-grounded characterization of LLM pre-training SDC resilience; shows critical-datapath and precision-format-dependent catastrophic divergence and NaN-detector blind spots [doi:10.48550/arxiv.2604.10390](https://doi.org/10.48550/arxiv.2604.10390)
32. Xie, Rui et al. (2025). *Making Strong Error-Correcting Codes Work Effectively for HBM in AI Inference*. arXiv preprint. Full text read. Proposes moving HBM reliability from fixed on-die ECC to controller-managed long codes, trading higher raw HBM BER for lower $/GB in AI inference [doi:10.48550/arxiv.2512.18152](https://doi.org/10.48550/arxiv.2512.18152)
33. Xie, Rui et al. (2025). *Breaking the HBM Bit Cost Barrier: Domain-Specific ECC for AI Inference Infrastructure*. arXiv preprint. Full text read. Companion HBM cost-reduction proposal eliminating on-die ECC entirely; quantifies accuracy/throughput headroom under relaxed raw BER for AI inference [doi:10.48550/arxiv.2507.02654](https://doi.org/10.48550/arxiv.2507.02654)
34. Chai, Duo et al. (2025). *Analysis of LLM Vulnerability to GPU Soft Errors: An Instruction-Level Fault Injection Study*. arXiv preprint. Full text read. First instruction-level fault injection study of LLM inference; quantifies DUE/SDC behavior and vulnerability factors by instruction, bit position, task and layer. [doi:10.48550/arxiv.2601.19912](https://doi.org/10.48550/arxiv.2601.19912)
35. Boixaderas, Isaac et al. (2020). *Cost-Aware Prediction of Uncorrected DRAM Errors in the Field*. arXiv. Full text read. Shows uncorrected-error prediction is practically actionable and that cost-benefit, not precision/recall, should evaluate predictors. [doi:10.1109/sc41405.2020.00065](https://doi.org/10.1109/sc41405.2020.00065)
36. Boixaderas, Isaac et al. (2024). *Reinforcement Learning-based Adaptive Mitigation of Uncorrected DRAM Errors in the Field*. arXiv. Full text read. First adaptive RL-based uncorrected-DRAM-error mitigation; quantifies achievable savings relative to an Oracle bound. [doi:10.1145/3625549.3658686](https://doi.org/10.1145/3625549.3658686)
37. Baseman, Elisabeth et al. (2018). *Physics-Informed Machine Learning for DRAM Error Modeling*. arXiv. Full text read. Physics-informed ML for DRAM error location prediction; evidence that spatial structure dominates DRAM error prediction and models transfer across systems. [doi:10.1109/dft.2018.8602983](https://doi.org/10.1109/dft.2018.8602983)
38. Mukhanov, Lev et al. (2019). *Workload-Aware DRAM Error Prediction using Machine Learning*. arXiv. Full text read. Quantifies workload dependence of DRAM error rates (8x) - key evidence that program behavior materially affects DRAM reliability. [doi:10.1109/iiswc47752.2019.9041963](https://doi.org/10.1109/iiswc47752.2019.9041963)
39. Mukhanov, Lev et al. (2020). *Revealing DRAM Operating GuardBands Through Workload-Aware Error Predictive Modeling*. IEEE Transactions on Computers. Full text read. Shows ML workload-aware models can safely relax DRAM guardbands (24% power saving) - reliability/power tradeoff evidence. [doi:10.1109/tc.2020.3033627](https://doi.org/10.1109/tc.2020.3033627)
40. Yong, Yaoguang et al. (2026). *Collaborative Prediction of Cloud DRAM Failures With Rules and Machine Learning*. IEEE Transactions on Computers. Abstract only. Shows CE storms, not just UEs, drive node unavailability in hyperscale clouds [doi:10.1109/tc.2026.3655008](https://doi.org/10.1109/tc.2026.3655008)
41. Wang, Chenglin et al. (2026). *Predicting DRAM Failures at Scale: A Two-Stage Approach for Heterogeneous Systems*. 2026 IEEE International Symposium on High Performance Computer Architecture (HPCA). Abstract only. Largest recent DIMM-level prediction study; cross-architecture error-pattern differences [doi:10.1109/hpca68181.2026.11408474](https://doi.org/10.1109/hpca68181.2026.11408474)
42. Chen, Jian et al. (2021). *CARE: Coordinated Augmentation for Elastic Resilience on DRAM Errors in Data Centers*. 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA). Abstract only. Cost-effective alternative to chipkill for datacenter DRAM error tolerance [doi:10.1109/hpca51647.2021.00052](https://doi.org/10.1109/hpca51647.2021.00052)
43. Nie, Bin et al. (2018). *Machine Learning Models for GPU Error Prediction in a Large Scale HPC System*. arXiv. Full text read. Large-scale GPU ECC soft-error prediction in production HPC; motivates dynamic ECC on/off via error prediction. [doi:10.1109/dsn.2018.00022](https://doi.org/10.1109/dsn.2018.00022)
44. Yoon, Doe Hyun & Erez, Mattan (2010). *Virtualized and flexible ECC for main memory*. arXiv. Abstract only. Enables chipkill-class protection even on non-ECC DIMMs at small overhead [doi:10.1145/1736020.1736064](https://doi.org/10.1145/1736020.1736064)
45. Udipi, Aniruddha N. et al. (2012). *LOT-ECC: Localized and tiered reliability mechanisms for commodity memory systems*. arXiv. Abstract only. Shows chipkill-level protection need not activate all chips on every access [doi:10.1109/isca.2012.6237025](https://doi.org/10.1109/isca.2012.6237025)
46. Patel, Minesh et al. (2020). *Bit-Exact ECC Recovery (BEER): Determining DRAM On-Die ECC Functions by Exploiting DRAM Data Retention Characteristics*. 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). Full text read. Shows on-die ECC obfuscates raw DRAM error patterns and provides a method to reconstruct bit-exact pre-correction error rates, complicating third-party ECC error studies [doi:10.1109/micro50266.2020.00034](https://doi.org/10.1109/micro50266.2020.00034)
47. Alam, Irina & Gupta, Puneet (2022). *COMET: On-die and In-controller Collaborative Memory ECC Technique for Safer and Stronger Correction of DRAM Errors*. 2022 52nd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN). Abstract only. Quantifies a new SDC vulnerability from on-die ECC + controller ECC interaction [doi:10.1109/dsn53405.2022.00024](https://doi.org/10.1109/dsn53405.2022.00024)
48. Xun Jian & Kumar, R. (2013). *Adaptive Reliability Chipkill Correct (ARCC)*. 2013 IEEE 19th International Symposium on High Performance Computer Architecture (HPCA). Abstract only. Exploits the observation that only a tiny fraction of memory faults during a server's lifetime [doi:10.1109/hpca.2013.6522325](https://doi.org/10.1109/hpca.2013.6522325)
49. Jian, Xun et al. (2013). *High Performance, Energy Efficient Chipkill Correct Memory with Multidimensional Parity*. IEEE Computer Architecture Letters. Abstract only. Decouples local-fault and device-fault correction to cut chipkill cost [doi:10.1109/l-ca.2012.21](https://doi.org/10.1109/l-ca.2012.21)
50. Lee, Young Seo et al. (2022). *Stealth ECC: A Data-Width Aware Adaptive ECC Scheme for DRAM Error Resilience*. Design, Automation and Test in Europe. Abstract only. Data-width-aware adaptive ECC leveraging narrow-width values for multi-bit correction [doi:10.23919/date54114.2022.9774775](https://doi.org/10.23919/date54114.2022.9774775)
51. Bae, H. et al. (2023). *Twin ECC: A Data Duplication Based ECC for Strong DRAM Error Resilience*. Design, Automation and Test in Europe. Abstract only. Exploits asymmetric DRAM error polarity via data duplication + bitwise OR [doi:10.23919/date56975.2023.10137096](https://doi.org/10.23919/date56975.2023.10137096)
52. Moon, Youngki et al. (2024). *A Novel Prediction-Based Two-Tiered ECC for Mitigating SWD Errors in HBM*. IEEE Transactions on Very Large Scale Integration (VLSI) Systems. Abstract only. Targets SWD errors, a primary HBM error cause, without extra check bits [doi:10.1109/tvlsi.2024.3474791](https://doi.org/10.1109/tvlsi.2024.3474791)
53. Kwon, Kiheon et al. (2023). *EPA ECC: Error-Pattern-Aligned ECC for HBM2E*. 2023 International Technical Conference on Circuits/Systems, Computers, and Communications (ITC-CSCC). Abstract only. Aligns OD-ECC symbols with observed HBM multi-bit error patterns [doi:10.1109/itc-cscc58803.2023.10212882](https://doi.org/10.1109/itc-cscc58803.2023.10212882)
54. Shin, Chaehyeon & Park, Jongsun (2025). *DBB-ECC: Random Double Bit and Burst Error Correction Code for HBM3*. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems. Abstract only. HBM3 on-die ECC must handle both burst and randomly scattered errors [doi:10.1109/tcad.2025.3544964](https://doi.org/10.1109/tcad.2025.3544964)
55. Shin, Jaeho & Kim, Jungrae (2025). *ROSE: Reliability-Optimized OD-ECC and S-ECC Enhancements for HBM3*. 2025 International Conference on Electronics, Information, and Communication (ICEIC). Abstract only. Shows combined OD-ECC/S-ECC can be made resource-efficient for HBM3 [doi:10.1109/iceic64972.2025.10879768](https://doi.org/10.1109/iceic64972.2025.10879768)
56. Liu, Yixuan et al. (2025). *CXL-ECC: an Efficient LRC-based on-CXL-Memory-eXpander-Controller ECC to Enhance Reliability and Performance of DRAM Error Correction*. 2025 62nd ACM/IEEE Design Automation Conference (DAC). Abstract only. First ECC design targeting DRAM reliability in the CXL memory expansion architecture [doi:10.1109/dac63849.2025.11133097](https://doi.org/10.1109/dac63849.2025.11133097)
57. Chen, Hsing-Min et al. (2018). *Configurable-ECC: Architecting a Flexible ECC Scheme to Support Different Sized Accesses in High Bandwidth Memory Systems*. IEEE Transactions on Computers. Abstract only. Flexible HBM ECC for mixed fine/coarse-grained GPU access granularities [doi:10.1109/tc.2018.2886884](https://doi.org/10.1109/tc.2018.2886884)
58. Kim, Yoongu et al. (2014). *Flipping bits in memory without accessing them*. ACM SIGARCH Computer Architecture News. Full text read. The original RowHammer discovery paper; demonstrates a circuit-level disturbance mechanism that defeats memory isolation without ECC visibility [doi:10.1145/2678373.2665726](https://doi.org/10.1145/2678373.2665726)
59. Mutlu, O. & Kim, Jeremie S. (2019). *RowHammer: A Retrospective*. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems. Full text read. Widely-cited survey establishing RowHammer as the canonical circuit-level failure mechanism with system security consequences [doi:10.1109/tcad.2019.2915318](https://doi.org/10.1109/tcad.2019.2915318)
60. Mutlu, Onur (2023). *Retrospective: Flipping Bits in Memory Without Accessing Them: An Experimental Study of DRAM Disturbance Errors*. arXiv (Cornell University). Full text read. Documents RowHammer prevalence (>80% of modules) - disturbance-induced bitflips that bypass ECC assumptions in commodity DRAM. [doi:10.48550/arxiv.2306.16093](https://doi.org/10.48550/arxiv.2306.16093)
61. Tatar, Andrei et al. (2018). *Defeating Software Mitigations Against Rowhammer: A Surgical Precision Hammer*. Lecture notes in computer science. Abstract only. Documents Rowhammer bit-flip exploitability and why DRAM reliability (ECC/mitigations) is a security concern [doi:10.1007/978-3-030-00470-5_3](https://doi.org/10.1007/978-3-030-00470-5_3)
62. Zhou, Longda et al. (2024). *Unveiling RowPress in Sub-20 nm DRAM Through Comparative Analysis With Row Hammer: From Leakage Mechanisms to Key Features*. IEEE Transactions on Electron Devices. Abstract only. Distinguishes RowPress from RowHammer disturbance mechanisms, informing the DRAM disturbance-error taxonomy [doi:10.1109/ted.2024.3418300](https://doi.org/10.1109/ted.2024.3418300)
63. Dixit, Harish Dattatraya et al. (2021). *Silent Data Corruptions at Scale*. arXiv preprint. Full text read. Industry evidence that SDC is a systemic, repeatable silicon-defect problem beyond DRAM ECC coverage, motivating system-level fault tolerance [doi:10.48550/arxiv.2102.11245](https://doi.org/10.48550/arxiv.2102.11245)
64. Dixit, Harish (2023). *Keytone: Silent Data Corruptions at Scale*. 2023 IEEE 29th International Symposium on On-Line Testing and Robust System Design (IOLTS). Abstract only. Meta fleet-scale evidence that SDC-affected CPUs exist in large numbers in production infrastructure [doi:10.1109/iolts59296.2023.10224872](https://doi.org/10.1109/iolts59296.2023.10224872)
65. Constantinescu, Cristian et al. (2008). *Silent Data Corruption &amp;#x2014; Myth or reality?*. arXiv. Abstract only. Early industry/academic framing of SDC as a real, costly phenomenon including HPC field evidence [doi:10.1109/dsn.2008.4630077](https://doi.org/10.1109/dsn.2008.4630077)
66. Gizopoulos, Dimitris (2025). *The Dark Side of Computing: Silent Data Corruptions*. Computer. Abstract only. Highlights open questions on SDC severity/frequency and mitigation economics [doi:10.1109/mc.2025.3554306](https://doi.org/10.1109/mc.2025.3554306)
67. Chatzopoulos, Odysseas et al. (2025). *Phoebe: Measuring the Unmeasurable—Demystifying Silent Data Corruptions in AI Accelerators Through Microarchitectural Modeling*. IEEE Micro. Abstract only. Provides a methodology for SDC vulnerability evaluation of AI accelerators, complementing CPU-focused reliability work [doi:10.1109/mm.2025.3646859](https://doi.org/10.1109/mm.2025.3646859)
68. Saxena, Nirmal et al. (2026). *Silent Data Corruption: Optimal Mitigation Strategies for Data Center Computing*. IEEE Micro. Abstract only. Argues ECC covers transient faults but residual SDC requires algorithm-based error detection in datacenters [doi:10.1109/mm.2025.3643799](https://doi.org/10.1109/mm.2025.3643799)
69. Vallin, Carlos et al. (2026). *Silent Data Corruption Challenges in Modern AI Public Cloud Deployments*. IEEE Micro. Abstract only. GPU-centric view of SDC screening challenges in hyperscale AI cloud deployments [doi:10.1109/mm.2025.3641148](https://doi.org/10.1109/mm.2025.3641148)
70. Altenbernd, Anton et al. (2026). *Exploring Silent Data Corruption as a Reliability Challenge in LLM Training*. 2026 IEEE 26th International Symposium on Cluster, Cloud and Internet Computing (CCGrid). Full text read. Controlled SDC study in LLM training with detection/recompute mitigation; carries industry SDC frequency anecdotes (Gemini ~1-2 weeks; Meta 6 incidents in 54 days). [doi:10.48550/arxiv.2604.00726](https://doi.org/10.48550/arxiv.2604.00726)
71. Pei, Hengzhi et al. (2026). *Connecting the Impact of Silent Data Corruption With Different Training Characteristics: An Empirical Study*. IEEE Micro. Abstract only. Empirically connects SDC impact to LLM training characteristics and tests a gradient-bounding mitigation [doi:10.1109/mm.2025.3642709](https://doi.org/10.1109/mm.2025.3642709)
72. Sangani, D. et al. (2024). *Possible Origins, Identification, and Screening of Silent Data Corruption in Data Centers*. 2024 IEEE International Reliability Physics Symposium (IRPS). Abstract only. Maps SDC to FEOL gate-oxide/RTN defect physics and proposes screening signatures [doi:10.1109/irps48228.2024.10529436](https://doi.org/10.1109/irps48228.2024.10529436)
73. Rojas, Elvis et al. (2022). *Exploring the Effects of Silent Data Corruption in Distributed Deep Learning Training*. 2022 IEEE 34th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD). Abstract only. Shows SDC sensitivity differences across DL libraries and between model vs optimizer state [doi:10.1109/sbac-pad55451.2022.00013](https://doi.org/10.1109/sbac-pad55451.2022.00013)
74. Gottscho, Mark et al. (2016). *Measuring the Impact of Memory Errors on Application  Performance*. IEEE Computer Architecture Letters. Abstract only. Quantifies the performance cost of the error-avalanche scenario in faulty memory, motivating error-reporting stack redesign [doi:10.1109/lca.2016.2599513](https://doi.org/10.1109/lca.2016.2599513)
75. Bridges, Patrick G. et al. (2012). *Cooperative Application/OS DRAM Fault Recovery*. Lecture notes in computer science. Full text read. Early cooperative application/OS recovery proposal for uncorrectable DRAM errors; supplies the 8% of DIMMs per year uncorrectable-error rate citation. [doi:10.1007/978-3-642-29740-3_28](https://doi.org/10.1007/978-3-642-29740-3_28)
76. Shimomura, Tsuyoshi & Yamada, Hiroshi (2022). *Hardening In-memory Key-value Stores against ECC-uncorrectable Memory Errors*. 2022 52nd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN). Abstract only. Addresses availability consequences of ECC-uncorrectable errors for in-memory key-value stores [doi:10.1109/dsn53405.2022.00057](https://doi.org/10.1109/dsn53405.2022.00057)
77. Lunardi, Caio et al. (2018). *On the Efficacy of ECC and the Benefits of FinFET Transistor Layout for GPU Reliability*. IEEE Transactions on Nuclear Science. Abstract only. Shows ECC is not a panacea for GPU radiation faults and that technology choice can match ECC benefits [doi:10.1109/tns.2018.2823786](https://doi.org/10.1109/tns.2018.2823786)
78. Oliveira, Daniel et al. (2015). *Evaluation and Mitigation of Radiation-Induced Soft Errors in Graphics Processing Units*. IEEE Transactions on Computers. Abstract only. Radiation experiments quantifying GPU memory neutron sensitivity and hardening tradeoffs [doi:10.1109/tc.2015.2444855](https://doi.org/10.1109/tc.2015.2444855)
79. Previlon, Fritz et al. (2020). *Characterizing and Exploiting Soft Error Vulnerability Phase Behavior in GPU Applications*. IEEE Transactions on Dependable and Secure Computing. Abstract only. Reveals phase behavior of GPU soft-error vulnerability and uses it to speed up fault injection [doi:10.1109/tdsc.2020.2991136](https://doi.org/10.1109/tdsc.2020.2991136)
80. Rahman, Md Hasanur et al. (2024). *Druto: Upper-Bounding Silent Data Corruption Vulnerability in GPU Applications*. 2024 IEEE International Parallel and Distributed Processing Symposium (IPDPS). Abstract only. Shows input dependence of GPU SDC rates; standard benchmark inputs can underestimate SDC risk by up to 74x [doi:10.1109/ipdps57955.2024.00058](https://doi.org/10.1109/ipdps57955.2024.00058)
81. Mukherjee, Shubhendu S. et al. (2003). *A systematic methodology to compute the architectural vulnerability factors for a high performance microprocessor*. Proceedings of the 36th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO-36). Abstract only. Foundational AVF methodology linking structure error rates to program-visible faults in microprocessors [doi:10.1109/micro.2003.1253181](https://doi.org/10.1109/micro.2003.1253181)
82. Sridharan, Vilas & Kaeli, David R. (2009). *Eliminating microarchitectural dependency from Architectural Vulnerability*. 2009 IEEE 15th International Symposium on High Performance Computer Architecture. Abstract only. PVF metric separates software from hardware fault masking, informing soft-error-aware software design [doi:10.1109/hpca.2009.4798243](https://doi.org/10.1109/hpca.2009.4798243)
83. Saleh, A.M. et al. (1990). *Reliability of scrubbing recovery-techniques for memory systems*. IEEE Transactions on Reliability. Abstract only. Foundational analytic treatment of scrubbing-based transient-error recovery in SEC-DED memories [doi:10.1109/24.52622](https://doi.org/10.1109/24.52622)
84. Reviriego, Pedro et al. (2010). *Optimizing Scrubbing Sequences for Advanced Computer Memories*. IEEE Transactions on Device and Materials Reliability. Abstract only. Shows scrubbing order matters when multibit errors dominate, with up to 2x MTTF improvement [doi:10.1109/tdmr.2009.2039481](https://doi.org/10.1109/tdmr.2009.2039481)