A large-scale study of failures in high-performance computing systems
Published 21 July 2006
Bianca Schroeder, Garth A. Gibson
Citations586
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
Designing highly dependable systems requires a good understanding of failure characteristics. Unfortunately, little raw data on failures in large IT installations is publicly available. This paper analyzes failure data recently made publicly available by one of the largest high-performance computing sites. The data has been collected over the past 9 years at Los Alamos National Laboratory and includes 23000 failures recorded on more than 20 different systems, mostly large clusters of SMP and NUMA nodes. We study the statistics of the data, including the root cause of failures, the mean time between failures, and the mean time to repair. We find for example that average failure rates differ wildly across systems, ranging from 20-1000 failures per year, and that time between failures is modeled well by a Weibull distribution with decreasing hazard rate. From one system to another, mean repair time varies from less than an hour to more than a day, and repair times are well modeled by a lognormal distribution
Keywords
Computer Science
Journal of the Royal Statistical Society Series A (Statistics in Society)Introduction to Probability Models.
4,596 Citations1995Flavia Jolliffe, Sheldon M. Ross
IEEE/ACM Transactions on NetworkingSelf-similarity through high-variability: statistical analysis of Ethernet LAN traffic at the source level
1,645 Citations1997Walter Willinger, Murad S. Taqqu +2 more
The superposition of many ON/OFF sources with strictly alternating ON- and OFF-periods and whose ON-periods or OFF-periods exhibit the Noah effect produces aggregate network traffic that exhibits the Joseph effect.
Disk failures in the real world: what does an MTTF of 1,000,000 hours mean to you?
773 Citations2007Bianca Schroeder, Garth A. Gibson
This paper presents and analyzes field-gathered disk replacement data from a number of large production systems, including high-performance computing sites and internet services sites, finding evidence, based on records of disk replacements in the field, that failure rate is not constant with age, and that, rather than a significant infant mortality effect, the authors see a significant early onset of wear-out degradation.
Why Do Computers Stop and What Can Be Done About It?
671 Citations2024Jim Gray
UC BerkeleyWhy Do Internet Services Fail, and What Can Be Done About It?
603 Citations2002David Oppenheimer, Archana Ganapathi +1 more
It is found that operator errors are their primary cause, operator error is the most difficult type of failure to mask, service front-ends are responsible for more problems than service back-ends but fewer minutes of unavailability, and that online testing and more thoroughly exposing and detecting component failures could reduce system failure rates for at least one service.
IEEE Transactions on ReliabilityA census of Tandem system availability between 1985 and 1990
324 Citations1990Jodi Gray
A census of customer outages reported to Tandem indicates that software is now the major source of reported outages, followed by system operations, a dramatic shift from the statistics for 1985.
Failure data analysis of a large-scale heterogeneous server environment
236 Citations2004R.K. Sahoo, Mark S. Squillante +2 more
This paper analyzes the empirical and statistical properties of system errors and failures from a network of nearly 400 heterogeneous servers running a diverse workload over a year and shows that the system error and failure patterns are comprised of time-varying behavior containing long stationary intervals.
ACM Transactions on Computer SystemsMeasurement and modeling of computer reliability as affected by system activity
190 Citations1986R.K. Iyer, David J. Rossetti +1 more
There are strong indications that permanent failures are both caused and discovered by increased activity, and more measurements and experiments are necessary to determine their respective contributions to the measured workload/failure relationship.
IEEE Transactions on ReliabilityError log analysis: statistical modeling and heuristic trend analysis
184 Citations1990Tse‐Chieh Lin, Daniel P. Siewiorek
The authors present the results of an analysis that demonstrates that the log is composed of at least two error processes: transient and intermittent, and it is shown that the DFT can extract intermittent errors from the error log and uses only one fifth of the error-log entry points required for failure prediction.
Lecture notes in computer scienceModeling Machine Availability in Enterprise and Wide-Area Distributed Computing Environments
182 Citations2005Daniel Nurmi, John Brevik +1 more
For all three data sets, it is found that a hyperexponential model fits slightly more accurately than a Weibull, but that both are substantially better choices than either an exponential or Pareto.
Failure data analysis of a LAN of Windows NT based computers
147 Citations2003M. Kalyanakrishnam, Zbigniew Kalbarczyk +1 more
The key observations from this study are: most of the problems that lead to reboots are software related, and the average availability is not a good measure to characterize this type of network service.
Networked Windows NT system field failure data analysis
140 Citations2003Jun Xu, Zbigniew Kalbarczyk +1 more
This paper presents a measurement-based dependability study of a Networked Windows NT system based on field data collected from NT System Logs from 503 servers running in a production environment over a four-month period.
A longitudinal survey of Internet host reliability
128 Citations2002Darrell D. E. Long, Andrew Muir +1 more
A second experiment using a fault-tolerant replicated monitoring tool directly measures TTF, TTR, and availability by polling many sites frequently from several locations and finds that failure and repair are unlikely to follow Poisson processes.
Experimental assessment of workstation failures and their impact on checkpointing systems
121 Citations2002James S. Plank, Wael Elwasif
A significant result of this work is that although the base assumptions of the theoretical research do not hold, many of the results are still applicable.
Lecture notes in computer sciencePerformance Implications of Failures in Large-Scale Cluster Scheduling
121 Citations2005Yanyong Zhang, Mark S. Squillante +2 more
A general failure modeling framework is developed based on recent results from large-scale clusters and then this framework is exploited to conduct a detailed performance analysis of the impact of failures on system performance for a wide range of scheduling policies.
Improving cluster availability using workstation validation
112 Citations2002Taliver Heath, Richard P. Martin +1 more
This work focuses on using the entire commodity workstation as a component because it possesses natural, fault-isolated interfaces, and defines a failure event as a reboot because not only is a workstation unavailable during a reboot, but also because reboots are symptomatic of a larger class of failures, such as configuration and operator errors.
Quality and Reliability Engineering InternationalMeasuring system and software reliability using an automated data collection process
95 Citations1995Thomas Brendan Murphy, Ted Gent
The on-system data logging process and analysis methodology used by Digital to measure system, product and operating system reliability with examples of the application of the techniques that provide insight into the causes of failures are described.
ACM SIGMETRICS Performance Evaluation ReviewA case for two-level distributed recovery schemes
86 Citations1995Nitin H. Vaidya
Munich Personal RePEc Archive (Ludwig Maximilian University of Munich)Workload, Performance, and Reliability of Digital Computing Systems.
85 Citations1980Xavier Castillo, Daniel P. Siewiorek
A new modeling methodology to characterize failure processes in Time-sharing systems due to hardware transients and software errors is summarized, which gives quantitative relationships between performance, workload, and (lack of) reliability for digital compiuting systems.
A case for two-level distributed recovery schemes
57 Citations1995Nitin H. Vaidya
This paper demonstrates that, it is often advantageous to use "two-level" recovery schemes, which tolerates the more probable failures with low performance overhead, while the less probable failures may be tolerated with a higher overhead.
Analysis of workload influence on dependability
52 Citations2003John F. Meyer, Lise Wei
A Markov renewal process model is established that represents the interaction among workload and fault accumulation ins systems for which fault tolerance can be characterized by fault margins and shows how the probabilistic nature of T/sub f/ can be formulated directly in terms of parameters regarding workload, fault arrivals, and fault margins.
Journal of Parallel and Distributed ComputingCheckpointing in Distributed Computing Systems
28 Citations1996Kenneth F. Wong, Mark A. Franklin
This paper examines the performance of synchronous checkpointing in a distributed computing environment with and without load redistribution, and develops performance models that are coupled with failure-free performance and the speedup functions associated with implementation of parallel algorithms.
Performance analysis of two time-based coordinated checkpointing protocols
23 Citations2002Gerard Patrick Kavanaugh, William H. Sanders
This paper analyzes and compares two time-based checkpointing protocols, and obtains a closed-form expression for the forward progress of the two protocols considered, and determines the checkpoint interval value that will maximize forward progress.