The structural cause of file size distributions
Published 1 June 2001
Allen B. Downey
Citations99
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
We propose a user model that explains the shape of the distribution of file sizes in local file systems and in the World Wide Web. We examine evidence from 562 file systems, 38 web clients and 6 web servers, and find that the model is a good description of these systems. These results cast doubt on the widespread view that the distribution of file sizes is long-tailed and that long-tailed distributions are the cause of self-similarity in the Internet.
Keywords
Computer Science
IEEE/ACM Transactions on NetworkingWide area traffic: the failure of Poisson modeling
3,719 Citations1995Vern Paxson, Sally Floyd
It is found that user-initiated TCP session arrivals, such as remote-login and file-transfer, are well-modeled as Poisson processes with fixed hourly rates, but that other connection arrivals deviate considerably from Poisson.
IEEE/ACM Transactions on NetworkingSelf-similarity through high-variability: statistical analysis of Ethernet LAN traffic at the source level
1,645 Citations1997Walter Willinger, Murad S. Taqqu +2 more
The superposition of many ON/OFF sources with strictly alternating ON- and OFF-periods and whose ON-periods or OFF-periods exhibit the Noah effect produces aggregate network traffic that exhibits the Joseph effect.
Generating representative Web workloads for network and server performance evaluation
1,438 Citations1998Paul Barford, Mark Crovella
This paper applies a number of observations of Web server usage to create a realistic Web workload generation tool which mimics a set of real users accessing a server and addresses the technical challenges to satisfying this large set of simultaneous constraints on the properties of the reference stream.
IEEE NetworkA workload characterization study of the 1998 World Cup Web site
765 Citations2000Martin Arlitt, Tai Jin
It is found that improvements in the caching architecture of the World Wide Web are changing the workloads of Web servers, but major improvements to that architecture are still necessary.
ACM SIGMETRICS Performance Evaluation ReviewWeb server workload characterization
753 Citations1996Martin Arlitt, Carey Williamson
ACM SIGCOMM Computer Communication ReviewProof of a fundamental result in self-similar traffic modeling
677 Citations1997Murad S. Taqqu, Walter Willinger +1 more
The superposition of many ON/OFF sources with strictly alternating ON- and OFF-periods can produce aggregate network traffic that exhibits the Joseph Effect, and this mathematical result can be combined with modern high-performance computing capabilities to yield a simple and efficient linear-time algorithm for generating self-similar traffic traces.
Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topicsHighly optimized tolerance: A mechanism for power laws in designed systems
550 Citations1999Jean M. Carlson, John C. Doyle
This work introduces a mechanism for generating power law distributions, referred to as highly optimized tolerance (HOT), which is motivated by biological organisms and advanced engineering technologies, and investigates the mechanism in the context of percolation and sand pile models in order to emphasize the sharp contrasts between SOC and self-organized criticality.
Characteristics of WWW Client-based Traces
529 Citations1995Carlos R. Cunha, Azer Bestavros +1 more
This paper presents a descriptive statistical summary of the traces of actual executions of NCSA Mosaic, and shows that many characteristics of WWW use can be modelled using power-law distributions, including the distribution of document sizes, the popularity of documents as a function of size, and the Distribution of user requests for documents.
ACM SIGMETRICS Performance Evaluation ReviewSelf-similarity in World Wide Web traffic
526 Citations1996Mark Crovella, Azer Bestavros
On the relationship between file sizes, transport protocols, and self-similar network traffic
439 Citations2002Ki‐Hong Park, Gitae Kim +1 more
A mechanism that gives rise to self-similar network traffic is examined, the transfer of files or messages whose size is drawn from a heavy-tailed distribution is studied and performance implications of self-Similarity are discussed as represented by various performance measures.
Dynamics of IP traffic
394 Citations1999Anja Feldmann, Anna C. Gilbert +2 more
It is shown that scaling analysis has the ability to extract relevant information about the time-scale dynamics of Internet traffic, thereby, it is hoped, making these techniques available to a larger segment of the networking research community.
World Wide WebChanges in Web client access patterns: Characteristics and caching implications
377 Citations1999Paul Barford, Azer Bestavros +2 more
This study compares two measurements of Web client workloads separated in time by three years, both captured from the same computing facility at Boston University and finds that for the computing facility represented by traces between 1995 and 1998, the benefits of using size‐based caching policies have diminished and the potential for caching requested files in the network has declined.
Heavy-tailed probability distributions in the World Wide Web
328 Citations1998Mark Crovella, Murad S. Taqqu +1 more
Evidence is presented that a number of le size distributions in the Web exhibit heavy tails, including les requested by users, les transmitted through the network, transmission durations of les, and les stored on servers, that are primarily determined by the distribution of les available on the Web.
ACM SIGCOMM Computer Communication ReviewThe changing nature of network traffic
292 Citations1998Anja Feldmann, Anna C. Gilbert +2 more
Preliminary results from an in-depth, wavelet-based analysis of a set of high-quality, packet-level traffic measurements, collected over the last 6-7 years from a number of different wide-area networks, show that the changing nature of WAN traffic can be naturally accounted for by self-similar traffic models.
The chaotic nature of TCP congestion control
291 Citations2002András Veres, Miklós Boda
It is shown that this argument is not necessary to explain self-similarity, neither is randomness required, and rather, TCP itself as a deterministic process creates chaos, which generates self-Similarity.
Self-similarity in World Wide Web traffic
262 Citations1996Mark Crovella, Azer Bestavros
It is shown that the self-similarity in WWW traffic can be explained based on the underlying distributions of WWW document sizes, the effects of caching and user preference in file transfer, the effect of user "think time", and the superimposition of many such transfers in a local area network.
Self-similarity through high-variability
262 Citations1995Walter Willinger, Murad S. Taqqu +2 more
This paper provides a plausible physical explanation for the occurrence of self-similarity in high-speed network traffic based on convergence results for processes that exhibit high variability (i.e., infinite variance) and is supported by detailed statistical analyses of real-time traffic measurements from Ethernet LAN's at the level of individual sources.
A large-scale study of file-system contents
215 Citations1999John R. Douceur, William J. Bolosky
It is found that file and directory sizes are fairly consistent across file systems, but file lifetimes vary widely and are significantly affected by the job function of the user.
Self‐Similar Network Traffic: An Overview
205 Citations2000Ki‐Hong Park, Walter Willinger
Methodology And Computing In Applied ProbabilityEstimating the Heavy Tail Index from Scaling Properties
123 Citations1999Mark Crovella, Murad S. Taqqu
Evidence that the scaling estimator appears to increase in accuracy as the size of the dataset grows is presented, which is particularly suited for large datasets, as are increasingly encountered in measurements of telecommunications and computing systems.
ACM SIGMETRICS Performance Evaluation ReviewWorkload characterization of a Web proxy in a cable modem environment
113 Citations1999Martin Arlitt, Rich Friedrich +1 more
It is found that when presented with faster access speeds users are willing to download extremely large files, which increases the importance of caching for ensuring the scalability of the Web.
M|G|∞ input processes: a versatile class of models for network traffic
86 Citations2002Minothi Parulekar, Armand M. Makowski
The M|G|/spl infin/ input process is suggested as a viable model for network traffic due to its versatility and tractability and the insufficiency of the short- vs. long-range dependence characterization of an input process as a means to accurately describe the corresponding buffer dynamics.
Heavy tails, generalized coding, and optimal Web layout
41 Citations2002Xiaoyun Zhu, Jie Yu +1 more
The results suggest that heavy tails are a permanent and ubiquitous feature of Internet traffic, and not an artifice of current applications or user behavior, and suggest new ways of thinking about protocol design that combines insights from information and control theory with traditional networking.
The adverse impact of the TCP congestion-control mechanism in heterogeneous computing systems
22 Citations2002Wu-chun Feng, Peerapol Tinnakornsrisuphap
This study illustrates how TCP modulates application traffic in such a way as to adversely affect network performance in a heterogeneous computing system, and develops a stochastic model for TCP Reno to demonstrate how the burstiness in TCP Reno can be modeled.