The unified logging infrastructure for data analytics at Twitter
Proceedings of the VLDB EndowmentPublished 1 August 2012
George Lee, Jimmy Lin, Chuang Liu, Andrew Lorek, Dmitriy Ryaboy
Citations95
SJR quartileQ1
SJR score1.83
SNIP1.80
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper presents Twitter's production logging infrastructure and its evolution from application-specific logging to a unified "client events" log format, where messages are captured in common, well-formatted, flexible Thrift messages.
Abstract
In recent years, there has been a substantial amount of work on large-scale data analytics using Hadoop-based platforms running on large clusters of commodity machines. A less-explored topic is how those data, dominated by application logs, are collected and structured to begin with. In this paper, we present Twitter's production logging infrastructure and its evolution from application-specific logging to a unified "client events" log format, where messages are captured in common, well-formatted, flexible Thrift messages. Since most analytics tasks consider the user session as the basic unit of analysis, we pre-materialize "session sequences", which are compact summaries that can answer a large class of common queries quickly. The development of this infrastructure has streamlined log collection and data analysis, thereby improving our ability to rapidly experiment and iterate on various aspects of the service.
Keywords
Computer Science
Journal of Molecular BiologyBasic local alignment search tool
94,539 Citations1990Stephen F. Altschul, Warren Gish +3 more
A new approach to rapid sequence comparison, basic local alignment search tool (BLAST), directly approximates alignments that optimize a measure of local similarity, the maximal segment pair (MSP) score.
Communications of the ACMMapReduce
18,538 Citations2008Jay B. Dean, Sanjay Ghemawat
This presentation explains how the underlying runtime system automatically parallelizes the computation across large-scale clusters of machines, handles machine failures, and schedules inter-machine communication to make efficient use of the network and disks.
Foundations of statistical natural language processing
9,996 Citations1999Christopher D. Manning, Hinrich Schütze
Mining sequential patterns
5,115 Citations2002R. K. Agrawal, Ramakrishnan Srikant
Three algorithms are presented to solve the problem of mining sequential patterns over databases of customer transactions, and empirically evaluating their performance using synthetic data shows that two of them have comparable performance.
Word association norms, mutual information, and lexicography
3,674 Citations1990Kenneth Church, Patrick Hanks
Speech and language processing
3,589 Citations2010Dan Jurafsky, James Martin +1 more
It is now clear that HAL’s creator, Arthur C. Clarke, was a little optimistic in predicting when an artificial agent such as HAL would be avail-able.
Accurate methods for the statistics of surprise and coincidence
2,688 Citations1993Ted Dunning
Pig latin
1,744 Citations2008Christopher Olston, Benjamin Reed +3 more
A new language called Pig Latin is described, designed to fit in a sweet spot between the declarative style of SQL, and the low-level, procedural style of map-reduce, which is an open-source, Apache-incubator project, and available for general use.
ZooKeeper: wait-free coordination for internet-scale systems
1,312 Citations2010P. G. Hunt, Mahadev Konar +2 more
ZooKeeper provides a per client guarantee of FIFO execution of requests and linearizability for all requests that change the ZooKeeper state to enable the implementation of a high performance processing pipeline with read requests being satisfied by local servers.
Hive - a petabyte scale data warehouse using Hadoop
925 Citations2010Ashish Thusoo, Joydeep Sen Sarma +7 more
Hive is presented, an open-source data warehousing solution built on top of Hadoop that supports queries expressed in a SQL-like declarative language - HiveQL, which are compiled into map-reduce jobs that are executed using Hadoops.
Proceedings of the VLDB EndowmentHadoopDB
856 Citations2009A. A. ABOUZEID, Kamil Bajda-Pawlikowski +3 more
This paper explores the feasibility of building a hybrid system that takes the best features from both technologies; the prototype built approaches parallel databases in performance and efficiency, yet still yields the scalability, fault tolerance, and flexibility of MapReduce-based systems.
Marketing ScienceModeling Online Browsing and Path Analysis Using Clickstream Data
565 Citations2004Alan L. Montgomery, Shibo Li +2 more
This work shows how path information can be categorized and modeled using a dynamic multinomial probit model of Web browsing and finds that purchasers can be predicted with more than 40% accuracy, which is much better than the benchmark 7% purchase conversion prediction rate made without path information.
Data warehousing and analytics infrastructure at facebook
433 Citations2010Ashish Thusoo, Zheng Shao +6 more
This paper presents how Scribe, Hadoop and Hive together form the cornerstones of the log collection, storage and analytics infrastructure at Facebook and enabled us to implement a data warehouse that stores more than 15PB of data and loads more than 60TB of new data every day.
Apache hadoop goes realtime at Facebook
432 Citations2011Dhruba Borthakur, Jonathan David Gray +10 more
The reasons why Facebook chose Hadoop and HBase over other systems such as Apache Cassandra and Voldemort are described and the application's requirements for consistency, availability, partition tolerance, data model and scalability are discussed.
Proceedings of the VLDB EndowmentBuilding a high-level dataflow system on top of Map-Reduce
413 Citations2009Alan Gates, Olga Natkovich +7 more
Pig is a high-level dataflow system that aims at a sweet spot between SQL and Map-Reduce, and performance comparisons between Pig execution and raw Map- Reduce execution are reported.
Practical guide to controlled experiments on the web
383 Citations2007Ron Kohavi, Randal M. Henne +1 more
This work provides a practical guide to conducting online experiments, and shares key lessons that will help practitioners in running trustworthy controlled experiments, including statistical power, sample size, and techniques for variance reduction.
Proceedings of the VLDB EndowmentHadoop++
368 Citations2010Jens Dittrich, Jorge-Arnulfo Quiané-Ruiz +4 more
This paper proposes a new type of system named Hadoop++: it boosts task performance without changing the Hadooper framework at all (Hadoop does not even 'notice it'), and shows the superiority of Hadoo++ over both Hadoops and HadoOPDB for tasks related to indexing and join processing.
LifeFlow
286 Citations2011Krist Wongsuphasawat, John Alexis Guerra Gómez +4 more
A novel interactive visual overview of event sequences called LifeFlow is introduced, which is scalable, can summarize all possible sequences, and represents the temporal spacing of the events within sequences.
RCFile: A fast and space-efficient data placement structure in MapReduce-based warehouse systems
258 Citations2011Yongqiang He, Rubao Lee +5 more
This paper presents a big data placement structure called RCFile (Record Columnar File) and its implementation in the Hadoop system and shows the effectiveness of RCFile in satisfying the four requirements.
A generative constituent-context model for improved grammar induction
253 Citations2001Dan Klein, Christopher D. Manning
A generative distributional model for the unsupervised induction of natural language syntax which explicitly models constituent yields and contexts is presented, giving the best published un-supervised parsing results on the ATIS corpus.
Large-scale machine learning at twitter
218 Citations2012Jimmy Lin, Alek Kolcz
A case study of Twitter's integration of machine learning tools into its existing Hadoop-based, Pig-centric analytics platform to provide predictive analytics capabilities that incorporate machine learning, focused specifically on supervised classification.
CHI 2009 Extended Abstracts
166 Citations2009Nicholas A. Knouf
A Comparative Evaluation of Collocation Extraction Techniques
134 Citations2002Darren Pearce
This paper describes an experiment that attempts to compare a range of existing collocation extraction techniques as well as the implementation of a new technique based on tests for lexical substitutability, and makes initial proposals for a common framework for evaluation of collocations extraction techniques.
Llama
126 Citations2011Yu-Ting Lin, Divyakant Agrawal +3 more
This paper proposes the design of a new cluster-based data warehouse system, LLama, a hybrid data management system which combines the features of row-wise and column-wise database systems and designs a new join algorithm to facilitate fast join processing.
Trojan data layouts
112 Citations2011Alekh Jindal, Jorge-Arnulfo Quiané-Ruiz +1 more
This paper proposes a new data layout, coined Trojan Layout, that internally organizes data blocks into attribute groups according to the workload in order to improve data access times and compares Trojan Layouts against Hadoop using Row and PAX layouts.
Efficient processing of data warehousing queries in a split execution environment
72 Citations2011Kamil Bajda-Pawlikowski, Daniel J. Abadi +2 more
This work considers processing data warehousing queries over very large datasets by analyzing the complexity of this problem in the split execution environment of HadoopDB, with particular focus on join and aggregation operations.
Distributed cube materialization on holistic measures
54 Citations2011Arnab Nandi, Cong Yu +2 more
This paper identifies an important subset of holistic measures and introduces MR-Cube, a MapReduce based framework for efficient cube computation on these measures, and demonstrates that, unlike existing techniques which cannot scale to the 100 million tuple mark for the authors' datasets, MR- Cube successfully and efficiently computes cubes with holistic measures over billion-tuple datasets.
Full-text indexing for optimizing selection operations in large-scale data analytics
46 Citations2011Jimmy Lin, Dmitriy Ryaboy +1 more
It is shown that it is possible to leverage a full-text index to optimize selection operations on text fields within records in Hadoop, and moderate improvements in end-to-end query running times and substantial savings in terms of cumulative processing time at the worker nodes are shown.
Information RetrievalModeling actions of PubMed users with n-gram language models
26 Citations2008Jimmy Lin, W. John Wilbur
This article presents a study of logs from PubMed, the public gateway to the MEDLINE database of bibliographic records from the medical and biomedical primary literature, which encode user actions as string sequences and model these sequences using n-gram language models.
LifeFlow
11 Citations2011Krist Wongsuphasawat, John Alexis Guerra Gómez +4 more
This work introduces a novel interactive visual overview of event sequences called LifeFlow, which is scalable, can summarize all possible sequences, and represents the temporal spacing of the events within sequences.
Funnel report mining for the MSN network
6 Citations2001Teresa Mah, Hank Hoek +1 more
A framework for funnel analysis is defined and a tree-based solution is provided successfully to extract all relevant funnels using only one scan of the data file.