IBDP: An Industrial Big Data Ingestion and Analysis Platform and Case Studies
Published 1 October 2015
Cun Ji, Shijun Liu, Chenglei Yang, Lei Wu, Li Pan
Citations15
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
In the IBDP, the platform is integrated HDFS, Spark, Hive, HBase, Flume, Sqoop, OpenStack etc, which works well for industrial data ingestion and analysis.
Abstract
The Internet of Things (IoT) brings traditional Internet industry and society with new trends and promising technologies. For industrial information with high amount and renewal speed characteristics, resulting in difficult data ingestion and analysis, this paper presented an Industrial Big Data ingestion and analysis Platform (IBDP). In the platform, we integrated HDFS, Spark, Hive, HBase, Flume, Sqoop, OpenStack etc. It works well for industrial data ingestion and analysis. In addition, we report some case studies on industrial big data processing flows respect to different data types.
Keywords
Computer ScienceBusiness, Management and Accounting
Communications of the ACMMapReduce
18,538 Citations2008Jay B. Dean, Sanjay Ghemawat
This presentation explains how the underlying runtime system automatically parallelizes the computation across large-scale clusters of machines, handles machine failures, and schedules inter-machine communication to make efficient use of the network and disks.
Communications of the ACMA view of cloud computing
8,997 Citations2010Michael Armbrust, Armando Fox +9 more
The clouds are clearing the clouds away from the true potential and obstacles posed by this computing capability.
ACM Transactions on Computer SystemsBigtable
3,432 Citations2008Fay W. Chang, Jay B. Dean +7 more
The simple data model provided by Bigtable is described, which gives clients dynamic control over data layout and format, and the design and implementation of Bigtable are described.
IEEE Transactions on Knowledge and Data EngineeringData mining with big data
2,395 Citations2013Xindong Wu, Xingquan Zhu +2 more
A HACE theorem is presented that characterizes the features of the Big Data revolution, and a Big Data processing model is proposed, from the data mining perspective, which involves demand-driven aggregation of information sources, mining and analysis, user interest modeling, and security and privacy considerations.
Cambridge University Press eBooksMining of Massive Datasets
2,301 Citations2011Anand Rajaraman, Jeffrey D. Ullman
Synthesis lectures on computer architectureThe Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines
1,746 Citations2009Luiz André Barroso, Urs Hölzle
The architecture of WSCs is described, the main factors influencing their design, operation, and cost structure, and the characteristics of their software base are described.
Proceedings of the VLDB EndowmentChallenges and opportunities with big data
1,056 Citations2012Alexandros Labrinidis, H. V. Jagadish
The controversies and myths surrounding Big Data are explored, to try to explore the controversies and debunk the myths around Big Data.
ACM SIGKDD Explorations NewsletterMining big data
676 Citations2013Wei Fan, Albert Bifet
This issue introduces four articles, written by influential scientists in the field, covering the most interesting and state-of-the-art topics on Big Data mining, and presents a broad overview of the topic, its current status, controversy, and a forecast to the future.
International Journal of Computer ApplicationsOpenStack: Toward an Open-source Solution for Cloud Computing
556 Citations2012Omar Sefraoui, Mohammed Aissaoui +1 more
A comparative study between some of the IaaS (Infrastructure as a Service) commonly used to select the best suited one for deployment and research development in the field of cloud computing is presented.
Synthesis lectures on computer architectureThe Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines, Second edition
491 Citations2013Luiz André Barroso, Jimmy Clidaras +1 more
The architecture of WSCs is described, the main factors influencing their design, operation, and cost structure, and the characteristics of their software base are described.
IEEE Cloud ComputingStreaming Big Data Processing in Datacenter Clouds
232 Citations2014Rajiv Ranjan
Government and business organizations are now overflowing with data, easily aggregating to terabytes or even petabytes of information.
login Usenix Mag.Fast and Interactive Analytics over Hadoop Data with Spark
162 Citations2012Michael J. Franklin, Thomas Siebel +7 more
Matei Zaharia is a fifth year PhD student at UC Berkeley, working with Scott Shenker and Ion Stoica on topics in computer systems, networks, cloud computing, and big data.
Survey of real-time processing systems for big data
133 Citations2014Xiufeng Liu, Nadeem Iftikhar +1 more
A survey of the open source technologies that support big data processing in a real-time/near real- time fashion, including their system architectures and platforms is presented.
ScienceAgencies Rally to Tackle Big Data
95 Citations2012Jeffrey Mervis
A federal effort to improve the nation's ability to manage, understand, and act upon the 1.2 zettabytes of electronic data generated each year and computer scientists welcome the spotlight that the White House is shining on big-data research.
Cloud Computing
94 Citations2011Wang, Lizhe, Ranjan, Rajiv +2 more
Comprehensive and timely, Cloud Computing: Methodology, Systems, and Applications summarizes progress in state-of-the-art research and offers step-by-step instruction on how to implement it.