Towards a Big Data Analytics Framework for IoT and Smart City Applications
Modeling and optimization in science and technologiesPublished 4 November 2014
Martin Strohbach, Holger Ziekow, Vangelis Gazis, Navot Akiva
Citations153
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This chapter shows how an integrated Big Data analytical framework for Internet of Things and Smart City application could look like and presents an initial version of such a framework mainly addressing the volume and velocity challenge.
Abstract
An increasing amount of valuable data sources, advances in Internet of Things and Big Data technologies as well as the availability of a wide range of machine learning algorithms offers new potential to deliver analytical services to citizens and urban decision makers. However, there is still a gap in combining the current state of the art in an integrated framework that would help reducing development costs and enable new kind of services. In this chapter, we show how such an integrated Big Data analytical framework for Internet of Things and Smart City application could look like. The contributions of this chapter are threefold: (1) we provide an overview of Big Data and Internet of Things technologies including a summary of their relationships, (2) we present a case study in the smart grid domain that illustrates the high-level requirements towards such an analytical Big Data framework, and (3) we present an initial version of such a framework mainly addressing the volume and velocity challenge. The findings presented in this chapter are extended results from the EU funded project BIG and the German funded project PEC.
Keywords
Computer ScienceEngineering
Communications of the ACMMapReduce
18,538 Citations2008Jay B. Dean, Sanjay Ghemawat
This presentation explains how the underlying runtime system automatically parallelizes the computation across large-scale clusters of machines, handles machine failures, and schedules inter-machine communication to make efficient use of the network and disks.
ACM SIGKDD Explorations NewsletterThe WEKA data mining software
17,849 Citations2009Mark Hall, Eibe Frank +4 more
This paper provides an introduction to the WEKA workbench, reviews the history of the project, and, in light of the recent 3.6 stable release, briefly discusses what has been added since the last stable version (Weka 3.4) released in 2003.
Computer NetworksThe Internet of Things: A survey
15,290 Citations2010Luigi Atzori, Antonio Iera +1 more
This survey is directed to those who want to approach this complex discipline and contribute to its development, and finds that still major issues shall be faced by the research community.
Future Generation Computer SystemsInternet of Things (IoT): A vision, architectural elements, and future directions
11,977 Citations2013Jayavardhana Gubbi, Rajkumar Buyya +2 more
This paper presents a Cloud centric vision for worldwide implementation of Internet of Things, and expands on the need for convergence of WSN, the Internet and distributed computing directed at technological research community.
Information Processing & ManagementTerm-weighting approaches in automatic text retrieval
9,532 Citations1988Gerard Salton, Chris Buckley
This paper summarizes the insights gained in automatic term weighting, and provides baseline single term indexing models with which other more elaborate content analysis procedures can be compared.
now publishers, Inc. eBooksOpinion Mining and Sentiment Analysis
6,745 Citations2008Bo Pang, Lillian Lee
Foundations and Trends® in Information RetrievalOpinion Mining and Sentiment Analysis
6,204 Citations2008Bo Pang, Lillian Lee
This survey covers techniques and approaches that promise to directly enable opinion-oriented information-seeking systems and focuses on methods that seek to address the new challenges raised by sentiment-aware applications, as compared to those that are already present in more traditional fact-based analysis.
Big data: The next frontier for innovation, competition, and productivity
5,228 Citations2011James Manyika
The amount of data in the authors' world has been exploding, and analyzing large data sets will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus, according to research by MGI and McKinsey.
Ad Hoc NetworksInternet of things: Vision, applications and research challenges
3,556 Citations2012Daniele Miorandi, Sabrina Sicari +2 more
A survey of technologies, applications and research challenges for Internetof-Things is presented, in which digital and physical entities can be linked by means of appropriate information and communication technologies to enable a whole new class of applications and services.
Encyclopedia of Machine Learning and Data MiningSentiment Analysis and Opinion Mining
3,319 Citations2017Lei Zhang, Bing Liu
Synthesis lectures on human language technologiesSentiment Analysis and Opinion Mining
3,241 Citations2012Bing Liu
This book is a comprehensive introductory and survey text that covers all important topics and the latest developments in the field with over 400 references and is suitable for students, researchers and practitioners who are interested in social media analysis in general and sentiment analysis in particular.
IEEE Transactions on Industrial InformaticsDemand Side Management: Demand Response, Intelligent Energy Systems, and Smart Loads
2,810 Citations2011Peter Pálenský, Dietmar Dietrich
An overview and a taxonomy for DSM is given, the various types of DSM are analyzed, and an outlook on the latest demonstration projects in this domain is given.
Hadoop: The Definitive Guide
2,714 Citations2009Tom White
This comprehensive resource demonstrates how to use Hadoop to build reliable, scalable, distributed systems: programmers will find details for analyzing large datasets, and administrators will learn how to set up and run Hadoops clusters.
ACM Transactions on Database SystemsTinyDB: an acquisitional query processing system for sensor networks
1,905 Citations2005Samuel Madden, Michael J. Franklin +2 more
This work evaluates issues in the context of TinyDB, a distributed query processor for smart sensor devices, and shows how acquisitional techniques can provide significant reductions in power consumption on the authors' sensor devices.
The MIT Press eBooksMap-Reduce for Machine Learning on Multicore
1,253 Citations2007Cheng-Tao Chu, Sang Kyun Kim +5 more
This work shows that algorithms that fit the Statistical Query model can be written in a certain "summation form," which allows them to be easily parallelized on multicore computers and shows basically linear speedup with an increasing number of processors.
The Design of the Borealis Stream Processing Engine
1,232 Citations2005Daniel J. Abadi, Yanif Ahmad +10 more
This paper outlines the basic design and functionality of Borealis, and presents a highly flexible and scalable QoS-based optimization model that operates across server and sensor networks and a new fault-tolerance model with flexible consistency-availability trade-offs.
TelegraphCQ
732 Citations2003Sirish Chandrasekaran, Owen Cooper +8 more
The current version of TelegraphCQ is shown, which is implemented by leveraging the code base of the open source PostgreSQL database system, which found that a significant portion of the PostgreSQL code was easily reusable.
Lecture notes in computer scienceSentiment Knowledge Discovery in Twitter Streaming Data
580 Citations2010Albert Bifet, Eibe Frank
To deal with streaming unbalanced classes, a sliding window Kappa statistic is proposed for evaluation in time-changing data streams, and a study on Twitter data is performed using learning algorithms for data streams.
SPADE
421 Citations2008Buğra Gedik, Henrique Andrade +3 more
Spade is the System S declarative stream processing engine that allows developers to construct their applications with fine granular stream operators without worrying about the performance implications that might exist, even in a distributed system.
Query Processing, Approximation, and Resource Management in a Data Stream Management System.
410 Citations2003Rajeev Motwani, Jennifer Widom +8 more
CERN Document Server (European Organization for Nuclear Research)PNG: The Definitive Guide
391 Citations1999G. R. Roelofs, Richard Koman
Mahout in Action
275 Citations2011Sean Owen, Robin Anil +2 more
This book presents practical use cases and illustrates how Mahout can be applied to solve them and places particular focus on issues of scalability and how to apply these techniques against large data sets using the Apache Hadoop framework.
Lecture notes in computer scienceAdvances in Database Technology - EDBT 2006
234 Citations2006Yannis Ioannidis, Marc H. Scholl +7 more
arXiv (Cornell University)Consistent Streaming Through Time: A Vision for Event Stream Processing
215 Citations2006Roger Barga, Jonathan Goldstein +2 more
An overview and discuss the foundations of CEDR, an event streaming system that embraces a temporal stream model to unify and further enrich query language features, handle imperfections in event delivery, define correctness guarantees, and define operator semantics.
Lecture notes in computer scienceBridging Physical and Virtual Worlds: Complex Event Processing for RFID Data Streams
210 Citations2006Fusheng Wang, Shaorong Liu +2 more
This paper develops an RFID event detection engine that can effectively process complex RFID events, and takes an event-oriented approach to process RFID data, by devising RFID application logic into complex events.
HAMA: An Efficient Matrix Computation with the MapReduce Framework
209 Citations2010Sangwon Seo, Edward J. Yoon +4 more
The state-of-the-art framework providing high-level matrix computation primitives with MapReduce is explored through the case study approach, and these primitives are demonstrated with different computation engines to show the performance and scalability.
Transactions in GISInferring the Location of Twitter Messages Based on User Relationships
192 Citations2011Clodoveu A. Davis, Gisele L. Pappa +2 more
This article focuses on the problem of enriching the location of tweets using alternative data, particularly the social relationships between Twitter users, and involves recursively expanding the network of locatable users using following‐follower relationships.
Carmen: A Twitter Geolocation System with Applications to Public Health
159 Citations2013Mark Dredze, Michael J. Paul +2 more
Carmen, a geolocation system that can determine structured location information for messages provided by the Twitter API, utilizes geocoding tools and a combination of automatic and manual alias resolution methods to infer location structures from GPS positions and user-provided profile data.
Event-based applications and enabling technologies
144 Citations2009Annika Hinze, Kai Sachs +1 more
The basic notions of event processing are introduced, the enabling technologies that are used for the implementation of event-based systems are presented, a wide range of applications are surveyed identifying their main features, and open research issues are discussed.
The PADRES Distributed Publish/Subscribe System.
121 Citations2005Ellen Fidler, Hans‐Arno Jacobsen +2 more
PADRES is a distributed middleware platform with features inspired by the requirements of workflow management and business process execution, which constitutes the basis for the next generation of enterprise management systems developed by Cybermation, Inc., including business process automation, monitoring, and execution applications.
Lecture notes in computer scienceEvent-Based Monitoring of Process Execution Violations
68 Citations2011Matthias Weidlich, Holger Ziekow +4 more
A formal technique to derive monitoring queries from a process model, such that they can be directly used in a complex event processing environment, and an approach to filter and aggregate query results to provide compact feedback on deviations are introduced.
The potential of smart home sensors in forecasting household electricity demand
53 Citations2013Holger Ziekow, Christoph Goebel +2 more
This paper uses state-of-the-art forecasting tools, in particular support vector machines and neural networks, to evaluate the use of disaggregated smart home sensor data for household-level demand forecasting and shows that having more sensors appears to be more valuable than increasing the time resolution of measurements.
Cybernetics & SystemsSMART SYSTEMS INTEGRATION: TOWARD OVERCOMING THE PROBLEM OF COMPLEXITY
43 Citations2008Edward Szczerbicki
The problem of information-based integration is introduced, a solution is proposed, and briefly future trends in this area are discussed.
Towards vulnerability-based intrusion detection with event processing
17 Citations2011Amer Farroukh, Mohammad Sadoghi +1 more
This work places event processing in the core of the IDS and proposes novel algorithms to efficiently match vulnerability signatures and is among the first to detect complex attacks such as the Conficker worm which requires correlating multiple protocol data units (MPDUs) while maintaining a small memory footprint.
DOAJ (DOAJ: Directory of Open Access Journals)Towards a stakeholder model for the co-production of the public-sector information system
12 Citations2005Zita P. Correia
A model is developed where the different groups of stakeholders are seen as groups of people and organizations with distinctive characteristics, playing different roles, but not mutually exclusive regarding their participation in the different subsystems.
Forecasting household electricity demand with complex event processing
9 Citations2013Holger Ziekow, Christoph Doblander +2 more
A system prototype for electricity demand forecasting based on highly disaggregated data from sensors deployed in homes is described and its performance both with respect to forecasting accuracy and ICT resource requirements is evaluated.
A Unified View on Data Path Aspects for Sensing Applications at a Smart City Scale
5 Citations2013Vangelis Gazis, Martin Strohbach +2 more
A unified view on the data path issues of sensing applications in terms of generic capabilities is provided,touching information granularity and semantic load concerns, discussing the escalation of detail from raw data to event indicators and, ultimately, deductions.
Running out of road
3 Citations2015Peter Kellner