The Challenges of Data Quality and Data Quality Assessment in the Big Data Era
Data Science JournalPublished 22 May 2015Open access
Li Cai, Yangyong Zhu
Citations977
SJR quartileQ2
SJR score0.35
SNIP0.97
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The data characteristics of the big data environment are analyzed, quality challenges faced by big data are presented, and a hierarchical data quality framework is formulates from the perspective of data users.
Abstract
High-quality data are the precondition for analyzing and using big data and for guaranteeing the value of the data. Currently, comprehensive analysis and research of quality standards and quality assessment methods for big data are lacking. First, this paper summarizes reviews of data quality research. Second, this paper analyzes the data characteristics of the big data environment, presents quality challenges faced by big data, and formulates a hierarchical data quality framework from the perspective of data users. This framework consists of big data quality dimensions, quality characteristics, and quality indexes. Finally, on the basis of this framework, this paper constructs a dynamic assessment process for data quality. This process has good expansibility and adaptability and can meet the needs of big data quality assessment. The research results enrich the theoretical scope of big data and lay a solid foundation for the future by establishing an assessment model and studying evaluation algorithms.
Keywords
Decision SciencesBusiness, Management and Accounting
Journal of Management Information SystemsBeyond Accuracy: What Data Quality Means to Data Consumers
4,330 Citations1996Richard Y. Wang, Diane M. Strong
Using this framework, IS managers were able to better understand and meet their data consumers' data quality needs and this research provides a basis for future studies that measure data quality along the dimensions of this framework.
Data Science JournalThe Challenges of Data Quality and Data Quality Assessment in the Big Data Era
977 Citations2015Li Cai, Yangyong Zhu
The data characteristics of the big data environment are analyzed, quality challenges faced by big data are presented, and a hierarchical data quality framework is formulates from the perspective of data users.
Big data: Issues, challenges, tools and Good practices
859 Citations2013Avita Katal, Mohammad Wazid +1 more
The various challenges and issues in adapting and accepting Big data technology, its tools (Hadoop) are discussed in detail along with the problems Hadoop is facing.
Addressing big data issues in Scientific Data Infrastructure
476 Citations2013Yuri Demchenko, Paola Grosso +2 more
The paper proposes the SDI generic architecture model that provides a basis for building interoperable data or project centric SDI using modern technologies and best practices and introduces the Scientific Data Lifecycle Management (SDLM) model.
Government Information QuarterlyPlanning and designing open government data programs: An ecosystem approach
398 Citations2016Sharon S. Dawes, Lyudmila Vidiasova +1 more
It is concluded that an ecosystem approach to planning and design can be widely used to assess existing conditions and to consider policies, strategies, and relationships that address realistic barriers and stimulate desired benefits.
Applied SciencesSwarm Intelligence Algorithms for Feature Selection: A Review
370 Citations2018Lucija Brezočnik, Iztok Fister +1 more
A unified SI framework is proposed and used to explain different approaches to FS and guidelines on how to develop SI approaches for FS are provided to support researchers and analysts in their data mining tasks and endeavors.
Informing Science The International Journal of an Emerging TransdisciplineDeveloping a Framework for Assessing Information Quality on the World Wide Web
353 Citations2005Shirlee-ann Knight, Janice M. Burn
This paper is part of a research project undertaken at Edith Cowan, Wollongong and Sienna Universities, to build an Internet Focused Crawler that uses "Quality" criterion in determining returns to user queries.
Government Information QuarterlyOpen data quality measurement framework: Definition and application to Open Government Data
343 Citations2016Antonio Vetrò, Lorenzo Canova +4 more
This work set up a framework of indicators to measure the quality of Open Government Data on a series of data quality dimensions at most granular level of measurement and provided technical and policy guidelines to overcome the weaknesses observed in the decentralized release policy, addressing specific quality aspects.
Information PolityUtilization of open government data: A systematic literature review of types, conditions, effects and users
279 Citations2017Igbal Safarov, Albert Meijer +1 more
The paper synthesizes the hypothesized relations in a multi-dimensional framework of OGD utilization and suggests four future directions for research: investigating the link between type of utilization and type of users, establishing a causal link between utilization and OGD outcomes, and investigating the conditions that moderate OGD effects.
Government Information QuarterlyBenchmarking open government: An open data perspective
273 Citations2014Nataša Veljković, Sanja Bogdanović-Dinić +1 more
A benchmark proposal for the Open Government and its application from the open data perspective using data available on the U.S. government's open data portal, the benchmark indicates the progress of government over time, the efficiency of recognizing and implementing new concepts and the willingness of the government to recognize and embrace innovative ideas.
Incorporating quality metrics in centralized/distributed information retrieval on the World Wide Web
223 Citations2000Xiaolan Zhu, Susan Gauch
The results show that incorporating quality metrics can generally improve search effectiveness in both centralized and distributed search environments.
Journal of the Association for Information SystemsMeasuring information quality of web sites: development of an instrument
219 Citations1999Pairin Katerattanakul, Keng Siau
This research proposes a framework and develops an instrument to measure the information quality of individual or personal Web sites and tests the proposed framework and instrument in an individual orpersonal Web site context.
International Journal of Information ManagementOpen data: Quality over quantity
140 Citations2017Shazia Sadiq, Marta Indulska
This research note outlines the challenges of dealing with data quality in open datasets, as opposed to datasets known to organisations, and sets an agenda for future research to address this risk to deriving value from open data investments.
Government Information QuarterlyComparison of metadata quality in open data portals using the Analytic Hierarchy Process
121 Citations2017Sylvain Kubler, Jérérmy Robert +3 more
An Open Data Portal Quality (ODPQ) framework that enables end-users to easily and in real-time assess/rank open data portals and reveals that today's organizations do not pay sufficient heed to the management of datasets, resources and associated metadata that they are currently publishing on their portal.
International Review of Administrative SciencesOpen data work: understanding open data usage from a practice lens
91 Citations2018Erna Ruijer, Stephan Grimmelikhuijsen +2 more
The findings show that an absence of a shared cognitive framework for understanding open data and a lack of high-quality datasets can prevent processes of collaborative learning and demonstrate how ongoing (digital) interactions over open data can result in shared understandings and meaningful practices.
ScientometricsThe advantages of an Ontology-Based Data Management approach: openness, interoperability and data quality
82 Citations2016Cinzia Daraio, Maurizio Lenzerini +4 more
Business & Information Systems EngineeringDiscovering Data Quality Problems
73 Citations2019Ruojing Zhang, Marta Indulska +1 more
The approach, named LANG, is developed through a Design Science approach on the basis of semiotics theory and data quality dimensions and is empirically validated in terms of soundness of the approach, its repeatability and generalizability.
Journal of theoretical and applied electronic commerce researchA Metrics-Driven Approach for Quality Assessment of Linked Open Data
64 Citations2014Behshid Behkamal, Mohsen Kahani +2 more
This paper proposes and validate a set of metrics for evaluating the inherent quality characteristics of a dataset before it is released to the Linked Open Data Cloud and establishes a starting point for a systematic inherent quality analysis of open datasets.
On the research frontiers of business management in the context of Big Data
29 Citations2013Guoqing Chen
Three promising research perspectives are identified, including socialized value creation, network-embedded business operations, and real-time market discovery, based on comprehensively examining the emerging challenges in business management in the context of Big Data.
Data quality in big data processing: Issues, solutions and open problems
27 Citations2017Pengcheng Zhang, Fang Xiong +2 more
This paper uses recommendation systems and prediction systems as typical big data applications and tries to find out the data quality issues during data collection, data preprocessing, data storage and data analysis stages of big data processing.
Expert Systems with ApplicationsA recommendation approach for consuming linked open data
26 Citations2016Jonice Oliveira, Carla Delgado +1 more
This work aims to create a LOD-consuming approach that uses recommendation techniques based on items’ description, their relations, users’ interests and social network.
An Analysis of High-Value Datasets: A Case Study of Thailand’s Open Government Data
22 Citations2018Piriya Utamachant, Chutiporn Anutariya
In this in-depth study, the high-value datasets announced by Electronic Government Agency of Thailand (EGA) were comprehensively and thoroughly analyzed.
Preserving the Benefits of Open Government Data by Measuring and Improving Their Quality: An Empirical Study
17 Citations2017Marco Torchiano, Antonio Vetrò +1 more
The objective is to define a set of basic metrics for public contracts OGD on the basis of the ISO SQuaRE standards family, with the goal of enabling the automated evaluation of dataset quality.
Journal of Systems ArchitectureUsing data visualization technique to detect sensitive information re-identification problem of real open dataset
15 Citations2017Chiun‐How Kao, Chih-Hung Hsieh +3 more
Demonstration proves that ODD Visualizer indeed uncovered one real vulnerability of record linkage attack among open datasets available on the internet.
Assessing data quality in open data: A case study
14 Citations2017John Ferney Mahecha Moyano, Nicolas Estefan Lopez Beltran +1 more
This article focuses on measuring the data quality extracted from the API www.datos.gov.co in the area of contracts, a platform that allows public entities to obey law 1712 of 2014 but does not have a validation system of minimum data quality.
Annals of Computer Science and Information SystemsData quality evaluation: a comparative analysis of company registers’ open data in four European countries
12 Citations2018Jānis Bičevskis, Zane Bičevska +2 more
Validation of an open data published by company registers in four different European countries shows deficiencies in the published data and demonstrates the applicability of the proposed methodology for data quality evaluation.
Data Quality for Decision Makers
9 Citations2013Guilherme Morbey
Data Quality Assessment in the Integration Process of Linked Open Data (LOD)
9 Citations2017Hana Haj Ahmed
It is postulate that data quality assessment can act both as a deciding factor for conflict resolution and as an indicator of low quality data which need to be improved.
Empirical Software EngineeringExperiences and challenges in building a data intensive system for data migration
8 Citations2017Marco Scavuzzo, Elisabetta Di Nitto +1 more
This paper reports the action research experience in developing-testing-reengineering a specific DI application, Hegira4Cloud, that migrates data between widely used NoSQL databases and highlights the issues it has faced and shows how cumbersome, expensive and time-consuming the developing- testing-reenGineering approach can be in this specific case.
Identifying and mitigating risks to the quality of open data in the post-truth era
7 Citations2017Adrienne Colborne, Michael Smit
The rise of post-truth in society, and the risks this poses to the quality, integrity, and authenticity of open data are described and approaches to identifying, assessing, and mitigating these risks are discussed.
Trabalho Educação e SaúdeCiência aberta, dados abertos: desafio e oportunidade
6 Citations2018Paula Xavier Santos, Paulo Cezar Vieira Guanaes
Information PolityHow to guarantee the right to use PSI in the age of open data: Lessons from the data policy of South Korea
5 Citations2019Sang Pil Yoon, Moon-Ho Joo +1 more
Xi'an Jiaotong Daxue xuebaoThe Challenge of Data Quality in the Big Data Age
4 Citations2013Zong We
The challenges and importance of assuring quality of big data and the according measures are discussed in the aspects of process, technology and management.
Microcomputer InformationResearch on Some Basic Problems in Data Quality Control
2 Citations2010Xiaoyi Zhang
The limitations and one-sidedness of existing data quality definitions are analyzed, and both management and technology are needed in data quality control.