Data Quality

Rupa Mahanti

Description

Good data is a source of myriad opportunities, while bad data is a tremendous burden. Companies that manage their data effectively are able to achieve a competitive advantage in the marketplace, while bad data, like cancer, can weaken and kill an organization.
In this comprehensive book, Rupa Mahanti provides guidance on the different aspects of data quality with the aim to be able to improve data quality. Specifically, the book addresses:
Causes of bad data quality, bad data quality impacts, and importance of data quality to justify the case for data quality
Butterfly effect of data quality
A detailed description of data quality dimensions and their measurement
Data quality strategy approach
Six Sigma - DMAIC approach to data quality
Data quality management techniques
Data quality in relation to data initiatives like data migration, MDM, data governance, etc.
Data quality myths, challenges, and critical success factors
Students, academicians, professionals, and researchers can all use the content in this book to further their knowledge and get guidance on their own specific projects. It balances technical details (for example, SQL statements, relational database components, data quality dimensions measurements) and higher-level qualitative discussions (cost of data quality, data quality strategy, data quality maturity, the case made for data quality, and so on) with case studies, illustrations, and real-world examples throughout.
About the Author
Rupa Mahanti, Ph.D. is a Business and Information Management consultant and has worked in different solution environments and industry sectors in the United States, United Kingdom, India, and Australia. She helps clients with activities such as business process mapping, information management, data quality, and strategy. Having a work experience (academic, industry, and research) of more than a decade and half, Rupa has guided a doctoral dissertation and published a large number of research articles. She is an associate editor with the journal Software Quality Professional and a reviewer for several international journals.
"This is not the kind of book that you'll read one time and be done with. So scan it quickly the first time through to get an idea of its breadth. Then dig in on one topic of special importance to your work. Finally, use it as a reference to guide your next steps, learn details, and broaden your perspective."
from the foreword by Thomas C. Redman, Ph.D., the Data Doc
Dr. Mahanti provides a very detailed and thorough coverage of all aspects of data quality management that would suit all ranges of expertise from a beginner to an advanced practitioner. With plenty of examples, diagrams, etc. the book is easy to follow and will deepen your knowledge in the data domain. I will certainly keep this handy as my go-to reference. I can't imagine the level of effort and passion that Dr. Mahanti has put into this book that captures so much knowledge and experience for the benefit of the reader. I would highly recommend this book for its comprehensiveness, depth, and detail. A must-have for a data practitioner at any level.
Clint D'Souza, CEO and Director, CDZM Consulting

Sujets

• Business
• Qualité, hygiène, sécurité, environnement
• Sciences formelles
• Mathématiques appliquées
• Mathématiques appliquées
• Data Anlytics
• Data Science
• Computers
• Informatique

Informations

• Publié par: ASQ Quality Press
• Date de parution: 18 mars 2019
• Nombre de lectures: 1
• EAN13: 9781951058678
• Langue: English
• Poids de l'ouvrage: 5 Mo

Extrait

Data Quality

Also available from ASQ Quality Press:

Quality Experience Telemetry: How to Effectively Use Telemetry for Improved Customer Success Alka Jarvis, Luis Morales, and Johnson Jose

Linear Regression Analysis with JMP and R Rachel T. Silvestrini and Sarah E. Burke

Navigating the Minefield: A Practical KM Companion Patricia Lee Eng and Paul J. Corney

The Certified Software Quality Engineer Handbook, Second Edition Linda Westfall

Introduction to 8D Problem Solving: Including Practical Applications and Examples Ali Zarghami and Don Benbow

The Quality Toolbox, Second Edition Nancy R. Tague

Root Cause Analysis: Simplified Tools and Techniques, Second Edition Bjørn Andersen and Tom Fagerhaug

The Certified Six Sigma Green Belt Handbook, Second Edition Roderick A. Munro, Govindarajan Ramu, and Daniel J. Zrymiak

The Certified Manager of Quality/Organizational Excellence Handbook, Fourth Edition Russell T. Westcott, editor

The Certified Six Sigma Black Belt Handbook, Third Edition T. M. Kubiak and Donald W. Benbow

The ASQ Auditing Handbook, Fourth Edition J.P. Russell, editor

The ASQ Quality Improvement Pocket Guide: Basic History, Concepts, Tools, and Relationships Grace L. Duffy, editor

To request a complimentary catalog of ASQ Quality Press publications, call 800-248-1946, or visit our website at http://www.asq.org/quality-press.

Data Quality

Dimensions, Measurement, Strategy, Management, and Governance

Dr. Rupa Mahanti

ASQ Quality Press Milwaukee, Wisconsin

American Society for Quality, Quality Press, Milwaukee 53203 © 2018 by ASQ All rights reserved. Published 2018 Printed in the United States of America 24 23 22 21 20 19 18 5 4 3 2 1

Library of Congress CataloginginPublication Data

Names: Mahanti, Rupa, author. Title: Data quality : dimensions, measurement, strategy, management, and  governance / Dr. Rupa Mahanti. Description: Milwaukee, Wisconsin : ASQ Quality Press, [2019] | Includes  bibliographical references and index. Identifiers: LCCN 2018050766 | ISBN 9780873899772 (hard cover : alk. paper) Subjects: LCSH: Database management—Quality control. Classification: LCC QA76.9.D3 M2848 2019 | DDC 005.74—dc23 LC record available at https://lccn.loc.gov/2018050766

ISBN: 978-0-87389-977-2

No part of this book may be reproduced in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of the publisher.

Publisher: Seiche Sanders Sr. Creative Services Specialist: Randy L. Benson

ASQ Mission: The American Society for Quality advances individual, organizational, and community excellence worldwide through learning, quality improvement, and knowledge exchange.

Attention Bookstores, Wholesalers, Schools, and Corporations: ASQ Quality Press books, video, audio, and software are available at quantity discounts with bulk purchases for business, educational, or instructional use. For information, please contact ASQ Quality Press at 800-248-1946, or write to ASQ Quality Press, P.O. Box 3005, Milwaukee, WI 53201-3005.

To place orders or to request ASQ membership information, call 800-248-1946. Visit our web- site at http://www.asq.org/quality-press.

 Printed on acid-free paper

Table of Contents

List of Figures and Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv Foreword: The Ins and Outs of Data Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxvii Chapter 1: Data, Data Quality, and Cost of Poor Data Quality . . . . . . . . . . . 1 The Data Age . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Are Data and Data Quality Important? Yes They Are! . . . . . . . . . . . . . . . . . 2 Data Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Categorization of Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Master Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Reference Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Transactional Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Historical Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Metadata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Data Quality: An Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 How Is Data Quality Different? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Data Quality Dimensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Causes of Bad Data Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Manual Data Entry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Inadequate Validation in the Data Capture Process . . . . . . . . . . . . . . . . 16 Aging of Data/Data Decay . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Inefficient Business Process Management and Design . . . . . . . . . . . . . . 18 Data Migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Data Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Data Cleansing Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Organizational Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 System Upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Data Purging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Multiple Uses of Data and Lack of Shared Understanding of Data . . . . 25 Loss of Expertise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Lack of Common Data Standards, Data Dictionary, and Metadata . . . . 26

v

vi

Table of Contents

Business Data Ownership and Governance Issues . . . . . . . . . . . . . . . . . Data Corruption by Hackers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Cost of Poor Data Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Butterfly Effect on Data Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Summary of Key Points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Key Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Categories of Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Cost of Poor Data Quality—Business Impacts . . . . . . . . . . . . . . . . . . . . Causes of Bad Quality Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Chapter 2: Building Blocks of Data: Evolutionary History and Data Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Evolution of Data Collection, Storage, and Data Quality . . . . . . Database, Database Models, and Database Schemas . . . . . . . . . Relational Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Normalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Warehouse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dimensional Modeling, Fact, Dimensions, and Grain . . . . Star Schema, Snowflake Schema, and OLAP Cube . . . . . . The Data Hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Common Terminologies in a Nutshell . . . . . . . . . . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Chapter 3: Data Quality Dimensions . . . . . . . Introduction . . . . . . . . . . . . . . . . . . . . . . . . . Data Quality Dimensions—Categories . . . . Data Quality Dimensions . . . . . . . . . . . . . . . Data Specifications . . . . . . . . . . . . . . . . Completeness . . . . . . . . . . . . . . . . . . . . . Conformity or Validity . . . . . . . . . . . . . Uniqueness . . . . . . . . . . . . . . . . . . . . . . Duplication . . . . . . . . . . . . . . . . . . . . . . Redundancy . . . . . . . . . . . . . . . . . . . . . . Consistency . . . . . . . . . . . . . . . . . . . . . . Integrity . . . . . . . . . . . . . . . . . . . . . . . . . Accuracy . . . . . . . . . . . . . . . . . . . . . . . . Correctness . . . . . . . . . . . . . . . . . . . . . . Granularity . . . . . . . . . . . . . . . . . . . . . . Precision . . . . . . . . . . . . . . . . . . . . . . . . Ease of Manipulation . . . . . . . . . . . . . . Conciseness . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

27 27 28 37 40 40 40 41 41 41

43 43 43 47 48 52 55 55 57 58 64 66 67

69 69 72 76 78 79 85 90 93 93 93 101 106 107 108 110 113 113

Table of Contents

Objectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Relevance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Interpretability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Believability, Credibility, and Trustworthiness . . . . . . . . . . . . . . . . . . . . Reputation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Time-Related Dimensions (Timeliness, Currency, and Volatility) . . . . . Accessibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Traceability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . How to Use Data Quality Dimensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Quality Dimensions: The Interrelationships . . . . . . . . . . . . . . . . . . . . . Data Quality Dimensions Summary Table . . . . . . . . . . . . . . . . . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Chapter 4: Measuring Data Quality Dimensions . . . . . . . . . . . . . . . . . . . . . . Measurement of Data Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Quality Dimensions Measurement: Subjective versus Objective . . . . . . What Data Should Be Targeted for Measurement? . . . . . . . . . . . . . . . . . . . . Critical Data Elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Metadata and Data Quality Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Quality: Basic Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Quality Dimensions: Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . Measuring Completeness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Measuring Uniqueness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Measuring Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Measuring Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Measuring Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Measuring Integrity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Measuring Volatility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Measuring Currency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Measuring Timeliness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Measuring Data Lineage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Measuring Data Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Measuring Relevance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Measuring Accessibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Measuring Data Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Measuring Data Reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Measuring Ease of Manipulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Measuring Conciseness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Measuring Objectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Measuring Interpretability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Measuring Believability, Credibility, and Trustworthiness . . . . . . . . . . .

vii

113 114 114 115 116 117 117 119 120 120 122 123 124 126 127

129 129 130 131 135 136 138 139 139 152 160 174 190 219 223 228 228 232 233 234 238 242 244 249 250 252 254 255

viii

Table of Contents

Assessing Reputation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . How to Conduct Data Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Manual Data Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Profiling Using Spreadsheets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . SQL Scripts for Profiling Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Use of a Data Profiling Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chapter 5: Data Quality Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . What Is a Data Quality Strategy? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Strategy versus Data Quality Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . Why Is a Data Quality Strategy Important? . . . . . . . . . . . . . . . . . . . . . . . . . . Data Quality Maturity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Quality Maturity Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Level 1: The Initial or Chaotic Level . . . . . . . . . . . . . . . . . . . . . . . . . . . Level 2: The Repeatable Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Level 3: The Defined Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Level 4: The Managed Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Level 5: The Optimized Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Scope of a Data Quality Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Quality Strategy: Preplanning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Phases in the Data Quality Strategy Formulation Process . . . . . . . . . . . . . . . Planning and Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Discovery and Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Prioritization and Roadmap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Quality Strategy Document and Presentation . . . . . . . . . . . . . . . . . . . . Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Assessment Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Findings and Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Initiative Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Implementation Roadmap and Financials . . . . . . . . . . . . . . . . . . . . . . . . Characteristics of a Good Data Quality Strategy . . . . . . . . . . . . . . . . . . . . . Data Quality Strategy Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Who Drives the Data Quality Strategy? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chief Data Officer (CDO) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . CDO Capabilities and Responsibilities . . . . . . . . . . . . . . . . . . . . . . . . . . Reporting Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Quality Strategy—A Few Useful Tips . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chapter 6: Data Quality Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Quality Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Quality Management—Reactive versus Proactive . . . . . . . . . . . . . . . . . Data Quality Management—Six Sigma DMAIC. . . . . . . . . . . . . . . . . . . . . .

259 260 260 261 262 271 280

283 283 283 284 285 286 287 288 289 290 291 293 294 296 297 298 299 301 303 303 304 304 304 306 308 308 308 309 310 313 314 316

317 317 319 323

Table of Contents

Define . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Measure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Analyze . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Improve . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Quality Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Access and Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Profiling and Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Results and Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Root Cause Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Cleansing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Parsing and Standardization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Matching, Linking, and Merging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Enrichment or Data Augmentation . . . . . . . . . . . . . . . . . . . . . . . . . Data Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Presence Check Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Format Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Range Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Lookup Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Type Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Quality Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Migration and Data Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . How to Ensure Data Quality in Data Migration Projects . . . . . . . . . . . . Data Quality Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Cleansing Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Solution Design—Data Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Coding, Testing, and Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Migration Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Migration Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Integration and Data Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Integration Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Warehousing and Data Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . Master Data Management and Data Quality . . . . . . . . . . . . . . . . . . . . . . . . . Master Data Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Master Data Integration and Data Quality . . . . . . . . . . . . . . . . . . . . . . . Data Governance and Master Data Management . . . . . . . . . . . . . . . . . . Metadata Management and Data Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . Six Sigma DMAIC and Data Quality Management—Example . . . . . . . . . . . Six Sigma Define Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Six Sigma Measure Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Six Sigma Analyze Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Six Sigma Improve Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Six Sigma Control Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Key Principles of Data Quality Management . . . . . . . . . . . . . . . . . . . . . . . . .

ix

323 325 325 326 326 326 328 328 329 330 333 335 338 341 343 343 343 343 344 344 345 346 348 348 350 351 353 353 354 354 355 362 362 366 368 369 370 372 372 372 379 379 380 380