Data quality is the foundation upon which data-driven culture rests. Before upper level management can embrace data-centric processes, before analytics and reliance on big data becomes pervasive, and before data stewardship can truly extend outside ITâs boundaries to become embraced by the business, organizations must trust their data.
Trustworthy data meets data quality measures for…
…and is the reliable, consistent basis for accurate analytics and the value data provide to optimize business processes.
By ensuring data quality, organizations are laying the foundation for becoming data driven both explicitly and implicitly. Explicit manifestations of this shift include an increased reliance on data, greater valuation of data as an asset, and an entrenchment of data as a means of optimizing business. Implicitly, the daily upkeep of data-driven processes become second-nature as aspects of data stewardship, provenance, integration, and even modeling simply become due diligence for everyoneâs job.
According to Tamr head of product and strategy Nidhi Aggarwal, the quintessential manifestation of a data-centered culture may be reflected in another way that delivers even greater pecuniary benefitsâthe utilization of all enterprise data. âPeople talk about the democratization of analytics and about being truly data driven,â Aggarwal commented. âYou cannot do that if youâre only using 20 percent of your data.â
Cognitive Data Science Automates Data Quality Measures
Data quality is largely conceived of as the output of assiduous data preparation. It is effectively ensured via the deployment of the machine learning technologies that are responsible for automating critical facets of data science: particularly the data cleansing and preparation that can otherwise become too laborious. Contemporary platforms for data quality establish machine learning models that map data of all types to specific measures for qualityâand which can also include additional facets of data preparation, such as transformation. âOur machine learning models are able to do that really fast, really cheap, and improve it over time as we see more and more data,â Aggarwal noted.
With machine learning, quality measures begin by mapping relevant data sources to one another to determine how their attributes relate. The cognitive prowess of these models are demonstrated in their ability to sift through individual records for these data sources (which may be bountiful). In doing so, they identify points of redundancy, relationships between data, names and terms, how recent data are, and many other facets of data quality. âItâs very difficult for a human to do that,â Aggarwal said. âWith machine learning, by doing statistical analysis, by looking at all of the attributes, by looking at these rules that some domain experts provide to the models, by looking at how the humans answered the questions that we presented as samples to them, it makes decisions about how these things should be de-duplicated.â
Reinforcing Trust with Provenance and Natural Language Processing
Competitive preparation platforms that facilitate data quality temper the quality measures of cognitive computing with human involvement. The result is extremely detailed data provenance which reinforces trust in data quality, and which is easily traced for the purposes of assurance. The decisions that domain experts make about how sources are unified and relate to each other for specific data typesâwhich is critical to establishing data qualityâare recorded and stored in platforms for traceability. Thus, there is little ambiguity about who made a decision, when, and what effect it had on the underlying machine learning model for how data was unified and defined to establish data quality. Natural Language Processing is involved in the data quality process (especially with unstructured text) by helping to reconcile definitions, different terms, and commonalities between terms and how they are phrased. The pivotal trust required for becoming data-driven is therefore facilitated with both machine learning and human expertise.
Metadata and Evolving Models
The granular nature of a machine learning, human tempered approach to data quality naturally lends itself to metadata and incorporating new data sources into quality measures. Metadata is identified and compared between sources to ensure unification for specific use cases and requisite data quality. The true value of this cognitive approach to data quality is evinced when additional data sources are included. According to Aggarwal: âPeople can do this manual mapping if they only wanted to do it manually once. But the trouble is when they have to add a new data source, itâs almost as much effort as doing it the first time.â However, the semantic technologies that form the crux of machine learning are able to incorporate new sources into models so that âthe model can actually look at the new data set, profile it really quickly, and figure out where it maps to all the things that it previously knows aboutâ Aggarwal said.
More significantly, the underlying machine learning model can evolve alongside data sets that are radically dissimilar from its initial ones. âThen the model updates itself to include this new data,â Aggarwal mentioned. âSo when a new data set comes in further down the line, the chances are that it will be completely new and that the models donât align with it go lower and lower every time.â The time saved from the expedited process of updating the models required for data quality underscore the agility required to further trust data when transitioning to becoming data driven.
Using All Data
When organizations are able to trust their data because of the aforementioned rigorous standards for data quality, they are able to incorporate more data into business processes. The mapping procedures previously outlined helps organizations to bring all of their data together and determine which of it relates to a specific uses case. The monetary boons of incorporating all enterprise data into business processes is exemplified with a use case from the procurement vertical. Were a company attempting to determine how many suppliers it had and whether it was getting the best payment terms from them, those that were not data savvy could only use a finite amount of their overall dataâlimited to particular business unitsâto determine this answer. Those that were truly data-driven and able to incorporate all of their data for this undertaking could incorporate the input of greater amounts of business units and, according to Aggarwal, who encountered this situation with a Tamr customer:
âThere were wildly different payment terms for the same supplies. When we dug into what parts they were buying from the suppliers and at what prices across the different business units, there were sometimes 300X differences in the price of the same part.â Unifying oneâs data for uniform quality measures is integral to identifying these variances, which translates into quantifiable financial advantages. âAn individual decision might save them a few hundred dollars here and there,â Aggarwal remarked. âCollectively, optimizing their decisions every single day has saved them millions and millions of dollars over time. Thatâs the power of bringing all data together.â
Citizen Stewardship and Business Engagement
The pervasiveness of data reliance and the value it creates for decision-making and business processes is intrinsically engendered through the trust gained from a firm foundation in data quality. By utilizing timely, reliable, data that is consistent in terms of metadata, attributes, and records management, organizations can transition to a datacentric culture. The products of such a culture are the foregoing cost advantages businesses attributed to improved decision-making. The by-products are streamlined data preparation, improved provenance, upper level management support, aligned metadata, and an appreciation of dataâs value and upkeep on the part of the business users who depend on it most.
Aggarwal commented that increased data quality processes facilitated by machine learning and human oversight result in: âA broader dialogue about data in terms of stewardship. Today stewardship is in the hands of IT people basically who donât have business context. What [we do] is take that stewardship and engage the business people who actually know something about the data much sooner in the process of data quality. Thatâs how they get to higher data quality, faster.â
AndÂ thatâsÂ alsoÂ howÂ theyÂ becomeÂ dataÂ driven,Â faster.