Historically, data discovery has existed at the nexus point between data preparation and analytics. The discovery process was frequently viewed as the means of gathering the requisite data for analytics while illustrating relationships between data elements which might inform them.
Today, data discoveryâs utility has considerably broadened. Aided by machine learning and data cataloging techniques, data discovery is playing an increasingly pivotal role in enablingâand solidifyingâdata governance for todayâs highly regulated data environments.
âWe now have the automated capability to see where data elements are showing up and what are the new instances of them that are being introduced [throughout the enterprise],â Io-Tahoe CEO Oksana Sokolovsky revealed. âNow, users can govern that as data owners and actually have this visibility into their changing data landscapes.â
The additional governance repercussions of data discovery (encompassing aspects of data quality, data stewardship, and data disambiguation), coupled with its traditional importance for enhancing analytics, makes this facet of data management more valuable than ever.
The expansion of data discovery into facets of data governance is rooted in the fundamental need to identify where data are for what specific purposes. Data cataloging immensely enriches this process by providing a means of detailing critical information about data assets that provide a blueprint for data governance. Moreover, discovery and cataloging systems which deploy machine learning are targeted towards business users, allowing them to âcreate business rules, maintain them, search for elements, define policies, and start providing the governance workflow for the data elements,â Sokolovsky said. The plethora of attributes imputed to data within catalogs is vast, including details about metadata, sensitivity, and access or security concerns. Another crucial advantage is that all of this information is stored in a centralized location. âThe catalog enhances the metadata and enhances the business description of the data elements,â Sokolovsky explained. âIt enables other business users to leverage that information. The catalog function now makes data discovery an actionable output for users.â
Exceeding Metadata Relationships
A number of data discovery tools are almost entirely based on metadataâproviding circumscribed value in situations in which there is limited metadata. The most common of these involve data lakes, in which data elements âmight not have any metadata associated with them, but we still need to tie them back to the same element which appears in your original sources,â Sokolovsky commented. Other metadata limitations involve scenarios in which there is not enough metadata, or metadata that applies to a specific use case. In these instances and others, discovery techniques informed by machine learning are superior because they can identify relationships among the actual data, as well as among any existent metadata.
According to Sokolovsky, this approach empowers organizations to ânow pick up 30 to 40 percent more [information about data elements], which used to be input manually by subject matter experts.â The disambiguation capability of this approach supports basic aspects of data quality. For example, when determining if data referencing âWashingtonâ applies to names, locations, or businesses, machine learning âalgorithms can narrow that down and say we found 700 Washington instances; out of that, X number is going to be last names, X number is going to be first names, X number is going to be streets, and X number is going to be cities,â Sokolovsky said.
The automation capabilities of machine learning for data discovery also support governance by democratizing the notion of data stewardship. It does so in two ways. Not only do those tools provide much needed visibility for employees in dedicated stewardship roles, but they also enable business users to add citizen stewardship responsibilities to their positions. The expansion of stewardship capabilities is useful for increasing data quality for data owners in particular, who âbecome more like stewards,â Sokolovsky maintained. âThey can now say okay, out of 75 instances 74 seem to be accurate and one is bad. Thatâs going to continue to enhance the machine learning capability.â
The capacity for disambiguating data, reinforcing data quality and assisting data stewardship that this approach facilitates results in higher levels of accuracy for data in any variety of use cases. Although a lot of this work is engineered by machine learning, the human oversight of data stewardship is instrumental for its ultimate success. âThe user should interact with the system to go and do the validation and say I accept or I reject [the machine learning results],â Sokolovsky said. âBecause of that not only are they in control of the governance, but also the system becomes smarter and smarter in the clientâs environment.â
Working for Business Users
The deployment of data discovery and data cataloging for data governance purposes indicates both the increasing importance of governance and machine learning. Machine learning is the intermediary that improves the data discovery process to make it suitable for the prominent data governance and regulatory compliance concerns contemporary enterprises face. It is further proof that these learning capabilities are not only ideal for analytics, but also for automating other processes that give those analytics value (such as data quality), which involves âworking directly with the business user,â Sokolovsky said.