Digital image representing Informatica data quality.

Seth Rao

CEO at FirstEigen

DataBuck and Alation Join Forces to Turbo Charge the Data Catalog with the Open Data Quality Initiative

Table of Contents
    Add a header to begin generating the table of contents
    Table of Content

      Quality data provides insights that organizations can trust when making big picture decisions. However, the weight of these individual records can often overwhelm even the biggest firms with a seemingly endless number of resources. The Open Data Quality Initiative from Alation aims to help companies deploy the latest tools and technologies like DataBuck to streamline their ingestion and validation of data across all organizational sources. 

      The data quality problem is a real concern for organizations. More than two-thirds of data professionals aren’t highly confident in their organization’s information quality while only 39% of data workers believe it was up to date and accurate. Because organizations make critical business decisions based on the reports, analytics, and statistical models derived from their datasets, autonomous data validation and sanitation arose as key technological enablers of modern businesses. 

      The Open Data Quality Initiative brings together vendors in this space to accelerate implementation when using Alation’s data catalog. Below, we unpack the need for this initiative, discuss the framework provided by Alation, and delve deeper into the capabilities of DataBuck. 

      Key Takeaways

      • DataBuck is part of the Open Data Quality Initiative recently launched by Alation
      • By making the integration of DataBuck and Alation easier, business users can improve their data quality without relying on IT teams as gatekeepers
      • With automated data validation, organizations gain trust in their reports, analytics, and dashboards
      • Businesses can make decisions confidently and investigate anomalous data issues easily using DataBuck and Alation

      What is the Open Data Quality Initiative?

      Addressing the issues of data governance in today’s organizations requires a concerted, unified effort by data professionals and the technology vendors that support them. The use of data catalogs is now an industry standard to structure organizational data into useful and meaningful assets by storing, indexing, and organizing records within a centralized repository. For companies that use data catalogs, the Open Data Quality Initiative gives them the flexibility of choice when selecting vendors for improving data quality and observability within operations. 

      The aim of the Open Data Quality Initiative is to:

      • Accelerate data governance within organizations
      • Improve accessibility and observability in data catalogs
      • Make it easier to deploy autonomous data quality solutions into the catalog
      • Provide a set of user interface capabilities to enable visualization for data intelligence

      Vendors who focus on data quality can now provide state-of-the-art data validation within the Alation data catalog as a single source of reference. 

      Data quality leaders that are uniting for the Open Data Quality Initiative with Alation

      Image Source

      Why is DataBuck Joining Forces with Alation and the Open Data Quality Initiative?

      When IT departments set the rules for data governance and quality validation, decision-makers often become frustrated due to the reliance on operational guardrails that stifle growth and innovation. Creating, modifying, and changing data quality thresholds for schemas, tables, and columns need to be in the hands of the users that rely on quality data to operate their departments successfully. 

      Data stewards can use DataBuck’s integration with Alation to drill down into data quality issues by using automated data validation rules and checks. A deprecation index will flag the data steward when records fail an automated test and fall below a threshold. They can then investigate the issues using DataBuck’s interface directly from Alation and see the exact columns where bad data is propagating into the system. 

      DataBuck is the only software that uses a 11-vector fingerprint to deeply understand the behavior, patterns, trends and relationships in the data assets. It tracks the evolution of data and organically evolves the fingerprint to flag deviant behavior. It also is the only software that can assign a Data Trust Score at an individual record level. Using Machine Learning DataBuck auto discovers which columns are contextually relevant for what types of data validation checks, and the thresholds for each of those individual checks. Some of the automated checks and thresholds that DataBuck generates include (but not limited to):

      • Length check – DataBuck auto discovers string patterns for each relevant column and will validate when these aren’t consistent.
      • Data completeness – Seemingly simple, but labor intensive to do even just half the required job. DataBuck auto learns contextually relevant columns for null checks and their acceptable thresholds for each of those columns individually. 
      • Duplicates – DataBuck automatically discovers multicolumn primary composite keys for any file. This is especially useful when new files arrive and the SME has no prior knowledge of the data or its structure. 
      • Record anomalies – This is a broad term that means many things to different people. DataBuck identifies different kinds of advanced anomalies, like volume anomaly, value anomaly, drift anomaly, inter-column relationship anomaly, entire microsegment behavior anomaly, and more.

      The high level of automation (with human guidance) makes this very rigorous and enables DataBuck to identify any issues in data due System Errors or otherwise. Inside Alation, stewards can track the lineage of tables to see how bad data propagated through different elements including business intelligence (BI) tools, reports, and dashboards. This shows users where exactly bad quality data currently resides and helps to determine whether they can trust the data used in different business departments. 

      What are the Benefits of Automated Data Quality Validation?

      Automating data validation allows companies to put the power back in the hands of the users who rely on quality data to make decisions every day. With IT teams already overwhelmed by operational issues, the responsibility of maintaining data quality may not be their highest priority. The sheer amount of data ingested can also overwhelm departments and render any manual validation futile due to the time it takes to assess, analyze, and correct records when generating reports and dashboards from the data pipeline.

      Data quality issues that enter the data pipeline during ingestion, propagation, and transformation in businesses

      Image Source

      Automating the validation of data using DataBuck helps you to establish a consistent approach to achieving data quality monitoring. It reduces the need for dedicated data quality managers and can save on the costs related to cleaning up bad quality data. Detecting system errors early and responding effectively allows the business to continuously improve data quality using automated checks that detect incorrect formats, anomalous string patterns, null values, and other inconsistencies. 

      Why DataBuck and the Open Data Quality Initiative are Great for Businesses

      The open integration provided by Alation will help businesses to execute automated data quality monitoring (DQM) checks quickly and start making better decisions from day one. It allows users to verify the legitimacy of each report and dashboard before using that dataset to make other business decisions. 

      With DataBuck, all data will receive a Data Trust Score (DTS), and you can use this to determine whether your analytics and reports are trustworthy from day one. As you ingest more data and expand your schemas, DataBuck will use its 11-vector fingerprint to evaluate all new records and alert you when there is a concerning element in your data pipeline. 

      To see this Open Data Quality Initiative integration between Alation and DataBuck, schedule a demo here

      Check out these articles on Data Trustability, Observability & Data Quality Management-

      Discover How Fortune 500 Companies Use DataBuck to Cut Data Validation Costs by 50%

      Recent Posts

      Major Banks Highlight Significant Compliance
      Recent Enforcement Actions Against Major Banks Highlight Significant Compliance Challenges Due to Data Integrity Issues
      Summary Banks face a high cost when data errors slip through due to inadequate data control. Examples ...
      Cloudera Data Lake
      Empowering Data Excellence: the Role of Cloudera Data Lake, Features & Benefits.
      In today's data-driven world, organizations are collecting more information than ever before. But the true value of ...
      Artistic representation of validating data on Databricks.
      Top 5 Challenges of Data Validation in Databricks and How to Overcome Them
      Databricks data validation is a critical step in the data analysis process, especially considering the growing reliance ...

      Get Started!