We have all heard the phrase “Garbage in, garbage out.” When it comes to AI, clean data is critical, but the reality is that the data is chaos. So what can you do to transform your data from garbage to gold?
Start With People, Not Technology
Let’s be clear that AI does not know what you and your colleagues know about your business. Much of the knowledge that it will take to clean up your data is undocumented. So you need to first codify that knowledge in order for AI to effectively play its part. In practice, this work sits at the intersection of people and process as much as technology. You need someone who knows the business (often a data steward or domain expert who can define what ‘correct’ looks like) and someone who knows how to structure data efficiently (typically a data architect or someone in a similar role). Sometimes resolving a data quality issue means going back to the source system administrator to fix it at the origin. There’s no automation shortcut for that conversation. So step one is identifying the right stakeholders and ensuring they are aligned on your process.
As an extension of your human team, technology can then automate your defined rules to ensure your data is organized, labeled, and enriched in a manner that is then easily understood and usable, both by humans and by machines. Tools like Boomi Meta Hub can help to codify that knowledge in business glossaries. This is where you put the data about your data, also known as the metadata. What does it mean when someone at your company refers to a ‘user’?
- To the authentication team, that might mean anyone with a valid account record.
- To the product team, it’s someone who logged into the system within the last 30 days.
- To the billing team, it’s anyone on a paid plan.
- To the data team running analysis, it’s a unique device or cookie.
That is the same word with four completely different meanings, each likely to produce different results when pulling from the same data source.
Getting to a Shared Definition of Your Data
To ensure the correct outcome for each use case, you need to account for each of these perspectives and definitions. And for that to be possible, you have to ensure you have the right humans and/or expertise assembled. Many organizations establish a data steering committee — a cross-functional group of business and functional leaders who align on these definitions before they’re ever codified in a glossary. AI might eventually take a pass at surfacing possible definitions, but human judgment still has to come first. There needs to be a way to ensure your AI agent understands this kind of nuance.
But semantic understanding is only the first layer. The next layer is the reliability of the data itself. Now that we know what we mean when we say ‘user,’ how do we know if we have identified the right user? There are two users in the system named John B. Smith. Usually the middle initial would be the differentiator, but in this case the middle initial is the same. How will we know that we are getting data on the RIGHT John B. Smith? There are other indicators that will tell us, such as user ID, phone number, email address, etc. These are user data fields we want to master, which will enable us to select the right user at the right time and get the data we need about that person. The same challenge applies to organizations, not just people. A vendor in your system might show up in three different records — one with the billing address, another with the shipping address, and yet another with the headquarters address. By aligning on data controls, you help your system recognize these records as belonging to the same company. Third-party services like Dun & Bradstreet can be connected to help fill gaps and enrich the data by validating supplier data against public information, giving your golden records a stronger foundation. We master entities such as these to ensure we can accurately and efficiently operate our businesses. This is mastering your data.
Mastering, Cleansing, and the Architecture That Connects Them
It’s important to note that mastering your data and cleansing your data are not necessarily the same. With an entity resolution system, you define rules that establish trusted “golden records.” These golden records contain just the information needed to identify and understand your business entities. This trusted data may feed into dedicated downstream systems, which hold information like transactions or support tickets for a specified customer. There may still be errors in the downstream system data, but the golden records can make them more apparent.
Boomi Data Hub, the platform’s entity resolution solution, operates on a hub-and-spoke model. The golden records are in the hub, and are fed by your various applications that connect to the hub through integration spokes. You decide which applications have read and write permissions to the hub to ensure those golden records always have the most up-to-date information. The data quality rules you set determine what data becomes part of a golden record, and what gets flagged for cleanup. As data comes in and is placed into quarantine for not meeting your established data quality standards, it becomes clear what data is incorrect, incomplete, or a duplicate. You can then create integrations to automate the cleansing of the data you’ve identified as needing transformation.
Consider a dataset of 10,000 records where Data Hub surfaces 2,000 duplicates. Knowing you have duplicates only gets you halfway from garbage to gold; you still need a mechanism to resolve them. Boomi Integration can take over here, transforming and routing that data through cleansing rules before it ever reaches the hub. Boomi Data Integration supports automated ELT workflows (with the transform step being particularly important) allowing you to reassign primary keys, rename fields, enforce data types, and resolve conflicts before records land in Data Hub for mastering.
Building on a Foundation of Trust
Trust is the key to success in this agentic era; trust that your agents are connected with the right systems, and the data within is organized, enriched, and up-to-date. And even once the agent has access to that data, it’s important to be able to to be able to hold both the agents and the systems accountable through tracing data origin. Data lineage becomes critical. Where did the data come from and how has it changed over time? Trust is built on transparency. This is what our Meta Hub service offers.
Governance extends to unstructured content too. The launch of Boomi Knowledge Hub provides a way to easily sort, organize, and store the unstructured data you need for your RAG (retrieval-augmented generation) pipelines. Knowledge Hub has governance built in, to authorize what data is visible and to whom. It pairs Boomi Data Integration with vector database storage to automatically and continuously prepare large volumes of unstructured content for AI activation.
Start Small, Build a Practice
There are multiple ways to approach data cleansing, but don’t try to fix everything at once. Choose one problem worth solving, identify the data sources that feed it, and start there. That single use case becomes your proof of concept, your template, and your argument for the next one.
It’s easy to get overwhelmed when the topic of clean data arises, and agentic transformation has made this an imperative, not a backlog item. But tackling it one use case at a time makes it manageable, seeming less like a data landfill and more like your steady path to golden data.
If you’re not sure where to begin, a good first step is understanding the shape of what you already have. Our data profiling course will walk you through exactly that.