Extract, Transform and Load (ETL) is the traditional approach for moving and transforming data for analysis, especially in large organizations working in regulated industries.
But modern, digital enterprise companies want their data right away. And they see no reason to invest time and money in ETL when a modern, cloud-native integration platform delivers the data they need far faster and more efficiently.
In this post, I’ll walk through the benefits and drawbacks of ETL and how an iPaaS (integration platform as a service) can easily eliminate the challenges of ETL and turbo-charge this integration method.
Lockstep Data Integration With ETL
ETL is a way of extracting data from a source system, manipulating and transforming it as it’s being moved between systems, and then loading it into a data warehouse or some other kind of data repository.
This is a tried and proven model of moving data, particularly for projects that require a data warehouse for storing data and making it available for analysis or enriching data sets in other applications.
ETL has been around for 30-plus years, and it’s something that enterprise architects all learn when we begin our careers in data integration and management.
“In many industries, we’re seeing young, cloud-centric companies skip ETL altogether and go straight to iPaaS. These companies are used to moving quickly, and iPaaS suits their needs for moving data and getting the answers they need on time.”
ETL is a fine approach for certain types of data integration and analysis projects. When I think of a typical ETL project, I imagine a financial services organization using an ETL tool to extract data from various core systems, transform it, and load it into a data warehouse for analysis and reporting.
There are a few key characteristics to note:
- A repeatable, predictable process
The process of extracting data, transforming it and loading it into a data warehouse is well-defined, stable and repeatable. The financial institution processes the same types of transactions, day in and day out. It knows which applications are storing the data and what sort of analysis it wants to get out of the warehouse in the end. The process is fixed, and the outcome is predictable.
- Focused on compliance
The process is being run in a highly regulated industry. The stability and predictability of the process are benefits. Regulators want to see that the institution is handling its data carefully and generating the required reports. Predictable ETL processes help meet those regulatory requirements.
- Slow is fine
Speed isn’t critical to building this workflow or running it on a regular schedule. It’s okay if an ETL integration takes three months to build, because once it’s built, it’s going to be used with few or no changes for several years. And the results don’t need to be rushed. They can be delivered on schedule after a week or a month. In ETL uses cases, slow and steady wins the race.
In this kind of scenario with a financial institution, ETL processes do fine. At Slalom, we are seeing large organizations continue to use ETL for data projects like this for the foreseeable future.
Fast, Fluid Data Integration With iPaaS
Now let’s contrast that example with a different kind of use case from another industry, such as consumer packaged goods (CPG) or retail. Compared to financial services, these markets move much more quickly.
For example, in the fashion market, manufacturers can design, create and launch new clothing lines in a matter of weeks.
Retailers in fashion and other fast-moving markets have to be nimble: rapidly analyzing trends, placing orders, designing and running sales campaigns, measuring results, and starting and stopping activities within weeks or even days.
In markets like this, slow, steady integration projects simply won’t do. They will fail to deliver the insights that today’s digital enterprises need, when they need them.
Developers must be able to integrate new sources quickly, load data into platforms for business intelligence and trend analysis, and get results back to business decision-makers as quickly as possible.
In these markets:
- Processes and analytics are in flux
Processes are constantly changing. Suppliers, products, partners and campaigns change from week to week, month to month, and season to season. When these things change, questions that business leaders ask and answers they need change as well.
- Compliance isn’t front and center
Repeatability and compliance aren’t as important. Many of the analytical reports being produced are not required as part of complying with regulatory practices. They’re intended to support decision-making.
- Speed is critical
Retailers, CPG companies and most other digital enterprises need answers as quickly as possible. Developers don’t have months or weeks to hand-code the integrations. They might have a day or even just an afternoon to move data to where it needs to go.
In situations like this, iPaaS shines. Using the Boomi unified platform for integration, data governance, API management and workflow automation, organizations can rapidly tie together their applications and data.
A low-code interface and ready-to-use connectors bring unprecedented speed to unite and transform your data.
- iPaaS solutions can complete projects in days or hours that might have taken a month with tradition hand-coding techniques and legacy middleware.
- Load data from any source quickly into a business intelligence (BI) application like Qlik or a cloud data warehouse like Snowflake.
- Export analytical results to whatever other application can use them.
Modern Integration Helps Digital Enterprises Thrive
At Slalom, we’re hyper-focused on our customers, and we’re dedicated to working in partnership to help them address their key business challenges and achieve their strategic goals.
We see other companies across industries — from education to telecommunications to retail — also increasing their attention on customers. That increased attention requires raw data and data analytics. How are campaigns working? Do customer like new features? How are they using our products? In which regions and in which demographics?
To answer questions like these, businesses need to be agile, curious and open-minded — you might not know exactly what you’re looking for at first. You might want to examine a data set six different ways. You might need to connect to Salesforce, a retail store real-time analytics application, a mobile service, and survey data. And you might need to run different types of analytic algorithms to find the answers you need, and then to follow up with more questions and find even more answers.
To do that, you can’t rely solely on ETL. It’s too cumbersome and inflexible. Instead, you need a unified platform like Boomi that lets you build integrations in days or hours. With Boomi, you get the business insights you need faster than the competition.
In many industries, we’re seeing young, cloud-centric companies skip ETL altogether and go straight to iPaaS. These companies are used to moving quickly, and iPaaS suits their needs for moving data and getting the answers they need on time.
ETL remains an important technology for large companies processing transaction data and other types of structured data as part of repeatable processes.
iPaaS is the right technology for many other types of data operations, where moving and analyzing data requires speed, flexibility and agility. Especially for cloud-centric businesses, we recommend iPaaS as an essential part of an IT architecture.
To learn more about all the ways the Boomi unified platform can help your organization do more, contact our partner, Slalom, or speak with a Boomi integration specialist today.