As part of my Integration 101 series, I published a blog on push versus pull integration. The idea was to clarify what those terms mean and why it’s important to understand how each type operates.
Like push versus pull, there’s a lot of misunderstanding around batch and real-time integration. Not so much about the meaning of the terms but about the pros and cons of the two methods. In this post, I’ll try to shed some light on that.
Are You Sure You Need Real-Time Integration?
It’s common when I’m talking to people about integration solutions to hear, “Oh, it’s got to be real-time. We need real-time integration.”
Now, don’t get me wrong: Real-time integration is great. But there are several things you need to consider when you’re choosing real-time integration. Most of the time, you’ll do just fine with a batch approach. It has definite advantages, which I’ll explain.
Let’s start with what a real-time data request from a system means. It means you want data to move transactionally. Anytime a transaction happens, information moves between your systems. There are certain cases where real-time integration is necessary, but most of the time, it’s not.
Real-Time Integration in a Multi-Tenant Environment
Your business probably works with cloud-based systems or at least has a cloud-based solution in the mix. Of course, whenever you do something that’s cloud-based, you’re (usually) using a multi-tenant environment. You have lots of people using the same processing power.
The economics of a cloud service provider work like this:
If I’ve got 10 units of processing power, I’m going to sell 15 units to different clients and bet that no one will need all their processing power at the same time. It’s a way for me to make a little more money and use my resources more efficiently.
A good service provider will have restrictions on the amount of computing power any single customer can use at one time. If you exceed the limit, you’re shut down or you’re charged for it.
One of the common limits in cloud-based systems is the number of API calls that can be made within a defined time period. Transactional, real-time integrations can drive your API limits through the roof.
Let’s say you have a big sales floor selling small-dollar items at high volume — $5 to $10 per sale. Your annual recurring revenue is $100 million. It takes many of those small sales to hit the $100 million level. So, you have tons of salespeople constantly logging transactions. And every time a sale is made, the point-of-sale (POS) system makes an API call to your ERP application.
It’s easy to do the math here. That’s a lot of API calls, which will either cost you a bunch of money, or, worst case, bring your system down because the service provider blocks access to your API.
The Race Car and the School Bus
Real-time integrations are meant to be small, light and quick. Think of a race car. It moves one person very fast. Batching is meant to be more like a school bus. It’s not as small, light and fast, but it can move a lot of people. And when you have a lot of people, it’s more efficient to put 50 people on one bus than it is to drive 50 small cars.
So, batching is the answer to the predicament described above. Take that transaction data and put it on the bus. Move it all at once instead of in many small trips. That’s a batch.
Batch integrations tend to be more complex than real-time or near-real-time integrations, which are usually simpler because there’s less logic involved.
For batch integrations, you need to architect your system so it can handle large volumes of data because responsiveness is still a consideration. There are architectural choices you can make that will significantly impact how long it takes to process batch jobs.
Batching Is Meant to Scale
Batching scales easily. It’s meant to scale. I work in Lehi, Utah, called the “Silicon Slopes” because we have so many venture-backed software as a service (SaaS) companies. And these companies are notorious for needing to scale quickly.
A common scenario we run into goes like this:
A company secures $30 million in Series A funding. In four or five months, it takes the sales organization from 20 reps to 200 reps. That kind of growth pushes systems to the limit. And the volume of data you’re working with will grow quite a bit.
When you’re in a growth mode, you don’t want your systems to limit you. You want them to accommodate growth and allow you to scale. And, often, batching is the answer. It doesn’t matter if I have five or 500 transactions. If I’ve designed the integration correctly, the batch process can handle all that data at once.
Boomi Can Handle Batch and Real-Time Integrations
When you’re looking at an integration platform like Boomi, you’re often looking at a batch solution. Boomi will run a batch process on a set interval. It can be as frequently as once a minute or as infrequently as every fiscal quarter. You control how often those processes run.
Boomi can also handle real-time requests. It allows you to turn processes into listeners. Listener processes wait for a system to announce that a change has been made and will then trigger the integration. In the development world this is often referred to as an “event based” action.
For example, if you run a brick-and-mortar retail store, you could use a batch process to handle your daily POS transactions. But what if you have a member program that offers a 10 percent discount at checkout and a customer signs up while checking out? Your POS system needs to verify that in real time with the member account system, so the register clerk can process the sale with the discount.
So, in this instance, you don’t need a real-time integration between your POS and ERP system, but you do need one between the POS and membership system. If your brand really values the customer’s in-store experience, it’s important.
“Boomi’s architecture can handle pretty much any scenario from the cloud or on-premise, batch or real time. You don’t have to switch solutions and you can scale as much as you need.”
Real-Time Integration Without the Cloud
Sometimes a business has a true need for real-time integration with high volumes of data. And it doesn’t want to rely on a multi-tenant, cloud model because it can’t take a chance on exceeding the limits set by the cloud provider. When it needs a lot of processing power, it needs it. Period.
A great example of this is the retail industry during Black Friday. If I’m a retail store, I can confidently expect significantly higher-than-normal volumes, and I can’t afford to miss any of those orders. So, I must have adequate processing power to take my orders. But in the cloud, I don’t have complete control over how much processing power is set aside for me to use. Worse yet, it’s not a stretch to conclude that a lot of other customers are experiencing higher-than-normal transaction volume. So how do I ensure my business needs are met? Boomi’s architecture allows you to take control of this situation and set up your own dedicated web servers to process those transactions — and you can scale pretty much to infinity.
At Boomi World, speakers from Nest Thermostats described using Boomi to process Nest’s Internet of Things (IoT) data — a half-billion transactions every 20 minutes! That takes astronomical processing power, but Boomi’s got the architecture to handle it in real time.
Boomi’s architecture can handle pretty much any scenario from the cloud or on-premise, batch or real time. You don’t have to switch solutions and you can scale as much as you need.
Three Factors Determine Your Integration Options
Integrations are a direct reflection of three things: the processes your company has, the systems you’ve deployed in your application ecosystem and the quality of your data. These things determine the kinds of integrations you can do — your integration “profile.”
Let’s look at real-time integration through this lens. If you need real-time integration, you must have those listeners I referred to. You must be able to trigger an action to push information to something that will receive it.
A lot of software has APIs available, but these APIs allow only a pull request. They cannot accept a push request. When you’re working with pull requests, your only option is a batch-based integration. So, in this case, real-time integration isn’t possible because your system can’t do it. It’s not a Boomi limitation, or an integration limitation. It’s a system limitation. That’s something to keep in mind.
Fortunately, applications like NetSuite and Salesforce are robust and allow you to build in push functionality. It doesn’t come out of the box, but you can build it. Then you can use Boomi as a listener.
Salesforce even has a low-code option to configure a push request that will send information to a listener in Boomi that immediately kicks off a process. Presto! Real-time integration.
This ability to have an analyst – not a developer – build a complete real-time integration is one of the key benefits of low-code platforms like Boomi and Salesforce. This approach gives you more agility and shorter development times, with whatever type of integration you want to design.
Batch and real-time integrations have different pros and cons — there is no right or wrong answer. I encourage our clients to look at each option carefully, and to choose the approach that works best for their businesses.
Start your integration strategy with a framework-based approach that defines your needs and priorities, and helps you make the right technology decisions with Integration Demystified.
To learn more about how integration can help transform your business, please contact a Boomi integration expert or reach out to our partner Eide Bailly.
Check out the previous posts in our Integration 101 series: