More and more enterprises are moving from on-premises enterprise service buses (ESB) to integration platforms as a service (iPaaS) like Boomi.
iPaaS is a key component of what Gartner calls a hybrid integration approach1, and as someone accustomed to creating ESB solutions, it’s been eye-opening to see how quickly and easily you can get them up and running. But as I create iPaaS solutions, one thought keeps running through my mind: this would be so much more powerful and elegant if we added an event broker to the mix.
iPaaS and Synchronous REST Calls
The aim of iPaaS is to simplify integration by creating a managed experience, particularly when deployed in the cloud. iPaaS typically includes:
- Graphical tools that show how transactions flow through the system
- Connectors that contain logic for popular protocols
- Built-in mapping tools that allow for data transformation
- Defined locations for code snippets that isolate customer-specific modifications, typically in a language like JavaScript
- Integrated source code repositories and deployment capabilities that are usually external in ESBs
Although there are adapters for many different information providers such as databases and EDI, the bread and butter of most iPaaSes (and most ESBs) has been synchronous REST calls between software as a service (SaaS) providers like Salesforce and SAP S/4HANA Cloud. As a result, iPaaS providers have focused on making synchronous API integrations as easy as possible.
Architecturally, this generally means the same process that receives an incoming request maps the data using a graphical tool, connects to the information consumer, and sends the transformed message.
Figure 1: Typical iPaaS process flow
Adding an Event Broker to Your iPaaS Solution
Adding an event broker allows you to separate (decouple) that process, such that the iPaaS receives information via one process and writes it to a queue or topic2, then a second process picks up the message and performs additional logic, typically including data transformation, before forwarding it to a consumer.
Figure 2: iPaaS flow with event broker
That decoupling can enhance traditional ESBs, but in some ways it’s even more important for an iPaaS. The architectural shift to decoupling using an event broker can improve an iPaaS’s resilience, agility and real-time capabilities — all without complicating its architecture or operation. The approach is a lot like breaking up a monolithic application into multiple microservices.
Some brief asides:
- Many iPaaS providers have begun to integrate lightweight event brokers into their products. While I think there are advantages to having an event broker live outside the iPaaS itself, I will save that for another day and focus on the benefits of all event brokers, including those they include.
- While event brokers provide great benefits for iPaaS architectures, they also present challenges. In a future blog I’ll discuss those challenges, the trade-offs they entail, and give guidance on architectural choices that can help you overcome them.
Boomi has partnered with Solace to extend the value of the industry leading Boomi Platform with enterprise grade event streaming and event management capabilities. Learn more here.
Planning for Failure – How an Event Broker Helps an iPaaS Stay Simple
One of the most complex and time-consuming tasks for any integration architect is making sure a system performs when things go wrong – in fact some experts say up to 25 percent of an integration project should be devoted to designing exception handling, and iPaaS solutions are no different. That isn’t surprising given that architects need to consider complex questions including:
- What if the iPaaS itself fails? Will I lose in-flight transactions?
- What should happen if a consumer goes down and I need to retry? How can I prevent it from decreasing performance and blocking incoming?
- What if a consumer can’t process my incoming message until a human intervenes (e.g. adding a value to a cross reference table)? How can I easily move on to the next transaction and replay it later?
You can try to solve these issues within the iPaaS itself, perhaps by creating a library of custom code snippets in a language like JavaScript that dictate architectural standards like manually compensating for failed transactions and using adapters to ensure that transactions are persisted to disk. However, in doing so you sacrifice some of the simplicity that makes iPaaS so advantageous.
Introducing an event broker as part of a decoupled architecture, on the other hand, can answer some of the questions while keeping the iPaaS simple.
As shown in Figure 3, by persisting messages to a queue or topic after arrival at the iPaaS, the event broker takes responsibility for guaranteeing that the message will not be lost in the event of a failure that brings down either your process or the entire iPaaS. Without an event broker, an iPaaS either requires custom persistence logic or accepts that messages will be lost in the event of failure, requiring the information provider to resend as necessary.
Figure 3: Event brokers allow for message persistence in the case of failure
Once the message is persisted by an event broker, the iPaaS receiver (not the end consumer) can acknowledge that the message has been successfully processed and becomes available for the next incoming message.
This ability for the iPaaS receiver to quickly and independently process messages comes in handy if the downstream information consumer isn’t available. Typically, an information consumer being unavailable results in multiple attempts to retry the transaction until it succeeds.
As shown in Figure 4, if the receiver and sender are on the same thread, this can quickly cause the iPaaS to run out of resources, creating a chain reaction that blocks incoming transactions. By buffering incoming transactions, an event broker helps to keep your solution up and running. This buffering is even more valuable as we add in additional consumers, which we discuss in the next section.
Figure 4: Event brokers conserve resources during retry
Lastly, if the error turns out to be one that will take a while to resolve (e.g. someone to add a value to a look up table), the message can be placed into an error queue. When the issue is resolved, the message can be requeued for processing.
Figure 5: Event brokers simplify replay after human intervention
The Value of Event-Driven Architecture
To this point, we’ve focused on the benefits of decoupling a single sender and receiver from one another, and the benefits of that decoupling are worthwhile on their own. But the value really adds up as you begin distributing transactions amongst multiple diverse consumers using a fully event-driven architecture. Making your integrations more agile lets your architects and developers innovate more freely, solving business needs and exploring new frontiers for your business.
Architects need to consider complex questions including:
- How can I quickly introduce new business functionality while making sure I’m not affecting existing functionality?
- How do I handle situations where one consumer processes information faster than another? Or one consumer rejects the transaction, while another processes it?
- How do I make sure I’m not locking myself into a deployment pattern (on-premises, cloud, etc), cloud provider, or even another iPaaS?
To aid agility, iPaaS providers typically feature integrated source control and deployment capabilities. Including these components reduces the amount of additional infrastructure needed to start building an iPaaS – no more setting up a Git repository and configuring Jenkins.
Most integration platforms as a service include the ability to call multiple sender processes from a single receiver process, but as Figure 6 shows, even with the code promotion infrastructure in place, if the solution tightly couples senders and receivers of information you still need to have a complete test cycle and deployment with every change.
That cycle is represented by a link icon in the diagram below. This tight coupling also makes error handling more complex, particularly in cases where one consumer rejects the transaction and additional consumers process it successfully.
Figure 6: iPaaS with multiple tightly coupled consumers
An event broker allows you to move into the realm of fully event-driven architecture. At the outset, the addition of an event broker may seem like it increases complexity — after all, there are now more components — but the addition of those components is more than offset by the fact that an event broker makes code more modular and reduces integration challenges.
By playing middleman between receivers and senders, an event broker lets you change a sender without redeploying and retesting the entire process. What’s more, you can very easily add new senders and consumers, and seamlessly integrate them into an existing solution.
With an event broker, you can design a new sender connecting to an innovative new application using iPaaS tools like graphical mapping, content-based filtering, and pre-built connectivity. Then once you add a new subscription to the event broker, events begin flowing to the new process and onward to the information consumer.
Figure 7: iPaaS with loosely coupled consumers using event broker
Learn more about the Boomi and Solace partnership in our blog post “Boomi and Solace Deliver Event-Driven iPaaS to Power Real-Time Integration.”
Handling Slow Consumers and Routing Information with an Event Mesh
At runtime, decoupling lets consumers process information independently. This can help to resolve several difficult architectural challenges:
- The first challenge decoupling alleviates is information consumers that process transactions at different rates of speed. An event broker creates a buffer that allows slow consumers to process messages at their own pace without affecting or constraining those applications that can process messages much more quickly.
- Another challenge is dealing with instances in which one consumer rejects an incoming transaction, but other consumers are still able to process it. An event broker can handle that scenario without resorting to ad hoc error handling. That said, error handling in an event-driven architecture can be tricky, and I look forward to diving into that topic in a future blog post.
- By linking multiple brokers in different locations to form an event mesh, event brokers can efficiently route information between on-premises and cloud applications, different clouds and even different iPaaSes — without having to explicitly route information between them within your iPaaS.
- Such an event mesh can also address the challenge of unreliable WAN connectivity by provide reliable delivery without each iPaaS application needing to handle this. If your deployment pattern changes, the event mesh dynamically reroutes events to the new topology.
Reacting to Real-Time Event Sources…in Real-Time
Beyond decoupling and event-driven architecture between iPaaS components, an event broker provides an on-ramp to real-time capabilities for your iPaaS. The internet of things (IoT) and other real-time information sources such as Workday and Salesforce present a massive opportunity to increase your business’s ability to react, but you typically need to act on such information in real-time to fully exploit its value. Real-time information sources can also send a massive spike in traffic during periods of high demand.
The need to process incoming events in real-time, and deal with sudden unpredictable bursts of data can be challenges for any iPaaS. In the absence of an event broker, an iPaaS is typically triggered either through a synchronous API request, or on a polling interval defined when the process is developed and deployed.
In the case of a process invoked by a synchronous API request, dealing with traffic spikes within the structures of an iPaaS typically means increasing the number of processing threads available to handle the incoming flow of information and over allocation of memory and processing power. In the case of a polling interval, it is difficult to predict how often to poll far in advance, especially if you can’t change that interval when conditions change.
Figure 8: Using polling for real-time event processing
An event broker helps you overcome these challenges by providing an entry point for event-driven applications. Messages that are placed on event broker topics can trigger your iPaaS immediately, without having to worry about selecting a static polling interval. An event broker can also buffer incoming events and provide a throttling mechanism that prevents downstream processes from being overwhelmed in periods of high traffic.
Figure 9: Using an event broker for real-time event processing
Conclusion
Event brokers are a great potential partner for your iPaaS, enhancing their resilience, agility and real-time capabilities. Recognizing those potential benefits is the first step in the journey. The next steps are recognizing and overcoming some challenges that event brokers can introduce. In my next post, I’m looking forward to answering questions such as:
- What are situations when it makes sense to use an event broker, and when is it overkill?
- What are the trade-off between using an iPaaS-provided message broker and an external one?
- What does error handling look like when using an event broker?
- What are some architectural “gotchas” associated with using event brokers with iPaaS?
- How do I make all this work in real-world environments?
Together, Boomi and Solace provide integration and data movement for any hybrid environment at scale, regardless of complexity. Find out more in our partner solution brief.