By Boomi
At its heart, event-driven integration means enhanced real-time customer engagement and better-informed, predictive decision-making. The Solace PubSub+ Connector for Boomi enables organizations to reap those benefits faster by creating a combined Boomi and Solace experience that’s greater than the sum of its parts.
Boomi developers get to stick with the UI they know and love, while getting added benefits through the event-driven Solace platform. For Boomi architects, the connector fits hand-in-glove with a time-tested methodology and comprehensive platform powering your enterprise’s event-driven evolution.
In this post, I’ll introduce the concept of API-first development and walk through the first three steps of a 6-step methodology that will help you get the most out of your event-driven integration investment.
(Async) API-First Development
Architects using synchronous connections like REST and SOAP have long acknowledged the importance of APIs. There’s a reason that tooling like API Management is such a crucial part of the Boomi AtomSphere Platform. API Management enables modeling synchronous APIs, establishing service level agreements, regulating security, enforcing SLAs, registering data formats, and ensuring that API users are notified of changes.
Despite their importance, traditional development methodologies (and the realities of rushed modern development) typically leave API definitions until late in the development process: after coding is complete, the developer exports the data structure schema and uploads it to API management.
Lately, thought leaders have turned this development process on its head. Instead of a late–in–the–game export, these thought leaders urge enterprises to define APIs at the start of the development cycle, then implement them using coding tools such as Boomi Integration. This philosophy is called API-first. Here is a summary of Boomi’s approach to API-first development:
- Create a schema using a graphical mapping tool like Swagger.io
- Test the schema
- Promote the schema to Boomi API Management
- Create a process in Boomi Integration, using import capabilities to grab the schema
- Complete the business logic implementation using standard shapes available in Boomi Integration
Until recently, asynchronous implementations haven’t had the tooling or methodologies necessary for an API-first development process. That is no longer the case when pairing Boomi and Solace. Much like synchronous API management, architects and developers can now develop event-driven, asynchronous APIs and model them using graphical tools provided in Solace PubSub+ Event Portal. The newly released Solace PubSub+ Connector for Boomi can then seamlessly integrate your freshly created APIs into Boomi and use them as the basis for creating processes that follow best-practices for event-driven integration.
The enhanced tooling aligns with a standardized, repeatable methodology tailored for use in event-driven integration, both with organizations new to the concepts of event-driven architecture (EDA), and those well along the journey to a mature EDA implementation.
Step 1. Formalize and Evangelize Your EDA Goals
Before you begin the journey, it’s helpful to review and solidify why you are moving towards event-driven integration with Boomi and Solace. These goals can serve as key performance indicators in your new architecture and provide a helpful list of reminders of why you pursued this initiative in the first place.
Adding Solace to an existing Boomi infrastructure brings:
- Operational simplicity: integration that adjusts to different topologies such as on-premises and cloud-based deployments, while offering built-in high-availability and load-balancing capabilities
- Agility for adding features: simplifies adding innovative features on the fly
- Simplified error handling
- Realtime, global responsiveness: integration that starts immediately, rather than using polling or batch processing
- Increased robustness: integration that absorbs a sudden influx of data and simplifies error handling
- Agility for a changing business: integration that distributes incoming events to multiple information consumers and makes it easier to add more information consumers as your business changes
- Scalability: integration that grows with your business, while insulating legacy systems from increased traffic
- Design simplicity: integration that componentizes your Boomi processes for simpler choreography
- Next generation innovations: integration that allows for easier adoption of technologies like microservices, cloud, artificial intelligence, and machine learning
- Asynchronous API management: management through a graphical interface
Adding Boomi to an existing Solace infrastructure brings the ability to:
- Transform data from any format to any format using standardized, visual tooling
- Add content-based routing and route events based on their content
- Enrich events with information from databases, CRMs, flat file look ups, caches cross reference tables and more
- Split events from a single incoming event into multiple events for more targeted downstream processing
- Combine multiple transactions: into a single transaction, providing a more contextualized view of an event
- Bridge information coming from Electronic Data Interchange partners into the world of events
- Enhance connectivity options to databases, flat files and others
- Accelerate your web and app presence by using low-code application development
- Create a single “golden record” of information within an enterprise using master data management
- Manage synchronous APIs through a graphical interface
While event-driven integration provides numerous benefits, enterprises need to exercise care in their evolution towards event-driven maturity. Making sure that your organization is on board, from executive stakeholders down to junior developers, requires thought leadership and persistence. The integration team needs to start thinking about choreography, rather than orchestration. The API teams need to start thinking about event-driven APIs, rather than just request/reply. Support personnel need to understand how events move through the enterprise. Leadership needs to grasp the benefits of real-time information flow across multiple clouds.
Fortunately, Boomi and Solace have collaborated on a wide range of materials to help you evangelize an organization. Here are a few recent posts:
- Four Factors to Consider When Choosing Between a Built-in vs. 3rd-Party Event Broker for iPaaS
- Supercharge your iPaaS for Better Agility, Resilience, and Real-time Capabilities with an Event Broker
- Simplify Event-Driven Integration With Solace and Boomi
It’s important to note that evangelism is not a one-time activity. Keeping an EDA evolution on track requires a continuing process of education and realignment to goals as the journey continues. As your company onboards more events and starts to see the fruits of your labor, the evangelism shifts from “why are we doing EDA?” over to “check out these events available within our enterprise, how can your group use them?”. An event portal plays a crucial role in that.
The remaining steps in the methodology ensure that your enterprise captures the benefits of event-based integration, while minimizing the uncertainly, resistance and pain associated with any large architectural shift.
Step 2. Understand your event-driven integration foundation
While nothing is more exciting than the possibilities of event-driven integration, bringing the concepts of EDA into reality take a firm foundation. While there are multiple options for EDA, this post assumes that you’ve decided to use the Boomi and Solace product stack. It’s a potent combination. But getting the most of your foundation requires the strengths of each of the components.
Boomi Components
Integration: Drag and drop UI, data mapping tools, and a comprehensive library of connectors. Coupling this with support for various integration patterns enables you to build any integrations with exceptional speed.
API Management: Supporting the full lifecycle of synchronous APIs in a hybrid environment. First, configure synchronous APIs and expose their functionality to both internal and external clients, then test and deploy synchronous APIs and enforce contracts and policies with an API gateway.
Flow: A low-code platform for apps that allows everyone to build workflow applications, without needing to be a software developer. Simply drag and drop your workflow as a flow diagram, and the Boomi Flow platform converts this into a fully functioning workflow application.
Master Data Hub: A cloud-native master data management (MDM) solution that sits at the center of the various data silos within your business and provides easy to implement, scalable, flexible, and secure master data management hub as a service.
Solace Components
Solace PubSub+ Event Broker: Modern messaging layer that can be deployed across every environment (appliance, on-premises software, cloud-based software, messaging-as-a-service) and component of the distributed enterprise.
Event Mesh: A collection of event brokers in different physical locations, woven together that dynamically routes events from one application or connected device to any other— no matter where they are deployed. Whether it’s in legacy on-premises systems or in the public cloud, private cloud, or somewhere in between, an event mesh takes the guesswork out of getting events where they need to be.
Event Portal: Event management toolset that enables you to create, design, share, and manage various aspects of your event-driven architecture. Event Portal brings to the async world the same functionality that lets enterprises create API-first architectures.
Cloud management: PubSub+ Event Broker: Cloud is a software as a service (SaaS) platform that gives you access to advanced features like the publish-subscribe messaging pattern, queuing, streaming, replay, and request-reply, while doing all the heavy lifting to dynamically deploy, upgrade, patch, and operate your event broker services. It is engineered to be secure, resilient, and cost-optimized so you can focus on building applications.
These components interact to create a complete ecosystem for event-driven interaction:
At the heart of the Boomi-Solace relationship is the PubSub+ Connector for Boomi. This connector lives inside the Boomi Integration GUI, both establishing runtime connectivity and letting you carry out common Solace tasks like discovering event definitions without ever leaving the friendly confines of Boomi. This integration creates a better experience for developers and keeps your EDA design vision on track. As part of the PubSub+ Connector for Boomi, there are multiple sample processes available in the Integration Process Library, which in turn link to extensive walk-through documentation. We’ll talk much more about the connector and process library later in Step 5 where we make our designs a reality.
The Basics of Events
While we are getting acquainted with the Solace and Boomi components, it might be helpful to review the terminology of event-driven integration. We’ll illustrate using a scenario where a fictional taxicab company hires a new driver through the SaaS provider Bamboo. In turn there are three downstream applications that are all interested in a new hire: Slack, Salesforce, and ServiceNow.
Note: there are many ways you can implement an event-driven architecture with Solace. However, the typical scenario and best practice is: publish to a topic, read from a queue. Some of us have tattoos that say exactly that. So, to simplify things, that’s what we’ll describe here. There’s much more in the documentation about other use cases.
Event: an event is a business moment where something changes somewhere in the enterprise. Here, the event is the taxi company hiring the driver. The key information about the hiring event is put into the payload (or content) of the event. For instance, the driver’s name, social security number, and address would be in the payload in a JSON or XML format.
Topic: a topic is a hierarchal string that briefly describes the content of the event to make routing more efficient. The topic is meta-information, separate from payload. The format of the string depends on the organization. The topic for this message is: /taxico/hr/bamboo/Driver/hired/v1/349034
Which can be broken down like this:
- taxico: Name of the company
- hr: Division within the company
- Bamboo: Application that created the event
- Driver: what the event is about (the noun)
- Hired: what happen (the verb)
- V1: the version of the event
- 349034: The person ID for the driver
Subscription: downstream applications indicate what events they are interested in using subscriptions. These subscriptions follow the same format at the topic string, but the true power comes in being able to use wildcards. So if Slack wants to know about all drivers that are hired, it can subscribe to the topic string: /taxico/hr/bamboo/Driver/hired/v1/* where * is the wildcard for “all”
And if Salesforce needs to know about drivers that are both hired, fired and promoted, it can use the subscription: /taxico/hr/bamboo/Driver/*/v1/*
Queue: When the event broker finds a subscription that matches the topic string of the incoming event, it creates a copy and places it on the queue. The queue holds messages until the downstream application is ready to process them. Some remarkable things about queues: they guarantee order, they preserve messages if there is an issue with the application and they serve as a buffer if there is heavy volume for a period of time and the application can’t handle it.
There might be many matching subscriptions, in which case all associated queues get a copy of the message, which makes publishing to a topic and reading from a queue a terrific way to spread information from one source to multiple interested consumers.
Step 3. Speed Date Pilot Candidates and Pick the Lucky Winner
No matter how great your foundation, biting off more than you can chew is sure way to sink a promising event-driven integration project. The key is a juicy pilot project that shows off event-driven integration without a big commitment. But with most enterprises featuring hundreds if not thousands of integrations, how do you pick a winner?
If your organization already has Boomi and wants to expand into event-driven integration with Solace…
One excellent feature of Boomi is that you can view deployed processes at any stage of the lifecycle through the “Deployments” view of the UI. This can give you a sense of how your organization is using Boomi, and which processes would be good candidates for Solace. These are signs that your Boomi process would be a strong candidate for introducing Solace for event-driven integration:
- Information incoming from a sole source using the process route node is distributed to multiple endpoints
- Long running processes
- Challenging error handling
- Processes that can use eventual consistency rather than strong consistency
- Processes that leverage multiple clouds or have cloud-based applications that need to access onsite resources
- Uses Internet of Things (IoT) or mobile phone applications
- Process where volume significantly varies over the course of a day or a week or is seasonal
- It’s using low latency mode to improve performance
- Already use messaging, or can use messaging technologies like JMS
If your organization already has Solace and wants to enhance your event-driven architecture with Boomi’s capabilities…
If you’re using Solace PubSub+ Event Portal, then you have an excellent catalog of applications and events moving through your enterprise. Use this as a starting point to locate good candidates event flows. These are signs that Boomi could enhance your existing Solace event integration:
- Data transformation requirements that are either currently unmet or addressed in an ad hoc fashion
- Desire to provide better context for an event using data enrichment from a lookup table, in-memory cache or other techniques
- Want to dig into the payload of the event to route more efficiently
- Splitting single events into multiple events with different structures to better pinpoint information
- Need API Management for synchronous interactions
- Using custom coding to connect to databases, legacy applications, flat files and more.
- Need MDM capabilities
- Need a low-code app
Applications featuring these characteristics (the more the better) should form a short list of potential candidates from which to choose. From there, winnow down further. The best event-driven integration pilot projects deliver a potent combination of:
- High business value, ideally through an improved customer experience
- Involves real-time data
- Rapid time from design to development to production
- Addresses a common use case within your organization (e.g. information coming out of Salesforce, distributed amongst multiple consumers)
- Resides in the happy medium between insanely complex and boringly easy
A quick win that delivers an improved customer experience can breathe optimism into both technical and business teams, relieve anxious stakeholders and provide momentum for additional projects. If you can simultaneously make developers and support personnel’s lives better by alleviating obvious technical pain (brittleness, performance, resiliency) so much the better.
In my next post, I’ll cover the final three steps to take to get the most out of your event-driven integration investment.
Boomi has partnered with Solace to extend the value of the industry leading Boomi platform with enterprise grade event streaming and event management capabilities. Learn more here.