We knew Data Integrations Patterns was a hot topic, so we wrote an eBook: Exploring Data Integration Patterns. Our follow-up webinar featured a live demo of integrations patterns in action, and we were thrilled to have an engaged audience, eager for more knowledge. We received over fifty questions for the webinar!
With 15 minutes dedicated for Q&A, we only could answer a handful of those questions during the event, but we did promise to answer them all and post in the Boomi Blog.
So, here are the answers to the questions that were submitted during the Exploring Data Integration Patterns webinar (duplicates have been removed), from myself and my colleague Kristina Avrionova.
Happy reading…
Q1. Suppose there are 100 million records we have to migrate from source to target, with complex transformation logic. How much time it will take to load data as per your experience?
That’s always a fun question! There are techniques and architectural and runtime deployment approaches to address those situations. It’s also how you go about extracting that data in parallel versus sequentially, how the workflow can be divided and conquered across multiple nodes in a molecule cluster, how those nodes are appropriately configured from infrastructure and resource perspective, CPU versus memory and disk, and so on. We would need more information on the scenario and the needs to provide recommendations for the optimal approach.
Q2. I have a requirement to post data from one source system to multiple targets. Each target needs a different dataset (e.g. source has 10 attributes). Target 1 needs only eight attributes. Target 2 needs only five attributes but I also need to apply filter criteria before posting the data to Target 2. Target 3 requires all 10 source attributes, but on the target side (source attribute 9 and 10 concatenated and mapped to attribute 9 on the target side). How do I configure this?
The flexibility that Boomi integration processes afford makes it easy to configure unique, complex routing and mapping requirements like these. Using logic shapes like branch allows the source data to be sent to multiple paths and to be filtered, mapped and sent to different destinations as needed. If further decoupling is desired, processes can be designed with subprocess flows, message queues or microservices to allow for even greater flexibility.
Q3. I’m assuming that I can use Boomi to enable a query from one cloud service (for example, a marketing automation system) to another system (on-premise CRM) to see if there is an existing lead or opportunity based on a contact ID and product ID? I’d like to return a message that I can write to a record to the first system (the marketing automation system) to branch my campaign flow. If yes, will this occur in real time or near real time?
Yes, you can automate your marketing workflows with Boomi, and automated processes will occur in near real time. This will mostly be event-triggered integration. Please refer to this article for more information.
Q4. Can you use Boomi to go from one blockchain system to another?
Yes you can. Read this article to understand more about the use case.
Q5. What is a difficult or complex integration (if any), since you explained all of them are easy?
Well, we said all integrations are easy to do with Boomi, because the Boomi Platform has it all. Again, Boomi will facilitate data mapping so you gain the benefit of format conversion. It will transform and send data to a publisher as well as convert data to read from an internal subscriber. Boomi will also expose data via our API management tool and help applications unlock the target data. And lastly, it also serves as an ETL process. And let’s not forget that we also do EDI/B2B integration. Really, Boomi will help you address any integration challenge.
Q6. Does Boomi support Snowflake connectivity?
While we do not have a branded Snowflake connector, we can and do connect to Snowflake. See Snowflake Database Integration.
Q7. How is your solution better than other solutions in the market, like Informatica, Pentaho, Talend, ODI, etc.?
Excellent question! What makes Boomi special, aside from our 9000+ customers, connection to over 1500 endpoints, six consecutive years as a Gartner MQ leader, and the fact that we actually pioneered the space, is that we offer a unified platform with baked-in intelligence and rich insights.
Q8. Can you explain how a case base would work? For example: I want to get on-demand info for a customer that will vary per case and drop that info into Excel or .csv and send an email to the requestor with this data.
With the integration process, you can perform lookups to external applications and data stores to enrich or retrieve additional information based on dynamic input values. Additionally, for more static lookup or translations, values can be configured within the Boomi integration process using a cross-reference table component.
Q9. Does the Boomi Platform include an API Gateway?
Yes it does. You can read more about our API Management solution here and here.
Q10. When either system changes, will you have to update these workflows or can they be dynamic?
In short, yes. If an endpoint changes in a material way and you want to take advantage of that change, you need to explicitly modify the integration process to incorporate it.
For example, a common use case might be to add a custom field to one application and map that across. To accommodate that, you might need to modify the source query logic and the mapping to the target. That can actually be a good thing if you need control over making conscious changes to those APIs or to the mapping rules vs. allowing dynamic mapping — especially in organizations that require change control policies around tracking changes and thoroughly testing before production.
Q11. When do those cool new icons make their way to our desktops?
Soon! We’re refreshing a number of things in the UI but in the meantime, you can preview these new shape icons for yourself if you want to check them out. Go to Setup > Beta Features
Q12. How does Boomi handle big files? What are the platform’s limitations when it comes to ETL volumes? Also, how can ELT be applied on the source and target side?
There are several strategies for handling large files and data sets. As demonstrated in the webinar, a divide-and-conquer approach can be achieved by splitting data and processing in parallel (e.g. multi-thread) through mapping and connector steps.
Many connectors have built-in paging and batching to handle large data sets. Boomi is optimized for files and data sets in the tens of gigabytes range. For ELT use cases, Boomi is typically used to orchestrate the loading and copying of data into data storage providers and then triggering subsequent actions to be performed on the target side. For example, you can load data into Amazon S3 and then trigger a COPY action to move the data to Redshift.
Q13. What is required to run an Atom?
At a technical level, there are system requirements. The Atom itself is a lightweight Java runtime engine that runs on Windows or Linux, whether that is on-premise or on a cloud hosting provider, like AWS or Azure, or even Docker. If you are running a containerized operation, we have Docker images for Atom runtimes. You can read all about Atom, Molecule, and Atom Cloud setup here.
Q14. Are Atom Queues JMS compliant?
While technically built using JMS-compliant technology, Atom Queues are not exposed
Q15. Is it possible to introduce human augmentation or changes to data within the flow components, or are you limited to approval?
Yes. You can develop Flow applications that can display document data to end users to review, correct and resubmit for further processes on the integration side. This is commonly done using a shared database.
Q16. I would like to understand more about error or exception handling.
Error handling is a necessary part of any production-quality integration. Depending on your needs and how fine-grained and how precise the control you need over the errors and the recovery is, you have a couple of different facilities at your disposal. There is a great module on error handling available in our free Boomi Training (Architect 1 course).
You can also read this article to learn how you can add error handling in Flow.
Q17. Can you share the process examples?
Yes, the examples are available here: Webinar: Exploring Data Integration Patterns.
Q18. You have mentioned parallel processing with Flow Control activity. With these four threads will the API be called in parallel with different 200 records on each call?
That’s correct. In this example, the source records were split into batches of 200 records per document, and then spread across four threads to be sent to the destination API in parallel.
Q19. Do Atom Queues guarantee message order (such as FIFO?)
Yes, Atom Queues can be configured to ensure FIFO by effectively single-threading reads from that queue. Read everything about Atom Queueing here.
Q20. Does Boomi has solutions for high-volume data streaming mechanism like i.e. Apache Kafka?
Yes, learn more about our newly released Kafka connector here.
Q21. Does Boomi provides full CI/CD and deployment/promotion tooling?
Boomi can connect any and all DevOps tools that you might have in-house. We already plug into your DevOps ecosystem through a rich set of APIs to automate many common, repetitive DevOps tasks such as deploying and promoting processes and APIs, or provisioning runtime engines and operational monitoring.
We are very serious about DevOps. Read more about our commitment to support our customers’ CI/CD journeys.
Q22. Can we build generic flows and add them to a specific business process/flow? For example, write an error handling flow that can send email notifications/SMS and then add this flow to multiple process flows?
Yes. You can configure generic flows such as error handling and notifications that can be called from any other process flow. Alternatively common logic can be decoupled via message queues or microservices.
Q23. Is there a schema validation for document/value transformation from source or destination?
There are several options and approaches for validating data. For basic structural validations, you can use the Cleanse shape. For more complex XML validation against an XSD schema file for example, you can use a Data Process Custom Scripting shape (example). Also, the Trading Partner framework automatically performs validations on incoming EDI documents.
Q24. For how long of a period do you allow to customers to use Boomi for their POCs?
We offer a free 30-day trial — and you should really give it a try!
Q25. Is the queue component used in sub/pub example not part of integration? Do you need Flow?
Atom Queues are an add-on feature included in some higher editions. Contact us for more information.
Q26. Do you offer data encryption in transit?
Yes. In addition to communication channels typically being encrypted using HTTPS, the data payloads themselves can be PGP encrypted, depending on what your destination supports.
Q27. Do you offer scheduling integrations (e.g. hourly incremental data)?
Yes, integrations can be scheduled within the platform, down to every minute. See Process Schedules.
Q28. Do you offer monitoring (amount of data going through or successfully completed) and alerting function (if any failures)?
Yes! We offer comprehensive process monitoring. Learn more here and here.
Q29. How do you do authorization / authentication when you are establishing connection between source and target?
When connecting to endpoints as a client, Boomi connectors support the authentication mechanism required by that endpoint such as basic authentication, certificates, OAuth, Kerberos and more. When publishing API endpoints, the Boomi web server supports basic authentication and client certificate, and can broker authentication with external identity providers to support many more.
Q30. Do you offer conversion (transformation) of data? For example, on the source side web service call, does the target side data need to be in a .csv file posted on a target site or need to be loaded into a database on the target side?
Yes. Boomi will facilitate the transformation of the data. This is done via the mapping component, which allows you to transform data from any format to any format. Moreover, with Boomi Suggest, you can rely on millions of indexed mappings to automatically recommend dataflows for new integrations based on the successful configurations that have been built by other users
Q31. On the single source to multiple target scenario, our target application owners need ability to review status of their data flow. Do you offer that function?
Yes. Application owners can be granted access to Boomi to view integration job results in the Process Reporting console. Alternatively, leveraging the Boomi Platform APIs, the process results can be exported and presented in a custom UI of your choosing, including using Boomi Flow for low-code application development
Q32. Do you have 3rd party security assessment for this system?
We have just completed FedRAMP authorization and have met the stringent security and reliability standards. Federal Risk and Authorization Management Program (FedRAMP) Authorization is a government-wide program that provides a standardized approach to security assessment, authorization and continuous monitoring for cloud products and services
Q33. Can we set return payload without using message shape?
When returning responses to a web services request, depending on the requirements of your web service you can use a message shape to create a simple payload, use a map component to create a complex payload, set a simple HTTP status code using a Set Properties shape, or simply return the response directly from the backend system.
Q34. Are you able to transform at the data level? For example, use functions like substring, replace, sum.
Yes. The mapping engine provides many prebuilt functions including scripting and the ability to configure custom multi-step functions.
Q35. In most of the integrations, we get the challenge of source sending the full file but on the target side only changes need to be considered. How does Boomi act as middleware to keep track of changes only for necessary updates?
Boomi integration processes have a built-in basic change data capture capability in the Find Changes shape. However a more robust option is Boomi DataHub, which is a full-featured synchronization engine that provides many controls over value differentials, data models, field-level contribution and survivorship rules, data quality, stewardship consoles and more.
Q36. We want to work as an FTP server / SFTP server to receive applications like as2 or web service requests. Do you have a server component/adapter, or plan to have one in future?
We do not currently offer FTP/SFTP server hosting as part of our Managed Cloud Services offering
Q37. Say we have developed the orchestration workflow using AtomSphere on our development (on-premise) environment. How can I package and deploy this same orchestration on a Production Cloud environment?
Versioning and deployment infrastructure is built into the Boomi Platform. After building and testing a process in a local dev Atom/environment, that exact same deployment version is copied or promoted to higher environments. Those environments and runtimes could be local or hosted.
Learn why Boomi is trusted by more than 9,000 leading brands: Sign up for our 30-day free trial today!