We built an amazing AI agent that few people used. Here’s how we fixed it.
Six months ago, our internal Enterprise AI Team built what we thought the Boomi Customer Success team needed. We designed a sophisticated, multi-agent chatbot that could handle any query from a customer success manager.
We invested significant resources, created an impressive demo, and released it with great fanfare. But the adoption rate was under 20%.
So, we decided to kill it.
And that’s not a bad thing. We’ve replaced it with a more diverse group of agents that operate collaboratively, handling bite-sized tasks and fitting more seamlessly into CSMs’ daily workflow. The early results for our 2.0 version confirm that we’re on the right track: 85% adoption in 30 days.
Our initial struggle is a microcosm of a larger problem: widespread AI fatigue throughout every enterprise. We’ve gone from “AI can do anything!” to “Can AI do something?!”
A much-discussed MIT study put some sobering numbers to the widespread disillusionment: 95% of organizations are getting zero return from generative AI despite billions in investment.
At Boomi, we don’t do AI hype. We focus on practical examples of agents that achieve quantifiable productivity gains. In that spirit of “keeping it real,” I want to share an honest account of where our original CS agent fell short, how we iterated on the fly, and why the new team of agents is now having the desired impact. I hope every enterprise AI leader can benefit from our experience as they navigate their agent journey challenges.
We didn’t fail with the original CS AI agent. We learned.
The Vision: Giving CSMs More Time
I lead the Enterprise AI Team, which uses the Boomi platform to build, manage, and secure agents that our employees use to automate repetitive tasks and make their work lives easier. To cite just a couple of examples, we have a widely successful internal chatbot called ChatB, which provides secure access to trusted company information, and a sales handoff agent that ensures promising leads always get kid glove treatment.
Customer Success is another area ripe for agent innovation.
There are never enough hours in the day for our CSMs, as Boomi’s global customer base has soared beyond 25,000 organizations. But it’s not just that sheer number. Boomi’s enterprise-grade platform is a foundational technology for our customers’ operations, and they need immediate support whenever issues arise.
We designed the customer success agent as an intelligent assistant to give CSMs back time. It was a tool that quickly helped them better understand accounts, perform administrative work, and allowed them to shift their attention to more of the personalized engagement that creates better customer experiences. That was the intent, at least.
The feedback we received proved our assumptions wrong. We heard from CSMs that the agent became another “thing” they needed to insert into their daily workflow instead of streamlining existing processes.
The problem wasn’t the AI. It was our expectations for the CSMs. We were demanding behavior change at scale. We expected our hardworking CSMs to adopt yet another tool in their already fragmented tech stack. We had left them even more overwhelmed.
That’s why we made a radical decision. We dissolved the monolith and adopted a more composable mindset.
The Superpower of Invisibility
My colleagues Matt McLarty and Stephen Fishman recently published a well-received book called “Unbundling the Enterprise.” Their primary thesis is that when you break down business capabilities into reusable digital assets, you can rapidly use them as interchangeable building blocks to capitalize on new opportunities.
So, we took their advice and “unbundled” the overly ambitious CS agent into smaller components. We retooled by deploying a team of agents, each of which handles a specialized task. These Account Research Agents are essentially invisible. They work behind the scenes, providing our CSMs with weekly, AI-generated summaries that offer a cohesive view of all their accounts.
The agents consolidate key signals from multiple data sources, such as Salesforce, Gong, and Gainsight, covering recent engagements, support case activity, and digital interactions like webinar attendance and content downloads. It all happens proactively. The agents run autonomously, analyzing account health signals, and send the reports to CSMs on a pre-scheduled basis. The formatted information eliminates the need for CSMs to ask an agent for help. It’s delivered to them automatically.
The solution isn’t perfect, but by working closely with our CSM colleagues, we’re improving it every day. We already have more agents in the pipeline for their use. Our expectation is that these Account Research Agents will remove four to six hours of weekly prep work for each CSM, giving them more time for higher-impact, revenue-generating activities.
Our agent tactics have changed, but our goal hasn’t. We want our CSMs to appear superhuman with their knowledge whenever they interact with customers.
Lessons for Every Enterprise AI Leader
I believe our story has broad implications for enterprise AI. At a recent AI Technology Leader forum, I heard another alarming statistic: 73% of enterprises remain stuck in proof-of-concept purgatory. The primary barrier isn’t technology or budget. It’s the inability to demonstrate a clear ROI quickly.
Yet organizations targeting 1,000-plus agents by 2026 share one pattern. They’re embedding AI into existing workflows rather than creating new ones. That’s a mistake. AI is not plug-and-play. Here are three insights for AI leaders that I’ve learned.
- Start with time audits, not tech specs. Map out where workers lose hours to repetitive research and tasks. That’s your ROI goldmine.
- Measure adoption velocity, not capability breadth. A simple agent with 90% adoption beats a sophisticated one with 10% use.
- Design for deletion. Every AI workflow should eliminate a manual task entirely, not just augment it.
There’s also a larger, more important takeaway. Those responsible for technology in business operations must roll up their sleeves and get into the game of agentic transformation. They must do that even knowing they’ll probably get knocked down a few times. That’s OK. You just have to get back up.
I’ve been building software for 20 years. There have always been tried-and-true processes that we know will work. But this emerging world of probabilistic AI models and stitching together autonomous agents is an undiscovered country. IT leaders need to realize that they don’t know everything. Feeling uncomfortable is just the world we live in today. We’re all searching for that razor-thin balance between moving too fast and not fast enough.
But you have to move.
You’ll never learn how agents can help if you’re too risk-averse. Waiting means that, sooner rather than later, you’ll see competitors with capabilities where you have no expertise or capacity to mimic and stay competitive.
Yes, we stumbled with our first CS agent. But if we didn’t have failures and learning loops, it would be a sign that we’re not trying hard enough. Innovation comes from exploring the jagged edges of what’s possible.
The enterprises that will dominate the agentic era won’t be those with the most agents. They’ll be the ones that, through trial and error, embed agents so deeply into daily work that employees forget they’re using AI at all.
Learn more about how Boomi uses its platform internally to improve productivity and efficiency in these posts about our “Boomi on Boomi” philosophy and the ChatB resource.