AI Architecture Strategy For A Multi-Agent Customer Support System
Client
A leading SaaS platform for the professional services industry, providing appointment management, point-of-sale, marketing and client relationship tools to thousands of small business owners worldwide.
About
Industry
SaaS & professional services software
Company size
250-500
Client Market
Global
Location
Ireland
Services Offered
- AI Alignment
- AI Strategy
- AI Workshops
- AI Agents
Challenge
Production Reliability Challenges Across A Growing Multi-Agent System
Intent detection, service recommendations and authentication all worked in testing, but broke down under real customer conversations as the system scaled to around fourteen agents.
The engineering team’s multi-agent system passed every test, with intent recognition scoring 100% on controlled datasets, yet real conversations exposed the gap: requests like rescheduling an appointment while adding a service bounced customers between agents that could not parse the unexpected phrasing. Vector store recommendations compounded the problem, producing illogical service and provider matches, particularly for new customers with no history, and eroding the predictability business owners expect from any recommendation.
Earlier attempts to fine-tune the underlying models on conversation data had already failed, overfitting to training examples and generalising poorly, so the team abandoned that route after weeks of work. Authentication added further strain: because each agent authenticated independently, customers were sometimes asked for a phone number late in a booking flow, breaking the conversational feel. With around fourteen agents and overlapping responsibilities, the system had grown hard to debug, without a clear pattern for structuring it reliably.
The engineering team had already built a working multi-agent prototype and knew what they wanted the system to do, but needed specialised AI/LLM architecture expertise to make it reliable in production.
Solution
One hands-on workshop, three phases
We began with a ninety-minute deep dive into the existing system
The team reviewed the client’s LangGraph architecture, agent design and orchestration flow, then examined real production failure cases and customer conversation logs rather than test results. This included an assessment of the existing testing methodology and observability infrastructure, establishing exactly where the system was breaking down in practice.
We whiteboarded alternative architectural approaches to the system’s hardest problems
Over ninety minutes, Zartis and the client’s engineers discussed trade-offs between complexity and determinism for each problem area, exploring specific tools and techniques suited to the existing stack. The session validated technical decisions the team had already made whilst challenging assumptions that were adding risk without adding reliability.
Turning the workshop’s technical findings into a concrete, sequenced action plan
The final sixty minutes set out specific next steps and deliverables, and established how Zartis and the client’s engineering team would collaborate on further analysis. The session also covered a phased production deployment strategy and the risk mitigation needed before any of the recommendations reached real customers.
The Zartis Approach
What we delivered
Graph-based information retrieval
The client needed reliability over sophistication. A knowledge graph approach was picked to improve reasoning - over explicit relationships rather than embedding proximity. This produces recommendations business owners can trace back to a specific reason.
Challenge: Vector-based recommendations were inconsistent and impossible for business owners to trust.
Recommendation: Replace vector similarity with a knowledge graph of services, staff and skills.
- Build a knowledge graph of services, staff, skills and relationships
- Use graph traversal to reason about the best match
- Trade embedding proximity for explicit, traceable service relationships
- Keep recommendations deterministic while handling complex service catalogues
Granular, single-purpose agents
Smaller agents create clearer failure points. When something breaks, the team can isolate the specific component responsible rather than untangling monolithic LLM behaviour across the whole system.
Challenge: Found fourteen agents with overlapping responsibilities were hard to debug or maintain.
Recommendation: Split complex agents into subgraphs with clearly defined, testable responsibilities.
- Split the booking agent into subgraphs with distinct responsibilities
- Use structured workflows with routers and defined control flow
- Add an agent handoff pattern for smooth transitions
- Design each agent around a single, testable purpose
Prompt engineering, not fine-tuning
Service definitions change constantly, and fine-tuned models go stale quickly. Prompt engineering iterates roughly ten times faster and produces more predictable behaviour without retraining.
Challenge: Earlier attempts to fine-tune models on conversation data overfit and failed to generalise.
Recommendation: Engineer versioned prompts instead of retraining models as business logic changes.
- Use existing chat conversation data to shape conversational style
- Write comprehensive system prompts with explicit, detailed instructions
- Template common elements like date, time and business context
- Treat prompts as versioned code, not playground experiments
Testing and observability infrastructure
Passing tests is not enough on its own. Measuring model confidence at each step of the workflow creates an early warning signal before failures reach real customers.
Challenge: The system passed every test yet still failed on real customer conversations.
Recommendation: Add unit, end-to-end and confidence-based testing across the agent workflow.
- Run unit tests on individual agents using LangSmith
- Test full multi-agent workflows end-to-end with mocked networks
- Analyse perplexity at each step to flag uncertainty
- Keep the architecture provider-agnostic to test different models quickly
Phased production deployment strategy
AI in production carries different risks to traditional software. A phased rollout with feature flags and kill switches means a problem affects a controlled subset rather than the whole customer base.
Challenge: The team had no agreed plan for safely rolling AI changes out to customers.
Recommendation: Roll out changes in three phases, with security checks built in throughout.
- Test internally with business owners and staff first
- Release to a feature-flagged subset of businesses next
- Monitor a wider rollout using A/B testing and metrics
- Red-team for prompt injection and add moderator nodes
- Trigger authentication only when needed, never upfront in the flow
Business Impact
Faster resolution of complex customer requests
More predictable, explainable service recommendations
Clearer, easier-to-maintain agent architecture
Earlier warning of failures and faster iteration without retraining models
Safer, phased rollout of AI changes
Results
A validated architecture roadmap
Resolved architectural debate
Questions that had stalled internal discussions, including whether to use vector stores, how granular agents should be, and whether fine-tuning was worth pursuing, were answered with specific, actionable direction.
A complete architectural roadmap
The client left with graph-based retrieval design, a granular agent structure, testing infrastructure recommendations, model selection guidance and a phased production deployment strategy with security considerations built in.
Knowledge transfer for the team
Zartis handed over frameworks for designing prompts for determinism versus creativity, deciding when to split or unify agents, debugging failures using perplexity analysis, and applying security considerations to production.
Tooling suggestions to act on
The team received specific tooling guidance: LangSmith for prompt versioning and A/B testing, specialised extraction models for structured data, the OpenAI Agents SDK, and perplexity analysis scripts for diagnostics.
