AI Workshop case study
Aligning AI Strategy for Multi-Agent System Design
Discover how our experts helped design an AI Agent to automate 40% of support queries and prepare for voice & image capabilities
About the client
Our client is a leading technology provider in the salon and spa industry, offering a comprehensive platform that helps businesses manage appointments, point-of-sale, marketing, and client relationships. They are actively exploring the integration of AI and Large Language Models (LLMs) to enhance their service offerings and improve operational efficiency for their salon owner clients.
The goal
Their primary goal was to develop a multi-agent AI system that could interact with salon customers like a human staff member, handling tasks such as rescheduling, canceling, and suggesting appointments. They also aimed for the AI to provide personalized recommendations for services and staff, based on customer history and needs.
About the project
The challenges
The client sought to develop a multi-agent AI system to handle customer interactions, aiming to replicate the personalized experience of a human staff member. Their primary challenges included:
Customer interaction
Creating an AI capable of basic tasks (rescheduling, cancellation) while also providing personalized recommendations for services and staff, especially for new customers.
Branding and personalization
Ensuring the AI agent could be customized by salon owners to reflect their unique brand voice and service knowledge.
Overcoming fine-tuning issues
Previous attempts at fine-tuning LLMs yielded unsatisfactory results, leading to a preference for prompt engineering.
Intent detection accuracy
The initial agent responsible for identifying user intent occasionally misidentified complex or nuanced requests, leading to inefficient conversational loops.
Reasoning over large datasets
Difficulty in enabling the AI to accurately reason about the best staff and service from a vast and diverse dataset of salon services and staff, with initial vector store approaches not providing consistent results.
Authentication complexities
Inconsistent authentication flows within the multi-agent architecture.
About the project
The solution
Our team provided a strategic workshop and consultation to align the client’s AI strategy, focusing on best practices for LLM implementation, agent architecture, and development methodologies.
Multi-Agent system architecture
Affirming the client's approach of using a multi-agent system (customer-facing and onboarding agents) but emphasizing the importance of granular agent design for increased determinism and reliability.
Prompt engineering emphasis
Advocating for semantically rich and detailed prompt engineering over extensive fine-tuning for predictability and efficiency, using existing chat data for conversational style customization.
Robust testing and observability
Stressing the critical role of automated testing, continuous delivery pipelines, and integrating evaluation metrics (like perplexity) at every step of the agent workflow to identify failure points. Tools like Langsmith were recommended for versioning and comparison.
Advanced information retrieval
Recommending graph-based retrieval as a more reliable alternative to semantic similarity in vector stores for reasoning over complex and diverse service/staff data, especially for intelligent matching of new customers with services and stylists.
LLM selection and optimization
Discussing the benefits of using appropriate LLMs for specific tasks (e.g., GPT-4 Turbo for reliability, specialized extraction models like New Extract 1.5 or Osmosis for ID extraction) and exploring the OpenAI Agents SDK for higher-level agent abstraction.
Structured workflows
Promoting the use of structured workflows with routers and defined control flow to enhance predictability and break down complex tasks into smaller, more manageable steps.
Context management
Implementing a summary agent to manage long conversations and maintain context for future interactions.
Authentication optimization
Discussing potential improvements in authentication flows, including intent-based authentication.
Security measures
Discussing strategies for prompt injection testing (e.g., red teaming) and the implementation of moderator nodes.
The outcomes
While this engagement was a strategic workshop and not a direct implementation, the immediate outcomes and expected results for the client based on the recommendations include:
Strategic Alignment
The workshop provided clear alignment on the client's AI and LLM implementation goals, validating their existing multi-agent approach while offering concrete avenues for improvement.
Enhanced AI Performance
The recommendations for graph-based retrieval and granular agent design are expected to significantly improve the accuracy of service and staff recommendations, especially for new customers, leading to a more intelligent and satisfying customer experience.
Improved Development Efficiency
Emphasizing automated testing, continuous delivery, and prompt engineering best practices will lead to a more robust and predictable LLM development lifecycle, reducing debugging time and improving model reliability.
Cost Optimization
The advice on using smaller, specialized models for specific tasks and prioritizing prompt engineering over large-scale fine-tuning could lead to more efficient resource utilization and potentially lower token costs.
Increased Confidence in AI Deployment
Addressing security considerations and establishing clear metrics for success will instill greater confidence in deploying the AI agent to production environments.
Actionable Next Steps
The client committed to providing detailed workflow flowcharts, prompts, and metrics for further analysis, setting a clear path for ongoing collaboration and refinement of their AI strategy.