When to use Rules vs AI algos?
How to break down AI Products (Part 3)
Build simply. That’s the most important lesson I’ve learned building AI models and products. The core of that? Is learning when to use rules vs. AI for a product.
In today’s article I’ll explain:
Why rules vs AI algos important
When to use rules vs. AI algorithms
Using rules and AI algos together.
This is the final part of a 3 series for breaking down AI product requirements.
My disclaimer: As with all articles, I am giving a high-level framework - the actual workflows may be different depending on use case, data, or organization.
1. Why is rules vs AI algos important?
As a lead data scientist, I tend to handle the building. While the product manager (client or in house) builds out the requirements or product vision.
But we both need to agree on the alignment of technical strategy and product goals. Rules vs. AI algorithms affect development time, profitability, and existing needs.
There are several reasons:
Development Time: Sometimes it's faster to build systems using simple rules. Other times, algorithms are better for handling large amounts of data. We need to choose the right approach to save time and effort.
Integration into Existing Systems: Simple rules (heuristics) are easier to integrate into existing systems because they are less complex. AI and ML algorithms can be more complicated to implement and might require significant changes to current systems.
Cost and ROI: Rules-based systems can be cheaper to develop, run, and maintain. For example, they might only need occasional updates. AI systems might need constant data input and retraining, which can be expensive. Lower costs mean higher profits.
Higher impact is deciding between rules and an AI algo for your product. Or if you should pursue an AI product. So, you need to think about rules vs. algo when building it.
But how do we know to create a rules-based product? And when do we know when to use AI?
2. When to use rules and when to use algorithms.
Rules versus algorithms you’ll start to face as you build an AI product. In my discussions with my product manager, we often debate this.
A simple set of rules (heuristics!) is a good start. But as a system matures, adding new heuristics can get frustratingly conflicting and complex. Managing these can be pain, and even harder to root cause. Nobody likes digging through all that.
Here’s a chart that highlights this difficulty:
Plus, as our data pool grows, our machine learning model begins to outperform the basic heuristics. So, we need a system that’s easier to maintain - an ML algorithm or AI model.
#1. When to use rules?
If the current rules are clear and limited, then use them instead of an AI algorithm. This saves time and resources during initial stages or when simplicity is important.
Patterns are repetitive and don’t change much historically. So, rules can capture them. As long as rules don’t get too complex, you can use it as a filter.
For example, recommending specific products to users with distinct traits. Or recommending the top 10 products on our app's homepage. In that case, rules are enough.
#2 When to use AI models or algos?
But when rules are unclear and cannot be fully listed, the product needs to learn some rules and features. So, we need to use algorithms.
Al models capture patterns that we cannot see easily. They analyze large amounts of data and identify relationships and structures that are not obvious to humans.
For example, determining the similarity between two sentences or finding pictures of cats from a set of 1000 images requires the use of algorithms.
Ask first if you’re not sure. Stakeholders have insights into business logic, context, and use cases. Knowing those will help you choose an AI algo vs. rules-based solution.
Check data. Structured data can be easier to use rules than unstructured data. As always, check. I’ve had way too surprises in my time. Never assume you know.
Don’t worry to much if you’re not sure. Start with business logic - then create rules. As use cases evolve, start thinking about AI algos. Many AI models and algos started as rules-based systems. As you advance, you’ll be able to spot this.
Now, let’s look at how to apply rules vs. algo with an example.
3. Case Study: Customer Service AI Agents
Let's look at this case study: AI agents in customer service. We’ll use an online education platform as an example.
Platforms can get spikes in traffic - especially during certain seasons. Customer service teams often get overwhelmed. This a great opportunity to use AI agents - to augment existing work.
Here’s the situation:
User activity is about to double
Customer service team is working at full capacity
Leadership would like an AI agent to handle requests
Relying only on business rules here would be difficult - its too complicated. The types of queries can vary widely - so this a clear case for an AI algorithms to support a product.
#1 What are the business requirements?
When users enter the inquiry page, we use their recent browsing history or orders to make our recommendations. We show activities or questions to match their interests.
Here’s them broken down:
So, we have several clear needs and responses here:
Pre-Inquiry Recommendations:
Recommend recent activities or likely questions.
Relate to the user's recent site behavior or orders.
Question Identification and Response:
Identify and respond to the user's question.
Provide empathetic support if negative emotions are detected.
Transfer to customer service if the AI agent can't answer the question.
Continuous Optimization:
Update the agent's datasets with new questions it can't answer, then retrain the model.
Improve the agent's range of responses.
Generate a report from feedback to monitor performance.
Here’s an example of the flow:
The AI agent here has multiple models for each stage of the customer journey. Its a combination of multiple models into an AI product.
The steps are long, but generally go like this:
Inquiry Page. We want to provide quick access to the user - especially the orders and items the user has viewed. It's also good to add past orders too. Give the user a range of options to help the AI algo behind the agent classify and get context for the need.
Suggestions can go a long way to a good AI product experience. Prompt and guide them . (Source: ViiDigital) Question & Answer Matching. After the user asks questions, recommend likely questions to follow up. They should be the ones they are most likely to click on or ask about. This can help label the data so AI agents can respond better.
Match Questions to Database. If you have previous questions and answers in a knowledge database, match it. If the user shows clear negative sentiment, you need a sentiment recognition step to comfort them. The agent needs to show next steps and empathize. User experience here is critical - you can build a customer connection. I find its a great place to gather product and customer insights.
Sentiment analysis can make a huge difference! Showing empathy matters. Source: Dashly Blog Question Classification. When the AI agent cannot solve the user's problem, we need to classify these questions. Then, we decide whether to add these questions into a knowledge dataset. The dataset can be used to update and retrain the model.
Source: Juji.io
It's then up to the product manager, team leads, and decision makers to decide how to curate these new datasets and dependent models to retrain. Curated datasets are the heart of AI products - they evolve as they address user and busienss needs.
#2 What are areas to add an AI algo?
Simple rules can’t capture all the rules and scenarios with user interaction. A customer reaching the page needs can’t be fully anticipated beforehand.
From analyzing the entire business flow above, we've identified four algorithms in this scenario:
Question Recommendation: When a user starts a conversation with the AI agent, we use recommendation algorithms to suggest relevant questions. You need recommendations that are relevant and context specific. For example, if the user just purchased a course, the algorithm might recommend questions about course content, access issues, or common post-purchase inquiries.
Question & Answer Matching: When a user asks the AI agent a question, an NLP algorithm matches this question with similar predefined question stored in a knowledge database. Once a match is found, the algorithm retrieves, ranks, then serves the most relevant questions to the user.
Sentiment Recognition: Sentiment analysis algorithms detect the user's emotional state. Pretrained data can help in providing empathetic responses tailored to the detected sentiment. If the customer is still unsatisfied, it can determine if it needs to refer to an agent.
Unrecognized Questions: When the AI agent gets many unrecognized questions, it can cluster similar questions by common keywords or topics. Then reference it to user’s history. The initial labels help decide which questions to add to the new training dataset for the AI agent. This process helps set the ground truth.
3. Can we add rules with AI?
Short answer: yes, we can! But the rules in this case aren’t the core of the AI agent. They augment the individual AI models - they can filter the data or responses before we talk to users.
Here’s how rules can enhance each AI algorithm:
Question Recommendation: Rules focus on course content, access issues, or common post-purchase inquiries, narrowing the user's needs. This makes the recommendation algorithm's job easier and more effective.
Question & Answer Matching: Rules match frequently asked questions using keywords or semantic search. They quickly filter out irrelevant questions before passing the refined list to the algorithm, helping the AI model focus on more relevant queries.
Sentiment Recognition: Rules detect and interpret the user's emotional state through their queries. Simple rules identify frustration, urgency, or confusion using keywords and punctuation analysis, aiding the sentiment analysis algorithm.
Unrecognized Questions: Rules group similar questions by keywords or topics like "refund" or "return," providing a starting point for clustering algorithms. They flag complex or frequent unrecognized questions for further review, aiding both customer service and AI retraining.
Notice how rules and algos seem similar? That’s because they’re going for the same goal. Using both have great benefits:
Efficiency: Rules excel at handling routine issues quickly. This frees up AI to focus on more narrow analysis.
Reliability: Rules can pre-filter data and responses, ensuring that the AI receives high-quality, relevant input. This boosts the overall precision and reliability of the AI product.
Scalability: As query volumes grow, rules help manage the load by taking care of simpler cases directly. This enables the AI to scale more effectively. Platform you host AI makes it really work.
Training and Improvement: Rules provide a structured way to collect and categorize data, which can be used to train and refine AI models. This creates a continuous feedback loop that enhances the AI's capabilities over time.
When I first started working with AI, I quickly saw that rules and algorithms each have their own strengths. I realized that data scientists and teams can save a lot of time on product requirements by combining rules with an AI algorithm or model.
By leveraging both, you create a more efficient and effective system. The business cares about the results, not how you achieve them. They want to see improved efficiency, reduced costs, and superior outcomes.
Takeaways
Rules versus AI algorithms isn’t a zero-sum game. In my experience, blending both can really elevate an AI product. By using them together, you can achieve better results and margins for an AI product
It takes a balance of product and tech. Which is highly dependent on industry and use cases. Here’s my key takeaways:
Use rules if you’re not sure. All AI products and features follow existing processes. Starting with AI can be confusing. Instead, ask about the business logic first. The business understands this better. Plus, it helps you get familiar with the logic needed to build an AI model.
Start simple for AI. If you want an AI product that has ROI? Start with a simple AI model and algo. Focus on the business and product needs first. Understand the core processes and goals. Align the AI model with real business value. Once the basic model works, you can build on it to enhance the product further.
Don’t be overly creative. Solve the use case. Find the product need. Solve for that first. Align on the business needs and benchmarks. Create for them. Rules work great here. Especially if you’re trying to sell AI products as next step.
And that’s it. That’s the final part of this entire series. Thanks for reading.
Before then, if you have any questions or comments, feel free to comment or reach out to me on:
Thanks for reading!