AI Product Asks - Part 2
How to break down a strategic AI product request
Hi there, it’s Matt! This is part of a larger series on AI product. If you haven’t read part1 of Breaking Down AI Product Requirements, check it out here.
Time to Read: 7 mins
One of the biggest eye openers during my transition to lead data scientist, was strategic AI product requirements. Starting out as builder doesn’t really prepare you for these. It takes a huge mental shift to handle them.
So, they’re often nuanced and complex - for both technical side (AI architect, lead data scientist/MLE) and the product side (product manager).
Unlike operational requests that I mentioned in the last article, the breakdown and requirements change. Visibility from the organization is critical factor.
In today’s article I’ll:
Highlight how to unpack strategic AI requests
Run a high-level AI strategic demand analysis
Get visibility for a strategic request.
This is the 2nd in 3-part series for breaking down AI product requirements. I’ll be starting this week with the business requests.
My disclaimer: As with all articles, I am giving a high-level framework - the actual workflows may be different depending on use case, data, or organization.
1. How to unpack strategic business requests?
This part refers to a scenario and its requests mentioned in Part 1. I talk about how to classify a request into operational and strategic ones.
Before we break down a strategic request, lets compare it and a operational request that I mentioned before. They’re very different in scope:
Comparing Operational vs. Strategic Requests
Operational AI product requests focus on improving day-to-day functions and workflows. The scope is typically narrow - its either team level or tactical - focused on immediate needs. Its usually department level or below, unless it’s part of a larger strategic initiative.
Examples:
Fixing issues in the AI product that affect user experience.
Augmenting an AI model with new data to improve its recommendations.
Performing enhancing and automating mundane tasks in a team workflow.
Impact: Short-term improvements and efficiency
Your scope is limited, and workflow focused. That’s very different with strategic requests.
Strategic AI product requests focus on: long-term goals, strategic business workflows, and growth. The scope here is broad, and often spans multiple business units. These enhance a business model, core products, etc.
Examples:
Developing new features in the AI product to meet emerging market needs.
Expanding the AI product's capabilities to enter new markets.
Conducting research to innovate and enhance the AI product's competitive edge.
Impact: Long-term value and competitive advantage.
Breaking down a Strategic Request
Now lets a look at a strategic request framework. I’m assuming at this point you’ve already classified a requirement as a strategic AI request. Here’s a graphic overview:
Now, lets break down the three steps:
Step 1: Determine whether this requirement is an AI requirement.
I find strategic requirements boil down to rules versus algo. You’ll be asking constantly if you need rules or to build an AI model. It can be a frustrating process.
But it doesn’t have to be. Some good starter questions to ask:
Can a non-expert easily decide without considering data size and scale?
Are the rules too many and uncertain?
Will the solution capture subtle patterns in the data?
For example, if a recommendation system has too many complex rules, it might not capture customer preferences accurately. In this case, using machine learning might be more effective.
Step 2: Unpack algorithm requirements.
Break out the requirements for algo from your use case discovery. The finer the granularity of the split, the better. Break up technical needs from product needs.
Identify Key Components: Break down the algorithm into its core parts of functionality. For example, a rec system might include data ingestion to collect user data, a ranking algorithm to sort recommendations, and a feedback loop to improve accuracy. Break it into essential parts that enable product functionality.
Specify Data Needs: Outline the data required for each component, including data sources, formats, and volume. This will be the hardest part. Data and business logic lives with different data analysts, engineers, and users. Set time limits on data collection to prevent bottlenecks. Go for good enough.
Figure out Last Mile. Platform and infrastructure support your AI product. Know the environment it will be deployed in. You have to serve an AI product - it doesn’t with the model. Last mile is often the hardest, and sound AI products often fail here.
Step 3: Sort out the algorithm requirements you understand.
Define the input and output of each algorithm. Then set clear business goals and metrics for the algorithms. Tying business metrics and evaluation metrics is a critical step.
Define Inputs and Outputs: Clearly outline what data each algorithm will take as input and what it will produce as output. This includes specifying the format, type, and structure of the data. For example, a recommendation algorithm might take user activity data as input and produce a ranked list of items as output.
Set Business Metrics: Establish specific business metrics that each algorithm will impact, such as conversion rates, customer retention, or revenue growth. These metrics should align with the overall business objectives and product requirements. For instance, a recommendation algorithm might aim to increase average order value or improve customer engagement.
Establish Evaluation Metrics: Determine the metrics that will be used to evaluate the performance of each algorithm. Define thresholds or benchmarks are considered acceptable. For example, in a predictive maintenance algorithm, metrics might include mean time between failures (MTBF) and prediction accuracy.
As workflow patterns become obvious, start creating intake forms and documentation. This helps clear up any confusion between builders and stakeholders.
With that out of the way, let’s talk how to break down a strategic AI request using a demand analysis.
2. Running a Strategic Demand Analysis
We talked about demand analysis before in Part 1, and its four foundations:
Understanding Customer/User Needs
Planning
Pricing
Marketing
Again, I’ll skip pricing and marketing. That’s a whole two articles on its own. I’ll stick to understanding needs and planning.
Let’s do a strategic demand analysis. I’ll break it down using the 3-step requirement framework. This time we’ll be focus on #3 and #4: the AI Generated Promotion Plan and AI Customer service.
A visualization is helpful, and here’s the one from the previous article:
Let’s assume that we have enough information from bottom up and top-down opportunity discovery.
Request #3: AI Generated Promotion Plan
So, let’s start with this request from marketing.
Original Demand: The marketing team wants an AI-driven promotion strategy to match each promotional article with the best audience.
Step 1: Is this an AI Requirement?
Since the goal is to match promotional articles with the best audience based on various data points (like user behavior and preferences), this can't be easily described by simple rules.
A layman would struggle to make these judgments manually due to the sheer amount and scale of data involved. This is definitely an AI requirement where algorithms can help in learning and predicting the best matches.
Step 2 : What are the requests needed for the algorithm?
We need to break down the requirements for the algorithm:
Data Inputs: User profiles, behavior data, past engagement metrics, and content details of the promotional articles.
Algorithm Tasks: Segment users based on behavior, predict user interests, and match articles to user segments.
Output: A ranked list of user segments for each promotional article, indicating the best target audience.
Step 3: What do we need to build and run the AI Algorithm?
We need to define the inputs and outputs clearly:
Input: Historical user data, content data, engagement metrics.
Output: Target audience segments for each promotional article.
Metrics: Accuracy of matches (measured by engagement rates), click-through rates, and conversion rates.
Conclusion: Find out how the business is matching articles currently. This gives you benchmarks and KPIs. Define the business metrics that the AI algorithm must exceed. Also define and source the data used in current benchmarks and KPIs. Consider filtering user behavior using rules, then feeding it to an AI algorithm.
Request #4: Intelligent Customer Service
Original Demand: The customer service team wants an AI system to handle a surge in user inquiries by providing the best answers to common questions.
Step 1: Determine if this is an AI Requirement
Handling a large volume of customer inquiries and providing accurate answers can't be managed efficiently through simple rules. There’s too many varieties in questions, sentiment, and users. An algorithm can learn from past inquiries and responses to give the best answers, making this an AI requirement.
Step 2: Unpacking Algorithm Requirements
Break down the algorithmic needs:
Data Inputs: Historical customer inquiries, responses, FAQ database, and context of the inquiries.
Algorithm Tasks: Classify inquiries, search for the most relevant responses, sentiment, and learn from new inquiries to improve over time.
Output: Suggested responses for each inquiry.
Step 3: Defining Algorithm Requirements
Clearly define what the algorithm should achieve:
Input: Customer inquiry data, response database.
Output: Relevant responses for each inquiry.
Metrics: Response accuracy, customer satisfaction scores, and reduction in response time.
Conclusion: A surge in business inquires might need a platform to handle the data volume from those requests. Assess if the current platform and data ingestion can handle user inquires. Then assess if the infrastructure and data maturity before building. Costs might be high.
This isn’t complete, it needs to be confirmed by decision makers and stakeholders. It’s a good idea to manage, understand, and clarify expectations early. You need to look at an org chart to check.
3. Get visibility
We’ve only scratched the surface of breaking down a strategic AI product requirement. But before you build:
Navigate the organization chart as early as possible.
Context is critical. I’ve seen it saves so much time and costs. It’s very easy for AI architects, team leads, and AI PMs to assume needs. After you’ve broken down requirements:
#1 Find who really needs it.
It’s easy to get stuck in a game of telephone. The requestor isn’t always the person who needs it.
Your AI request is strategic - so you should be aware of chain involved. Let's get a list of key people to find:
Economic Buyer. The decision maker who holds budget decisions. They’re trying to leverage AI to create a profit center. First link in the chain.
Sponsor. A decision maker(s) sponsoring the project. The higher the advocate up the organization chart, the better. Second link in the chain
Advocates. Managers or senior/staff ICs in business units. They’ll be building or using it day to day. Their feedback and support is valuable for iteration. They lend mass to advocates support.
#2 Discover opportunities that exist.
Talk with leaders and managers to understand what the company wants to achieve with AI. This aligns your AI project with the company’s goals.
There’s two types:
Top-Down Opportunity Discovery. Talk with leaders and managers to understand what the company wants to achieve with AI. This aligns your AI project with the company’s goals. It helps you find advocates, which is crucial for getting resources and visibility.
Bottom-Up Opportunity Discovery. Connect with users who will use your AI product. Get to know the builders. You need to uncover who’s building and who’s using it in their workflows. If an AI product is customer facing, get to know ppl who understand their workflows.
Start with opportunity discovery. You can find the deeper needs for a strategic request. What I find useful is it defines data you need. A common pitfall is to start with the data then build. Or worse - ignore the data for an AI solution.
#3 Keep Momentum.
Knowing who uses it, builds it, and uses it to support strategy is half the battle. AI requirements have more impact if you keep momentum. Excitement and alignment is a force multiplier.
Decision Criteria: Determine what standards, conditions, and requirements allow adoption of an AI product. Remember, these requirements and expectations can vary greatly between departments!
Decision Process: Understand the process to get a request built into a breathing AI product. This includes presentations to key stakeholders, detailed testing, and final approval from decision-makers.
Expectations: Clarify what outcomes or improvements everyone anticipates from implementing the tool. Not all expectations are possible. Check these before and after meetings. Its important maintenance.
Front end your work to keep the momentum. For the best results, you need to give time for users, builders, and stakeholders to ask questions, and coordinate work.
Takeaways
Breaking down a strategic AI product request is a lot of work and a major team effort. It’s never the same every time - and there is a ton of ambiguity. There’s data, algorithms, platform, and metrics - and the infrastructure to support all that.
It’s not easy, and I’m still learning. I’ll probably still be learning 5 years from now as our field shifts. Here’s my key takeaways:
Clarify ambiguity early. Strategic AI product requests can be ambiguous. As you unpack a request, get clarity around the desired business outcome. This can radically change data and algo product requirements. Path dependency happens when you don’t clarify it.
Strategic demand analysis needs advocates. It's hard to clarify a strategic AI request without support. Find advocates who can connect you to groups that provide valuable insights. They help you find the right stakeholders. These stakeholders refine the goals, metrics, and business alignment of the project. Align and make requests relevant.
Keep up momentum. Momentum keeps teams talking to you about a strategic request. Top of mind requests helps you develop great AI products and features. Strategic requests aren’t one and done. They need constant support, alignment, and constant familiarity with needs as they change. Keep your finger on the pulse, and keep the enthusiasm going.
In the next article, I’ll talk about the when to use rules versus an AI algo in your product. AI products augment or enhance - so its very critical to know.
Before then, if you have any questions or comments, feel free to comment or reach out to me on:
In the next article, I’ll talk about how to break down an AI product strategic requests. It’s going to be fun!
Thanks for reading!
Really liked your breakdown of building AI product requirements, a lot of these concepts align with my and my team's learning for a health tech product. Also, a few of your points would have saved us a fair few headaches too!