Introduction

In the world of Product Management, the one question that probably bothers the Product Leaders the most (apart from probably a laundry list of hundreds of other questions!) is; ‘Should we build the product or not?’. As fundamental as it sounds, finding the answer to this question is not just important but actually at the very core of its (the product) existence. Be it a ‘Cool’ new product idea or an ‘Amazing’ feature for the existing product; making that decision is as multifaceted as it is critical for business success. And, to make it worse there’s no ‘thumb rule’, there’s no ‘one-size-fits-all’ mantra and there’s definitely no ‘magic wand’.

You have the risk of ‘flying blind’ knowing fully well that the road ahead might head straight into a dead wall! However, by following a structured approach we might just increase the chance of us ‘hitting the brake’ right in that ‘knocking on the heavens door' moment, the proverbial ‘moment of truth’. I will not be surprised if you are thinking right now, ‘what happens if the brake fails?’. Well, great thought, and all I can say is that, ‘brace for impact’ and most importantly ‘pray’!

I, over the course of my career have witnessed this process a good number of times. And, seeing the action unfold both from a ‘top of the hill’ and ‘up close and personal’ perspectives, it was but evident that,

‘Bigger the problem to be solved means bigger is the build objective and bigger the build objective means bigger is the risk of failure’.

Hence, conducting a thorough analysis when the stakes are high becomes absolutely crucial. In this blog, I’ve tried to summarize and bring forth a structured approach to conducting both qualitative and quantitative analysis for making the all-important ‘build’ or ‘no-build’ decisions. However, this by no means is a prescriptive method rather a ‘practical go-to-approach’ that one can use ‘in the time of need’ as a viable ‘point of reference’.

And, it is to be noted that the approach mentioned below has been structured largely keeping a B2B software product in mind. However, many of these steps and the underlying activities could very well be curated to fit into a B2C product scenario as well. You are advised to use your discretion and apply it as you deem fit.

What are the specific reasons for conducting a thorough analysis?

Let us take a look at a few specific reasons (not conclusive) for conducting thorough qualitative and quantitative analysis both for new and existing products.

A. Qualitative

i) Understanding User Needs and Pain Points – New Product

Qualitative analysis helps in gathering in-depth insights into the challenges and pain points faced by potential customers or users. This knowledge is critical for identifying the specific features or products that would address these pain points effectively.

Say for an example, in a B2B accounting software, qualitative analysis might reveal that users struggle with complex reporting. This insight can inform the decision to build a robust reporting module.

ii) Making Improvement Decisions - Existing Product

Qualitative analysis allows you to gather feedback directly from users regarding their experiences with the existing product. This provides valuable insights into what is working well and what needs improvement.

For example, through user interviews, it may come to light that users find a certain workflow in the software cumbersome and time-consuming.

B. Quantitative

i) Market Validation – New Product

Quantitative analysis on the other hand provides data-driven evidence regarding the potential demand for a new feature or product. It helps in assessing the market size, user preferences, and the likely adoption rate.

For example, conducting surveys or analyzing market research data may show that there is a high demand for integration with a popular CRM platform.

ii) Data-Driven Insights – Existing Product

Quantitative analysis provides objective data on how users are interacting with the existing product. This includes metrics like user engagement, conversion rates, and feature usage patterns.

For example, analytics may show that a specific feature is rarely used, indicating a potential opportunity for improvement or removal.

C. Both – Qualitative and Quantitative

i) Risk Mitigation

Both types of analysis are necessary for risk assessment. Qualitative analysis can uncover potential usability issues or user experience challenges, while quantitative analysis can provide statistical evidence about market trends and potential adoption rates.

For example, Qualitative feedback may reveal that users find a proposed feature confusing. Quantitative analysis can then confirm if a majority of users share this sentiment.

ii) Competitive Benchmarking

A combination of both types of analysis is necessary for comparing the existing product with competitors. Qualitative insights can uncover strengths and weaknesses not captured by quantitative metrics alone.

For example, while quantitative data may indicate that your product has a higher user retention rate, qualitative analysis might reveal that competitors offer a more intuitive user interface.

iii) Prioritization of Enhancements

Both types of analysis play a crucial role in prioritizing which improvements should be addressed first. Qualitative insights can highlight critical pain points, while quantitative data can help assess the overall impact.

For example, Qualitative feedback may emphasize the need for a specific feature, while quantitative analysis can estimate the potential increase in user engagement if it's implemented.

As we can see, qualitative and quantitative analysis work in tandem to provide a holistic understanding of user needs, market demand, and areas for improvement. This comprehensive approach helps in making informed decisions about building new features or products, as well as refining existing offerings to better serve the user base.

Now, let us delve a little deeper and put all the fragments together to frame a structured approach around conducting qualitative and quantitative analysis for new and existing products that is ‘usable’, ‘easy to follow’ and can provide a ‘practical guard rail’.

Building a Structured Approach to Conducting Qualitative and Quantitative Analysis

As we have seen in the earlier section that conducting a comprehensive analysis for product specifications, features, and improvements involves a combination of qualitative and quantitative methods. It is to be noted that elaborate explanations of the specific activities have been kept to minimal as it is beyond the scope of this blog article; readers are requested to do further study to know more about any of those activities mentioned below.

Stage 1: Define Objectives and Scope

New Products

Activities

o Define Goals: Clearly state what you aim to achieve with the product analysis (e.g., user likeability, user pains, user adoption etc.). This is very important as the whole analysis rests on this definition. Try to frame the goals around questions; I found that approach to be more effective. And, I always like to start with knowing the ‘Why?’ then everything else.

o Market Research: Analyze market trends, customer purchase behavior, explicit and implied customer needs, and competitor offerings to identify gaps and opportunities in the market.

o Concept Testing: Gather feedback on initial product ideas and concepts from potential users or target audience. Starting with internal stakeholders to gauge the response is a good idea if the product is a novel concept; however, going to external stakeholders might give us an outside-in perspective to gauge if the concept stands a chance at making it big.

o Define Minimum Viable Product (MVP): Determine the core features and functionality necessary for the initial product or MVP release. MVP is a saleable product; it just has lesser features compared to the full version. Hence, and MVP has to portray the best of what we intend to offer; because first impression could be the last impression in this case. And, we should try and make a good impression.

Do's and Don'ts

o Do: Focus on understanding the unique value proposition your new product brings to the market. The product has to solve something and then give something more. Differentiation and positioning are the key here.

o Don't: Overcomplicate the initial concept with too many features. Start with the basics and build from there. Not focusing on usability is a big mistake; because Utility is important but Usability is what makes the product click with the audience. Simplicity is not ‘over hyped’; it indeed can be a powerful way to make the product shine.

Should not shy away from getting that ‘first love’, getting a strong early push to build a strong momentum later.

Existing Products

Activities

o Stakeholder Interviews: Engage with existing customers, internal teams, and gather input from key stakeholders to understand their expectations and requirements. This can be done as 1-o-1 or in groups. But ensure that the stakeholders shortlisted from the interview are notified early to ensure they understand the expectations and can prepare in advance. Usually stakeholder interviews could be 1 to 2-hour long.

o Competitive Analysis: Evaluate how your existing product compares to competitors and identify areas for improvement. There are two ways to look at it; do what competitors do but do it better; or do what they don’t but in a way that they probably can’t. Hence, either you ‘copy better’ or ‘innovate smarter’.

The point is, we should always try to be either ‘at par’ with our competitors or ‘ahead’ of them but never behind, because that seriously impacts our ‘market relevance’ and ‘desirability’.

o Review Historical Data: Analyze customer feedback, support tickets, and user behavior data to identify recurring issues or opportunities. This helps us understand our past performance so that we can consider making appropriate course corrections for the future. There is a wealth of information hidden in plain sight and we must leverage that to our advantage.

Do's and Don'ts

o Do: Leverage existing customer feedback to inform potential enhancements or feature additions. Ascertain if the really wants what he says he does or is it something else, maybe he is seeing it differently. May be there’s a way we can give him more; we can help him ‘get things done’. Charge up the ‘empathy quotient’ and try to look at it from the customers’ perspective, get your ‘hands dirty’ if need be but keep looking until something as bright as a ‘sunny day’ come out. Trust me when I say this, Customers love it more when we show our love for what they do; and, when we sincerely show that ‘we care’.

o Don't: Assume that the current product is perfect. There is always room for improvement, there is always something more that the customer might need, there are always different ways to ‘wow’ the customer. Here, in this context ‘ignorance is certainly not a bliss’.

The objective might be to identify such features which will enhance user experience and maintain a competitive edge and leadership position in the market.

Stage 2: Qualitative Analysis

New Products

Activities

o User Surveys and Interviews: Gather feedback from potential customers regarding their preferences, pain points, and desired features. From the background color to the placement of a button and from the number of clicks needed to perform an action to the difficult to read fonts; everything is important. User responses are powerful for us to drive the usability factor of our product. The idea must be to make the users ‘fall in love with our product’. (Note: this activity can also be performed for existing product enhancement scenarios.)

o User Persona Development: Create fictional characters that represent your target audience and conduct interviews or surveys to understand their preferences. Developing the user persona can be as detailed as needed to truly get the character to shine. As relatable and real world the persona gets then better it is.

o User Story Mapping: Map out the user's journey from discovery to purchase, identifying pain points and opportunities for improvement. Story mapping can consider user frustrations or difficulties in completing an action for e.g. placing the order or making the payment etc. And, then we need to go to the root of it and find and fix the problem to ensure that the user journey from start to end is seamless and enjoyable.

Do's and Don'ts

o Do: Be open to pivoting based on early user feedback to ensure the product meets market needs. Build small, test and pivot. The idea is to give something that customer sees as ‘valuable’ and will thus not be so willing to ‘replace’ easily.

o Don't: Rely solely on internal assumptions about user behavior. Assumptions of any type could be derailing unless tested out. We should not try to ‘generalize’ certain ‘human behavior’ rather look at ways to find a ‘reasonable pattern’ so that an appropriate strategy could be devised.

Existing Products

Activities

o Usability Testing: Observe how users interact with the product and their reactions (confusion, frustration etc.) to identify areas of improvement for a better user experience. Good user experience is a careful blend of utility plus usability.

o Heuristic Evaluation: Evaluate the product based on established usability principles (e.g., Nielsen's 10 Usability Heuristics). This helps to identify usability problems in the user interface design evaluating against a set of guidelines. In simple words, it is a way to determine whether our product is ‘user friendly’ or not.

o Feedback Loops: Establish mechanisms for ongoing customer feedback through surveys, support channels, or user communities. We should maintain a constant connect with the customer at all time through multiple channels so that the customer can reach to us in the event of a ‘difficulty’. Because in the modern digital World that we live in, it is better for the customer to talk to us directly and early about his problem than use the social media for ‘solution mining’ or even worse ‘causing bad PR’.

Do's and Don'ts

o Do: Engage with long-time users who are familiar with the product to gain deep insights into its strengths and weaknesses. Idea is convert these ‘users’ into ‘ambassadors’. Their ‘familiarity’ can be our ‘strength’ if we do it right.

o Don't: Ignore user feedback, especially if it consistently points to specific pain points. This under no situation is a good practice. User feedbacks are ‘gold mine’ of useful information that are waiting to be ‘unearthed, understood and acted upon’.

Qualitative analysis might involve gathering feedback from hosts and guests on their experiences, preferences, and pain points.

Stage 3: Quantitative Analysis

New Products

Activities

o Market Sizing: Estimate the potential market size and customer base for your new product category. Market size typically is the total number of potential buyer for our product in a given market. Which also means, the potential total revenue that may come from it. We can do this in two ways ‘Top Down’ or ‘Bottom Up’. We can calculate the TAM (Total Addressable Market), SAM (Serviceable Available Market) & SOM (Serviceable Obtainable Market); where in, SAM is a subset of TAM and SOM is a subset of SAM. To get an overall feel of the size of the market we can also use two simple formulas such as,

· For number of potential users: {Target Market x Penetration Rate = Market Size}

· For potential revenue estimate: {Market size x Average Value = Market Value}

o Landing Page Testing: Run tests on landing pages or concept pages or apps pages to gauge interest and conversion rates. This helps in optimizing the page and make small changes instead of changing the whole page in one go. The focus for this must be on call-to-action (CTA) and whether the user is able to complete the desired action smoothly or not. Based on that necessary optimization can be implemented.

o A/B Testing: Conduct experiments to compare different versions of a page, content, design, feature etc. to see which one performs better that the other. We can create two or more versions of the design and run this test against a set of potential customers or stakeholders where the 100% of the traffic can be split into all the versions to run this test. Reason why, this test is also known as Split testing or bucket testing.

o Surveys: We can use objective type questionnaire, pools and surveys to collect specific data from the customer. Using Likert scales we can quantify user preferences, opinions, behaviors and satisfaction. Since the user responses are captured in continuum (Lowest to Highest) using Likert scale can give a more nuanced understanding than a generic objective type questionnaire with just yes or no answers.

Do's and Don'ts

o Do: Leverage online analytics tools to gather insights about your target audience's online behavior. There are several tools out there in the market which can with this. Recommending any particular tool is outside the scope of this article since it requires further assessment and fitment depending upon the exact nature of the requirement and the desired objective.

o Don't: Make assumptions about market demand without data to support them. Depending on untested assumptions is highly risky as it leaves a large part it ‘hanging in the balance’. Every assumption must be tested and data-driven insight must be drawn to bring it a conclusion.

Existing Products

Activities

o User Analytics: Analyze user behavior data and monitor metrics like (e.g., session duration, click-through rates, user retention, feature adoption rates etc.) using tools like Google Analytics or Mixpanel (these tools are stated here as mere examples and not as recommendations).

o Churn Analysis: Identify reasons for customer churn and use this data to inform feature improvements. Customer churn can prove to be a huge drainer for all things done well. It is essential to observe user engagement, overall time spent by the user over a given period, activities performed, purchases (if any) made etc. Customer churn can be calculated using a simple formula;

{Lost customers / Total customers at the start of time period x 100}

Churn can also be calculated for revenue lost known as Revenue churn.

Do's and Don'ts

o Do: Segment your user base for more granular insights, as different user groups may have distinct needs. The better we understand the different needs the better we can frame the solution strategy around those needs.

o Don't: Focus solely on high-level metrics. Dig deeper to understand the underlying trends and behaviors. Micro-level view is essential for us to get to the root and decipher deeper meaning than just ‘scratching the surface’.

Quantitative analysis could involve tracking user engagement metrics like time spent on the platform, content preferences, and user ratings.

Stage 4: Competitive Analysis

New Products

Activities

o Market Mapping: Create a visual representation of existing products in your space, highlighting their key features and value propositions. Market mapping is a useful approach to identify our competitors in a given market by collective details regarding their revenue, pricing, products, number of customers, growth rate, industries they serve etc. and then creating a graph or a plot using that data for all the competitors to visually analyze the data. It gives out the relative positioning of respective competitors and their products and helps us devise an appropriate strategic direction. Market mapping can effective reveal existing gaps in the market proving it to be tool to showcase potential new business opportunities.

o Differentiation Analysis: Identify areas where your new product can offer something unique or superior. The word ‘differentiation’ by definition signifies that there are other companies offering similar products or solutions or features. This analysis helps us ascertain ways that our product or feature or offering can be unique and different from other competitors in the market but at the same time being unique or standing out is just not good enough as we also need to ensure that we deliver real value to our target customers. We can create differentiation is many different ways such as through design, features, support, pricing, value added offerings etc.

o Benchmarking: Similar to the differentiation analysis we can also use benchmarking method to evaluate our product's performance against industry standards and peers i.e. competitors. This is a continuous process and can be performed throughout the product lifecycle for accelerating innovations and continuous improvement and creating a strong product adoption and performance. Benchmarking can be essentially performed in two ways; external (compare against competitors) and internal (analyze based on historical product data) benchmarking. This method is equally applicable and effective for both new and existing products.

Do's and Don'ts

o Do: Look for underserved niches or opportunities for disruption within the market. A word of caution though, disruption is a tricky place to be in and disruption without an innovation mindset may not yield the desired result.

o Don't: Simply replicate what competitors are doing. Seek ways to provide added value. As mentioned earlier in this article, being a ‘copycat’ is never a good idea. We should not ignore our very own ‘Why’, and position ourselves in a way that our strategic goal is not honored.

Existing Products

Activities

o Feature Gap Analysis: Compare your product's feature set with direct competitors to identify areas for improvement. This is undoubtedly one of my preferred analytical methods as the benefits of this are multifold. Through this analysis we not only get to know who are real competitors are but also what they offer, we get to compare our features head-to-head with that of our competitors to identify where we lack and what we need to improve. This analysis provides an efficient way for us to come out of our slumber to be introspective about our own product. It helps us taking a critical look at what we offer to understand what do we need to develop more so that we can cover those gaps. This gives us an opportunity to build what our customers truly wants which in turn increases our relevancy and value in the mind of customer.

o SWOT Analysis: This needs no introduction; one of the oldest trick in the business world. Through this we can assess our product's strengths, weaknesses, opportunities, and threats in comparison to competitors. Continuously update your SWOT analysis to stay aware of evolving market dynamics. We need to be mindful that strengths and weakness are internal aspects hence should look at the business from inside-out perspective while the opportunities and threats are external aspects and must look at the outside-in perspectives.

Do's and Don'ts

o Do: Be aware of emerging technologies and trends that could impact your product's competitiveness. Changing technology landscape could be a boon or a deterrent. We must be vigil and watchful at all times.

o Don't: Neglect the importance of regular competitive analysis, as the market is constantly evolving. A strong competitor can make us stronger, we shouldn’t miss considering this as an opportunity to constantly improve what we do to the point where competition can no longer or easily strike a dent into our shield. But we must not forget the fact that there can always be a new tough competitor rising tomorrow.

Competitive analysis could involve analyzing aspects like functionality, pricing, strengths, weaknesses, value propositions or USPs and specific capabilities etc.

Stage 5: Prioritize and Validate Findings

New Products

Activities

o Prototype Testing: Create low-fidelity prototypes to validate proposed changes with users. A good product or feature or design must always be ‘at home’ with ‘customer needs and expectations’ by continuously being built on ‘customer feedback’. Testing a prototype early gives us an advantage of rapid learning and early improvement. This testing must be done before development and can be done after launch as well to ensure that we constantly keep working on improving our product to satisfy customer needs.

o MVP Testing: Develop and launch a Minimum Viable Product to gather real-world user feedback. And MVP can be tested in multiple ways such as through a real product which has the most important features in a very simplistic but fully functioning condition, or it can be done through explainer videos or landing pages or A/B testing or concierge MVP etc. MVP testing can lead to several benefits such as validating demand, securing buy-in, testing design and usability, build monetization strategy etc.

o Lean Canvas: Use a Lean Canvas or similar framework to prioritize features based on their impact on the core value proposition. The primary objective of the lean canvas is to derive a holistic look at the concept or the idea so as to make it more actionable and doable. The focus is on creating an efficient process that can save resources and reduce any form of wastage by optimizing the nuances of the idea at every stage of the idea evolution.

Do's and Don'ts

o Do: Focus on the core value proposition and prioritize features that directly contribute to it. It’s simple, build those first that give the highest value. Everything else can wait. Every single dollar that we invest in building a product or a feature must give a return in the form of real value for the customer which would in turn help us get returns in the form of increased adoption, increased revenue, better ratings etc.

o Don't: Rush to add every suggested feature. Start with the essentials and build from there. Let us understand that when we try to satisfy everyone, we satisfy no one. Hence, when we try to offer everything under the sun we hugely compromise on the user experience and value proposition.

I tend to look at it this way, if we offer just one feature in our product that we are building then it has to be ‘the ultimate pinnacle of a feature’ that has the potential to ‘wow the customer’ and the one that can make them ‘come back again’.

Existing Products

Activities

o Impact-Feasibility Matrix: Use a prioritization matrix to assess the potential impact and feasibility of proposed improvements. This analysis is done using two primary aspects i.e., Impact and Feasibility. Impact signifies the value that can be created and Feasibility denotes how easy is it to build and test it. This is a hypotheses testing framework which is spilt into four quadrants representing the following combinations;

• High Impact + High Feasibility: 1st priority since value is high and these are easy to build

• High Impact + Low Feasibility: 2nd priority since value is still high although a little challenging to build

• Low Impact + High Feasibility: 3rd priority since the value is low but it is easy to build

• Low Impact + Low Feasibility: Lowest priority since value is low and building is hard

o Cost-Benefit Analysis: Cost-Benefit analysis is one of the most popular business analysis methods that essentially compares the projected or estimated costs as against the benefits or opportunities associated with an idea or concept. To ascertain whether or not it makes business sense to build something we can estimate the costs associated with building it and compare them with the expected benefits that we anticipate we would derive by building it. The following two scenarios will emerge from this;

• Benefits outweighing the Cost: The concept can prove to be beneficial if built; this supports going ahead with the build if there are no business risks identified and validated

• Cost outweighing the Benefit: Rethink and further evaluate; ascertain if there’s any other way the benefit could be increased; if not pivot as needed or drop the idea altogether

o Customer Satisfaction Surveys: Customer Satisfaction or CSAT surveys are extremely important to gather customer feedback to validate the impact of proposed changes with the existing user base. Customer satisfaction is like the ‘prize money’ that a we strive to earn. We want our customers to happy and satisfied with our products and offerings yet a significant amount of attempts that we make fail to yield any progressive result at all.

Do's and Don'ts

o Do: We should consider the long-term impact of proposed changes on the overall user experience.

o Don't: We should not disregard the potential technical complexities or resource requirements of proposed improvements.

Prioritize features based on user feedback, usability testing results, and the potential impact on user satisfaction.

Stage 6: Iterate and Refine

New Products

Activities

o Agile Development: Adopt an agile development approach to make small and manageable incremental improvements to maintain flexibility based on user responses, quickly incorporate user feedback and adapt to changing market dynamics.

o Pivot or Persevere: Continuously gather feedback and be willing to pivot if user feedback or market conditions suggest a change in direction is necessary.

Do's and Don'ts

o Do: We must stay nimble and responsive to user feedback, even if it means making significant adjustments.

o Don't: We should not be overly attached to initial ideas if they don't resonate with the market. We must be ready to pivot or drop the idea at any point if need be.

Existing Products

Activities

o Regression Testing: Ensure that new features or changes do not negatively impact existing functionality. Thoroughly test every new feature or improvement so that they don't introduce unintended issues.

o Version Control: Implement version control practices to track changes and easily roll back if needed. Version control helps us keep things in check and step in whenever calamity strikes to resolve or restore as needed.

Do's and Don'ts

o Do: Continuously monitor the impact of changes on key metrics to ensure they are delivering the intended benefits.

o Don't: Rush to implement changes without proper testing and validation. This is extremely important that we follow a diligent approach before implementing any change to ensure we are aligned with the objective at all time.

It is a good practice to regularly gather feedback and update the product based on user feedback, making small iterative changes and observing the impact on user engagement.

Conclusion

Both qualitative and quantitative analysis are extremely crucial for finding product market fit and achieving long-term product success. The framework outlined in this blog and the different stages and relevant activities described for each stage would depend largely upon the industry, product, target market, audience profile and resource availability. However, there seems to be a common baseline of sort. And, thus many of these could be used regardless of the industry and the domain as such. Hence, it is up to us to understand the actual requirement and then decide on the viable plan of action. Having said that this blog could very well be used as a reference tool for the planning purpose as it captures some of the most vital aspects of this exercise (qualitative and quantitative analysis). The activities shown in each stage could also be curated to suit the unique circumstances under consideration for getting the desired outcome.

- End

Disclaimer: The above article is based on personal understanding and views of the author. The concepts discussed are solely for informational purposes and should not be considered as professional advice or guidance. The author does not take responsibility for any positive or negative impact resulting from the application of these concepts or ideas. Readers are advised to exercise their own judgment and discretion when implementing any information provided in this article. The author recommends seeking professional advice or conducting further research to verify and validate any concepts or ideas discussed. The author shall not be held liable for any consequences or damages arising from the use of the information presented in this article.