One of the most difficult decisions product managers make is how to measure customer success. There are myriad different ways, but ultimately the success of your digital initiative is at stake. We recently did this on one of our key projects for an industry-leading non-profit – and hopefully sharing our decision making process and results can be helpful in your endeavors. In this article I will dive into what we decided to measure (and what not to measure), how we adapted Google’s “HEART” framework to our specific tools and needs, and specifics on how we measured as well as our learnings.
NEW RESEARCH: LEARN HOW DECISION-MAKERS ARE PRIORITIZING DIGITAL INITIATIVES IN 2024.
Background
We partnered with our client, one of the leading US nonprofits, to create a rich ecosystem serving American consumers over age 40 on their financial journey, with the objective of creating positive, impactful, enduring behavioral change in their financial lives. This platform provides tailored information, resources and tools driving consumers to take specific actions that will improve their financial well-being. So far, we have collaboratively created two solutions that focus on helping customers deal with and recover from a financial shock — such as unexpected expenses — and manage their debts, especially if they are having trouble making all of their payments. A key part of driving successful outcomes is to set Key Performance Indicators (KPIs), or goals we can track with real numbers. A key activity, therefore, was finding the right KPIs that would align with these objectives and find a way to measure those effectively.
What We Won’t Measure
We wanted to make sure to focus on metrics that are specific and interpretable, actionable (or helpful for other purposes, such as marketing) and ultimately measurable using the tools we have in hand.
As one specific example, while “time on-site” is a very common metric, it’s meaning isn’t often clear. Let’s say we determine that spending 3 minutes on a specific page is a long time (which by itself can be a feat), it’s not clear whether it’s good or bad: was the user. finding our information super interesting, or are they confused on how and what to do, and spending time trying to figure it out? Similarly, while we have a lot of interest in aggregate customer numbers, especially as indicators of changing behavior (for example for the debt app, how much debt customers carry and how it decreases over time), our current analytics and database tool sets require significant development work to measure these effectively, so we opted out from measuring it in the short term, and can implement better tools to do so later on.
What Are We Measuring and Why?
Our immediate goal for these newly developed tools was to influence each customer’s financial situation and habits in a positive way, but not necessarily in acquisition or retention numbers. Ultimately, creating the best experience that solves customer’s problems in an impactful way can serve both of these goals. So, we focused on measuring the app’s impact in a meaningful, actionable, and feasible way. The next step: find specific drivers to rally our initiatives around and measure success of the app overall, as well as evaluate, prioritize, and measure success of features and user tests moving forward.
Our two tools are different enough that each requires its own measures. We focused on two large categories for each as a guide for choosing our metrics: First is customer progress through the app, as without successful user engagement and using the tool extensively, our impact would be limited. This includes on-boarding and major user flows’ success rates, and engagement with the tool’s key pages and key tasks completion. Second is the tool’s usefulness to the user, as this is our ultimate goal once they access the app. This includes immediate on-site indicators (like surveys and registrations) and whether our customers find the app useful enough to return. Having decided on these key metrics, themes, and top metrics within these themes, we then applied Google’s HEART framework to deepen our understanding of a customer’s success with the app.
Applying the “HEART” Framework to Measure Customer Success
HEART is a user-centered and data-driven framework that was created and tested by Google on their own products. Metrics are organized around five themes (thus spelling the acronym): Happiness, Engagement, Adoption, Retention and Task Completion. It helps identify main user behaviors we are looking for in each of the themes, then looks for signals that these desired behaviors are taking place, and finally helps determine the most efficient metrics to measure these signals. HEART framework allows us to really focus on customer experience as we evaluate success of the app.
The image below shows how we applied this framework to the debt tool, and specifically to the H (Happiness) theme. The following section goes deeper into each one of the themes and goals, signals and metrics for each one of them.
HEART: Goals, Signals and Key Metrics
While there are dozens of metrics that can be measured for each of the signals, we narrowed them down to the most meaningful, measurable and feasible, for focus and efficiency when applied to the Debt tool.
(H) Happiness: We decided that the best indication of customer’s happiness with the tools is them indicating it directly as well as actually following the recommendations stated in the app.
We used a survey to ask customers directly whether they found the tool useful. Surveys marked ‘Very Happy’ indicate ultimate customer satisfaction. Additional questions captured more information, including great ideas about potential improvements from customers. We also collected customer information for follow-up to gain even more insights into the customer’s journey and opportunities for improvement
We also used some secondary metrics such as ‘% of customers who emailed themselves the assessment results’. Once we are able to track customer’s data from the database side instead of just using Google Analytics, we can track customer’s success in paying off their debt by actual dollars. We are working on the “goals” feature which will help customers track their progress (staying on track with creditors) within the app, and once that’s developed, overall goal completion can be tracked as one of the measurements of happiness and success
(E) Engagement: Our customer goals are centered around customer’s usage and engagement with the main tool’s features – primarily the main ‘action plan page’ for both features – as well as customers taking desired actions. Such actions include customers calling their creditors when the plan recommends to, marking certain actions as ‘complete’, or taking some actions that help their debt situation to ultimately move them into a better financial state. This could happen when someone who didn’t have enough funds for the tool to allocate the payments is now able to do so after they called their creditors to negotiate their debts.
(A) Adoption: Adoption is important because before a customer can really experience our tools’ value, they need to go through on-boarding. We measured on-boarding from the main landing page in each tool, and through the tool’s on-boarding process to the main ‘my plan’ page. For one of our products we got a bit more granular in our definition and looked at adoption by category (e.g. Medical versus Home related unexpected expenses), since each funnel is unique.
(R) Retention: A true indicator of a tool’s usefulness over time is whether customers return. As a more immediate indicator of a customer’s potential to return we also looked at registration rates and overall numbers of new users (our tool doesn’t require registration, but it’s helpful to save information for future use, so those who find it useful are more likely to do so).
(T) Task Success: This theme measures a customer’s ability and willingness to complete major tasks in the app, so they get the most value out of what we believe are the most important features. In the case of our debt app, adding customer’s accounts successfully and eventually completing a set of explicitly listed ‘goals’ (such as following the plan’s payment recommendations) within the app are key indicators of the customer’s app task success.
Note that some of the metrics can fit into multiple themes pretty seamlessly, while others are unique to a specific theme. This indicated to us that these were a good way to measure customer success. By design, these metrics were attainable within Google Analytics with minimal customization.
Our Learnings
By choosing meaningful and measurable KPIs up front, we have been able to monitor and report on our client’s apps’ successes and challenges consistently across the organization. We will be using these metrics when designing A/B experiments and to evaluate growth potential of features, as well as understand the overall efficacy and helpfulness of the tool. At current, by these metrics, 78% of our app’s users are happy with the tool. Almost a fifth of our sessions come from return users. Because they are actionable, knowing these metrics allows us to compare with other tools and dive deep into specific flows and customer segments to help us make meaningful product decisions with the client.
