UX — WHAT TO MEASURE, HOW TO MEASURE, WHEN TO MEASURE
Establishing the ROI of UX
“The reality is that great UX design can differentiate products, attract customers, and drive superior business results, so there is no inherent contradiction between investing in User Experience and a business focus on creating customers and profits. — Pabini Gabriel-Petit, UX Matters
The focus towards user experience(UX) has experienced huge attention & growth over the past five years. As management thinker Peter Drucker is often quoted saying that “you can’t manage what you can’t measure.”, more than ever, teams are now turning to user experience metrics to understand how to improve the holistic customer or user experience.
In itself, the term user experience is qualitative. It’s thought of as “making something delightful for customers.” Customer delight isn’t directly measurable and probably won’t make you or stakeholders dive in wallets first.
When designers begin to think, process, and write about design it leads to ideas, it leads to concepts and ideas that you wouldn’t be able to create otherwise. Similarly, when designers observe how their design add value to the people’s life, when they make an effort to learn about how the design has empowered people, made their lives easier, how is the design being used and most importantly when designers analyze the end result of their design solutions and follow up on it, this analysis certainly makes them, “design-wise”.
All that wisdom but how do you explain that with numbers? How do we measure UX design? What value does UX design bring to a company?
These are common questions and at some point, every designer has had to advocate the value of design in terms of ROI (Return of Investment). To measure the impact of UX design, designers have to choose the right metrics and develop an ecosystem to evaluate designs and track progress over time.
What is Return on Investment (ROI)?
Return on investment (ROI) is a financial scale used to analyze the efficiency of an investment. ROI = profit from an investment / investment cost, it is usually expressed in percentage.
In design projects, an investment is often money or time, but the benefit could be increased user satisfaction and improved user efficiency, or the depletion of user error — all of which will indirectly increase the profit. ROI of user experience involves determining and demonstrating how changes in design can affect the business goals like revenue, cost savings, or other key performance indicators (KPIs).The calculation of ROI can show that UX is not only helpful for users, but also beneficial for the business.
However, many UX teams do not calculate the ROI for their work, however it’s always a good practice to evaluate tangible benefits of UX.
Here are three popular myths about calculating the ROI of UX according to the Nielsen Norman Group :
1.The ROI of UX is all about monetary value
2.The ROI of UX has to be perfectly error-free
3.The ROI of UX has to report for every detail
The ROI of UX Is all about monetary value
When calculating return on investment, the final goal commonly is to convert the UX impact into a monetary amount however this does not happen always. People often get stuck because they are trying really hard to figure out how to put a dollar amount on something. But measuring the ROI of UX is more about showing the design that improves the customer experience and has a positive impact on our business goals — whatever those goals are. Business goals often come down to money, but not always.
The ROI of UX has to be perfectly error-free
We want ROI calculations to be absolutely accurate and error-free. But, this is not always possible because ultimately, these are only estimates. As a result of that, they cannot (and don’t have to be) perfect.
The ROI of UX accounts for every detail
The third myth that is often encountered is that ROI calculations must be extremely detailed and that they take ample amount of time to complete. Sometimes this is true, but it doesn’t mean this happens always.
ROI calculations can be as detailed and comprehensive according to your wish. However, it is recommend doing only as much work on these calculations as is required to get your point across to your audience.
The six areas of business that can improve ROI from UX are:
- Revenue • Customer Retention • Team Productivity • Support Costs
- Development Costs • Development Time
Essentially, these comes down to two main components that is time and money.
Here are some interesting stats:
- According to IBM, “every dollar invested in ease of use returns $10 to $100.”
- Staples increased their online revenues by 500% after their UX-focused site redesign. (Human Factors International)
- Jared Spool, the founder of User Interface Engineering, has projected that a simple tweak to the UI of Amazon has contributed roughly $2.7 billion in revenue. (UIE Articles)
- “88% of online consumers are less likely to return to a site after a bad user experience.” (InvisionApp Blog)
- The Rockefeller Corporation, in a study on consumer behavior, found that 68% of consumers will switch to a company’s competitor when they don’t feel valued. (Smashing Magazine)
- Mozilla saw a 70% decrease in support calls after spending about 14-weeks focusing on usability and iterative design. (Nielsen Norman Group)
For understanding the quantitative benefits of UX and how it can be enhanced, it is important for us to understand the different metrics that represents effectiveness of UX.
The goal for UX metrics
As per your expectation, UX metrics are used to calculate, compare, and track the quality of the user experience over time. They are also used to calculate the effectiveness–consequences and success–of the UX design strategy. It’s really important to note: The metrics you select must be focused on the user and not on the business. Because we are discussing about tracking and measuring the user’s experience, not what it means for the business. Yes, that’s important as well, but UX metrics are not business metrics; they must be linked, but they are not synonyms. For instance, a user-focused metric is “task success rate;” the business-focused version of that might be “conversion rate.” The user and his success should be at the forefront.
There are two main types of UX metrics:
Attitudinal & Behavioral.
Attitudinal metrics focuses on how the users feel or what they say about the product, and behavioral metrics measure what users do with or how they interact with the product.
Why you should measure both behavioral and attitudinal metrics
Measuring both behavioral and attitudinal metrics are necessary because it shows us the whole picture of the user experience.
Behavioral metrics can be tracked automatically without intervening the user experience–you can capture the breadcrumbs of data that users leave behind as they interact or transact with your product, website, or app. However, this only provides one piece of the entire user experience equation. While you might see data that shows that the experience broke down, you won’t know why, how the user felt about it, what impact it had on them, etc.
On the other side, attitudinal metrics are measured through surveys, feedback buttons/ forms, or interviews, where the user can tell you what happened or how they felt about the experience. Often the surveys or feedback buttons function as intercepts, popping up in a minimally intrusive way while the user is interacting on your website or app. Combining the two provides a comprehensive overview of the experience and puts you in a much better position to understand what’s going well and what isn’t. Here are some top behavioral and attitudinal metrics used in UX research and design.
How to measure behavioral metrics
Many UX designers use PULSE metrics, which are:
- Page views: the number of pages viewed on your website by a visitor
- Uptime: the total amount of time users can access your site or your app (meaning, it’s not down or offline).
- Latency: it is the hang time or the lag time. i.e., when you click the button, how much time it takes to go to the following page.
- Seven-Day Active Users: the number of unique active users on your site or app within the last seven days.
- Earnings: revenue generated by the site, app, or product.
Other popular behavioral metrics include:
- Time on Task: the amount of time taken by the user to complete a task.
- Task Success Rate: number of tasks completed correctly divided by the number of attempts. (It’s important to define what success looks like first!)
- User Errors/Error Rate: the number of times a user makes a wrong entry.
- Abandonment Rate: how often users left the task incomplete, e.g., filled the cart but didn’t make the purchase.
How to collect behavioral metrics
By their nature, user behavioral metrics can be collected automatically without the interference of an interviewer or observer, this makes it a simple way of collecting UX metrics. These are collected through site analytics, user session data, app analytics, search log, bug review/tracking, and more. But behavioral metrics can also be collected through other methods, such as ethnography or observation, A/B testing, eye tracking, and usability testing. While it is important to collect these metrics but they don’t provide a complete picture or the context needed to understand the “Why?” behind each metric. That’s where attitudinal metrics come in.
How to measure attitudinal metrics
Attitudinal metrics collect what people’s feelings about the product. There are far fewer of these metrics than there are of behavioral, but they are equally important. Given below are some of the top attitudinal metrics you should capture.
System Usability Scale (SUS): System Usability Scale is a popular metric used by UX researchers and designers. It consists of ten questions answered on a five-point agreement scale about the experience with the product or website. These are ease-of-use type questions, but don’t provide any sort of diagnostic details. The score itself is difficult to calculate, but it has become an industry-standard, popularly used metric.
Satisfaction: Satisfaction measures how satisfied the user is with the experience of the product or website, down to the features and functionality. Satisfaction with the UX can be measured in the same way as customer satisfaction: using the Customer Satisfaction Score (CSAT)
Net Promoter Score (NPS): Net Promoter Score states an intention to do something based on how users feel. If someone is likely to recommend your product, your app, or your site based on the experience they had using it, then the experience might have been a good one.
How to collect attitudinal metrics
There are various ways to capture attitudinal metrics. Nowadays, this data is typically collected through different types of surveys, interviews, or feedback buttons.
How to choose the right user experience metrics
There not a single general accepted list of metrics to use — there’s only the accepted classification of these metrics: behavioral or attitudinal. That’s why you need to narrow your focus and select metrics that matter to your brand.
Clearly, the first thing to note is that they have to be metrics that matter to your users, to your business, and to the experience that you’re measuring. Beyond that, many folks are leaning on the Google HEART framework to hone in key UX metrics. In 2010, Kerry Rodden, Hilary Hutchinson, and Xin Fu wrote a paper about the HEART framework of metrics that they had developed after applying it to 20 products while working at Google. The purpose of the framework is to help you integrate both attitudinal and behavioral metrics. Given that it’s based on work at Google, as you can imagine, its focus is on web-based user experience, but the framework can be applied to any user experience.
They developed the framework after seeing the shortcomings in PULSE metrics (Pageviews, Uptime, Latency, Seven-day activity, and Earnings) for measuring user experience quality and reaping actionable data. HEART stands for Happiness (satisfaction), Engagement, Adoption, Retention, and Task success. If you look at each description below, you’ll find that, in the end, they are all either attitudinal or behavioral metrics.
• Happiness: includes attitudinal measures like satisfaction, likelihood to recommend, and ease of use.
• Engagement: includes some usage metrics, like the number of visits per user per week or the number of photos uploaded per user per day.
• Adoption and Retention: include metrics like the number of unique users per time period to distinguish between new (adoption) and existing/returning users (retention).
• Task success: includes behavioral metrics, like efficiency (time to complete), effectiveness (ability to complete, percentage completed), and error rate.
You should select metrics that the business cares about and that truly reflect what the user is trying to achieve. All of these metrics are useless, unless they are tied to some goal. Just because a visitor spent an extended period of time on your site does not mean the site experience is great; it could mean quite the opposite — that they’ve been muddling through it for minutes or hours just to do what they are trying to do. So the next step in the HEART framework is to identify user goals. What are they trying to achieve? How does the product help them accomplish this?
Here is a process that may help you:
1. State the goals (ex: user outcomes).
2. Identify the signals that indicate success (ex: survey ratings, user data).
3. Select the identifying metrics (ex: satisfaction) to track on your dashboard.
How Google starting to see User Experience — Page Experience
Last year November it was announced by Google that the page experience ranking change will go live on Google Search this year, in what Google calling the “page experience update”. This is done to help publishers and site owners improve their page experience and prepare
On May 5th, 2020 Google announced the introduction of Core Web Vitals, a set of metrics designed to help the search giant quantify user experience and its impact on your users when they visit your site.
• LCP Largest Contentful Paint
• FID First Input Delay
• CLS Cumulative Layout Shift
May 28th, 2020 Google further confirmed in an article on their blog that they were going to build on Core Web Vitals and start using these page experience metrics to find out how and where to rank you in SERPs this year, in 2021.
These three key metrics were designed to measure the user experience a website provides and brings three simple questions to our mind :
How fast is the speed of the website or webpage? How stable is the website visually? How fast can a user interact with a page?
What are Core Web Vitals?
As mentioned above, Google now measures the user experience (UX) provided by your website using three user-centered metrics called Core Web Vitals. You can test your websites largest contentful paint, first input delay and cumulative layout shift programmatically using Google’s Pagespeed insights tool.
1. Largest Contentful Paint : Largest Contentful Paint is a metric used to interpret how long it takes for the largest web page elements to load. It’s an accurate test of the overall load speed of your website’s pages.
2. First Input Delay : First Input Delay is a user-centric, page experience metric introduced by Google to measure how responsive a website is to a user. Essentially, it measures how long it takes for your website to be fully interactive to the user.
3. Cumulative Layout Shift : Cumulative Layout Shift is a page experience metric introduced to measure a webpages stability when loading. Lighthouse looks at how elements such as images, adverts and containers shift when loading. For example, if you don’t set explicit heights to images, then they would start 0px later ‘shifting’ to the height determined by the image itself.
Google announced on the 15th June 2021 that they have started rolling out their page experience update. The process is further finalized throughout June & July 2021 and potentially into August.
Prove the ROI of your user experience efforts
User experience improvement efforts are no different from any other project or initiative within your organization in that, if you want to get the resources–time, human, capital, etc.–to design or to improve the user experience, you must link the work to outcomes and show a return on investment (ROI). Outcomes matter so don’t get so deep into the weeds on the many UX metrics that you forget to consider the outcomes–for the user and for the business.
What could those outcomes be for the user?
- Ease of use
- Goal completion rates
- Satisfaction/great experience
What outcomes would you like to see for the business?
- Increased conversion rate
- Increased purchases/reduced cart abandonment
- Reduced support volume/costs
- Reduced development costs
- Increased retention and referrals
Calculating the ROI
So finally, how is this calculated?
To get to the exact gain following the UX investment, you need to eliminate the investment from the economic gain. But you should also take into account the years it will take to pay off the investment as well as inflation.
To calculate the ROI I will recommend to use the ROI calculators from Human Factors International:
https://humanfactors.com/thinking-favorite.aspx
Here you can calculate how much money you earn from increasing productivity, conversion rate, decrease calls to help desk etc. Just pick the right calculator for your specific project.
Design shouldn’t be looked at as an expense, the investment of a good design can also be valued through its capability to engage, influence, and function. Design can also be seen in good content, aesthetics, and in its capability to meet the user’s needs. A design that unifies a brand’s message, strengthens business with clarity, and provides solutions affects the overall profitability through memorable experiences, and as we all know good experiences are the whole basis of a good UX design.
— — — — — — — — — — — — — — — — — — — — — — — —
Further Readings & References:
· Three Myths About Calculating the ROI of UX — Nielsen Norman Group
· A Guide to Measuring the ROI of UX — playbook UX
· UX Collective — List of studies and cases (https://uxdesign.cc/ux-roi-i-can-prove-it44ded1b0e25b)
· Interaction Design Foundation
· Google Search Central
· The Complete Guide To User Experience Metrics — Usabilla
