Quality is a vague concept, which makes it hard to measure directly. Here are 5 steps to more easily measure the quality of anything.

We don’t measure a recipe; we measure each of its ingredients. And the ingredients will vary depending on the recipe and the tastes of the people who will be fed by it. Some ingredients need to be measured because they matter, while others don’t (like lettuce in a salad, broth in a risotto, or decorations on a cake). A recipe is a concept and it’s only the ingredients that can be measured meaningfully.

Because quality is a concept, and too vague to measure directly, to measure quality meaningfully, we have to unpack it first. Inspired by Douglas Hubbard’s clarification chain, from his brilliant book How To Measure Anything (and discussed in this article about measuring design success) is this basic guideline I use to make vague concepts measurable:

1. unpack the multi-focus concept into singular-focus attributes
2. reword the intangible concept into observable language

Quality is both intangible and multi-focus. We can’t meaningfully measure it until we make it unpack its attributes and make each attribute observable. Here’s how…

STEP 1: What entity do you want to measure the quality of?

Quality relates to so many things. Fundamentally, quality is about how good something is. And even more specifically, it’s about how good something is relative to how good it should be.

So it can be useful to measure the quality of all kinds of things we work on, like these:

• Data quality e.g. financial data or performance data or HR data
• Product quality e.g. a vacuum cleaner or gourmet chocolate
• Service quality (or customer experience) e.g. of a train trip or legal advice
• Software quality e.g. a meditation app or KPI dashboard app
• Process quality e.g. technical support process or delivery process or manufacturing process

Our first step, then, is to clearly identify the entity we want to measure the quality of. What’s yours?

STEP 2: Who defines the quality of that entity?

I first learned how to measure quality in customer service. My job was to create a system of consistent customer service quality surveys across the railway I worked for. Managers had been defining the attributes of service on behalf of customers. They spent tens of thousands of dollars on surveying customers based on these manager-defined attributes. And they got it wrong.

On-time running of trains was not the most important attribute to customers, as managers had assumed. And the attributes that were most important to customers – safety and security on trains and in stations – the managers were not even aware of. It’s customers that define the most important attributes of quality for the entity of customer service. Not managers.

And for the quality of your entity, you need to let the experts define the quality attributes. And often the experts are the end users, for example:

• Data quality might best be defined by data scientists or others who both analyse data and understand the structure of data
• Software quality might best be defined by the users of the software
• Service quality is best defined by the customers that directly experience that service

Who is the end user of the entity you want to define the quality of?

STEP 3: What’s their overall assessment of quality?

Don’t add up attributes to create an overall. This a composite measure or index and it’s a very unreliable and inaccurate way to measure something. Especially when you can actually measure it directly. And often, we’re too quick to assume we can’t measure something directly, when in fact we can. Here are some examples:

• Overall Perfect Data Rate: using samples of data records, take the percentage of all data items that are either missing, incorrect, incomplete or irrelevant
• Overall User Perception of Software Quality: the average rating provided by users, on a scale of 1 to 10, of the degree to which the software meets their known requirements
• Overall Customer Perception of Quality: the average rating that current customers give, on a scale of 1 to 10, of how satisfied they are with overall service quality

How could you get a direct measure of the overall quality of your entity?

STEP 4: Which specific attributes of quality matter most to them now?

When I was involved in the railway customer surveys, we used focus groups of customers to discuss what quality of service meant to them. And from the qualitative discussion, we listed about a dozen or so quality of service attributes.

They were different for each service: commuters versus freight customers versus tourists. So we can’t rely on finding the “one right answer” someone has already published. But it’s okay to use existing frameworks as a good starting point, or some prompts to start the discussion:

• Data quality – start with these 5 dimensions of data integrity
• Software quality – the ASQ have this neat checklist of software quality attributes
• Service quality – a tool called SERVQUAL uses 5 widely accepted service dimensions that most customers care about

What method could help you get an objective list of quality attributes for your entity?

STEP 5: Measure the most important 3 attributes.

When you have worked out which are the top 3 attributes of quality for your entity, and have made sure you express them measurably, you can go ahead and design your own performance measures of those quality attributes.

I recommend designing your own measures, rather than copy them from published lists of what others measure. That’s because you want the right KPIs that will fit your situation the best. But just for inspiration:

• Software quality – check out Rainforest’s map of Essential QA Metrics for software, including test coverage, time to test, time to fix, escaped defects, and Net Promoter Score
• Data quality – Christopher Tozzi wrote this list of 7 metrics to measure data quality, which I don’t think is comprehensive but can at least give you a start
• Service quality – for a freight company’s billing team, how they translated one of the most important quality attributes for their customers into
  quantitative KPIs

Just like for any weasely, vague, broad or intangible concept, quality can be measured meaningfully. The trick is to take a deliberate approach to unpacking its defining attributes, expressing each quality attribute measurably, and designing measures as direct evidence of each quality attribute.

Quality is too vague to measure directly. But we can unpack quality into attributes that can be directly measured.
[tweet this]

DISCUSSION:

Use this Make Quality Measurable Template to plan your mini research project to unpack the measurable attributes of quality for the domain you’re interested in.

1. Greg Armstrong says:

   January 5, 2019 at 4:47 am

   This is useful. I spend a lot of time in indicator workshops trying to get people to explain what they mean by “quality”. Both for this, and for indicators in general I agree also with two points in particular:
   a) Not using composite indicators
   b) Getting a group to develop its own indicators rather than using someone else’s standard indicator

   Reply
   • Stacey Barr says:

     January 15, 2019 at 4:21 pm

     Thanks Greg – it’s always nice to find kindred spirits when it comes to measuring vague concepts like quality. Should you use these tips with a future group, please let us know how it goes.

     Reply
2. Stephen says:

   May 23, 2020 at 4:49 am

   In Step #2: Who Defines the Process Quality?

   Reply
   • Stacey Barr says:

     May 27, 2020 at 11:23 am

     Stephen, I’m not sure if you finished typing your question, but I’ll take it literally as you’ve typed it… Ultimately it’s the customer, who consumes what the process produces, that defines quality. The process owner and process improvement team will design a method to find this out from the customer, and prioritise which attributes of quality are the most important for most of the customers.

     I’ve had a lot of experience where process owners or leaders *think* they know which quality attributes are most important for the process, and when compared to what customers wanted, they were far from accurate.

     Reply
3. Five Blogs – 10 May 2021 – 5blogs says:

   May 10, 2021 at 2:59 pm

   […] 5 Steps to Measure the Quality of Anything Written by: Stacey Barr […]

   Reply