ARE YOUR SURVEYS ONLY SUITABLE for WRAPPING FISH?

www.C3excellence.com Page 1 of 9

ARE YOUR SURVEYS ONLY SUITABLE

for WRAPPING FISH?

The shortcomings of customer satisfaction surveys, and what you can do about it.

By Ken Miller

_____________________________

There's no question that the popularity of surveys is on the rise. You can't go anywhere without

being asked to complete one. They are in every hotel room and restaurant, and on every airplane. I

fully expect to come home from work one day and find a survey from my wife on the kitchen

counter.

If paper and pencil surveys are the panacea for building long-term relationships as some suggest,

then why don't people survey their spouses? After all, wouldn't it be valuable to know that this

year my wife rated me a 9 out of 10, which is 10% higher than last year? Surely I could do some

kind of regression analysis to figure out what I'm doing right.

The fact is, people don't survey their spouses. Why? Because they intuitively know surveys are not

very effective and there is a better, simpler way to learn how to satisfy one's spouse.

Why use surveys?

So if people don't survey their spouses, why do organizations survey their customers? Consider a

typical survey and some of the assumptions people make when they create or use them.

Assumption No. 1: Surveys tell companies that their customers are satisfied. This would be

true if the organization knew that the questions on the survey were truly related to what customers

want. Are they? Where do survey questions come from? If your surveys are like the ones I've seen,

they are developed by a committee of people in a series of meetings without a customer in sight.

The end result is a survey to determine how satisfied customers are with a number of factors that

the organization believes customers think are important.

One improvement to surveys over the years has been the inclusion of questions such as: "Overall,

how satisfied are you with…." Even if the questions being asked are poor, organizations will at

least have some indication of a customer's general satisfaction level.

I recently worked with an organization that had just finished surveying its customers. Members of

the organization developed the questions without really knowing what customers wanted and

needed my help interpreting the results. When I asked why they needed my help they said, "We

don't think the customers understood the questions." It turns out that the organization's customers

said they were satisfied with the organization overall, but were not satisfied with the specifics

asked about on the survey. The customers were satisfied but the organization had no idea why. It

could have been worse; the customers could have been satisfied with all the factors listed but

unsatisfied overall.

www.C3excellence.com Page 2 of 9

Look at the survey in Figure 1.

Figure 1: Lawnmower Satisfaction Survey

Please circle the number that best represents your satisfaction with the factors listed below (5

equals highly satisfied and 1 equals highly dissatisfied).

Highly

Satisfied

Neutral

Highly

Dissatisfied

Easy to operate

Safe

Affordable to

maintain

Affordable to buy

Fuel consumption

Quiet

Reliable

Easy to start

Overall, how

satisfied are you

with your mower?

10.

What suggestions do you have for improving the mower?

Is it possible to mark "dissatisfied" for each of the questions and still be satisfied with the

lawnmower? Perhaps the customer is satisfied simply because the mower is bigger and faster than

the neighbor's mower. Is it possible to circle "highly satisfied" for each of the characteristics listed

and be dissatisfied with the mower? Perhaps the mower is fine, but the dealer was rude. The results

of many surveys can leave people scratching their heads.

If companies remove all the things that dissatisfy their customers, will the customers be satisfied?

If I suddenly let my wife use the remote control and start putting the toilet seat down and asking for

directions when I'm lost, would she be satisfied? I would argue that she would only be less

dissatisfied. The remote control, the toilet seat, and my stubbornness are not satisfiers, they are

dissatisfiers. Look instead for the satisfiers in your relationships. What are your customers'

satisfiers? Is your organization improving the satisfiers or dissatisfiers? It is not safe to assume that

the elimination of dissatisfaction creates satisfaction.

Assumption No. 2: Surveys reveal what to improve. If assumption No. 1 is wrong, assumption

No. 2 has major problems. In Figure 1, for example, if 85% of respondents answer questions 3, 4,

and 5 as dissatisfied, what would you do? The natural approach is to try to improve these factors.

But what if those factors are not the most important to the customers? What if they are not

www.C3excellence.com Page 3 of 9

important at all?

Surveys rarely reveal how companies can improve satisfaction. If customers rate the lawnmower a

4 (satisfied) on question 1 (easy to operate), what would it take to rate a 5? Do all of the company's

customers have the same definition of "easy to operate"?

I recently met with a manager who was quite proud of his survey results. He oversees four

divisions and requires each of them to survey its customers. The manager was ecstatic that each

division was scoring higher than 4.2 on a five-point scale. I asked him, "What do these results tell

you?" He replied "That we're doing pretty well." So I asked, "What do you have to do to do better?

What does each division have to do to rate a 5? Why are some divisions rating 4.7 while others are

just at 4.2?" In short, he had no idea.

Assumption No. 3: If 95% of the organization's customers are satisfied, the organization is

doing okay. Besides the problems pointed out previously, there is a far more dangerous problem:

A company could satisfy 100% of its customers but have very few of them. For example, Apple's

Macintosh computer has extraordinarily satisfied customers, yet the company's market share

continues to decline.

How often do organizations survey their competitors' customers? Companies will never figure out

why customers choose competitors if they only survey their own satisfied customers. Similarly,

how do government programs know why potential participants choose not to use their services

even when they're free?

Assumption No. 4: Customer satisfaction equals customer loyalty. A problem that has received

a lot of attention in quality circles lately has been the issue of customer loyalty. Is it possible for

customers to be completely satisfied with a product and still switch to something else? It happens

all too frequently. So what drives customer loyalty?

Peter Drucker summed it up best when he said "Customers don't buy products, they buy results."

That is, people don't buy drills, they buy holes. They don't buy cars; they buy transportation (or

status). They don't buy light bulbs, they buy light. Customers are loyal to whatever product best

helps them achieve their desired outcome.

If customer loyalty is based on the product's ability to help customers achieve a desired result,

shouldn't that be the first question on every survey. Surprisingly, it is rarely asked at all.

Consider an example. One of the most common places to find a survey is in a hotel room. Figure 2

shows a typical hotel survey.

www.C3excellence.com Page 4 of 9

Figure 2: Hotel Satisfaction Survey

Excellent

Good

Poor

How would you rate our staff:

Front desk

Housekeeping

Room service

Restaurant servers

How would you rate our:

Front desk services

Check-in efficiency

Check-out efficiency

How would you rate:

Bathroom cleanliness

Bedroom cleanliness

Television

Heating/air conditioning

Lighting

Overall, how would you rate

your stay?

What could we do to make your stay more pleasant next time?

It is possible to mark "satisfied" with all the questions listed and still not be satisfied. What is the

No. 1 outcome a business traveler would like from a hotel stay? It is probably a good night's sleep.

(This is why it's important to segment your customers. A good night's sleep is unlikely to be a high

priority for honeymooners.)

Where is the good-night's-sleep question on the survey? The fact is, you could have a clean room

with friendly service and a nice television, but if housekeeping knocks on the door at 5 a.m. to

clean the room you will not be satisfied. The next time you stay in a hotel, glance at the survey on

the night stand and see if the good-night's-sleep question is there.

Assumption No. 5: The right people are being surveyed. Who mows the lawn at your house?

You, your spouse, or your eager and grateful teenager? Should the company survey the person who

bought the mower, the person who uses the mower, or both? Will their priorities be the same? I

know that when I mowed my father's lawn, a satisfying mower would have been light, fast, and

self-propelled. Unfortunately my father opted for the $99 Immovable Force Deluxe.

www.C3excellence.com Page 5 of 9

Identifying exactly who the customer is can be challenging. Even organizations that understand

who their customers are rarely segment them properly. These companies are missing tremendous

opportunities to differentiate themselves to different markets. A one-size-fits-all approach often

misses the opportunity to create customized products for different market segments.

Consider another example. A health insurance company wants to know how satisfied its customers

are. Does it survey the insurance agent who distributes the policy, the employer who pays for the

policy, the employer's benefits administrator, the employee, or the employee's sick spouse?

Likewise, should a government agency survey taxpayers, citizens, lawmakers, or program

participants? Surveying the right people requires thought.

To summarize, there are four problems with most surveys:

1) They ask the wrong questions

2) They ask the wrong people.

3) They are given at the wrong time.

4) They don't enable focused action.

Yet it seems that every organization that embraces the "focus-on-the-customer" mantra

immediately begins by surveying its customers. Unfortunately, most of these surveys are only

suitable for wrapping fish. The organizations' intentions are admirable, but if companies are going

to better understand what customers want (not just how happy they are with what they receive) they

need to take a different approach. That approach is one that will allow companies to ask the right

questions, in the right way, of the right people, and at the right time so they know what to change.

The right time

Wouldn't it have been nice to know what you're spouse expected of you before you got married?

Wouldn't an organization like to know what would satisfy its customers before designing its

products and services? Surveys are an acceptable tool if what customers want is already known.

Organizations rarely ask, however, and if they do, they don't ask the right questions.

The right way to find out what customers want is to ask them, but not in a survey. Organizations

can learn more about their customers in a 90-minute focus group or a 15-minute interview than

they will ever find out in a decade of surveys. The right time to survey is after determining what

customers want, designing the product or service to meet those expectations, and determining that

what the customers got is what they wanted.

Total quality management became popular in manufacturing because it challenged a common

practice - trying to inspect quality into a product. By the time the defects made it to quality

assurance, it was too late. While most organizations have learned this lesson on the shop floor, they

are still applying the old logic to surveys. Quality cannot be surveyed into a product. Rather,

organizations must take the steps to build quality into their products and services from the

beginning.

www.C3excellence.com Page 6 of 9

The right people

Who are the right people to ask? To just say "customers" doesn't cut it. As was pointed out earlier,

there is a lot confusion about the word "customer" and who the customer is in a given situation. To

put an end to this confusion, consider the two key roles customers can play: end users and brokers.

End users are customers who use a given product or service to achieve some desired outcome.

They are the customers the company had in mind when it designed the product.

Brokers transfer the product to someone else who will use it. They may act either as agents of the

end user or agents of the producer. As an agent of the end user, the broker makes the product more

accessible, easier to use, or more appealing. As an agent of the producer, the broker encourages the

end user to accept the product.

Going back to our lawnmower example, the end user may well be the teenager, while the parents

and retailer play the role of brokers. The parents may be acting on behalf of the teenager. The

retailer is acting on behalf of the manufacturer. Both the parents and the retailer can also be acting

on their own interests, as is often the case with brokers.

Consider a TV program as another example. The end users of a given TV show can be numerous.

Who would the brokers be? They might include the local television station that broadcasts the

signal and advertisers that pay for the programs Who do you think has the most power over what is

on television - the local station, the advertisers or the viewers?

In a perfect world the viewers would have the power, but reality has the power in the hands of the

brokers. The aim should always be to satisfy the end users. Unfortunately, companies may

inadvertently focus on satisfying brokers' needs. This can happen by not being clear about customer

roles.

It's not that brokers are bad people. Everyone acts as brokers from time to time. Just make sure that

as you set out to discover what it will take to satisfy customers that you first ask the end users, then

consider the brokers - especially the brokers acting on behalf of the end users.

The right questions

Once you know what customers want and have properly identified the right people to survey, it is

helpful to know what questions to ask to drive action.

Figure 3 depicts a survey from a spouse done in the typical style discussed earlier. On the left side

is a list of variables believed to be important, and across the top is a rating scale.

www.C3excellence.com Page 7 of 9

Figure 3: Spouse Satisfaction Survey (Typical Approach)

Highly

Satisfied

Neutral

Highly

Dissatisfied

Trash removal

Personal hygiene

Lawn maintenance

Romance

Thoughtfulness

Listening skills

Faithfulness

Respect for mother-in-

law

Overall, how satisfied with your marriage?

Look at question 1. Suppose I scored a 2 out of 5 on taking out the trash (a rather lofty goal for

myself). What does this tell me? I would argue it tells me very little. For example, does it tell me

what my wife expects? Better yet, does it tell me what she wants? You'll notice these two terms,

expect and want, do not mean the same thing. My wife may want me to take out the trash daily, but

she's come to expect that it will only be removed when it's placed in the path between me and my

golf clubs.

I also have no idea how important this is, especially when compared with question 4. Which should

I try to improve first? In short, the survey as written does not allow me to make intelligent

decisions about what needs to be improved. Good survey questions should find out four things:

1. What was expected or wanted

2. What was experienced

3. The level of satisfaction with the product or experience

4. The degree of relative importance of this variable

Compare Figure 4 with question 1 of Figure 3.

www.C3excellence.com Page 8 of 9

Figure 4: Spouse Satisfaction Survey (The Right Way)

Trash removal

a. How often do you expect the trash to be taken out by your spouse?

____ Not at all

____ Daily

____ When its full

____ When reminded

____ When the stench arouses the anger of the neighbors

b. How often would you like the trash to be taken out by your spouse?

____ Not at all

____ Daily

____ When its full

____ When reminded

____ When the stench arouses the anger of the neighbors

c. How often is the trash taken out by your spouse?

____ Not at all

____ Daily

____ When its full

____ When reminded

____ When the stench arouses the anger of the neighbors

d. How satisfied are you with your spouse's trash removal?

____ Very Dissatisfied

____ Dissatisfied

____ Neutral

____ Satisfied

____ I fantasize about it

On a scale of 1 to 8, please rank the importance of the following to the happiness of your

marriage, where 1 equals most important.

____ Trash removal

____ Thoughtfulness

____ Personal hygiene

____ Listening skills

____ Lawn maintenance

____ Faithfulness

____ Romance

____ Respect for mother-in-law

Which approach will yield the most improvement action? If all the questions from Figure 3 were

structured this way I would know what is important to my spouse, how well I am performing, how

well I need to perform, and where I should focus my improvement efforts. Which figure looks most

like your surveys?

The magic question

Do you know what outcomes your spouse is hoping to achieve in your marriage? Try asking him or

her to complete the phrase "A satisfying marriage is one that results in…" Do you know how he or

she would answer? Would you agree that the answer to that question is far more important than

taking out the trash or mowing the lawn? If your company were to ask its customers to complete a

similar phrase, would it know how they would answer?

If I could add one question to everybody's existing surveys it would simply be this: "What results

were you hoping to achieve by using a certain product, service, or organization?" A follow-up

question would be: "How successful have you been in achieving these results?" These simple

www.C3excellence.com Page 9 of 9

questions are a doorway into knowing what drives customer satisfaction today and what will

impact customer loyalty and innovation tomorrow.

Customer satisfaction surveys that ask the right questions of the right people (end users properly

segmented) at the right time (after determining what customers want and designing the product or

service accordingly) can be an effective and economical way to drive improvement action. By

following this advice, organizations can change their surveys from fish wrapping to gift wrapping

(and maybe save a marriage or two).

Ken Miller is a partner with International Management Technologies. He has a master's degree in

business from Northwest Missouri State University in Maryville. Miller is a member of ASQ.

Figure 5: Common Survey Problems

The wrong people are surveyed.

The wrong questions are asked.

The questions are asked the wrong way.

The questions are asked at the wrong time.

Satisfaction and dissatisfaction assumed to be equally important.

Those who did not buy or use the product/service are not surveyed.

Surveys are conducted for the wrong reasons.

The results are generalized to groups not surveyed.

Surveys are used as a substitute for better methods.

The results do not direct improvement actions.

Bibliography:

Hayes, Bob E., Measuring Customer Satisfaction: Survey Design, Use, and Statistical

Analysis Methods (Milwaukee, WI: ASQ Quality Press, 1998).

Kessler, Sheila, Measuring and Managing Customer Satisfaction, Going For the Gold

(Milwaukee, WI: ASQ Quality Press, 1996).

Lawton, Robin L., Creating a Customer-Centered Culture, Leadership in Quality,

Innovation, and Speed (Milwaukee, WI: ASQ Quality Press, 1993).

Lowenstein, Michael W., Customer Retention, An Integrated Process for Keeping Your

Best Customers (Milwaukee, WI: ASQ Quality Press, 1995).

Naumann, Earl and Kathleen Giel, Customer Satisfaction Measurement and Management

(Cincinnati, Ohio: Thomson Executive Press, 1995).

Rea, Louis M. and Richard Parker, Designing and Conducting Survey Research, A

Comprehensive Guide (San Francisco, CA: Jossey-Bass, 1997) .