Tip #6: UNDERSTANDING YOUR SURVEY RESULTS:  IT'S ALL ABOUT CONTEXT

 

A group of senior executives was discussing the survey results for their organization.  They were looking specifically at the item "I have a good understanding of the needs of my external customers," which had received a 79 percent favorable response.

Almost everyone in the group saw this finding as extremely positive.  After all, the numeric guidelines given to them by their consultant had said that anything above 70 percent favorable could be considered a strength.  However, one executive, after pondering the response for a few minutes, stated that if this result were applied to HIS employees, he would view it negatively and immediately set it as a priority action opportunity.  Who was he?  The Vice-President of Sales.  According to his way of thinking, a 79 percent favorable response would mean that 21 percent of his sales force did NOT understand the needs of their external customers - an unacceptable and potentially damaging finding for him. 

That VP was applying context to the results to make them more meaningful for him.  And he was correct.

It's Not Just the Facts

Surveys provide us with data - lots of data.  Reports full of different numbers.  Percentages.  Frequencies.  N-counts.  Mean scores (with or without standard deviations).  And you have to make sense of it all.   

Just knowing what the numbers say is not enough; you have to figure out what the numbers mean in order to use survey results.  When survey reports are distributed, one question is on everyone's mind:  Is this result good or bad?  The answer is almost always "That depends."  There are several ways to interpret survey data, and all of them involve context.  It turns out that meaning - to co-opt an old saying - is in the eye of the beholder.

 

Numeric Guidelines

One of the easiest ways to understand survey data involves numeric guidelines.  Guidelines define your "strengths" and "opportunities" in terms of numbers, usually percentages or mean scores.  Most consultants use them to help you get started.  But it's important to remember that these numbers are meant to guide your thinking, not replace it.  There are no absolutes in data interpretation - no hard-and-fast rules, nothing carved in stone.  That's one reason why numeric guidelines are neither universal nor permanent. 

In one organization, it may be perfectly appropriate to say that an item achieving a score of 65 percent favorable is a strength.  But what if, in another organization, every item on the survey surpasses that score?  How meaningful is that guideline?  In a situation like that, the guideline should be revised upward, perhaps requiring a score of 70 or 75 percent favorable to define a strength.  Conversely, if an organization tends to have lower scores, the guideline might be revised downward (though generally not below 50 percent).

It's also important to remember that guidelines are generally set for organizations - but results are often reported for departments or units.  So what if the organization has set a numeric guideline and none of the results for your department meet it?  Does that mean you have no strengths?  No.  That's where context comes in.  The issues that your employees are most favorable about are your strengths, even if they do not meet the numeric guideline. 

And if all of your items exceed the numeric threshold for strengths, that's doesn't mean you still don't have issues to work on.  The issues that your employees are least favorable (or most unfavorable) about are your opportunities for improvement, even if they exceed the numeric guideline.  For example, if every item in one category is over 80 percent favorable except for one, which is at 67 percent, that item is your opportunity for improvement even though it may be higher than your numeric guideline.

 

Relative Comparisons

Most people love to compare themselves to others.  Survey results are no different.  That's why organizations set up their reporting so that managers can see how their department results compare to their division results, or location results, or total organization results.  We compare our current results with historical data.  Sometimes we use an external norm like an industry comparison.

The problem with relative comparisons is that we tend to think of them in absolute terms.  But this is a place where context is more important than ever. 

Several things can affect how meaningful a comparison may be.  Suppose you compare your department of 20 people to the total organization of 10,000.  That means a five percent difference on an item might be due to the response of one person.  Is that meaningful enough for you to take action?  In one company, a division that had undergone a merger (and doubled in size) nevertheless compared its results to the pre-merger results from a year earlier.  Enormous changes were seen - but was it meaningful change or just the result of an influx of new employees with different expectations?  And even if a large organization is relatively stable, small work groups rarely are.  None of this is meant to imply that comparing your results to others can't be beneficial.  But it's important to keep context in mind. 

Some organizations include statistical significance in their reports, assuming that if a difference is significant, it means something.  Well, it does - but not what most people think.  In the English language, the word "significant" usually means "important," but in statistics, it simply means "not due to chance."  If you have a large organization of several thousand employees, your current results will differ significantly from your historical results on virtually every item.  Does that mean all of those differences are important for taking action?  No.  

The most critical thing to remember when using relative comparisons is that your own results must always come first.  It is very tempting to discount a low score simply because it is more favorable than someone else's score.  Resist that temptation!  Deal with your results as they are.  Use comparisons as points of interest along the road, but don't let them detour you from your final destination.

 

Personal Standards

This is the most important way to look at your survey results.  The personal standard is all about context.  It involves asking - and answering - questions such as:  Are the survey responses what I expected?  What I hoped for?  Do they reflect my group's priorities and recent initiatives?  Did anything take me by surprise? 

Think about the role context played in the following:

• If you headed up a Communications department, a score of 50 percent favorable in the area of communication might be cause for alarm.  But in one company, the head of IT was ecstatic at that result.  He had made internal communication a priority and that 50 percent favorable showed progress from the previous survey.

• The VP of Sales in the initial example used his personal standard to determine that a 79 percent favorable result was not good enough.  In another company, an 80 percent favorable response to a quality issue was deemed unsatisfactory by the CEO because the company was in the second year of a huge quality push, and he had expected the number to be much higher.

• A manufacturing firm had survey results that were very high.  However, senior management knew that there were problems and were concerned that employees did not trust the process enough to be honest.  They implemented survey follow-up to try to learn why - when morale and teamwork problems were evident - employees had chosen to respond favorably.  They worked hard to address concerns about trust in the process.  When they next surveyed, they were pleased to find specific areas being marked more realistically.  In an odd turn of events, they felt that lower scores from the previous survey indicated success!

These situations show the importance of context - examining carefully the circumstances of both individual groups and the organization as a whole, and applying personal standards to better understand the results. 

What Does This Mean for Me?

Data interpretation is partly science and partly art.  The science comes from the numbers.  Certain things are obvious, e.g., if you achieve a 95 percent favorable score on something, chances are that it is a strength that you can leverage in addressing other lower-scoring areas.  If one-third of your employees respond unfavorably to something, chances are that it's an issue you need to deal with.

The "art" comes in interpreting things that aren't so obvious.  Then you must rely on context.  Only you as a manager know what has been occurring in your group both before and during the survey.  Your understanding and expectations have merit in analyzing your data.  You also know what your personal and organizational priorities are.  These will help determine your group's strengths and areas of opportunity, no matter what the numeric guidelines or relative comparisons say.