Why You Need to Analyze Survey Comments

I'm putting the finishing touches on the second edition of my book, Getting Service Right. The book was originally called Service Failure, and I've now updated both the title and some of the research.

The cover is one of the most important sales tools for a book, so I worked with Anne Likes Red to come up with a few designs. I then launched a survey to ask readers for their feedback on three cover options. The survey was up for just a few days and a 135 people responded.

Here were the results:

Option A (28%)


Option B (52%)

Option C (20%)

Picking cover option B should be a no-brainer, right? After all, more than half of all survey respondents picked that option.

Without qualitative information, I might have made that mistake. Fortunately, I also included a comment field in the survey. When you analyze the comments to learn why someone chose a particular option, a new pattern emerges.

Searching for Themes

I recently hosted a webinar with Alyona Medelyan, CEO of the customer insight firm Thematic. Medelyan brought actual client data to reveal some interesting insights that a survey score alone wouldn’t show:

  • A cable company found customers with modem issues were impacting overall NPS by -2 points.

  • Another company discovered one variable that caused customers to spend $140 more per year.

  • An airline learned passengers were 4x angrier about missed connections than delayed flights.

The point Medelyan made is we usually get deeper, more actionable insights when we analyze the comments and not just the scores. So I applied this concept to my book cover survey and found two significant themes contained in the comments.

The first was quite a few people chose B because they liked the subtitle below the title better than the way it was shown in option A and C. So it wasn't just the color that's drove people to option B.

The second theme was quite a few people who selected option B mentioned they liked the title arrangement of option B, but preferred the color of option A. There were even a handful who picked B but mentioned they liked the color on option C best.

Suddenly option B isn't such a clear and convincing winner. Here's what happened when I revised the survey results to account for color choice alone:

Option A (40%)

Option B (39%)

Option C (21%)

Now I have two insights:

  • People prefer the blue cover shown option A

  • People like the title arrangement in option B

Keep in mind I only made adjustments where respondents were explicit in their survey comments. If someone didn't explain why they chose B, they may have done it for the title arrangement, the color, or pure whimsy.

Making a Final Decision

I did a similar survey with my last two book covers, and both times I ended up choosing elements from different options. I did the same thing this time.

Going with option B's title arrangement was a pretty easy decision. There were numerous comments describing option B as the preference without any support for the layout of options A and C.

I ultimately chose the blue color from option A. 

Several survey comments mentioned color theory, and my friend Jim even shared this helpful resource from Quick Sprout. According to the guide, the color blue symbolizes tranquilty and peace and has more positive associations across various cultures than purple and green.

The kicker is the blue is my personal preference. I really like it, and it's important for an author to really like the cover of their book! Here's the final cover:

It was also important to consider how the cover will look when shown together with my other books on Amazon, in a bookstore, or at a trade show. Here's how it will look displayed next to my other books:

Take Action

You can gain so much more from a survey if you combine the fixed choices (ex: option A, B, or C) with comments. Try analyzing one of your own surveys to see what hidden insight is revealed.

You’ll find a lot of simple analysis techniques in the webinar with Alyona Medelyan from Thematic.

You can also get more help with your survey on this survey resource page.