Why Gemba is the Best Way to Solve Service Failures

Advertising disclosure: We are a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for us to earn fees by linking to Amazon.com and affiliated sites.

The CEO called me with an urgent training project.

Our parking management firm was in danger of losing an important contract at a hotel where we managed the valet and self-parking operation. The client was unhappy about poor service quality and gave us thirty days to improve.

The CEO told me to go to the hotel and train the staff and the manager. He wanted me to show them how to deliver service the right way, and then make sure they did it. This was going to be my priority for the next 30 days.

I decided to meet the parking manager and take a gemba walk. It was fortunate that I did, because it quickly became clear that training was not the problem.

A valet parking attendant is opening the car door for a guest.

What is a gemba walk?

A gemba walk involves going to where the work is done and observing it first-hand. It requires you to approach the situation with an open mind and ask questions to gain a better understanding of how people do the work and why they do it the way they do.

The term gemba (or genba) is a Japanese word that means "the actual place." It's a principle closely associated with lean manufacturing, but I've always found it to be a great way to diagnose service failures.

My gemba walk with the hotel parking manager was revealing.

Our client, the hotel's general manager, was unhappy because our valets frequently failed service audits conducted by mystery shoppers. These were auditors who posed as guests and evaluated the hotel on a comprehensive list of service measures. 

Graph showing actual mystery shopper audit score of 78% compared to a goal of 85%.

The mission was to find out why our valets were failing the audits.

I spent about an hour with the manager reviewing the valet operation. Unlike a mystery shopper, a gemba walk involves directly observing the work and talking to the people doing it.

  • We watched the valets serve guests.

  • I asked questions to learn about what they were doing and why.

  • We walked through the entire operation, include back-of-the-house areas.

This was just one shift, and the hotel was a 24/7 operation. I came back several times on nights and weekends to observe other valets in action and did another gemba walk with the supervisor who reported to the manager. 

We quickly discovered the root cause of the service issues.

What is the purpose of a gemba walk?

A gemba walk allows you to see insights that might otherwise be hidden. Leaders can be misled by data and easily jump to the wrong conclusions without seeing the full picture.

Remember the call from my CEO?

He had assumed the problem was training. Using this guide for diagnosing training issues, I knew that employees need training when they lack one of three things necessary to do their job:

  • Knowledge

  • Skill

  • Ability

The gemba walk with the parking manager quickly revealed the valets had the knowledge, skill, and ability to do their jobs. None of this was guesswork. 

  • I observed the valets providing excellent service to guests when the front drive was busy.

  • I observed them getting bored when the hotel's front drive was slow.

  • The valets told me exactly why they got bored and goofed off during slow times.

So it wasn't a training issue.

The problem was they didn't do the job consistently. The valets got bored when work was slow, lost focus, and started goofing off. They were also missing some critical information:

  • Mystery shopper reports weren’t shared with the team.

  • The valets didn’t realize the contract, and their jobs, were at risk.

I didn't have to suggest the solutions. The valets came up with some on their own, and the manager created a simple, but brilliant, plan to tie it all together.

When should you do a gemba walk?

A gemba walk is useful whenever you need to identify the root cause of a problem. There are a number of benefits to going directly to where the work is being done.

  • Test assumptions

  • Verify procedures are being followed (often, they aren't)

  • Talk to the people actually doing the work

Observing the work being done is one of the best quick fixes for solving performance challenges of any kind.

Throughout my career as a trainer, customer service manger, and a consultant, I've often seen gemba walks lead to very different conclusions than the initial diagnosis:

  • A "problem employee" was actually being victimized by a toxic coworker.

  • An incentive program designed to improve service made service worse.

  • A "short-staffed" team improved productivity by 25 percent without adding staff.

Gemba walks can also help you identify customer service icebergs.

An iceberg looks like a small issue on the surface, but a much larger and dangerous problem is hidden below the surface. For example, when the pages fell out of one of my books, I investigated the problem and discovered thousands of defective books had been shipped to retailers.

The author holding up a defective copy of his book, Service Failure.

How to do a customer service gemba walk

There are a few techniques that can make your gemba walk successful. Do a little bit of upfront planning, ask questions to approach the work with an open mind, and show respect to the employees doing the work.

Planning for a Gemba Walk

You don't need to do a lot of planning to prepare for a gemba walk, but a few simple steps will make the process much more useful.

  1. Clearly identify the objective. What are you trying to discover?

  2. Let people know you're coming.

These steps will help you get the cooperation and buy-in from the employees you observe. You'll learn a lot more, and get more forthright cooperation, if you avoid coming across as someone who is merely there to catch people doing it wrong.

I did a few things to prepare for my visit to the hotel.

First, I asked the CEO to let the president of the hotel division know what I would be doing, and that he had requested it. The president was a very hands-off leader, but I also knew he could easily get defensive. After all, the CEO was coming to me because the hotel division president had failed.

Second, I called the hotel parking manager. Fortunately, we already had a good relationship, so I was very candid about my project. I knew the contract was in jeopardy and it was my goal to help him save it.

Finally, I reviewed the mystery shopping audits. I wanted to make sure the mystery shoppers were looking at the same service standards we were training our valets to perform. (They were.)

Ask Questions

Keep and open mind and ask questions to reveal insights that you might otherwise miss. Resist jumping to conclusions. Even if you see an employee doing something wrong, asking why they're doing it can be revealing.

I asked a lot of questions when I observed the hotel valets. I even asked them why they were goofing off when I saw them get bored and start to stand in a circle and talk to each other. It wasn't an accusatory question— I really wanted to know.

The valets were very forthcoming about the reasons for this. The valets found it hard to stay focused and alert when nothing was happening. Most were young and inexperienced, and they enjoyed an easy camaraderie with each other, so goofing off was almost second nature.

They also had some suggestions for improvement.

  • Provide small tasks they could do in between guests.

  • Rotate positions during slow times to reduce boredom.

  • Share the results of the mystery shopper audits.

That last point was key.

The manager hadn't been sharing the audit reports with the valets. They knew the hotel's general manager was unhappy, but they had no idea the contract was at risk. And they didn't realize that losing the contract would mean losing their jobs.

Show Respect

Employees will generally be candid about how they do their job if you ask honest questions with an intention to help. Keep in mind that you're there to help them, not catch them doing something wrong.

With the hotel valets, I was careful not to come off as some corporate guy who was there to catch them doing wrong. I tried to convey to each one that I appreciated the work they were doing and wanted to help.

It was also important to show respect to the manager and the supervisor. Once we discovered the valets needed more information about the mystery shopper audits, I asked the manager what he thought could be done.

His idea was brilliant.

Additional Resources

Here are a few resources to help you plan your first gemba walk.

This short video provides some nice visual examples.


The CEO requested training, but I didn’t do any.

What I did was work with the valets, the valet manager, and the valet supervisor to understand the root cause of the problems. I then facilitated their ideas for improving service and keeping the contract.

The valets had made several suggestions for improvement. The valet manager tied it all together with a simple tactic.

He cleared a bulletin board in the parking office and mounted a piece of string horizontally across the board. Then he put a sign on the string that read "85%" to represent the target score for mystery shopper reports.

The manager began posting each mystery shopper report on the board as it came in. 

  • If it passed, it went above the string.

  • If it scored below 85 percent, he posted it below the string.

The valets immediately got the message. 

Nobody wanted to let the team down and fail an audit. They encouraged each other to stay sharp and implemented their ideas. The manager gave praise and recognition with each passing audit, and offered coaching each time an audit was failed.

The hotel's general manager was very happy with the results by the end of the month.

Graph showing the improvement of mystery shopper audit scores after one month.

My CEO was happy, too. He didn’t really care whether or not we did training. His goal was to save the contract, which is exactly what the gemba walk helped us do.

Cover image of Getting Service Right book.

In my book, Getting Service Right, I detail a number of service failures where the solution wasn't immediately obvious. The book also captures candid responses from employees:

  • Why an employee lied to customers.

  • Why an employee deliberately provided poor service.

  • What an employee really wanted to do when confronted by an angry customer.

Finding the solution to these problems often requires a gemba walk.

How Unresolved Service Failures Can Cost You Big Time

Sally's Mom loved the bouquet ordered through ProFlowers.

Sally's Mom loved the bouquet ordered through ProFlowers.

Service failures happen.

They’re unfortunate. Many are preventable. But make no mistake, they happen to every business. After all, we’re human.

Unresolved service failures are something else. They’re annoying and unnecessary. Continuing to fail at resolving a problem shows customers you don’t care, you can’t handle your business, or both.

Service failures also get far more expensive the longer they take to resolve. A 2013 Zendesk white paper estimated that waiting 48 hours to fix a problem costs companies 66 percent more than if they had solved it in just one business day. Way back in 2008, I wrote a post detailing how recovery costs rose rapidly when there wasn’t a quick solution. The longer it takes you to fix something, the worse it gets.

FTD recently provided another example. 

The costs of their service failure will be hard to spot on their profit and lost statement. Nobody in senior management may ever even notice.

That’s too bad for them. Hopefully, this post will ensure it doesn’t happen to you.


FTD's Service Failure

My wife, Sally, ordered some flowers via FTD’s website for her Mom. Her Mom was staying in a hotel for a few days and Sally thought the flowers would bring some extra cheer to her room. She paid extra for Sunday delivery so they’d be there when her Mom arrived.

An automated confirmation email arrived on Sunday signaling the flowers had been delivered.

Except they weren’t. That was service failure number one. It got worse. 

Sally called on Monday morning to see if the flowers could be delivered that day. She was told that someone would have to do some research and call back. No one ever called. That was service failure number two.

Sally called again late Monday afternoon. She still couldn’t get an update on the status of her delivery, so she asked to speak with a supervisor. The supervisor couldn’t guarantee the flowers would even be delivered on Tuesday. He was completely un-empathetic and did not apologize. 

That was service failure number three. Three strikes and your out. 


How it Cost FTD Money

This service failure caused FTD to lose money in a number of ways:

The first was labor. Two customer service representatives and a supervisor spent time talking to Sally on Monday about FTD’s service failure.

The second was revenue. Sally cancelled the order.

The third was a customer. Sally spends an estimated $300 a year with FTD. Now, she’ll spend that $300 with one of FTD’s competitors.

The fourth is goodwill. I wouldn’t take the time to write this blog post if FTD had fixed their error. It was their inability to correct a service failure that inspired me to write.

It may seem like FTD hasn’t lost too much if Sally takes her business somewhere else. After all, she’s just one customer.

But, what if this situation is just the tip of the iceberg?

Companies like FTD don’t spend much time looking for icebergs. They should. The cost of losing one customer like Sally might be small. But like an iceberg, what lurks beneath the surface might be bigger and more dangerous. 

How many customers has FTD lost because of unresolved service failures? 


ProFlowers Nailed It

Sally didn’t want to do business with FTD, but she still wanted to send some flowers to her Mom. She decided to give ProFlowers a try.

They delivered a beautiful bouquet without a hitch. Sally’s Mom loved it. 

ProFlowers earned Sally’s future business in the process. They did what they promised they’d do. No drama, no issues, just service. 

And, next to FTD, they look like rockstars. ProFlowers delivered flowers faster than FTD could issue Sally a refund.

Live Experiment: A breakthrough with Whirlpool?

I think I finally have a resolution after contacting Whirlpool 16 times to update an expired credit card. And, I've also confirmed my suspicions that there was a broken link in the chain. As I've written before, your service is only as good as the weakest link in the chain.


My wife and I had a subscription where Whirlpool automatically sends out a new water filter for our refrigerator every six months and bills the credit card they have on file. Our credit card recently expired, but so far we've been unable to give Whirlpool the updated information. 


Yesterday, I exchanged direct messages on Twitter with Chris, a Whirlpool employee who monitors their customer service Twitter feed @WhirlpoolCare. This led to a phone call where I explained that between my wife and I, we had now contacted Whirlpool 16 times in an effort to update an expired credit card. Chris listened, apologized, but like everyone else we had interacted with, he told me he was unable to help me. However, unlike everyone else so far, Chris offered an alternative solution and explained why he was unable to fix my expired credit card (more on the credit card in a moment).

The alternative we agreed upon was that Chris would send us a complimentary water filter as a gesture of goodwill. It would then be up to me to re-establish a new online account with my updated credit card as a workaround to the problem. (I could also find an alternative source for the water filter.)

This is huge because, as I explained to Chris, I have a house full of Whirlpool appliances. Before this incident, I wouldn't consider another brand. Now, I wouldn't consider Whirlpool unless this was resolved. If the filter arrives as promised I'll consider Whirlpool back on my list of preferred appliance brands (their appliances are really, really good).

The Broken Link


Chris also revealed the broken link in their chain. The water filters are fulfilled by a third party, so Whirlpool customer service employees have no access to that company's fulfillment system. The only tool they are given is the instructions on using the website that they can relay to customers. This explains why each customer service representative we've encountered has been unable to help. Apparently, the system's designers never imagined the system could break so there were no contingency plans for handling this sort of situation.

Unanswered Questions

I didn't want to press my luck by asking Chris too many question since I was his last customer of the day and he had stayed a little late to talk to me. My top priority was getting a resolution and I had that now. However, there are a few unanswered questions that could be instructive.

What is the escalation procedure? If a system is broken, someone should be able to escalate. Why couldn't (or wouldn't) Whirlpool's customer service employees escalate this issue to someone who was empowered to fix it?

Where is the process broken? The specific problem was technical, but was it on Whirlpool's end, the fulfillment company's end, or both? When two parties encounter a problem, the instinct is often to point the finger at the other party, which means nothing gets resolved.

What's the full impact? I have to imagine my wife and I aren't the only ones to experience this problem. Is this problem really an iceberg? In other words, how much business is Whirlpool losing due to situations like ours?

Jeff Toister is the author of Service Failure: The Real Reasons Employees Struggle with Customer Service and What You Can Do About It. The book is available in paperbook, e-book, and audio book formats.

You can learn more about the book at www.servicefailurebook.com or purchase a copy online at AmazonBarnes & Noble, or Powell's Books.

Service recovery from Heitz Cellars

Earlier this week, I wrote a post about three wineries that all handled a missing or delayed wine shipment in different ways. (See Good, Bad, and Ugly ways to handle the same problem.) Since then, Heitz Cellars has made a bit of recovery.

Heitz Cellars was my "ugly" example in the post because I had called three times to check the status of some missing wine and they had short shipped my order twice. Yesterday, the last two missing bottles finally arrived. The modest recovery came from the refund they issued to my credit card. This means the end result was I finally had my delicious wine ('07 Zinfandel) and I didn't have to pay for it. Heitz Cellars makes some terrific wine and this gesture was enough to keep me as a customer.

This also serves as another installment in my collection of stories that prove the longer you take to solve a customer service problem, the more expensive recovery will be.

Related posts on expensive service recovery:



Unexpected customer answers reveal "moments of truth"

There are certain stock phrases used so often in customer service situations that they've almost lost all meaning. They've become perfunctory and the responses they illicit from customers almost seem scripted. "How are you today?" asks the customer service rep. The answer, of course, is "I'm fine."

But what happens when the customer goes off script and says something unexpected? You can earn an "A" for service if you are recognize these moments of truth and are ready for a little improv. On the other hand, you might get a "C" or even an "F" if you don't seize the moment.

Here are some examples:

Did you find everything OK?
Expected Answer: "Yes"
Moment of Truth: "No". My wife, Sally, got this one at the bookstore last night. She was looking for some note cards but didn't find anything she liked. She bought a few other items and the associate at the cash register asked her "Did you find everything OK?" Sally said "No", but the associate missed the opportunity to make some suggestions. She simply ignored the response and continued the transaction.

How is your stay at our hotel so far?
Expected answer: "Good"
Moment of truth:"It's OK." There's a subtle difference here, but a savvy hotel associate will catch it and take action. I once gave this answer to a hotel associate while sharing an elevator. Instead of following up with "How can I make your stay better?" it got uncomfortably quiet until we got to her floor and she quickly exited the elevator.

How are you today?
Expected answer: "I'm fine"
Moment of truth: "I'm terrible". I must admit I dropped the ball on this one when I was a teenager working in a retail clothing store. I really didn't know what to say until the customer, seeing my surprised look, followed up with, "Well, you asked!" Yes, I did ask, but I also realized I hadn't cared what the answer was. From that point forward, I was ready for those moments of truth and knew to respond with, "I'm sorry to hear that -- what can I do to make your day better?"

What are your moments of truth?

Frame-up! The exciting conclusion...

And now, the exciting conclusion to my incredible misadventures getting some frames made at Aaron Brothers. Special thanks go out to all the people who left comments (mostly on Facebook) wishing me well and especially my friend Marjorie, who wondered what I might have done to deserve bad customer service karma.

The big day

In action movies there's often a montage scene where the hero is shown training for that inevitable big moment. I imagined a montage all last week where Mr. Clean-up was busy building my frames while "Eye of the Tiger" or some equally corny rock anthem was playing in the background. At first, it was very difficult for him and he almost quit. But, through determination and hard work, he got better and better at it until finally he was ready to take on the forces of poor customer service and deliver my order.


My phone rang on Friday afternoon. It was Mr. Clean-up. I held my breath.

"Your frames are all ready for pick-up," said Mr. Clean-up. I could hardly believe it. Are you sure? "Yes - they're ready to go. We close at 9, so you can pick them up this evening if you'd like."

Plot twist?

My wife, Sally, and I headed down to Aaron Brothers later that night to pick up our frames. In the back of my mind, I was anticipating a plot twist. The kind where you think the super villain has been vanquished, but suddenly he comes back to life for one last-ditch attempt at chaos. When we walked in, we were greeted by the Ball Dropper, that fiendish customer service villain who shirks responsibility and never seems to get things dong. Aaaaarrgh!

Bracing ourselves for the inevitable conflict, we were surprised to find the Ball Dropper contrite and eager to please. He brought out our frames (they were all great) and apologized for the hassle. A few minutes later we were walking out of the store with our frames in hand. Somehow, this really was the happy ending we had hoped for.

Frame-up! Service hero, villains, and not-so-innocent bystanders.

My favorite framing store in San Diego has evidently disappeared, so my wife (Sally) and I recently decided to give Aaron Brothers another chance. We'd received a lot of disinterested service from Aaron Brothers in the past, but they have a store near our house and we didn't feel like doing too much searching for a new place to get a couple of prints framed. Little did we know our adventure would read like a customer service comic book, complete with a cliffhanger ending!

The Villains

The Ball Dropper
The Ball Dropper shirks responsibility. Sometimes intentionally and sometimes just because, well, he's the Ball Dropper. That's his stupor-power -- not getting things done. In our case, the Ball Dropper struck twice. First, he mis-measured the matting for our frames, causing a delay since it resulted in the last piece of that matting being cut to the wrong dimensions. Zap! Pow! Backorder! The second instance was worse -- he neglected to call us and let us know there would be a delay.  Aaaargh!

Apathy Girl
We dropped by Aaron Brothers on Sunday to buy some additional frames and check on our prints. They were due on Monday, so we figured we'd see if they were ready a day early. That's when Apathy Girl materialized and informed us there had been a delay and our prints wouldn't be ready on Monday as planned because the evil Ball Dropper and bungled the order.

Apathy Girl's favorite phrases are "I don't know", "That's not my job," and "That sucks for you." The last phrase sometimes sounds like, "I'm sorry", but that's because she has a thick Apathetic accent. She's really saying, "That sucks for you."

In this case, Apathy Girl told us she didn't know when our order would be ready. We wanted to order an additional frame to match the first one, but she told us it was out of stock. She also didn't know when it would be in (maybe February?!). We asked if we could pick out a similar frame and get it all done by Friday since the original order was delayed. Apathy Girl didn't know. The new frame was also more expensive, so we asked if they would give it to us at a discount since we were inconvenienced. Apathy Girl didn't know that either. "I only work here one day a week," she said.

Apathy Girl's evil forces were so powerful that Sally and I decided to give up on Aaron Brothers for this frame job. We took our prints, got a refund, trudged out of the store, and then shook our fists at the sky. (Shaking your fist at the sky is what you do when you feel powerless because an Evil Customer Service Villain took advantage of you.)

Not-So-Innocent Bystander

The store manager witnessed all of Apathy Girl's show. She didn't say or do anything. Perhaps it was because she was assisting another customer and didn't want to cause a scene. Perhaps it was because she felt powerless to stop the awesome power of the evil super villain Apathy Girl. Or, perhaps she was secretly Apathy Woman and Apathy Girl was her prodigy. Whatever the reason, she didn't do much store managing on this day.

The Super Hero

Mr. Clean-up
On Monday, the date when our framed prints were originally due, I received a phone call from a Service Hero, Mr. Clean-up. He told me he was the framer at Aaron Brothers and was surprised to come into work to find our order had been cancelled. Mr. Clean-up was calling to see if he could do anything to win back our business or at least find out what went wrong. I told him our story.

"Ahhhh, the Ball Dropper and Apathy Girl strike again!" Mr. Clean-up then offered to do what he does best - clean-up a bad situation.

I asked Mr. Clean-up if he could frame our original prints plus make the additional frame we wanted at a discount, and get it all done by Friday. He explained that was a tall order because he had to get permission from the store manager (who might very well be Apathy Woman), but he would see what he could do. Could he give me a call back by Tuesday and let me know? Sure.

I can only imagine what happened next as Mr. Clean-up took on the evil forces of indifference and poor customer service that were part of the Aaron Brothers company culture.

Biff! Pow! Boom!

The rest of the story...

Mr. Clean-up left a message for me yesterday - we got the deal! All I had to do was bring the prints back in that day and everything would be ready by Friday. At a discount. I grabbed the prints and hustled out the door as soon as I got the message.

When I got to Aaron Brothers a few minutes later I was told Mr. Clean-up had already left for the night. (Has anyone actually seen Mr. Clean-up?!) However, another employee offered to help me. Unfortunately, I could tell she was another not-so-innocent bystander who was lured by the call of Apathy. "Mr. Clean-up has already left for the night." And, "We don't usually do that." And, "I couldn't guarantee it would be done by Friday." Aaaaargh!

I felt the urge to shake my fist at the sky again, but I had one last card to play. "Let's call Mr. Clean-up and see what he has to say about it," I said. She agreed and made the call. I only heard her end of the conversation:

"Uh huh. Uh huh. Well, we don't normally do.. Oh, I see.  Oh, OK. OK."

She came back to the counter, smiling. "We can do it all by Friday and give you that discount. Mr. Clean-up says it's OK."

Cliffhanger Ending...

I left Aaron Brothers last night feeling confident that I'd get my order by Friday. Or will I? Will Mr. Clean-up finish the job? Or, will the Ball Dropper intervene and drop the ball? Will Apathy Girl swoop in on her day off (she only works Sundays, remember?) and spread apathy across the store? I won't know until Friday, but I'm excited!



Avis service recovery - he DID try harder (eventually)

I had an interesting experience renting a car from Avis yesterday. Initially, it seemed like just another example of poor service.  At the end, I was pleasantly surprised to see the agent make an attempt at recovery.

This situation was a "moment of truth" because I had just signed up as an Avis "Preferred" member where I could skip the rental counter and go straight to my car.  Unfortunately, my name wasn't on the "preferred" list when I got to the rental car area, so I was a bit deflated. I noticed an "Avis Preferred" office, so I decided to go in and see if they could get my rental straightened out.

The agent who helped me definitely didn't live up to the "we try harder" motto.  He tried every trick in the book to make me go away. "You'll need to go to the rental counter." (Well, can't you at least look up my reservation?) "I'll have to stop serving you if a Preferred member comes in." (Aren't I a Preferred member?) "You're not in the program until you get your card in the mail - it takes 2 to 3 weeks." (I DID get my card in the mail, now what?)  "You'll have to call customer service." (I'll HAVE to call?! Can't you help me?)

Finally, after trying everything he could to get me to go away, the agent found my reservation in the system, verified I WAS a Preferred member, and printed out my contract. No apology was offered, but at least I had my rental car and I didn't have to go all the way back to the rental counter.

The surprise happened when I got in the car and started exiting the rental car garage.  I saw the agent leave the office and motion to me to roll down my window.  I stopped and he said with a smile, "I think I figured out why you weren't on the Preferred list!" He went on to explain that since my reservation was made before I became a Preferred member, the reservation didn't automatically upgrade to Preferred status. He assured me that I would be all set going forward with any new reservation.

All told, it wasn't a great experience, but I did appreciate the agent trying to turn things after our initial encounter. To me, that's the true lesson.  We'll all have moments when we say the wrong thing or drop the ball in some way.  What matters most is what we do after that moment has passed.