The Kneed for The Knack

The following was originally posted on the Avaya site here.

In a product support organization, your people are, without a doubt, your most critical resources. While you can use knowledge bases and tools to prevent re-discovery, new problems will always arise and you need engineers who can quickly get to a resolution. The skill I value the most in engineers is their ability to troubleshoot. By that I do not mean their ability to recall from memory what an Error 529 is on a specific product (that’s what knowledge bases are for); rather I’m referring to their ability to look at a technical problem, narrow down possible root causes, and logically find the solution. At first glance this may be confused with basic critical thinking, but it is something beyond critical thinking; it’s The Knack.

There is no better way for me to convey the concept of The Knack then to play the following embedded Dilbert cartoon. If you’re short on time, just watch the first two minutes. (The rest of this episode can be found here and here.)

Like most humor, the writers of that comic took a truth and moved it to an extreme for a laugh. I’ve yet to meet an engineer that could sense a battery going bad in a TV remote, but I have met many that can stare at a problem and solve it without any direct experience. For many people, changing the time on a car radio can be irritatingly complex without the manual for reference. An engineer with The Knack, howeverunderstands the design limitations of the user interface of the clock, has used one before, and can surmise the proper steps in almost any car.

Several years ago, I was involved with a critical outage of a web-based application experiencing a timeout issue. An engineer was brought into the escalation that had no experience with this solution at all. He asked a series of questions to get a mental image of the architecture similar to the one below:

He reasoned (correctly so), that if we submitted a unique series of letters at the website, searched the logs on each server for that phrase, and noted the timestamps of the relevant log messages, we could determine which component was adding the delay and tripping the timeout. Once identified, we could get the relevant expert on the phone and get the system back up within minutes. This engineer didn’t get caught up in the reality that he didn’t know how this solution was written nor the underlying technologies well enough to troubleshoot any of them in great detail. Instead, just like with the clock, he combined solid critical thinking with a working knowledge of web architecture to quickly narrow down the possibilities.

While this diagnostic skill has always been respected, it is getting more and more valuable as technology evolves and become more intricate. While new solutions bring improved value to users, the underlying technology that makes those products work is becoming more complex to troubleshoot.

Ten years ago, enterprise phone systems were analog and digital. The phones had their own wiring that connected them directly to the PBX, creating a contained environment where all diagnostic information was in one place. With modern SIP solutions, the phones have evolved to be multi-functional, offering complex features including presence and video. To make this possible, the infrastructure has become distributed, making the diagnostic work distributed as well. To trace a single SIP call requires diagnostic information from several servers and endpoints. The Knack – this art of troubleshooting – is needed now more than ever.

At Avaya, I’ve had the great pleasure of working with not only individuals, but entire teams of engineers blessed with The Knack. Watching one of them attack a problem is like seeing an artist paint. They use the same paintbrushes and paints as others, but the way they wield them makes them true artists. Having one of these engineers working a support issue can mean the difference between resolution times measured in hours versus weeks. While we have amazing automated tools and knowledge bases (ex. Avaya Mentor), it is these talented engineers who bring the most value to an Avaya maintenance contract.

In my next post, I’ll write about how to identify engineers with The Knack as well as how to nurture this skill within your organization. Until then, remember that with great power comes great responsibility, and in this case, often social ineptitude.