Way of the Retainer

I don’t claim to be a Top Gun consultant by any stretch of the imagination. Sadly, the Val Kilmer of IT consulting (whoever that might be) is unlikely to offer to let me ride his tail any time. (seriously, how could you deliver that dialogue with a straight face? Even in the 80s?)

But, I have gotten a lot of wins on client site by following one deceptively simple rule: read the log files. When strange things are happening intermittently, the first and best port of call is the log files for the system you’re debugging. Now, I don’t claim to be a genius in having discovered this rule. But it’s somewhere between surprising and depressing the number of times I’ve worked with really smart people and get a blank stare when I’ve asked ‘what do the log files say?’

And it’s wisdom I’ve come by the hard way myself. One of my first jobs was installing a HRMS system on client sites, and while troubleshooting a major hiccup with an install on client site, I was on the phone to our guru in the office. He asked if I’d checked the log files. I realised… I didn’t even know where the log files were. It turned out, if I’d checked them, I would have saved myself and the client a lot of grief, including a day’s work restoring from an old backup.

Log files are the forensic evidence of technical troubleshooting. And if we want to be the Grissoms of our own CSI, then there’s a few pointers it’s worth following.

Know where your log files are

It might sound like an obvious rule of thumb, but cases often have more than one crime scene, and the most important one might not be particularly obvious. You might be driving requests to an application through an interface that’s not giving you much more than an ‘Error occurred’, but chances are a log file somewhere is dutifully scribing reams of text about the problem that’s occurring.

For JEE applications, for example, you’ve got your default server log file, and application-specific log files at the very least. In the case of WebLogic Server, you’ve got your domain log file as well, and if you’re having problems with your back-end database, then the real answer might be in XA trace logs on your database server.

Once you’ve established where your log files are, understand where you can go to get additional context on problems that are occurring, as well. If you’re fortunate enough to have a unique message id associated with the error that’s occurring, understand where to find the specific documentation for your product stack so you can get more information on that specific error. And if that doesn’t give you a satisfactory answer, your next step might be a good solid Googling.

Don’t search on text strings

If there’s one thing my years of intensive crime show watching have taught me, it’s that forensics are all about understanding the problem that’s there, not the one you think you have. Almost invariably, if you search on a text string to try and troubleshoot, you’ll miss something.

A bit part of troubleshooting is understanding specific problems or errors in the context of anything else that might be happening at runtime, as well. So searching for keywords or text strings will catapult you past pages of logging that may contain that vital clue that will crack the case.

Look for patterns

Individual error messages by themselves may not mean much, but combining error messages and application behaviour can be a telling sign. It could well be that a specific server-side component is causing database connection leaks. Or, maybe it’s accessing a particular page in your web application that is causing all of those JNI crashes.

Understanding the patterns of behaviour in an application (if you’re lucky enough to have several days of log file information to debug from) can be crucial to finding the core problem.

Know how to turn on the fire hose

Once you’ve gotten a fairly clear idea of what the problem is, knowing how to configure more verbose levels of debug information can be very important. Keeping a reference handy on the debug switches supported for the product you’re troubleshooting is never a bad idea. In the case of WebLogic Server, for example, from WebLogic Server 9.0 onward, a number of configurable debug switches can be selected from inside the Administration Console. Handy! (it’s important to note though that these switches are far from the complete list you can use)

I’d like to talk about some rules of thumb that I think are good to follow to make a log file to be proud of as an application developer, and also how to troubleshoot some specific technologies, but that’s enough for now. I’ll come back to the other stuff.

Got any log file experiences? Bizarre error messages involving processes being sent insane by zombie children or the like? Times where you’ve saved the day and come out a hero, all thanks to a few choice log file messages? Share them in the comments, or just drop me a line and let me know what you think!

1 Comment »

Hello! Once again, profuse apologies for the silence. Moving to Perth, technical issues on the project I’m working on, and a crunch for deadlines for one of the products I’m involved in developing has meant that things have been a little disrupted for a while. But I’ve been thinking big thoughts… mostly about Tuxedo and ALSB at the moment. They’re a rarified taste, but I’ll try to pass on some of what I’ve been dealing with in time.

For the moment though… now that I’ve found my notebook again, here’s a post I’ve had queued for a little while now and have been meaning to get around to.

---

This is a much shorter post than the lengthy screeds on the material and delivery aspects of training, for three key reasons.

• One, I’m trying to learn my lesson about long posts.
• Two, I have less to say about this aspect of training delivery.
• Three, admittedly it’s the aspect of training you have the least amount of control over.

Fundamentally, a good training venue is one that can keep the trainer mobile, and the students awake. And, given that keeping the instructor mobile is about keeping students awake, it fundamentally boils down to keeping the students awake. The rest largely takes care of itself.

Air conditioning and lighting

There’s no greater recipe for sending students off to sleep than a warm, dimly lit room. Except maybe if you handed out pillows first. A good training venue should have reliable air conditioning. Without creating a welcoming environment for the penguins, err on the side of cooler. Also, bright lighting helps it feel less like Sleepytown.

Snack time!

It’s understandable that training venues often don’t allow students to eat food, given the mess it makes. But, even with the possibility of spills, good training venues let students (and the trainer) have drinks in the room. Water helps concentration, and coffee and tea helps students (and you as the trainer) fight through the tiredness as the week draws on.

Also, while sugar has a crash to go with its rush, the better training venues provide bowls of lollies for students to nibble on. Lollies like Minties or Fantales give students sugar to keep them awake, something to do besides listening, and something to eat that doesn’t make a mess.

Remote mouse and projector

If you can stay mobile as a trainer, you’ll be more awake and alert, and you’ll be able to be more engaging and dynamic in your delivery as a result. Being able to move around the room lets you interact with students directly – to be more dramatic, and vary how your delivery is physically arranged. But that requires two things.

First and foremost, the venue should use a projector to display the slides being talked to at the front of the room. That way, you can address the course material while still moving about. A laser pointer can help here as well, but that’s a matter of personal preference.

Second, some sort of remote to be able to interact with the PC displaying the slideware. I’ve used both a radio mouse and a laptop remote to good effect in the past. Being able to interact with the slideware remotely stops you having to continually dash back to press a key to move between slides – it lets your presentation flow more smoothly.

The other big bonus from having the slides projected to the front of the room is that it gives everyone a shared, central focal point, rather than staring at individual monitors.

Internet-capable, well-spaced PCs

The student’s PCs should, in the interests of comfort, be spaced far enough apart that students aren’t cramped together. A gap the size of a monitor between student’s monitors is a good starting rule.

Also, and this is a bit of a divided topic, the students should have internet access, and be able to use their PCs while you’re delivering training. It can mean that student’s minds will wander the internet rather than listen to you, but students are more likely to stay awake if they’re not forced into a passive role. I’ve delivered training for a partner where the students PCs were turned into passive broadcasters of the training material while the trainer is presenting, and it’s not great. It makes staying focused and awake more difficult for students, not to mention preventing them from tinkering or playing catchup with exercises.

Well, that’s it for thoughts on training venues, and training overall. Hope you’ve enjoyed the posts, as slow as they’ve been to arrive. Here’s to a better posting schedule in 2009 now that I’m settled down!

No Comments »

I will get back to looking at training venues in short order, but I wanted to talk about customer service, based on some experiences on the weekend.

I had what you might call an “interesting” weekend. After flying back from Perth to Melbourne and getting home at ten past midnight, I couldn’t get into my apartment building. The security keypad was completely dead at the front door. After leaving a message with my real estate agent (that wasn’t returned, despite the “urgent” setting on the voice mail) I had no other option but to check into the hotel next door.

I spent most of the next day getting hand-balled between my real estate agent and the body corporate for my building, driven almost to tears with frustration and still unable to get into my own apartment thanks to the absence of a miracle key. Based on the experience, I wouldn’t recommend either my body corporate or real estate agent as great businesses.

Situations like these – adverse circumstances where there are questions around who is responsible for what, are great opportunities to win over customers. Personally, all I’m ever looking for in these situations is the exact same thing I try to provide our customers professionally: an advocate who is apologetic, sympathetic and will help get a solution to the situation. Someone who’s on your side.

Maybe it comes out of working in the integration space, where as the glue between applications you’re most often guilty until proven innocent. But I don’t think that’s quite it. I was talking with my partner while all of this was happening, and while we work in vastly different fields, we have very similar views on commitment to quality service.

We were boggling at the time at the language people were using over the phone – not our responsibility, not part of our service, not going to do that for another hour or so. People spent more time telling me what they weren’t going to do for me, rather than what they could do for me. That, and pointing all care and responsibility at the other party.

The problem was solved eventually, but not before a slew of phone calls ensuring that both parties would actually do something to solve the issue rather than ignore it. And I’m not looking forward to the bugfight ahead to get my hotel bill sorted.

I could go into a myriad of tiny details about the incompetence and indifference I pushed through just to be able to get into my apartment, but that’s not the point. The point is, next time you have a disgruntled customer to deal with, while you may have no control over what’s happened up to that point, you have complete control over what happens next. People remember customer service that goes above and beyond the expected norm, and there is no better opportunity to create an advocate for your company or service.

It’d be nice to think in our Kevin Rudd / Barack Obama world, we can be better than hand-balling problems to avoid culpability.

So take that opportunity to be extraordinary, and sell your company on amazing quality of service.

No Comments »