Way of the Retainer

Things have been a little quiet around here, my dear imaginary audience, and for that I apologise profusely. I’ve been finding most of my spare time taken over by a pet programming project which I’m perched on the edge of revealing and quite excited about, despite its tiny scope. (more on that soon, I promise!)

As I find myself for the first time in quite a long time on a plane without the girl as company, it seemed high time to furnish you with a blog post. Post-production has taken a LOT longer than I’d hoped in making this available post-flight, but here we are finally.

My last post was about the general principle of reading the log files, so I thought I’d jump into specifics about where to find useful information with Tuxedo. I had intended this to be one complete post, but due to the wonderful fractal quality of explaining things, I’ll need to break this up into a series.

For those of you unfamiliar with Tuxedo, this post is unlikely to have much interest or benefit. Before we possibly part ways though, I’ll take a moment to explain exactly what Tuxedo is, as it’s somewhat rarified knowledge nowadays, but the market seems to be pushing to make it and similar products popular again. From my less cynical vantage point, it looks at the moment like a healthy step backwards from making the solution to EVERYTHING a web service, or SOA.

So what is Tuxedo?

Tuxedo is what’s referred to as an Online Transaction Processing system. (OLTP, if you’re looking for a sexy acronym for your new tattoo) It’s also commonly referred to as a Transaction Processisng system, but that’s a shorter version of the same thing fundamentally.That means that it processes a huge volume of small operations. Because those operations are things that need to either work or fail consistently, we call those transactions. Transactions can be anything from processing a credit card charge, to updating the location of a package in transit for delivery, to transferring funds between accounts.

One of the big advantages of Tuxedo in particular is that it offers clients a set of available services, each with a unique name. Those services can be provided by a large, complex group of interconnected machines, or all of those services can be provided by one single machine. Clients don’t know and don’t care about the exact location of an available service. They connect to a central point, and request services by name. The exact process that provides that service is unknown to them, which lets you be much more flexible with how you structure the internals of a Tuxedo application.

Because OLTPs are so good at processing these high volumes of transactions (and typically have other technological benefits that make them attractive) they’re used by some big companies, often in preference to newer, ‘cooler’ technology like web services or JEE applications, for example. Just to drop two names, last time I heard both American Express and Fedex were Tuxedo users.

Thus ends the general part of the post. Thanks for listening!

Now… how to get more useful information out of Tuxedo?

ULOG

The pimp-mack-super-daddy of information in Tuxedo is the ULOG file. This is where Tuxedo logs all of its own information, and also where any calls to the function write data. if you’re having problems with Tuxedo, this should always be your first port of call. By default, the ULOG file lives in the directory pointed to by the APPDIR environment variable. If you haven’t set APPDIR explicitly, then the ULOG file lives in the same directory as your TUXCONFIG file.

The ULOG file is actually a series of files, one for each day. The format of ULOG files is as below:

ULOG.mmddyy

where mm is the current month, dd is the current day of the month, and yy is the last two digits of the current year. Frustratingly, that leaves those files just short of sorting perfectly in chronological order. But I’m sure there’s a solid historical American reason for that convention, possibly involving orange cheese. As it stands, they’ll sort fine inside one year.

You can also control the location of ULOG files with the ULOGPFX environment variable. If you want to keep log files somewhere specific, you can specify a directory and filename like the following:

export ULOGPFX=/var/log/tuxedo/VERYSPECIAL

With that convention, you’d get a file created on the 12th of April 2009 called /var/log/tuxedo/VERYSPECIAL.041209.

Every message that Tuxedo places in the ULOG file comes from a message catalog. You can find a starting point for the Tuxedo message catalog reference here. Nine times out of ten, if you have some cryptic message in the ULOG file, the message catalog documentation will shed some light on what’s going on, or at least give a hint of the general area of the problem. Unfortunately, one time out of ten, you’ll get a “Contact BEA Support” message, or a repeat of exactly what you’re seeing in the ULOG file.

One thing that gets overlooked sometimes when troubleshooting behaviour between a client and server is that the client gets their own ULOG file too. The same variables (APPDIR and ULOGPFX) control where it’s created. Oftentimes, especially with connectivity issues or situations where nothing appears to be happening in the Tuxedo application itself, it’s the logs for the client and server together that form the complete picture.

TMTRACE: The Firehose

Now, the standard information in the ULOG file is often enough to solve simple problems. But, sometimes a problem comes along that’s complex enough that you need more. That’s where tmtrace comes in. It’s a facility in Tuxedo that allows you to provide more detailed information about what’s happening at runtime. It’s a very lightly documented feature, and I have to confess that my understanding of it borders on hoodoo. But I’ve found that even that hoodoo level of understanding is tremendously useful.

It works on a trace specification, which essentially allows you to specify three things:

• A trace category, identifying what area of Tuxedo you want more information on.
• Where you want information to go, which has to be the ULOG file.
• Whether you want your particular service request to ‘dye’, meaning that all work done as part of your service request will use the same trace configuration you’ve specified, or just use the current configuration for that server.

I’ll explain each of those in a little more detail to shed some light:

The trace category can be as broad as a number of key areas that the documentation provides, or specific as an individual function. Personally, I tend to use the broader areas that are documented, because it means not having to adjust trace specifications constantly. I’m a big fan of the firehose.

The documented categories are:

• atmi – Specific ATMI functions
• iatmi – work done implicitly on behalf of an ATMI function, or as part of administration. This is a superset of the atmi category.
• xa – work done to manage XA transactions
• trace – work done related to managing trace functionality

The first three areas tend to be the most useful. In fact, you can just set the firehose to pulverize, use ‘*’, and get everything.

In terms of destinations for trace information, there’s an exciting new change with the more recent versions of Tuxedo: you can now specify utrace as a destination, which can send trace information to a custom destination. That was certainly new to me. The documentation is a little unclear on the specific limits of utrace, and seems to indicate that it can only be used with tmtrace information.

Now, dyeing behaviour. I can set trace configuration as surgically as on a server by server basis. So individual processes can have either their own or no current trace configuration. They could be the quiet, introspective server that lives next door.

Because Tuxedo works on the concept of services, the work done in order to process my service call to process a credit card charge might not be as simple as I think. The one server that takes my service call might make two service calls of its own. One of those service calls might turn into another three service calls. The flow could end up looking like this:

If we only want to use tracing for a specific flow of logic, to find out what happens with the full flow of processing, using dye allows us to follow processing not just for server A (where we’d set trace) but all the way through to servers D, E and F.

You can set trace configuration either using the TMTRACE environment variable, or using changetrace from inside tmadmin. The simplest way to enable tracing is flick everything on, using changetrace on, or you could just enable tracing for ATMI functions and send trace info to the ULOG file, as follows:

changetrace "atmi:ulog"

For more information on trace specification formats, the best reference is the documentation for tmtrace.

So that’s the ULOG file, and TMTRACE. Both of these together will form the first and and best line of investigation when something is going wrong with your Tuxedo application. Typically, your ULOG file is kind enough to provide some idea of what is going wrong with your application without any additional work. If not, you can coax more useful information out via TMTRACE.

Next up, we’ll have a look at txrpt, and the .TMIB service in Tuxedo – both great tools that provide us with the basics of performance and runtime monitoring in Tuxedo.

1 Comment »

What if I told you right now that you could have a free developer on your team? No cost except interview time, and this mythical dogsbody would diligently do nothing other than compile the latest code in source control, perform anal-retentive checks against Sun’s coding standards (or your own company coding standards, if you’re one of the rare examples implementing standards beyond Sun’s) and run unit tests. And they’d do this twenty four hours a day.

If you’ve done time doing Java development, you can probably see where this is going already.

Automated builds.

Nothing incredible or revolutionary here. Sure. Nothing new. And yet, the number of times I’ve had conversations with really smart people that have gone along the following lines:

“So, do we have automated builds?”
“Er, no… we didn’t have time to set up automated builds”

Really smart people, and yet there’s a contradiction here that so frustrating that it makes steam shoot out my ears. Automated builds don’t cost time, they save time. Mentally tally up:

• Time lost across the whole team because one developer either checks in a broken file, or forgets to check in something critical to the build (that’s my personal cardinal sin)
• Time spent across the whole team hand-cranking unit test suites (or, even worse, manually testing functionality repeatedly) to validate code
• Time spent by your poor harried release manager lovingly handcrafting artifacts to release to controlled environments.
• Time spent before releasing code to prying eyes spent polishing and refinding and whoops!… I just broke a unit test.

  …and that’s just off the top of my head. Now compare all of that to a day’s solid work for one developer putting together a workable, basic automated build process in place, and then realistically spending another 2-5 days throughout the project tinkering and tuning.

  For my money, I don’t understand how you can make do without automated builds. An automated build is like a canary in the dank mines of team development. Is it safe to update from source control? Possibly not if the canary’s been pinballing itself bloody back and forth against the bars of its cage for the last five hours. Time to go on a developer witchhunt instead…

  I’m helping out with a development project that is being run out of our Brisbane office at the moment, and even on days when I don’t have time to code, firing up CruiseControl helps me keep an eye on what’s happening up north. Oh, and hey – speaking of CruiseControl, the instigator for writing about all of this was getting a chance to check out the latest release of CruiseControl – 2.7.1. The latest at the time I downloaded it, at least. You know all those magical transformations in teen rom-com-dramas where the smart, funny, practical girl you’d date anyway suddenly gets seriously sexy on the other side of a montage sequence?

  Yeah. CruiseControl has always been a fantastic tool, but with the help of some hardcore GUI development and some serious Ajax love, the automated build dashboard has been enhanced beyond belief. Just a quick look at some of the information CruiseControl puts at-a-glance: (click images for a larger view)

  Current build status

  

  CVS modification logs – a summary of checkin messages left by developers

  

  Test suite results

  

  One to six days work over the length of the project, and you add the most diligent developer you’ll find to your team. And you can scale that commitment in time to what makes sense to your project. You only get to add that magical free developer once, but they’ll never complain, never steal your Jolt, never steal your girlfriend with their more impressive range of Jinx.com T-shirts, or steal your precious torrenting bandwidth.

  I can’t think of another way to get that significant a return for that same amount of time.

No Comments »

I have a confession to make. I’m a huge data geek. I love data. It’s one of the things that’s kept me in IT. I love games like HeroClix and World of WarCraft because they are delicious soups of raw information, leading in the case of WoW to sites like Thottbot.com. In fact, a lot of the time I like the raw potential of these games more than sinking time into playing them. (time, it must be said, is clearly the most precious resource here)

One of the reasons why I work in integration is that integration is primarily about getting data from A to B. It’s like saying you like water, so yes please, I will wrap my mouth around a fire hose to enjoy the torrent of information ensuing.

Part of being a data geek is also being a huge geek for representation of data. The denser your data, the more of a challenge it is to present that information in a compelling and transparent way. Recently I splashed out on Edward Tufte’s works, which I’m going through presently.

The point of all of this is that Google recently released a charting API. Charting’s always been something you could do manually, or programmatically if you were willing to go through some onerous (for my money) Apache configuration, but now you can do it with a simple URL.

Here’s the Google Chart API documentation, as well. Y’know. Could be handy.

This one popped up courtesy of Lifehacker Australia, which is a really great extension to the Lifehacker family and well worth reading. I’m addicted to productivity porn.

On a side note: Generated charts are limited to 50,000 views per day. I’m pretty confident I’m not in danger here, but to add some value I’ll follow this up soon with a PHP script that caches the result of a generated page (using the URL to determine uniqueness, given that all the information is in there) and serves up the cached image instead.

No Comments »