Packets Don't Lie

My nickname was given to me a while back, thepacketmaster.  It was because I love using packet tracers to solve problems.  I sometimes wonder if I'm unique in this sense or if other network/security admins use packet tracers as often as I do.  Time and again I solve problems using packet dumps when nothing else can work.  It's not that using packet tracers is an action of last resort.  It's simply that you can't argue with a packet traces.

Packets don't lie. I'm not saying that people are lying when they come to me with problems that they can't resolve. However, I've seen far too many instances where an administrator of an application, database, or server will come to me with a problem that is based on a false premise given to them by a poorly written error message.  More often than not it starts with, "I think there's a problem with the network".  Fair enough, it could be there is a problem with the network.  We in the network/security realm are not without problems and I never have a problem admitting that.  In my case I'd have to say the security side comes up more than the network side, simply because I prefer firewall rules that are as strict as possible and deny everything that isn't necessary.  Most of the other administrators aren't necessarily appreciative of that fact.

After making sure I understand the problem and checking that it isn't related to the network or security systems, I move onto trying to help the person understand what other problems it could be.  In many instances it could simply be that they've gotten some cryptic error message from a piece of software that says "Network error".  The bane of my existence are developers that write software with generic error messages that always mention the network.  Let this be a lesson to developers to always be specific in your error messages.  Instead of saying "Network error" put something useful like "Cannot bind to port" or "Unable to establish TCP connection to remote system".  Anything but "Network error"!

Once I have a feel for the problem I often try to suggest what other causes it could be.  In my previous lives, before I was a Network and Security Architect, I have been a UNIX admin, Network admin, Windows admin, Application admin, Security admin, database admin, and even a backup admin.  I've written in assembler, C, Perl, C++, php, bash, BASIC and COBOL.  In short, I have a holistic view of IT.  That's not to say I know it all.  Nobody, NOBODY, ever knows it all.  Those that claim to are full-of-it.

If a solution to the problem is still eluding detection, that's generally when I fallback to the packet capture.  This can tell so many things about what's actually going on.  Packet captures are mind altering experiences.  Previous conclusions that seemed unbreakable can be completely shattered or reinforced by a packet capture.  Problems and solutions never even imagined can be envisioned with complete clarity.  Packet captures can be a humbling experience.

The great thing is that nowadays you don't even need help interpreting what a packet capture says.  Tools like Wireshark are so user friendly that I can put it up on a projector to an entire room of people that have never seen a packet capture in their lives, show them what's I'm seeing and what is happening, and they'll understand the problem completely.