“Help! The website is down! The server is down! The Internet is broken! Help!!!” If you’ve ever been on the receiving end of this panicked cry for help, you might read this quote and identify with my feelings of frustration. I’ve gotten this phone call so many times, and while I know the website or web server might actually be “down” and the Internet connection might be “broken”, neither is probably the case. Generally, what’s happening is that the user is opening a web browser, and the web page they are trying to access is not responding for some reason. In my experience, it seems that the server is rarely, if ever, down, and even more rarely is the Internet connection failing. If you’ve been blessed with the job of handling such support requests, I’m sure you have some techniques in your tool belt that you use to run through your troubleshooting process. In this article I am going to speak about one of my most trusted and essential tools – the Unix shell account. If you’re looking for an introduction to Unix, this article isn’t the one. This article is more of ode to the shell account, one of the last vestiges of an era when computing wasn’t all buzzwords and marketing, and the power of a computer wasn’t hidden beneath a bedazzled interface.
Even in 2011, a time when computers are in our pockets that can act at the swipe of a finger or simple words spoken from our lips, the Unix shell, with its antiquated command line interface and cryptic commands, is still one of the most reliable and essential tools to aid in troubleshooting. How could this be? How could access to a 40 year old operating system account be relevant in this era? It’s quite simple actually, and while it may seem self deprecating, it all boils down to this: sometimes the best computer to use for testing and troubleshooting is a computer that is not your own. In fact, I’ve learned that it’s better if the computer isn’t even on your network. Why not your computer on your network? Because there’s a chance that whatever is happening is your fault to begin with, and it’s nice to know there’s a computer that there’s essentially no way you could have messed up!
It might be hard to face this fact, but if you, like me, are a “tech / admin / programmer / the one people call when things need fixing”, then you’ve probably had a hand in the installation and configuration of your own machine and possibly even the network at your office. You’re smart – no, brilliant! You think on your feet, outside the box. You plan ahead. You’re clever. You multi-task. The truth is that all of these attributes you possess can be liabilities, just as much as they can be qualities. Take for example the following anecdote, which may have happened to me, or to you:
Joe is a support tech. He get’s a ring on the phone from Sara – “Hey, so I opened my web browser and nothing’s happening.” Having heard this a million times, Joe replies, “What do you mean nothing’s happening? What page are you trying to get to?” Joe knows the Internet isn’t down, he’s been reading reddit all morning, or rather, working diligently all morning… So he opens a browser and tries to access the URL the caller can’t reach, “www.youremployer.com.” This can generally go a couple of ways at this point: If Joe can access the site, Sara is probably having some kind of issue local to her machine, but in this case, Joe can’t access the site either. So what now? Joe calls another staff person, the tech-savvy receptionist, Sherine. It turns out that Sherine can’t access the site either. Interesting. At this point there are 3 known machines that can’t access the site. This could be really bad. Is the server down? Joe’s starting to feel a sense of urgency here (He never panics.). What Joe doesn’t know yet is that the server is up, but there’s a local DNS problem. A problem he caused, by implementing a clever fix while “cleaning up the DNS server.” Luckily for Joe, he has a shell account, so the next step he takes is to log in to his remote shell, fire up Lynx, and access www.youremployer.com. The site is up, so it must be a local network issue. It only takes Joe a few minutes to figure out what’s really going on, because his shell not only has a web browser, but has DNS lookup utilities, and entire set of Unix power tools that he could wield. A bit of dig and nslookup and it’s revealed that the local DNS server on his network had a typo in the new www alias that he added. 5 minute fix.
This could have been much worse. Joe’s friend, Mark doesn’t have a shell account. A few weeks ago, he faced a situation like this and the only way to test from outside was to call his buddy Larry and see if he could open a browser and try it from his office. These days, Larry is so busy that he probably won’t even be able to pick up his phone. It might seem like a fringe case, but believe me, it’s not uncommon that the issues that arise day to day on your network are caused by the same techs who are charged with resolving them. Mark could be at it for hours before finding the problem, because Mark assumes that everything on his office desktop and office network are guaranteed to be in perfect working order. Mark doesn’t see the connection between the clever hack he implemented on Tuesday morning last week, and the “that’s weird” problem he’s facing on this sunny Friday afternoon (And if he doesn’t solve it soon, means missing happy hour!). These are the times when having a shell account are so handy. The techs at my office all have a hand in our network environment and server configuration. It’s our house, and we all pitch in to take care of it. None of us are ever doing one thing at a time (Who can get away with that luxury?) and mistakes happen. Often, the results of these mistakes don’t surface for some time after they’re made, which makes finding the cause even harder. If you have a trusted machine on a trusted network that you can rely on, it makes life so much easier.
Today, there’s a lot of talk about “the cloud.” What is the cloud? We’re familiar with the cloud, because there’s usually a cloud on our network diagrams – it’s the Internet. What’s so good about the cloud? Why would someone want to, for example, keep their music collection on some cloud service, instead of having it all on their local machine? It’s not hard to see why – there’s just less responsibility involved in keeping it working. We can generally trust that the folks at Amazon aren’t going to accidentally erase the hard drive, or kick the plug out of a switch, or select shutdown instead of restart when trying to reboot the server remotely. The odds are that the “server” will be up, and since you can’t get your brilliant hands on that server, there’s little you (or your brilliant colleagues) could do to ruin it. So we trust our email to the cloud, our documents to the cloud, our photos and music to the cloud. Some of the younger generation doesn’t realize that before the cloud was something spoken about in an IBM television commercial, and before most of us could afford a server of our own, one could access a powerful server, on a trusted network, for little or no cost. Shell accounts are the original cloud computing.
If you don’t have a shell account, you need one. There’s really no debating this. There’s no excuse for not having at least one. A great place to start is the Super Dimension Fortress, which you can access at sdf.org. They have been offering low-cost and free shells for over 20 years – it’s what they do. Their mission, “…is to provide remotely accessible computing facilities for the advancement of public education, cultural enrichment, scientific research and recreation.” I use my SDF shell for troubleshooting, testing scripts, storing files, tunneling ssh connections, accessing IRC support channels, and just about anything else. Sure most of the time I could use my own computer, but as you may know by now – sometimes my computer is broken, because I broke it.
I’ve found that while SDF is great (and I can’t expound on that enough), there are times when I need to have root access. Sometimes I need to test some software SDF doesn’t have installed, or I need to access some privileged ports (ports numbered lower than 1024.) Here is where services from a place like RootBSD are terrific. From RootBSD you can get a virtual private server that runs Unix (even NetBSD or OpenBSD, which is not easy to find anywhere else), and you can do whatever you want with it. I recommend having a Unix VPS around, because they are indispensable for serious testing and troubleshooting. Obviously, I still recommend having a shell account somewhere else to test from, and by now I hope you’ll agree that it’s sensible. Let’s just hope the Internet isn’t really broken… 🙂
— Michael Hernandez