Flutterby™! : Danny's vacation. 2008-09-16 11:34:07.027538+00

Danny's vacation.

2008-09-16 11:34:07.027538+00 by meuon 3 comments

"Danny" was the linux nut at a regional large company with many locations. Had been for years, and while IT had a decent budget, Danny kept it down by using a lot of FOSS Solutions where he could. A Linux mail server here, a few OpenVPN connections via public internet.. a couple of Samba servers for odd file servers and a backup of the main MS one. Had even ported a few business apps to LAMP, they just worked. Because things just worked and Danny didn't complain much, Danny lost his job. Management's perception was he wasn't doing anything hard and wasn't spending gobs of money on problems... They let Danny go, he smiled, took early retirement and hit the road with his wife (he's also a caver/explorer). That was about two months ago.

I just got done proxying some emails to Danny while on the road. He's had a nice vacation and is heading back. He was surprised it only took two months, he expected it to take 6+, as he left things in good shape. Seems they called in some "experts", the proposals were starting to add up as others started to inherit Danny's world and didn't understand it. Seems they needed an Exchange server, lots of Cisco VPN gear, some .Net developers, a DBA. The licenses alone were adding up to a few years of Danny's formerly modest salary.

Familiar sounding story... Seems to speak to the greatness of FOSS - but it also highlights one of the great failings of FOSS. Part of what you get when you buy Exchange is a structure for training and support and legions of geeks with experience. The same with Cisco...

"Cathedral" style development creates single solutions. The "Bazaar" results in dozens of solutions to the same problem. So FOSS training outfits focus on one solution stack - but every geek implements their own favorite solution stack.

The real failure here was the company not realizing the true cost of what they had. We saw the same thing at Q-Debit. We produced astounding results at essentially zero cost. While the sales people were blowing startup funds in Vegas, we were making things work with duct-tape and bailing-wire.

Smart companies know that the ability to keep things running even if their prize uber-geek falls into a cave and dies, is more important than any license expense. I saw this at McKee Foods. They write monster checks to Oracle every year. Why? Because they know that the business will keep running without Oracle (or IT) but they calculated the increased profit to be greater than the cost. They wouldn't be able to make the same calculation with FOSS because there's no standard way to measure the cost. It's easy to figure out how much a new Oracle geek will cost - but how much does a geek cost that can figure out a random stack of FOSS? McKee Foods isn't an IT business - they make snacks. They make money off of snacks. If they can measurably find a way to make more money selling snacks, then they'll do it. If they can't measure the potential profit they won't bother.

Danny broke Scotty's (from Star Trek) rule and worked miracles without ensuring that they seemed like miracles. The company got so used to miracles happening that they thought it was normal.

Makes me wonder if FOSS generates a vicious cycle: we all want FOSS to succeed, so we put miraculous efforts into making sure it works. Because we aren't Cisco/Microsoft/Oracle certified, we accept lower pay or we work longer hours (afterall, our worth to the company can only be measured by our performance on the job - something we tend to revel in). Because FOSS, at the hands of a talented geek, IS BETTER than closed-source software, miracles happen. So we teach companies that they can hire us for less money, spend less on software and training, and get "magical" results. It's really demeaning...

I used to see the same pattern in the game development world. Getting a job as a game programmer was seen as so cool that people accept crappy offers. Of course, games are like hit records. You never know when some group of kids in a garage will turn out the next Quake (used to be more common than today). Those kids in the garage weren't even making a salary!

I'd have to lean toward Eric on this one. FOSS developers are FOSS advocates, the same way that .NET developers will tell you all the benefits of the win platform. We work well with the tools available to us, regardless of what we're given. Given the choice, we use what we prefer.

The main part that I agree with is that "Danny" made miracles seem commonplace. His miracles happened often and every time they were needed, his miracles became everyday thing. I don't think that's so much a reflection of FOSS, but a reflection on "Danny" Since "Danny" worked at a reduced rate and used software that costs nothing and made it work every day the value was incredible, but without the regular miracle worker, it still failed just like any system will without the little man who comes in to feed and satiate the monsters on a regular basis before they group up and revolt (log files, stuck db transactions, code maintenance, etc).

I tend not to wish bad things on others, but I'm actually waiting to see that happen to another company soon. I try to keep the place I work for very quiet. I quietly fix bugs in the background, quietly satiate some of our biggest "pain in the butt" customers, quietly deal with quality control, code reviews, regression testing, and take care of emergencies when they happen to make my own little miracles. I'm wondering what's going to happen in November when I'm not around to feed and placate the monsters any more... I wonder how long before they revolt.

I think there's likely a strong correlation between miracle workers and FOSS, even if it's not his favorite tool; the guy who knows what a dozuki is is more likely to cut fine dovetails than the guy who thinks hand saws end at the crosscut back saw from Ace Hardware.

However, I'm currently trying to make a few suggestions to a company that's a .NET shop, and that doesn't see how it can justify hiring top-end people ("why pay $150k/year when we can get kids out of college for $48k?"), and I'm going supported high cost products all the way, mostly as a form of butt covering. If it were me running the project I'd get a couple of PHP hotshots and re-code everything currently in play, but the idea of "yeah, it looked like a good idea to me, too, so talk to the vendor about why it isn't working" is strong.

As Eric and JT have also pointed out, you've gotta have two of those top-end people, because their function is critical to the operation of the company. However, you also have to have management smart enough to realize that good combination ops and slight coders should be working two hours a day, tops, because they're not hanging around for when things are going well.