Programmers as blacksmiths

Recently I stumbled across a three part article: The Blacksmith and the Bookkeeper, Part 1, Part 2, Part 3. The premise is that programming as a profession is doomed to go the way of the blacksmith, that it will create the tools that eventually bring about its end. In the article, Max Goff presents a hypothesis about why blacksmiths faded away while bookkeepers thrive today: In summary, blacksmithing was easily automated, streamlined, or assembly-lined, while bookkeeping required creative application of knowledge. Goff then posits that programming (except for embedded programming) falls into the same general category as smithing: Something that can be easily described and automated. But I think there are essential skills of programmers beyond being able to decipher assembly language or Java or Perl, and that those essential skills will be necessary for quite a long time.

I can easily see how he can come to his conclusion--the IT industry seems to be shrinking at an alarming rate, H1-B's and exported jobs are displacing native workers, and the premium that the average programmer once commanded has dwindled and become much more modest. In fact his conclusions are not unique: Another article in USA Today (Endangered species: US programmers) puts forth similar conclusions. It's not the first time I've encountered this concept. I've fielded questions on this topic from various friends and family members for years, in fact. And every new programming automation tool seems to herald the end of programming; there have been enough "silver bullet" software products and methodologies touted in the last twenty years to slay an army of werewolves, and yet none has lived up to the promises of its press releases.

As someone who has been a professional programmer and software engineer since 1987, I personally do not see the demand for my skills decreasing any time in the near future. Why am I not worried? In part because, as a video game programmer, my skills are in the minority--Goff makes reference to "Embedded Java" programmers in his article as being an exception to the rule, but I would put forth that many more programming specializations are going to need programmers for the foreseeable future.

With the dot com boom, the demand for programmers went way up along with their salaries, and many new programmers were being cranked out by universities. Many more companies than the supply of programmers could satisfy "needed" to have a Web presence as soon as possible, and so the demand created as if by magic a non-sustainable number of extra IT positions. With the bust, the demand necessarily goes down--partly because of market saturation, in that most companies now have a Web presence, and partly because putting an application up on the Web is a mature, understood process, and for a vast majority of typical Web sites, is not innovation--it's often just assembling the correct puzzle pieces at this point. I would guess that even in the heyday of smithing that during, e.g., wartime many more blacksmiths were trained and highly paid to produce weapons and armor than could be supported when the demand for hammered iron was lower. So part of the problem is certainly related to this: There are simply too many programmers for the positions that are available.

So will there be enough jobs for everyone who wants to make a living as a programmer? No, probably not. Which sucks for the thousands who lined up to get CS degrees for guaranteed high pay and job security--another broken promise of the "New Economy." The secret to getting steady work in the field is to find a niche where the programming that needs to be done isn't one that isn't easily generalized. I think this is a corollary to Goff's premise, in fact: If the job you're doing is something repetitive and predictable, if there's no (or very little) adaptation necessary for each new project, then it's going to be in the greatest danger of being automated or eliminated. There's more skill in adapting to a dynamic situation than in assembling components by rote. And having a skill that's in high demand is what raises your paycheck, not something magical about the titles "Programmer" and "Software Engineer". We're not just entitled to a high paycheck due by virtue of being able to speak Java or (in my case) C++. The high paycheck comes from providing a lot of value to a business.

This argues for restricting H1-B access, since the supply is truly greater than the demand for some positions. It turns out, though, that in niche industries like video game programming, where you really do need highly specialized talent that really is hard to find, even today, the H1-B applicant will often be the only one qualified for the position. Not just the cheapest one, but the only one. So it's not a simple issue--preventing H1-B access can prevent some companies from finding employees with the right skills. Which hurts the economy, etc. For a similar reason, video game outsourcing isn't thriving: I know that it does exist, though for the most part it exists as complete game development houses or component providers.

I think that another real problem facing the army of programmers out there right now is that many are working on similar problems, and that those problems are predominantly solved. Most Web sites and custom applications use the same features, many use off-the-shelf back ends with similar collections of components organized in slightly different ways, and one or two programmers can now do in a short time what it once took a small army of programmers many months. Note though that it still does require those few programmers--for the simple reason that programmers are trained to break a problem down analytically and debug it when it doesn't work as expected. Even if the programmers aren't manipulating anything like code, they will still be needed--maybe they'll be much more efficient, but they'll be around. Of course when they're more efficient, there won't be as many jobs...see above.

When a problem domain is well understood, however, it can often be analyzed, packaged, and automated into a wizard that you don't need a programmer for--as Paul Graham describes in his essay Beating the Averages, you can automate the process of creating an entire E-Commerce site. Now his software is used in Yahoo Stores, and while most of the Yahoo stores tend to have a similar look, they look professional and, with a custom graphics layer, would rival many custom-generated sites. In one wizard on a Web site, in under an hour, a person who knows nothing about programming can assemble an entire custom store with inventory management, payments, shipment tracking, and whatever other features Graham's group was able to squeeze in--with no direct programmer interaction at all. You might think I'm undermining my own argument here. But note this is a single problem domain, and however well it solves the Generic E-Commerce Site problem, it won't help in the slightest with creating a piece of software that tracks shipping containers for an export company, or that analyzes communication traffic to optimize switching, or any number of unrelated problems that still will need programmers until those problems are well understood, at which point they could potentially be automated and the programmers move on to new problems. And the Yahoo Store generating software still needs a team of programmers--who maintain 20000+ web sites rather than just a handful.

I do believe that companies are right to not want to pay premium salaries for programmers who are not really creating anything new. So programmers should always endeavor to keep their minds sharp, and always be in the position of creating new things rather than just pushing components around. I'm not against components--use components or any other tool that gives you greater leverage by all means!--just be sure you're personally adding value and not just running wizards. If a programmer isn't flexible enough to be able to adapt to a new industry or specialization, then unfortunately that programmer has become a blacksmith of programming, a specialist fit for one job on a programming assembly line, and unfortunately that's the position that's easiest to outsource or otherwise replace. So what you need to do is either reinvent yourself as a programmer, or reinvent yourself as something else. But don't just feel sorry for yourself that the bandwagon that you jumped on is overcrowded and that people keep falling off as its carrying capacity shrinks.

There are an infinite number of potential problem domains out there--and we're nowhere near putting together a tool that can solve any large fraction of them. To be able to generate a completely new program (presuming that the program is non-trivial, and understanding that what is meant by "trivial" evolves over time), you need to understand the motivations and needs of the people who will be using that program, distilling those concepts into a specification, sub-divide the specification into whatever logical units you have at your disposal, refine the implementation to improve its usefulness, and debug it when it doesn't work quite correctly. And these are the skills that epitomize a programmer or software engineer. I don't care if the "programming" is done by writing lines of code, dragging boxes around, talking into a microphone, or projecting thoughts into a brain-scanning UI. Programming will evolve. The only way to eliminate programmers from the equation entirely is to replace them with human-like AI entities with a breadth of understand of human endeavors, which, if created, would certainly do more than just put programmers out of business. Programming won't be the major that people flock to for easy money, certainly, but its oft-reported imminent death has been greatly exaggerated.

I can talk about my job now!

A few weeks ago I joined a startup that had been funded, but its official goals hadn't been broadcast to the world yet. Well, the time for secrecy is over!

The San Jose Mercury News ran this today:

http://www.mercurynews.com/mld/mercurynews/business/9770737.htm

...and according to Google News it was syndicated in at least 13 other papers. (This one doesn't require a subscription.)

And Gamespot is running this story and interview with our CEO:

http://www.gamespot.com/news/2004/09/27/news_6108681.html

After reading the latter, I was reminded why I decided to join Playfirst to begin with--the excitement and enthusiasm generated by the top tier of this group will carry it quite a ways. $5 million in funding will keep the burden light. And it feels to me like a real market that's just opening up, so the destination is attractive, especially with stock options in the company.

Finally, the Playfirst web site is up, and it looks great!

In any event, that's where I'll be working for the forseeable future. I'm a lead programmer ("Senior Software Engineer") on an internal title right now. It's been fun so far!

Server Insecurity

This article attempts to chronicle my recent upgrade adventure installing Gentoo Linux/SElinux on my home server. I'm a technical kind of guy, so I'm likely to use highly technical references without realizing it, but I'm trying to keep my tale within reach of non-Linux-experts. My objective is to keep the article accessible to anyone interested, while illustrating why someone might be crazy enough to try to install a hardened Linux build despite the extra effort required.

Set the wayback machine to mid-1998. I was looking through my home Linux server, which was then running RedHat 5.0, if memory serves, when I discovered files on the hard disk that I didn't put there. Since no one else had access to the server (legitimately) , I had been hacked. As a result, I immediately started reading everything I could get my hands on about computer security. The first thing I learned was that the applications I was using with RedHat--SendMail, WS-FTP, and Bind, three very common server applications that were installed by default on that old version of RedHat--had some of the worst security hole records out there. I turned off two of those applications, replacing them with more secure versions: Qmail, a powerful and free email server, and ncftpd, a proprietary but also free FTP server. I didn't find a good replacement for Bind (at the time), so I patched it and crossed my fingers.

About a year later, I found another file I hadn't put there. It looked like I'd been hacked again. I dropped RedHat and switched to a little-known hardened distribution called Trustix. I also found djbdns, a Bind replacement, and managed to get rid of that last major security risk.

Fast-forward to the present: Trustix didn't upgrade to the newest Linux kernels quickly enough, so I decided to try other distributions and settled on Mandrake, which had been good to me so far; whenever I heard about a new exploit or security hole, Mandrake would supply the patch and I'd quickly update my box. Unfortunately, some patches would break things, and it would take a few hours to get things up and running. And because there was that window between when the exploit is discovered and when I patched it, I always had to wonder if someone had broken in and left a trojan on my box. As a result, every new install meant starting over practically from scratch with a virgin system (so that nothing a hypothetical hacker had left behind would persist to the new system). And with a full time job, it's not fun to have to spend that much free time rebuilding my system every year or two.

For over a year I ran (on my private home server) a Mandrake 9.0 Linux box with as much security as I could turn on while still being able to use it. The box did a lot of good work for me -- it stored my MP3 files, operated as a mail server and webmail client, served my domain names, and blocked evil attacks from the outside world from my vulnerable Windows boxes.

However, Mandrake wasn't supporting 9.0 any more, so it was time to upgrade. This time, however, I wanted it to be different. I wanted this upgrade to be the last major upgrade I would need to perform, at least for many years, and I wanted the server files to be safe without me having to worry about whether someone snuck in with an exploit leaving trojans throughout my system.

I'd read a lot about Gentoo Linux, so I thought I'd give it a try. A few features recommended it to me, including the fact that there isn't really a discrete "version" number of Gentoo--when new package versions become available, you can just upgrade--and you can rebuild packages with custom options for the exact system you're building. I also discovered that Gentoo supported SElinux, a package developed by the NSA to minimize (specifically to localize) the potential damage that an intrusion can do. For instance, if your Web server can only read web files and someone breaks in through the Web server, the most that hacker can do is read web files. An over-simplification, but that's the basic idea. Gentoo also has other options for a hardened kernel build (features that prevent some attacks to begin with), which I also turned on.

The only stumbling block I hit getting started was that the normal Gentoo Live CD (the CD that you use to boot up the first time) didn't work with the SElinux build, and it didn't tell me why it wouldn't work--but a little Google searching came up with the alternate SElinux Live CD, and I was off and running. There was also a bit of confusion where my Google search had picked up an outdated HOWTO with broken links, but once I found the correct HOWTO things went a bit more smoothly.

So with an SElinux Live CD, I got a build together and installed my favorite packages--which worked fine until I turned on the SElinux protections. I had enabled the download of "development packages" so I could get SElinux components--not realizing at first that I would get the development packages for everything. These new development packages didn't have updated SElinux rules, so it took a lot of reconfiguration to get the ultra-paranoid SElinux kernel to work with all of these packages. Worse, I had the temerity to want to install packages that had no existing SElinux configurations available at all. It took a lot of work to get all of the settings tuned.

In a nutshell, when you want to make a new service work under SElinux, it's a kind of whack-a-mole game of granting access to each specific feature that the service needs. It's not quite that easy, though, because you also need to create new "security domains" for the new service, and map out how each domain can interact with each other domain...and if you do it wrong, you've just opened up a new potential security hole. I would say that this process is not for the faint-of-heart, and certainly not for the newbie.

Once all of your services have SElinux packages, and you're using the right versions of each, it will be hard to beat the security of SElinux, though. If the Gentoo folks are interested, I'll put together a few packages with my changes in them to help out future SElinux installers.

In any case, my new server is finally up and running. I also have a lot of new features almost for free because of the great Gentoo package system: SpamAssassin helps to deflect spam from my system, ClamAV helps to block viruses before they enter my email box, and Poptop will soon allow me to VPN into my home box to get to my file shares.

I won't say that the new server is bulletproof, because that would be stupid--it would certainly attract more attention than I want trying to prove me wrong, and I would probably be wrong. But considering it's a completely uninteresting personal server without as much as a single credit card number, the level of security I have on it now is enough to quell my fears.

Besides which, just to be extra paranoid, I did a baseline checksum scan of all of the applications on the server, so if someone does someday get past all of these barriers I put up, at least I'll be able to know. Until then, I'm going to sleep a bit more easily.