Programmers as blacksmiths

Recently I stumbled across a three part article: The Blacksmith and the Bookkeeper, Part 1, Part 2, Part 3. The premise is that programming as a profession is doomed to go the way of the blacksmith, that it will create the tools that eventually bring about its end. In the article, Max Goff presents a hypothesis about why blacksmiths faded away while bookkeepers thrive today: In summary, blacksmithing was easily automated, streamlined, or assembly-lined, while bookkeeping required creative application of knowledge. Goff then posits that programming (except for embedded programming) falls into the same general category as smithing: Something that can be easily described and automated. But I think there are essential skills of programmers beyond being able to decipher assembly language or Java or Perl, and that those essential skills will be necessary for quite a long time.

I can easily see how he can come to his conclusion--the IT industry seems to be shrinking at an alarming rate, H1-B's and exported jobs are displacing native workers, and the premium that the average programmer once commanded has dwindled and become much more modest. In fact his conclusions are not unique: Another article in USA Today (Endangered species: US programmers) puts forth similar conclusions. It's not the first time I've encountered this concept. I've fielded questions on this topic from various friends and family members for years, in fact. And every new programming automation tool seems to herald the end of programming; there have been enough "silver bullet" software products and methodologies touted in the last twenty years to slay an army of werewolves, and yet none has lived up to the promises of its press releases.

As someone who has been a professional programmer and software engineer since 1987, I personally do not see the demand for my skills decreasing any time in the near future. Why am I not worried? In part because, as a video game programmer, my skills are in the minority--Goff makes reference to "Embedded Java" programmers in his article as being an exception to the rule, but I would put forth that many more programming specializations are going to need programmers for the foreseeable future.

With the dot com boom, the demand for programmers went way up along with their salaries, and many new programmers were being cranked out by universities. Many more companies than the supply of programmers could satisfy "needed" to have a Web presence as soon as possible, and so the demand created as if by magic a non-sustainable number of extra IT positions. With the bust, the demand necessarily goes down--partly because of market saturation, in that most companies now have a Web presence, and partly because putting an application up on the Web is a mature, understood process, and for a vast majority of typical Web sites, is not innovation--it's often just assembling the correct puzzle pieces at this point. I would guess that even in the heyday of smithing that during, e.g., wartime many more blacksmiths were trained and highly paid to produce weapons and armor than could be supported when the demand for hammered iron was lower. So part of the problem is certainly related to this: There are simply too many programmers for the positions that are available.

So will there be enough jobs for everyone who wants to make a living as a programmer? No, probably not. Which sucks for the thousands who lined up to get CS degrees for guaranteed high pay and job security--another broken promise of the "New Economy." The secret to getting steady work in the field is to find a niche where the programming that needs to be done isn't one that isn't easily generalized. I think this is a corollary to Goff's premise, in fact: If the job you're doing is something repetitive and predictable, if there's no (or very little) adaptation necessary for each new project, then it's going to be in the greatest danger of being automated or eliminated. There's more skill in adapting to a dynamic situation than in assembling components by rote. And having a skill that's in high demand is what raises your paycheck, not something magical about the titles "Programmer" and "Software Engineer". We're not just entitled to a high paycheck due by virtue of being able to speak Java or (in my case) C++. The high paycheck comes from providing a lot of value to a business.

This argues for restricting H1-B access, since the supply is truly greater than the demand for some positions. It turns out, though, that in niche industries like video game programming, where you really do need highly specialized talent that really is hard to find, even today, the H1-B applicant will often be the only one qualified for the position. Not just the cheapest one, but the only one. So it's not a simple issue--preventing H1-B access can prevent some companies from finding employees with the right skills. Which hurts the economy, etc. For a similar reason, video game outsourcing isn't thriving: I know that it does exist, though for the most part it exists as complete game development houses or component providers.

I think that another real problem facing the army of programmers out there right now is that many are working on similar problems, and that those problems are predominantly solved. Most Web sites and custom applications use the same features, many use off-the-shelf back ends with similar collections of components organized in slightly different ways, and one or two programmers can now do in a short time what it once took a small army of programmers many months. Note though that it still does require those few programmers--for the simple reason that programmers are trained to break a problem down analytically and debug it when it doesn't work as expected. Even if the programmers aren't manipulating anything like code, they will still be needed--maybe they'll be much more efficient, but they'll be around. Of course when they're more efficient, there won't be as many jobs...see above.

When a problem domain is well understood, however, it can often be analyzed, packaged, and automated into a wizard that you don't need a programmer for--as Paul Graham describes in his essay Beating the Averages, you can automate the process of creating an entire E-Commerce site. Now his software is used in Yahoo Stores, and while most of the Yahoo stores tend to have a similar look, they look professional and, with a custom graphics layer, would rival many custom-generated sites. In one wizard on a Web site, in under an hour, a person who knows nothing about programming can assemble an entire custom store with inventory management, payments, shipment tracking, and whatever other features Graham's group was able to squeeze in--with no direct programmer interaction at all. You might think I'm undermining my own argument here. But note this is a single problem domain, and however well it solves the Generic E-Commerce Site problem, it won't help in the slightest with creating a piece of software that tracks shipping containers for an export company, or that analyzes communication traffic to optimize switching, or any number of unrelated problems that still will need programmers until those problems are well understood, at which point they could potentially be automated and the programmers move on to new problems. And the Yahoo Store generating software still needs a team of programmers--who maintain 20000+ web sites rather than just a handful.

I do believe that companies are right to not want to pay premium salaries for programmers who are not really creating anything new. So programmers should always endeavor to keep their minds sharp, and always be in the position of creating new things rather than just pushing components around. I'm not against components--use components or any other tool that gives you greater leverage by all means!--just be sure you're personally adding value and not just running wizards. If a programmer isn't flexible enough to be able to adapt to a new industry or specialization, then unfortunately that programmer has become a blacksmith of programming, a specialist fit for one job on a programming assembly line, and unfortunately that's the position that's easiest to outsource or otherwise replace. So what you need to do is either reinvent yourself as a programmer, or reinvent yourself as something else. But don't just feel sorry for yourself that the bandwagon that you jumped on is overcrowded and that people keep falling off as its carrying capacity shrinks.

There are an infinite number of potential problem domains out there--and we're nowhere near putting together a tool that can solve any large fraction of them. To be able to generate a completely new program (presuming that the program is non-trivial, and understanding that what is meant by "trivial" evolves over time), you need to understand the motivations and needs of the people who will be using that program, distilling those concepts into a specification, sub-divide the specification into whatever logical units you have at your disposal, refine the implementation to improve its usefulness, and debug it when it doesn't work quite correctly. And these are the skills that epitomize a programmer or software engineer. I don't care if the "programming" is done by writing lines of code, dragging boxes around, talking into a microphone, or projecting thoughts into a brain-scanning UI. Programming will evolve. The only way to eliminate programmers from the equation entirely is to replace them with human-like AI entities with a breadth of understand of human endeavors, which, if created, would certainly do more than just put programmers out of business. Programming won't be the major that people flock to for easy money, certainly, but its oft-reported imminent death has been greatly exaggerated.