Curse of Choice
Today I was lucky enough to see a video of Barry Schwartz talking about the paradox of choice. This really struck a nerve for me, because it really nicely explains the problems I’ve been having, more or less, since college.
You see, the problem is that once you have more than a certain number of choices, you are bewildered by the possibilities. Like trying to get in the line that goes the fastest at the supermarket. Your odds of picking the fastest line are actually 1/N, where N is the number of lanes. Your odds of picking the best anything are the same, presuming you have enough criteria to make a total ordering.
The result is what Josh used to call “analysis paralysis.” And suddenly a lot of things are making sense that didn’t used to. Especially the content management situation. Ordinary humans are only going to evaluate one or two. Knowing the ins and outs of a handful or more leads you to believe they’re all crap. Same story with programming languages. Once you know (to some degree or another) about thirty languages, it can be quite hard to pick the “right” one for a given problem.
These weren’t ever really big problems for Michael, and he’s much happier as a result. He didn’t wonder if he should use Lisp or Ruby or PHP or Python or Haskell or Erlang or something weirder for his website. He had PHP on the server, so he was going to learn and use PHP. He didn’t wonder if he should use PostgreSQL or MySQL; he had MySQL on the server and so did the clients. Michael just recently managed to underbid my company by an astonishing amount of money—not because he cheated or we overbid, but because he has three years of infrastructure built on his CMS Voltaire, and we don’t have any infrastructure built on anything so it would all have to be custom-written. We have spent so much time trying to decide, we have no reusable infrastructure.
People using PHP routinely spend, say, an hour or two doing something that would take me 10 minutes or so in a modern programming language like Lisp. But it doesn’t matter, because the other 7 hours of that day, they were making progress and I was reading Reddit. Trying to make sure I don’t make the wrong move, instead of moving in any direction at all.
The worst part is, even knowing this doesn’t seem to be enough to solve the problem. After listening to Barry’s talk, we sat there trying to decide what to do about it. I’m going to have buyer’s remorse no matter what. Our programming tools are fabulous now, so great we’re completely befuddled.
I still think unnecessary abstraction is a big part of the problem. All those layers of abstraction, they all have names, they all have different purposes and ultimately, they all add not just intellectual burden in terms of learning them, but tremendous intellectual burden in terms of comparing them to decide on one.
An often-heard complaint about programmers is that we take our tools too seriously; carpenters don’t develop preferences for saws or hammers. But this view is quite outdated when it comes to programming, just because, in programming, all the tools have the same power if they are Turing-complete. You simply cannot do the same things with a hammer and a saw. And hammers and saws don’t expand in capability over time. But programming languages and abstractions do expand over time.
I believe today that a major function of CMSes and programming language frameworks is simply to remove choice to enhance productivity. I doubt now that Merb will significantly affect Rails’s popularity, simply because Merb forces you to make more decisions than Rails does. ActiveRecord doesn’t need to be a great ORM, it just needs to be good enough to help you avoid learning SQL, to keep you from making another decision. Merb will be there for Rails the way the BSDs are there for Linux users: a better alternative that requires more thinking.
I am still torn on what my company is going to use for our major platform but there can’t be any doubt that a major platform is needed, if only to create an environment in which code sharing can occur so as to compete with Michael. My major choices are really PHP and Lisp. My fear in choosing PHP is that the language is too weak to express the things that I’ll be capable of thinking of and that programming in it will feel like drudgery. My fear in choosing Lisp is that tool support will be poor making it hard on Reid, that it will make it hard to sell, and, not insignificantly, that if I fail at it I will have to take more responsibility. But it seems clear that there will be no forward motion until there is a platform to be dedicated to (and stuck with), and only then will buyer’s remorse really begin to set in.
It would be nice if Barry had some suggestions on how to fix that but I don’t think he did. But this may be the root of my programming despair—and why Michael has succeeded and been much happier. The choice of tools actually may have been the problem.