Hiding Complexity

Posted by Daniel Lyons on March 6, 2009

The eternal language wars aren’t going to end until we start to understand the ways in which we are talking past each other. We seem to be fond of posturing for the sake of having opinions, especially seemingly unpopular opinions. Underneath every language there is a philosophy. I think there are probably few enough of these philosophies we can catalogue them and get further than we would by necessarily studying language families.

I believe the main question is, where should we stuff the complexity?

In the days of yore, the answer was, “in the head of a programmer.” Assembler is the simplest because there’s no abstraction. If you want a complex structure, you have to create it in your mind; only a few wisps of it will ever wind up presented to the computer. Obviously everyone doing assembly these days is obsessed with performance. Today, that’s the only reason to learn it or use it. Don’t forget that it forces you to be that way. Many of the people you meet who extol its virtues today only see things that way because it was what they had when they learned.

Another classical answer is “in the operating system.” The major OSes of the day have truly huge APIs. All of that complexity is ostensibly being lifted from the shoulders of the application developer. You don’t have to worry about different kinds of hard disk, or even the hard disk directly at all, thanks to the OS. OSes like Windows boast gigantic application programmer interfaces, both within and without the kernel, but traditionally supplied a powerful yet complex language for programming. The major app for Windows, Office, had to supply its own programming platforms.

Unix took the OS approach much further, providing a pipeline-of-simple-utilities metaphor. Under Unix, if you needed something that wasn’t supplied with the system badly enough, you were expected to make a special-purpose tool in C to do it which you could then script with the shell. Unix teaches that this cooperation happens not just in the kernel and the libraries, but also in the applications themselves and even in user customizations. The price for the flexibility is a linguistic interface which is harder to learn. Plan 9 extends the Unix approach to its logical conclusion by destroying most of Unix’s accumulated system APIs and unifying them into the filesystem, and then bridging the network in the same manner, thus providing a system where Unix’s strengths can be leveraged across complex networks of computers transparently. Both OSes implicitly argue that the language does not need to be featureful if it is sufficiently powerful to implement the kernel, userland and augment the system.

Prolog took a different approach, demonstrating that computation itself can be performed with radically different evaluation strategies. Ostensibly simplifying things, in practice it shows that in general people prefer a simple, stupid system which can be made smart over having to out-think a smart system that in some cases behaves counterintuitively.

Lisp, Forth and their ilk suggest that complexity should evolve in the syntax and library of the language itself. Lisp allows the programmer to take part in the transformation of the parsed code which normally occurs deep within the compiler below all of the libraries. This makes the programmer a peer with the compiler writer from the perspective of other languages. API design in Lisp takes on a flavor of language design. Alas, this also makes Lisp code hard to enter into the middle of, and requires that the programmer do part of the work that a compiler normally would do, in reading and writing code that is less intuitive for the brain and more intuitive for the computer.

Haskell pushes the notion that mathematics has mastered much of complexity itself and that managing its manifestations is largely the job of the compiler. Other languages such as C++ make the compiler work as hard but nowhere near as smart. Haskell is extremely terse and produces extremely reliable software but depends on the programmer having a very deep understanding of the mathematics of what they are doing. In a certain sense, it places the burden on the programmer again as in assembler, but in a completely different domain.

Java and friends assert that the complexity should be managed and maintained through the help of intelligent editors and other tools. Again as in Unix, there is a kind of linguistic conservatism at work, this time borne of maximizing the help provided by the assistive tool chain rather than out of frugality or a minimalistic philosophy. A key difference between the two is the amount of learning foisted upon the programmer versus the amount of effort expended by the tool developer.

Io, Lua and Scheme are all almost negative statements about complexity, but each asserts that the language must be kept small and out of the way of the programmer to reduce complexity. Minimalism is almost the dominant theme of the software world, each movement seemingly a reaction to a lack of minimalism in some aspect of the movements that came before: structured programming a reaction to a lack of maintainability (too much code), object-oriented programming a reaction to a lack of reusability (too specific code), functional programming a reaction to a lack of terseness (code too far removed from mathematical roots), XML-based programming a reaction to an abundance of low-quality trivial parsers, “low ritual” languages a reaction to languages or libraries with much boilerplate or XML usage, type inference a reaction to verbose typing.

The central paradox of all this being that our one and only tool is to create a new abstraction, and each new layer makes the complete system much more complex. The only solution to that would be a complete re-imagining of computer science and programming from the ground up. But to do such a thing would be costly, as one would have to throw out all of the intellectual capital invested in the system we already have.

We are seeing some cost-effective attempts at merging these “simplifying” layers. ZFS is a strong example, merging several different layers into a few subsystems and a group of utilities. However it is still quite complex itself, perhaps more complex than the things it replaces. More is needed. Another example is .NET, which replaces some five or six brittle API interfaces with linguistic interfaces that are much less brittle. This comes at no small cost in performance, both on the CPU and terms of storage and memory resources.

Where do we go from here? I think some honest questions are called for.

  1. Is there an alternative to creating new abstractions for progress in programming and computer science? Is appropriately merging abstractions the best way forward?
  2. Will there come a point at which the cost of rewriting all of these layers will be lower than putting up with their detrimental effect on performance (and sanity)? Is merging layers a viable option for growth into the future, or does that still increase complexity too much? Administering ZFS, for example, is harder than administering a regular filesystem but also much easier than managing RAID, a filesystem, and a physical disk abstraction layer. Is the code complexity increase justified?
  3. As software evolves, dependent software breaks. Zero-defect programming notwithstanding, is there a way to manage change in such a way that grants backwards compatibility without sacrificing huge amounts of time, energy, and performance? Can we avoid the combinatorial explosion of software dependency versions and problems without resorting to unversioned, perfect, unchanging Dijkstra-like programming (which appears to be beyond reach of normal humans)? I think the Ruby community (especially the Rails subset thereof) are going to run into this issue in a way that conservative (by today’s standards) Unix programmers haven’t, and I halfway wonder if they will find a clever solution.
  4. To what extent are these problems we are witnessing today manifestations of the limits of the textual-linguistic basis of computer programming? How do you manage change and interdependency in a non-linguistic programming environment such as Subtext? What about abstraction? Hopefully Jonathan Edwards will produce something in which a large test can be attempted and some of these questions can be answered.