Daniel's blog

Tablify CSV Utility

2009-09-11T00:26:40-06:00

Tablify is a tool to render CSV into tables of various formats, including HTML, tbl, and character art (both ASCII and Unicode). What motivated me to write this was discovering the way sexy cool UTF8 box characters. They appear to render really well in Apple’s new Menlo font—at least substantially better than they do in Microsoft’s Consolas font which was my previous fave. They seem to have a lot in common.

Obligatory screenshot:

It also does HTML and tbl(1) output.

For kicks, I decided to write it in Haskell. I was actually contemplating what language I am the most fluent in, and I found myself running a lot of sloccount and wondering how exactly to compare my output in various languages. Two ways that seemed promising were to compare by number of files I've created and by number of lines of code. Here’s what I found:

Language	# of Files
PHP	196
Python	158
Ruby	146
Haskell	144
OCaml	88
Prolog	75
Lisp	63
C	44
Erlang	41
Clojure	30

Language	# of Lines
PHP	12,749
Python	10,476
Ruby	7,051
OCaml	5,817
Lisp	5,058
Haskell	3,256
C	1,412
Prolog	1,349
Clojure	1,177
Erlang	964

I think there are some interesting natural divisions in here. For example, PHP, Python and Ruby are at the top in both categories. This reflects the fact that most of my professional programming experience has been in these languages. Towards the bottom, there’s my old friends C, Erlang and Clojure. Clojure undoubtedly is there because it’s the language I'm the newest with; the others because I don’t really like them that much (no offense, Erlang, I think you’re great).

Funny thing about Prolog is, I think I start a lot of programs in Prolog but I don’t finish very many of them. Lisp has the opposite phenomena; it looks like my average Lisp file is fairly large, or else I have some fairly large Lisp files. I think I've only ever really written one small-to-medium sized app in Lisp and it certainly took more code than I expected it to.

I think what’s interesting about this table is the stuff in the middle: OCaml and Haskell. I haven’t really written an OCaml program in some time but I still think of it as comparable in power and utility to C++, whereas Haskell is more on par in terms of complexity (thankfully neither is syntactically). So I started looking through my little files and I found a surprising spread of functionality amongst my Haskell programs. I had a few network servers, several parallel calculations, and some other stuff. Another interesting thing is that I didn’t really see any Haskell programs sitting in a broken incomplete state. There were several incomplete programs but few broken ones. I even found a program called “ritual.hs” which I had apparently started but abandoned for doing nothing but exactly this kind of linguistic archeology for me.

As I was making my little chart I happened on the Unicode box drawing characters and the idea was born. So I decided to try doing it in Haskell and see if my attitude towards the world’s most useless programming language has changed.

As usual, Haskell made some of the program trivial and other parts rather difficult. I didn’t find the I/O to be particularly hard. Most of the hair pulling came from changing interfaces and learning more about the module system, compounded by some issues having to do with the update to Mac OS X 10.6.

Initially, I thought I would want my internal data structure to be an array, so I used Data.Array. I've since changed it to a list of lists, which would have been the obvious choice, and removed quite a bit of code in the process without (to my knowledge) affecting the runtime characteristics much.

There was one rather nasty stack-blowing bug. I found a fair amount of advice online for fixing it but on reflection all I had to do was replace foldr with foldl' in one place. This took an hour or two.

There are many places I feel the code is obtuse or poorly factored. Reading Real World Haskell gives me the impression that I'm still an early grade of Haskell programmer. The book does a good job of showing the evolution of code from horrifying to mediocre to grand. Mine’s on the horrifying side in places, on the grand side in others, but not very far from the middle anywhere. I notice in addition to isolating the monadic code, I tend to try and limit the amount of it. There were several places, particularly in the HTML generation facility, where I thought I could eschew some complexity with the appropriate monad if only I knew which one and how to go about it.

There’s also a built-in HTML combinator library I could be using instead of the homebrew one I made, as well as a third-party library that provides a ByteString-based CSV parser. I tried to port over to it but ran into issues with ByteStrings and String literals, then found the Data.ByteString.Char8 module and the third-party Data.ByteString.UTF8 module, but none of these seem to be using the same ByteString. For example, if I import Data.ByteString.UTF8 and use its fromString function, I can’t seem to pass it to the output function Data.ByteString.putStrLn. This is currently one of the other big issues I have.

So essentially, while I have understood monads for some time at least conceptually, I seem to lack both the library knowledge of which ones to use when, and the experiential knowledge of how to apply them and perhaps create my own. Obnoxiously, I see a fair number of little examples of monads in the programs I wrote to learn Haskell but I don’t see how to apply them inside an actual program effectively.

I guess you could say my relationship with Haskell continues to be characterized by love of the ideals and hate of the hardships in getting to proficiency. I suspect a big part of the problem is trying to learn Haskell on one’s own. Yet, even if I write poor Haskell, I can be assured that if it compiles, it’s likely to be correct.

Also next on my list of things to learn for Haskell is QuickCheck. My eyes glazed over when I read about coarbitrary: “The coarbitrary method interprets a value of type a as a generator transformer. It should be defined so that different values are interpreted as independent generator transformers.” Sure. I’ll get right on that.

Highlights of my code:

-- from FixedWidth.hs
-- given a table, return a list of the maximum widths of each column
columnWidths :: Table -> [Integer]
columnWidths table = map (maximum . map length) $ transpose table

-- from HTML.hs
-- escapes the bare minimum of characters for safe embedding in HTML
htmlEscape :: String -> String
htmlEscape html = foldl' process html mappings
    where
        mappings = [("&", "&amp;"), ("<", "&lt;"), (">", "&gt;")]
        process text (find, replace) = subRegex (mkRegex find) text replace

Pretty much everything else is a mess. I welcome any advice.

In Defense of Unreason

2009-08-21T01:13:34-06:00

This is my analysis of Republicans, religion and the triumph of unreason by Johann Hari.

First of all, in broad strokes, I agree with the description of the symptoms in the essay. The American right is basically living in another universe. When Bush was in power, the liberal conspiracy theories were plausible: that Bush wanted to invade Iran, for example. The new conspiracy theories are just insane. I am reasonably certain that Obama’s first health care reform bill will be a major disaster, but only because they are rushing to get it through and with a bunch of democrats in power it’s like to be padded with tons of pork. The fears that are being passed around are completely absurd. Isn’t it enough to be afraid of the likeliest case, that we’ll just be saddled with more useless government? I hope Obama can do better, but statistically his odds are not great.

However, I think it’s silly to blame religion for this problem. With atheism at around 15% of the population, Obama is not in power because of some kind of nationwide repudiation of religion. This is the chief vanity of the left, that it represents a harbinger of the kind of “rationality” espoused by certain intellectuals who happen to be atheists.

It’s tempting to put the blame at religion’s doorstep because it was supposedly the rallying cry of the right. But the truth about the American right is that it is a massive hodgepodge of vastly different groups with almost nothing in common. And guess what? So is the left. Politicians use propaganda to acquire power, not because they actually believe it. How can you accuse Bush and his cronies of corruption and then simultaneously blame their ideals? Obviously if their ideals meant anything they wouldn’t have been corrupt in the first place. Instead we hate them for what they've done and then we hate them more for being hypocrites. This is a game two can (and will) play.

It’s also tempting to blame religion because it is a handy target and has few defenders. Especially when couched in gerrymandered terminology like “organized religion.” What makes this article’s kind of analysis so disgusting to me is the assumption that the essence of religion is faith. In American Christianity, the foundation of the religion is faith, but this is like saying the problem with Congress is that they meet in a building with a floor. The problem with the American right is that they’re credulous, but guess what? That’s the problem with the American left as well! The common factor I see is the American education system. The 2.2% that are homeschooled would be just as statistically irrelevant to the counter-argument as the atheists above. If what you hate is Christianity, why not just come right out and say it and why?

For every Bible-thumping redneck we love to hate, there’s a suburban driving a gas guzzling SUV with a bike rack on top who lives 20 miles from where they work. Evidence from psychological studies that abortions are highly detrimental to the mother’s mental health are discarded from the abortion debate because that debate is about “rights,” but they then turn around and spin arguments about the environment’s health being grounds for restricting rights. Which one is it? You can argue both rationally. It would be better economically to legalize weed and prostitution and regulate them. Which rational argument do you want to make?

This infatuation with rationality is handy in a debate, but what do you do if you have contradictory facts? Upon what do you base a rational debate about ethics? Assertions? Religions supply a foundation for these kinds of debates that cannot otherwise be deduced. There is no fundamental particle that carries good or evil. This means science is of limited use in many of the most important debates. You'd think the author of this essay encourages people to marry on the basis of empirical evidence from observation and calculations! Apparently mechanical hearts can also bleed!

The best part is, this author doesn’t have a single statistic to back up his assertions about religion. It’s all speculation. Angels dancing on the head of a pin, if you will. But because it is so well argued, it will further reinforce in the mind of the intellectual left that they’re fighting a war against religion rather than a war against propaganda and irrationality. In other words, this article is itself leftist propaganda that distracts from the real issues with made-up facts! Isn’t that the very thing this article is against? This self-certainty about ones interpretation of reality is groupthink exactly and it is what causes the continual pendulum sway of American politics.

Groupthink is certainly preventing the health care debate from taking shape the right way. I think intentional meddling might be a bigger problem. Responding to redder groupthink with bluer groupthink isn’t a solution. Mr. Hari would do better to analyze both sides of the debate with the same kind of cynicism.

PostgreSQL 8.4: Windowing Functions

2009-08-12T23:18:03-06:00

PostgreSQL 8.4 is the first version to introduce OLAP functionality in the form of the windowing functions. These functions are new to me too but they look quite interesting and powerful. They will greatly simplify combining many large SQL statements into simpler ones for complex reporting, as well as make it possible to push some “fixup” processing that depends on ordering back into the database.

As an example, I'm going to be using the orders table from the (Dell store example database)[http://pgfoundry.org/projects/dbsamples/]. A few rows from this table look like this:

SELECT 
  orderid, orderdate, customerid, 
  netamount::text::money, 
  tax::text::money, 
  totalamount::text::money 
FROM orders LIMIT 10;

orderid	orderdate	customerid	netamount	tax	totalamount
1	2004-01-27	7888	$313.24	$25.84	$339.08
2	2004-01-01	4858	$54.90	$4.53	$59.43
3	2004-01-17	15399	$160.10	$13.21	$173.31
4	2004-01-28	17019	$106.67	$8.80	$115.47
5	2004-01-09	14771	$256.00	$21.12	$277.12
6	2004-01-11	13734	$382.59	$31.56	$414.15
7	2004-01-05	17622	$256.44	$21.16	$277.60
8	2004-01-18	8331	$67.85	$5.60	$73.45
9	2004-01-06	14902	$29.82	$2.46	$32.28
10	2004-01-18	15112	$20.78	$1.71	$22.49

Without getting into the new features, it’s possible to generate interesting summary information. You might want to see how much you made each month, for example:

SELECT
  EXTRACT(MONTH FROM orderdate) AS month, 
  EXTRACT(YEAR FROM orderdate) AS year, 
  SUM(totalamount)::text::money AS total 
FROM orders 
GROUP BY year, month 
ORDER BY year, month;

month	year	total
1	2004	$215,898.76
2	2004	$216,792.09
3	2004	$210,564.40
4	2004	$216,642.01
5	2004	$213,395.19
6	2004	$216,412.59
7	2004	$206,707.38
8	2004	$210,115.92
9	2004	$213,034.71
10	2004	$211,952.17
11	2004	$216,373.17
12	2004	$219,498.40

What we’re doing here is just grouping on some computed data, in this case, EXTRACT is giving us the information we need from the DATE to use it for grouping. An interesting fact about grouping is that it restricts you to relational calculations. You cannot, for example, do something different depending on where you are in the calculation. Grouping also applies to the whole statement, making it hard to make a complex report without using many subselects.

The window functions were created to support OLAP or Off-Line Analytical Processing, as opposed to OLTP or On-Line Transaction Processing which is what SQL RDBMSes are generally thought of. Despite this nobody has managed to give me an example of what OLAP is like in reality, so it must be a pretty specific and rare topic. However, the nutshell of windowing is that it lets you combine multiple different groupings in the same statement, or do calculations with respect to your current location in the generated result relation.

What if you want to also calculate the year-to-date earnings for each month? Without the window functions I'm not sure this is possible in straight SQL. What if you want to rank each month relative to the other months in terms of income?

Windowing essentially is a modification of an aggregate function with a new clause, OVER. There are some new functions too, like rank() which is essentially the row number. The OVER clause itself is built out of two optional expressions, an ORDER BY expression which is the same as the kind you already know, and a PARTITION BY which works much like GROUP BY with some extra features.

The rank of a given month is clearly related to the total amount made in that month, so its OVER clause will be ORDER BY total DESC since the largest amount should come first. Let’s add that to the query:

SELECT 
  EXTRACT(MONTH FROM orderdate) AS month,
  EXTRACT(YEAR FROM orderdate) AS year, 
  SUM(totalamount)::text::money AS total,
  rank() OVER (ORDER BY total DESC)
FROM orders
GROUP BY year, month 
ORDER BY year, month;

month	year	total	rank
1	2004	$215,898.76	6
2	2004	$216,792.09	2
3	2004	$210,564.40	10
4	2004	$216,642.01	3
5	2004	$213,395.19	7
6	2004	$216,412.59	4
7	2004	$206,707.38	12
8	2004	$210,115.92	11
9	2004	$213,034.71	8
10	2004	$211,952.17	9
11	2004	$216,373.17	5
12	2004	$219,498.40	1

This is already giving us some information we ordinarily would have had to infer or calculate on the other side. Now let’s say you want to also see the earnings year-to-date for each month. Our first try isn’t going to work:

SELECT 
  EXTRACT(MONTH FROM orderdate) AS month,
  EXTRACT(YEAR FROM orderdate) AS year, 
  SUM(totalamount)::text::money AS total,
  rank() OVER (ORDER BY total DESC),
  SUM(total)::text::money OVER (ORDER BY year, month) AS ytd
FROM orders
GROUP BY year, month 
ORDER BY year, month;


ERROR:  column "total" does not exist
LINE 6:   SUM(total)::text::money OVER (ORDER BY year, month) AS ytd
              ^

What this message is showing us is that our SUM(totalamount) AS total isn't visible inside the OVER clause of the other column. Fortunately, this is easy to work around by making the original query a subselect of the new query like so:

SELECT
  *,
  SUM(total) OVER (ORDER BY year, month)::text::money AS ytd
FROM
  (SELECT 
     EXTRACT(MONTH FROM orderdate) AS month,
     EXTRACT(YEAR FROM orderdate) AS year, 
     SUM(totalamount)::text::money AS total,
     rank() OVER (ORDER BY total DESC)
   FROM orders
   GROUP BY year, month 
   ORDER BY year, month) AS q
ORDER BY year, month;

month	year	total	rank	ytd
1	2004	$215,898.76	6	$215,898.76
2	2004	$216,792.09	2	$432,690.85
3	2004	$210,564.40	10	$643,255.25
4	2004	$216,642.01	3	$859,897.26
5	2004	$213,395.19	7	$1,073,292.45
6	2004	$216,412.59	4	$1,289,705.04
7	2004	$206,707.38	12	$1,496,412.42
8	2004	$210,115.92	11	$1,706,528.34
9	2004	$213,034.71	8	$1,919,563.05
10	2004	$211,952.17	9	$2,131,515.22
11	2004	$216,373.17	5	$2,347,888.39
12	2004	$219,498.40	1	$2,567,386.79

As an aside, you really shouldn’t use the money type in PostgreSQL. I only used it here to make the formatting a little prettier. In general you should use NUMERIC(X,2) where X is a number larger than you expect to ever have in your database. If you have to deal with multiple countries (or might someday) you should also track the currency code and perhaps even the conversion rate to your native currency code on that particular day. It gets complex in a hurry.

Fibonacci in PostgreSQL

2009-08-12T09:54:17-06:00

I should have brought this up in the previous post. The question on everyone’s mind must be, if SQL has recursion, how do you compute the Fibonacci sequence? Here’s how:

SELECT fib FROM
(WITH RECURSIVE fibonacci(fib, fib1) AS (
  SELECT 1, 0
  UNION
  SELECT 1 :: NUMERIC, 1 :: NUMERIC
  UNION
  SELECT fib + fib1, fib FROM fibonacci
)
SELECT * FROM fibonacci) AS fibs LIMIT 1000;

I couldn’t see a way to refer to the previous two rows, so instead I just carry along the previous row’s value inside the current row, so the next calculation can see the previous two values. I wrapped the whole thing in a SELECT to hide fib1. The NUMERIC type gives you arbitrary-precision math (or at least to 1000 decimals). Without it, the maximum number you can calculate is 46 because 1134903170 + 1836311903 overflows the normal integer.

Postgresql 8.4 Recursive Queries

2009-08-11T11:17:47-06:00

Apparently I missed the news: PostgreSQL 8.4 came out on July 1st! This release brings two new features and greatly improves one old feature. This post will be about recursive queries using WITH.

A classic problem in SQL is how to model trees. In web development you often have to store nested subcategories or model pages as files inside folders. The simplest proper way to model this relationship in SQL is to add a self-referencing column to the table to point to a row’s parent row. For example, you might represent the file path /foo/bar/baz.html with the following table:

id	name	parent_id
1	/	NULL
2	foo	1
3	bar	2
4	baz.html	3
`fs` table

This structure has a number of pleasant features from a relational perspective. For one thing, you can rename a folder by modifying just one row. You can insert a new node anywhere in this hierarchy by just knowing the parent folder’s ID. You can also move a node anywhere in the hierarchy by modifying that one ID.

Unfortunately, from an implementation perspective, this structure sucks for doing many common operations: computing a string version of a path, for example, or listing all the descendants under a given node. There are many simpler, less normalized methods that are better at these common operations at the expense of some relational normality. (One I have used is to have outside software maintain a text path in the database.)

With recursive queries it’s now possible to recreate the string path, find all descendant nodes, and get all of the parent nodes up to the root. I'm going to show you how to do each of these things with just one query apiece.

All of the magic comes down to the WITH statement. This statement is provided both to support recursive queries and also for common table expressions or CTEs. If you find yourself repeating the same table definition in your query you can pull it out and put it above the statement in a WITH clause. You can think of this as a way to make a temporary view just for a single SQL statement.

The basic structure you’re looking at is this:

WITH temp_view(col1, ...) AS (SELECT ...)
SELECT ...;

If you use temp_view inside the SELECT within the WITH statement, you have to add RECURSIVE:

WITH RECURSIVE temp_view(col1, ...) AS (SELECT ...)
SELECT ...;

All three of the queries I'm going to show you rely on the recursive flavor in order to do their work. There is a design pattern in the inner SELECT whenever a recursive query is written, which is related to the notion of recursion. You must define a base case and an inductive case. The base case sets up the initial conditions for the recursion, usually the first set of rows for the result. The inductive case uses the rows which have already been computed to compute more rows and decide whether or not the recursion is complete.

For example, to find the breadcrumbs for a given path, the base case is the initial path. The inductive case is just to find the parent path. If there’s no parent path, the recursion is complete. Inside the SQL, you’ll do a naïve SELECT and UNION it with a SELECT that uses the recursively defined name. You can read more about this in the (PostgreSQL Documentation)[http://www.postgresql.org/docs/8.4/interactive/queries-with.html]. So the SQL will look like this:

WITH RECURSIVE breadcrumb(id, name, parent_id) AS (
  SELECT id, name, parent_id FROM fs WHERE id = 4
  UNION
    SELECT fs.id, fs.name, fs.parent_id FROM fs, breadcrumb
    WHERE breadcrumb.parent_id = fs.id)
SELECT * FROM breadcrumb;

The base case gets all of the information for the row we’re interested in, which is the row with ID 4. The inductive case takes the previously generate row and gets the row from fs with the ID of the parent ID of the row with ID 4, which in this case is the row with ID 3. The next iteration finds the row from fs with the ID of the parent ID of the row with ID 3, and so on.

If you’re following along at home, run this SQL to set up our initial conditions:

CREATE TABLE fs (
  id SERIAL PRIMARY KEY, 
  name VARCHAR, 
  parent_id INTEGER REFERENCES fs(id)
);

INSERT INTO fs (name, parent_id) 
VALUES ('/', NULL), ('foo', 1), ('bar', 2), ('baz.html', 3);

Now you should get this output from the above SQL:

id	name	parent_id
4	baz.html	3
3	bar	2
2	foo	1
1	/	NULL

Let’s work on the other two problems: getting a string path and listing descendants of a given node. We’ll use the root node for the sake of fun and add a few extra nodes so that the tree is fuller and tests our query a little better:

INSERT INTO fs (name, parent_id) 
VALUES ('under foo', 2), ('under bar', 3), ('under bar 2', 3), 
       ('baz', 2), ('bazzle.html', 8);

Now, to do this query the base case would seem to be the root of the tree. In thinking about what kind of result we want, I think for convenience we might want the current node’s name and the full path of this node in addition to the regular columns. We’re going to assume that the root of the tree is the node with ID 1, which gives us this base case:

WITH RECURSIVE path(name, path, parent, id, parent_id) AS (
  SELECT name, '/', NULL, id, parent_id FROM fs WHERE id = 1

Now the inductive case just takes that case and computes the descendant nodes of that node by looking at their parent_id:

  SELECT
    fs.name, 
    parentpath.path || 
      CASE parentpath.path 
        WHEN '/' THEN '' 
        ELSE '/' 
      END || fs.name,
    parentpath.path, fs.id, fs.parent_id
  FROM fs, path as parentpath
  WHERE fs.parent_id = parentpath.id

Calculating the path, the CASE statement you see there is a hack to prevent extra ‘/’s at the start of the path because of the UNIX convention of paths starting at /. Otherwise you see I'm just taking the name of this node and concatenating the parent path with my name, separated by /’s. So this is the full version:

WITH RECURSIVE path(name, path, parent, id, parent_id) AS (
  SELECT name, '/', NULL, id, parent_id FROM fs WHERE id = 1
  UNION
  SELECT
    fs.name, 
    parentpath.path || 
      CASE parentpath.path 
        WHEN '/' THEN '' 
        ELSE '/' 
      END || fs.name,
    parentpath.path, fs.id, fs.parent_id
  FROM fs, path as parentpath
  WHERE fs.parent_id = parentpath.id)
SELECT * FROM path;

Let’s see what it does!

name	path	parent	id	parent_id
/	/		1
foo	/foo	/	2	1
bar	/foo/bar	/foo	3	2
under foo	/foo/under foo	/foo	5	2
baz	/foo/baz	/foo	8	2
baz.html	/foo/bar/baz.html	/foo/bar	4	3
under bar	/foo/bar/under bar	/foo/bar	6	3
under bar 2	/foo/bar/under bar 2	/foo/bar	7	3
bazzle.html	/foo/baz/bazzle.html	/foo/baz	9	8

Looks like it worked! But now that we have this statement it seems like it would be handy to make it a function so you can find the descendants of any particular node. The path might not look right but at least you could get relative children. Let’s try it:

CREATE FUNCTION children_of(root_id INTEGER) 
RETURNS TABLE (name VARCHAR, path VARCHAR, parent VARCHAR, id INTEGER, parent_id INTEGER)
AS $$
WITH RECURSIVE path(name, path, parent, id, parent_id) AS (
  SELECT name, '/', NULL, id, parent_id FROM fs WHERE id = $1
  UNION
  SELECT
    fs.name, 
    parentpath.path || 
      CASE parentpath.path 
        WHEN '/' THEN '' 
        ELSE '/' 
      END || fs.name,
    parentpath.path, fs.id, fs.parent_id
  FROM fs, path as parentpath
  WHERE fs.parent_id = parentpath.id)
SELECT * FROM path;
$$ LANGUAGE 'sql';

All I've done here is wrap the previous statement with some stuff to make it into a function and replaced 1 with $1, which makes the value the first parameter of the function call. For extra convenience, if I were using this as the foundation of a CMS or something where I would expect to need the path frequently, I might also make a view:

CREATE VIEW paths AS SELECT * FROM children_of(1);

It’s worth noting that recursive queries probably do not have excellent performance characteristics (I'm not certain but it seems likely.) If you do run into performance problems with this, you might consider looking into (materialized views)[http://tech.jonathangardner.net/wiki/PostgreSQL/Materialized_Views] which can be implemented manually in PostgreSQL using triggers.

If you prefer, you could instead create a function to find the path of a given ID:

CREATE FUNCTION pathname(id INTEGER) RETURNS VARCHAR AS $$
SELECT '/' || path FROM
(WITH RECURSIVE pathto(path, id) AS (
  SELECT name, parent_id FROM fs WHERE id = $1
  UNION
  SELECT fs.name || '/' || pathto.path, parent_id
  FROM fs, pathto WHERE fs.id = pathto.id)
SELECT * FROM pathto) AS pathto(path, id)
WHERE id = 1;
$$ LANGUAGE 'sql';

I was then tempted to make an index on this function, but PostgreSQL won’t allow computed indices using functions not marked IMMUTABLE, which is a way of certifying to PostgreSQL that it is a pure function. Because this function will return different values depending on the content of a table, it isn’t immutable, and the index could get out-of-sync with reality without PostgreSQL knowing it. Intuitively, if you rename a node, all of its children will then have different paths reflecting the new name but it’s unclear how you could make PostgreSQL aware of that fact.

Plan 9's Advantage

2009-08-07T02:01:07-06:00

Plan 9 is Unix sans castration.

The central idea behind Plan 9 is that a filesystem is a dandy interface to all kinds of things. Take a look at the list of system calls in Plan 9, all 51 of them.

What kinds of things can be filesystems that typically aren’t? All your processes—post a note (remember signals in Unix?) by writing to their control file, see what files they have open, etc. Your network interface. Need NAT? Mount the gateway’s network devices locally. Get a connection and listen or connect to a remote host by opening a file and writing to it. Access a database through the filesystem. Access the web through the filesystem. Access FTP or any other remote filesystem through the local filesystem. Manipulate this window on the screen through the filesystem. Draw graphics through the filesystem. Access your keychain through the filesystem. Access your sound card through the filesystem.

Exposing everything through the filesystem has three principle benefits:

Now any aspect of your machine’s hardware, display, or any running software can be accessed from other places on the network. Or vice versa.
Want to improve security by jailing a process? Remove the interfacing files from the namespace of that process.
Access to your abstraction is democratized and language-agnostic.

The first two points get made a lot in Plan 9 circles and are the backbone of a secure distributed environment. But the third point is one that’s been fascinating me. The echo service on Plan 9 is a shell script that runs cat. You can use regular shell scripts to connect programs and devices across machines. If you find a filesystem that provides access to a given device, protocol, or whatever, every language on your machine gets uniform access to that device, protocol or whatever without any bias towards a particular one.

This also means that in Plan 9, writing a filesystem is better in many cases than writing a library. A library can only be used from C; a filesystem can be used from C, the shell, awk or any other language. Furthermore, a filesystem can be exported over the network or imported from another host. Hooking up a file to another file doesn’t sound very interesting, but if one file is the output of a progress bar application and the other is your soundcard’s control file, it becomes interesting. Or if one file is your mail and the other is your Jabber account, and between them is a little AWK script that pulls out the subject. Or if one file is your hard drive’s SMART status and the other is your SMS gateway. The mere thought of being able to combine these things with trivial shell scripts in the grand Unix tradition is quite alluring.

How to Manage a Website, Destructively

2009-07-13T00:29:57-06:00

For a little while I was thinking about managing my website with this before I decided to use Webby instead. I changed my mind for three reasons:

My system sucks at syntax highlighting. With Webby, you just use Ultraviolet.
There isn’t a good way to do escape characters in m4, so some blog posts break spontaneously.
Webby somehow manages to be faster.

I’m not sure I’m going to abandon this idea completely though. It might be a good project to do with Plan 9. If I remake the site in Clojure then I probably will go for something more dynamic though.

Anyway.

Conception

This project is a hate crime.

I hate websites. Especially I hate software for managing them which is too limited and doesn’t build on the fundamental flexibility of the platform. I also hate reinventing the wheel.

It occurred to me as I was playing with Webby that I hate Rake, because it’s slower than real make and less powerful and because I hate Ruby now too.

Why not use make? Make has a great advantage over other systems: you can write pattern rules that generate some file type from some other file type and now the whole thing knows how to transform all your files from one thing to another. That’s handy.

Thinking about what everyone needs in a website, they really only need a system to get some text into a giant ball of HTML and stuff that into their template somehow. Perhaps also doing some other template-like crap at the same time. There are lots of great tools for managing your HTML glorp, unfortunately many of them are Ruby specific, but there’s Textile, Markdown, Haml and others.

Interestingly, you only really want to deal with Haml for the template and Textile or Markdown for the structure. Textile looks more like plain text to me than Markdown but Markdown is better at dealing with code and nested stuff—I seem to have to resort to raw HTML a lot less, and it’s bi-directional. I like having choices.

So this seemed like my formula: glorp + template = HTML.

A Twist

How do you deal with things like the title? The title ostensibly appears in the glorp somewhere but it also needs to wind up at the top of the file, in the <title> tag.

This seems like metadata to me. So the new formula looks like this: glorp + metadata + template = HTML.

A Taste

‘bin/build-rules.sh sites
make


    [fusion@Anatolia Makesite] % make rsync
    haml template.haml > template.m4.pre
    cat header.m4 template.m4.pre > template.m4
    rm template.m4.pre
    redcloth site/about-me.textile > site/about-me.phtml
    ruby bin/metadata.rb site/about-me.phtml > site/about-me.meta.m4
    m4 -D __metadata__=site/about-me.meta.m4 -D __body__=site/about-me.phtml template.m4 -D __dir__=site > site/about-me.html
    ruby bin/webby2ms.rb m4 site/how-it-works.webby > site/how-it-works.meta.m4
    ruby bin/webby2ms.rb content site/how-it-works.webby > site/how-it-works.textile
    redcloth site/how-it-works.textile > site/how-it-works.phtml
    m4 -D __metadata__=site/how-it-works.meta.m4 -D __body__=site/how-it-works.phtml template.m4 -D __dir__=site > site/how-it-works.html
    perl bin/Markdown.pl --html4tags site/index.markdown > site/index.phtml
    ruby bin/metadata.rb site/index.phtml > site/index.meta.m4
    m4 -D __metadata__=site/index.meta.m4 -D __body__=site/index.phtml template.m4 -D __dir__=site > site/index.html
    cp -r static output
    bin/install.sh output  site/about-me.html site/how-it-works.html site/index.html
    site/about-me.html -> output//about-me.html
    site/how-it-works.html -> output//how-it-works.html
    site/index.html -> output//index.html
    rsync -vrzp -e ssh --chmod=u+rwX,go+rX output/* clanspum.net:sites/beta.storytotell.org
    sending incremental file list
    about-me.html
    how-it-works.html
    index.html
    css/style.css
    downloads/
    downloads/makesite.tar.gz
    images/.DS_Store
    images/accept.png
    images/accept50.png
    images/bg_body.png
    images/bullet_bottom.png
    images/bullet_right.png
    images/content_bg.png
    images/email_open.png
    images/email_open50.png
    images/h1_bubble_bg.png
    images/h2_bubble_bg.png
    images/meta.png
    images/page_edit.png
    images/page_edit50.png
    images/sidebar_section_bg.png
    images/sidebar_section_bg_over.png
    images/tag.png
    images/title_bg.png
    images/user.png
    images/user50.png
    images/world_link.png
    images/world_link50.png
    
    sent 94105 bytes  received 1851 bytes  38382.40 bytes/sec
    total size is 234097  speedup is 2.44

Templates

For the final substitution, I don’t need much; all I really need is a way to substitute some metadata variables and swap in the content into the big hole in the middle of the template. Suddenly I remembered M4, the ancient Unix macro processor. Yeah, it’s fugly and gross, but like make, it isn’t going anywhere and it’s definitely powerful enough.

Using make

If you’re going to use make effectively, you need to figure out how to encode your information in the actual filename of the file. Make is going to use the extension of your file to figure out what to do with it.

Since I’m using m4, I decided to make a separate file for the metadata and make it m4 format. This is the extension .meta.m4. Also, pre-processed HTML has the extension .phtml. Now all I had to do was come up with the m4 command line to process a template with the body and the metadata. I came up with this:

%.html: %.meta.m4 %.phtml template.m4
	m4 -D __metadata__=$< -D __body__=$(word 2,$+) $(lastword $+) -D __dir__=$(dir) > $@

I decided to use foo for all my m4 variables to reduce the chances that it runs into some text I’ve already defined in my files. body is the filename of the HTML glorp, metadata is the filename of the metadata and dir is the path we’re at when we invoke m4 (important for calculating relative paths).

Template Structure

Our template must begin with this:

include(site/header.m4)

These two lines do two things: import the metadata file using the variable from the command line and define the relative path back to the root (handy for links to CSS and whatnot in the base of the site).

Later on I decided that template.m4 ought to be built out of parts, I extracted this to header.m4 and define template.m4 to be built out of Haml using this rule:

template.m4: template.haml
	haml $< > $@.pre
	cat header.m4 $@.pre > $@
	rm $@.pre

Now I can just use my m4 variables from within Haml instead of worrying about filling in the template with Ruby! :D

Making Glorp

To make the HTML glorp, I have two patterns for handling textile and markdown:

%.phtml: %.textile
	redcloth $< > $@

%.phtml: %.markdown bin/Markdown.pl
	perl bin/Markdown.pl --html4tags $< > $@

(I have the markdown depend on the Perl file because it’s in the distribution and may change.)

Both of these just render the text and produce the interior HTML.

Handling Webby

I had already written a utility to convert my Typo blog posts to Webby format. Webby’s format is interesting; basically it’s a YAML wrapper on top of plain text. The wrapper specifies the format of the text and the YAML contains the metadata. So at this point I created a utility just to deal with Webby files and renamed all of them to end in .webby.

The utility takes a file and a single command, which is either ‘m4’ or ‘content’ depending on whether it’s generating the content or the m4 metadata. To make the content, it just slices it out and prints it; to make the metadata it parses the YAML and converts it to a series of define() statements in m4.

Make was informed:

%.meta.m4: %.webby bin/webby2ms.rb
	ruby bin/webby2ms.rb m4 $< > $@

%.textile: %.webby bin/webby2ms.rb
	ruby bin/webby2ms.rb content $< > $@

In a future version, it might be able to determine the file type to create, but for right now it was convenient just to specify Textile since all my old blog posts are in Textile anyway.

Generic Metadata

It occurred to me that you could find some metadata just by introspecting the file, such as to look for the first h2/3/4 for the title or the ctime of the file for the creation time. So I wrote a short script to create a .meta.m4 from a .phtml. It’s important to keep the Webby lines above these rules or it will use these rules to produce .meta.m4 from .webby files when there is a better method available.

%.meta.m4: %.phtml bin/metadata.rb
	ruby bin/metadata.rb $< > $@

Non-Recursive Make Structure

At this point, I could generate whatever html file I wanted from my templates and my old content, but I’d have to ask for it by name. I needed some kind of make rules that would find all the old content and build it.

As it turns out, there’s still recursion in non-recursive make, it’s just recursion of reading make files and processing their contents before doing all the work.

First of all we need to understand how to invoke make non-recursively. I read the famous paper, Recursive Make Considered Harmful and ran into it the other day and saw the link off to Emile van Bergen’s implementation.

I had to follow this in a very regimented fashion, so I wrote a script which recursively builds Rules.mk files for all of the subdirectories under some directory. This is very tailored to this app and will need to be extensively overhauled if I want to generate something other than HTML or use it in another app. But it’s also quite short and illustrates recursive calling in the shell.

#!/bin/zsh

# bring in the header
function build_rules() {
	...
}

function build_rules_rec() {
	build_rules $1 > $1/Rules.mk

	for dir in $(find $1 -type d -maxdepth 1 | sed '1d'); do
		build_rules_rec $dir > $dir/Rules.mk
	done	
}

build_rules_rec $1

The essence of build_rules is that it looks in a directory and creates recursive loading of Rules.mk in each of the subdirectories, and adds all of the .webby files in the current directory to $(TGTS_HTML) with the extension changed. We’re being declarative here, folks.

Rock and Roll

The rest of the makefile is pretty much cake; define all to be targets, define targets to be $(TGTS_HTML), define clean to be rm -f $(CLEAN). Run make -j4 and watch as your website gets built with SMP!

Of course now you have the problem of installation. I didn’t want the installation method to be decoupled from the list of targets we’re building so I wound up writing a shell script which does some path nastiness to copy the target files preserving their path parts into the output directory. The output directory is first made by making a copy of a static folder which contains things that don’t need to be expanded like images and CSS files.

Then I added an install target and an rsync target which copies it up to the website.

That’s it in a nutshell.

The Code

I’m sure you’re all quite desperate to give this a try. Download the source and hopefully it’ll still be working for you then.

On Optimization

2009-07-10T15:22:09-06:00

Frequently on the Clojure list somebody shows up and wants to know what’s the most optimal way to do something, you know, without any of that costly functional stuff, just pure speed. I have a feeling this comes up in other forums too so I thought I'd spend a minute and give the world my nasty opinion about optimization.

Every excellent programmer knows that correctness is harder to achieve than performance and more important. You can’t profile unfinished code. Optimally performing code is also usually larger and less clear—which means it has more places to hide bugs. It’s also usually less flexible. If possible, you'd rather have an obviously correct function to compare results with as you’re making an optimal version (I consider this more useful than unit tests, because it can generate unit tests).

Performance worrying is the last refuge of the lousy. Nobody takes broken but fast code over correct code, unless they don’t care about the program. Optimizing correct code is easy. You find the two lines that are slow with your profiler and fix them. This is much easier when you haven’t pre-written optimal-looking but enormous code. You have less code to change.

Optimizing code before you absolutely have to is also a violation of YAGNI. Great programmers don’t just spill out reams and reams of code, they also work hard to contain the codebase and remove redundant code. If my two line function performs within a factor of three of your ten line function, mine wins until you have proof that it is the bottleneck in this program.

And yes, every program has a bottleneck. But for most programs ordinary folk like us are going to write, that bottleneck is actually I/O or the network.

I once worked on a program that had to deal with building a bracket for playoffs in various sports. I spent a little time thinking about the problem and how I wanted to approach it and realized it’s just a binary tree and pulled out my log base 2s. By some fluke, the other programmer hadn’t thought of this yet and was just going to hard code it for 16 or 32 teams. Hard-coding it would definitely have been more lines of code, slower, and less flexible.

The major, overlooked way to achieve better performance in all your software is to do less work. In order to do less work you have to understand your tools and your fundamentals. Math is in your fundamentals! You should understand algebra, logarithms, a little calculus, a little probability, if you want to program well. SQL is one of your tools! Nine times out of ten when a webapp performs poorly it’s because of the N+1 query problem and it can be fixed with better, more intelligent SQL. If you’re storing data in a relational database, you should use it like a relational database rather than an object graph! If you want to store an object graph, find an OODB and make that your back end! Your performance and your LOC will thank you.

To sum up:

Think about the problem. Try to avoid doing some or all of the work. Use math!
If you have to do it, write the code as clearly and correctly as you can
See if you notice an actual performance problem.
Profile the code.
Rewrite the critical section. Preferably, it’s already a function and you can swap this in as the new version.
Ensure that your new version returns the same results as the old.
Don’t touch anything unrelated to the performance problem.

If you follow this pattern, you’ll get more done, your code will be higher quality and it will perform well with only a few minor sacrifices, and you won’t lose correctness.

Bonus tip: Don’t fight the language! Don’t write Java in PHP, or your PHP’s performance will go to hell. Don’t write C in Java or Java’s performance will go to hell. Don’t write Java in Clojure or Clojure’s performance will go to hell. Use the language the way it was meant to be used: you’ll hit all the paths that the implementor has optimized rather than the weird corner cases he hasn’t anyway. Plus, usually the extra work you do to try and obtain better performance will wind up costing more than just doing it the easy way. And don’t even start with this without profiling!

On Prolog

2009-07-10T15:03:19-06:00

For a couple days this week I was working on a clone of bc(1) in Prolog.

My original thought was that this would be quite perverted and cool, because you can do some interesting things with Prolog, like DCG’s and you can create and destructure Prolog terms pretty easily. For example:

?- foo(bar,baz) =.. X.
X = [foo, bar, baz].

?- Y =.. [foo, bar, baz].
Y = foo(bar, baz).

The key here is the =.. operator, which unifies a list of terms on the right with a Prolog-style term on the left. I was hoping to have a DCG that looked something like this:

term(Expr) --> "(", whitespace, expr(Expr), whitespace, ")".
term(-Expr1) --> "-", whitespace, expr(Expr1).
term(N) --> number(N).

expr(Left ^ Right) --> powterm(Left), whitespace, "^", whitespace, powterm(Right).
% and so forth

Problem is, Prolog’s DCGs don’t really want to do this and I found a mailing list article about it that took the wind right out of my sails. Basically, it says, to do this you can’t use a DCG without passing along some extra information.

Anyway, I just wanted to share that for some reason. I have a grammar right now that looks like it should be working, but it has associativity backwards. To wit:

> 2 ^ 1 + 1 * 2
2^ ((1+1)*2)
16

The correct answer would be (2^1) + (1*2).

If any of you want to take a crack at this and brighten my day, the code’s attached.

Thoughts About Plan 9 and Inferno

2009-07-10T14:38:22-06:00

I've been thinking about paradigms of programming lately. Prolog always seems to bring that to mind and I'm not sure why, but Prolog has been on my mind. And Plan 9 and Inferno.

There’s an interesting comparison between Inferno and Java I looked at. It seems misguided to compare these two systems but perhaps they should be compared. They differ at the very lowest level, as in, what is the virtual part of this virtual machine? Java uses the VM strictly as an execution architecture for a single program. Inferno gives you a whole OS on top of that VM; you get multiple processes, each potentially having multiple threads. Yet it seems to somehow be thinner than Java in many respects.

Inferno is really just another way to run Plan 9, just a friendlier, sneaky way of getting people who use normal OSes to use it. Of course it can run on the bare hardware too. Which means there’s really six ways to run Plan 9:

Plan 9 on bare hardware
Plan 9 from Userspace + Unix of your choice
9vx, user-mode virtualized Plan 9
Inferno on bare hardware
Inferno under the OS of your choice
Glendix (port of Plan 9 userland to Linux)

And of course that’s omitting traditional virtualization software like VirtualBox and Parallels.

I'm suspicious of ideas that come in six or ten boxes. Seems like the idea doesn’t really have a home. Which one of those is the preferred way to get the Plan 9 experience? Hard for me to say. I have tried all of the virtual methods I can find and still don’t find myself using it, even though I love the theory.

The other weird thing about Plan 9 is that it’s a complex thought. I don’t mean the namespacing and 9fs for all servers, that makes sense, it’s the whole Unix pantheon that has to come along too. I guess it’s a testament to the portability of the idea.

The last time I looked at the programming language for Inferno, Limbo, I thought it looked really bizarre. This time it looks like C with a big dose of ML and a small dose of C++. The thought of programming in a language with so much syntax in an editor that doesn’t support syntax highlighting is pretty intimidating. But nobody seems to be using anything but Acme though or there would be an Emacs mode and a TextMate mode, right? So maybe it’s manageable. Presumably the compiler does a great job of telling you how you screwed up.

Another weird problem for me is the parsing problem. The advantage of networking with XML or JSON is that you already know the object model; you can pretend you’re dealing in structured data (objects) instead of bytes. Not that it’s particularly hard to write a parser with the old school tools, but is yacc the best we get to have? When I see the man page for iyacc I get a woozy feeling. Plan 9 was made by and for people who thought Unix was perfect at everything except networking. Inferno seems almost to have been made sarcastically to make a version of Plan 9 that competes with Java. But Java tries to solve the whole problem of portability inside a language where the primitive concept is the object, whereas Inferno tries to solve it with Plan 9, which means the whole OS has to come along for the ride.

I really like the idea of the Plan 9 environment. But to run it on a single computer is a little pointless. Plan 9 is really intended to take a whole network of computers and make it behave like one computer split across a network. You give every user their own Plan 9 box and then set up a big file server for the common filesystems, get a bunch of CPU boxes for running CPU-intensive stuff, you’re basically done. You can do cool things like wire up all of the CPU server’s audio to go back to yours over the network. Or move the display around on various computers. You could do some really neat stuff. With just one box… you’re already there. It feels a little futile.

I have the sense that if I just got a couple dedicated computers to run it I could be having a lot of fun and really make some progress. Is it an illusion? Maybe someday I’ll find out.