Cloje 0.1 Post-Mortem

Last weekend I announced version 0.1 of Cloje, a clone of Clojure built atop Scheme/Lisp, my latest side project. Developing it was an interesting experience, so I wanted to take some time to write out my thoughts. Cloje is still an ongoing project, but this is something of a post-mortem of its first two months of development.

Table of contents:

Preface: Regarding criticism of Clojure

Before I get to the post-mortem, I need to say a few words regarding criticism of Clojure.

Parts of this post (and occasionally my Twitter feed over the past two months) are critical of Clojure. I am somewhat conflicted about criticizing Clojure. After all, is it not a labor of love, made available to you and I at no charge, out of the generosity of Rich Hickey's heart? I know I would feel pretty bummed if random strangers were going around publicly skewering my hobby projects. Should I not extend the same courtesy to Clojure?

But Clojure is not Rich Hickey's personal toy language anymore. It hasn't been for years. As of this writing, the Clojure website lists nearly a thousand people who have signed Clojure contributor agreements, over a hundred of whom have contributed patches to Clojure. There are multiple conferences around the world dedicated to Clojure, and Hickey and others often speak about and promote Clojure at other technology conferences. Cognitect (of which Hickey is CTO) use their status as Clojure's stewards as a marketing point, and they describe part of their mission as:

Enhanced community support: We are dedicated to the developers who have adopted our open source tools. Cognitect is positioned to advance the platform and engage the community to continue improving it for all developers.

In short, Clojure is a topic of widespread public interest among developers, and its stewards actively encourage and intend for it to be. Therefore, I think it is fair for it to receive a proportionate degree of public scrutiny.

Given that I am voluntarily spending some of my free time making a clone of Clojure, it should go without saying that I consider Clojure to have many excellent and worthwhile qualities. I sincerely thank and applaud Hickey et al for creating and maintaining it. That said, there are aspects of Clojure (as a language and a project) which raise concerns about the priorities of Clojure's stewards.

One gets the impression that Clojure's stewards wish people to treat Clojure as a mature language, but have given little priority to the work needed to actually make it a mature language. Marvelous, innovative, advanced features continue to be added to Clojure. Meanwhile Clojure still has no specification, the documentation is a shambles, and basic features needed in everyday programming (such as math and string operations) have been neglected for years. They might as well put a big banner on the Clojure website telling new and prospective developers: "We don't care about your needs".

Don't mistake me, I am not suggesting that Rich Hickey should drop whatever he's doing and go write a bunch of docstrings and trivial functions himself. But the fact that these same issues have remained unaddressed for years, despite being pointed out many times before, suggests that he considers them so low priority that they're not even worth delegating. I am sure there must be many members of the Clojure community who care about such issues, and who may even be trying to address them. But without support from the top, it probably feels like pushing rope uphill.

One can't help but notice that the most impactful efforts to address the basic needs of Clojure users — such as the Clojure cheatsheet, ClojureDocs, and leiningen — were conceived, implemented, and maintained by community members, not by Clojure's stewards, and not as an official part of the Clojure project. (From what I can gather, the cheatsheet was created independently, then was adopted by the project.) Apparently, creating and maintaining a sophisticated web app or build tool on your own, is considered less of a hassle than trying to improve the Clojure user experience through official channels.

When I criticize Clojure, understand that I do so not because I enjoy bashing the hard work of Hickey and the other Clojure contributors. I do so because it saddens and frustrates me to see Clojure falling short of its potential, to see basic needs and issues being neglected for years due to apparent lack of leadership.

I would love to be proven wrong. To see even half as much "hammock time" given to the long-neglected basic needs of Clojure developers, as has been given to advanced new features like transducers. For example, the stewards could:

Efforts along those lines would demonstrate that they meant what they said about "engaging the community to continue improving Clojure for all developers".

P.S. This post is not an invitation for random Clojure fans to attempt to debate me, nor Clojure foes to express agreement with me, about the virtues or shortcomings of Clojure or its stewards. The comments are moderated, and no comment about Clojure (pro or anti) will be published unless it adds to the conversation in a substantial and insightful way. (Constructive comments about Cloje, on the other hand, are very welcome.)

Now then, with that out of the way...

Change of Goal

I initially intended to make Cloje emulate Clojure as faithful as possible, with the goal that existing code written in "pure Clojure" should eventually run in Cloje. (By "pure Clojure" I mean Clojure code that does not rely on Java interop.) I developed version 0.1 of Cloje with this goal in mind, but even within a few weeks of starting, I had already begun to suspect this was an impractical and undesirable goal.

Clojure's Undocumented Behavior

Clojure has no formal specification, and the official documentation is... well, let's call it "frugal". Plus, I have set for myself the restriction that I am not allowed to look at Clojure's source code (that ruins the fun and learning). As a result, the only good way for me to understand the behavior of any Clojure feature well enough to clone it, is to carefully observe how it behaves in the official Clojure implementation.

But, the official Clojure implementation is not a static thing. It is constantly growing and changing, and in the absence of a specification or adequate documentation, it is difficult to determine which behaviors are significant and worth cloning, and which are incidental and might change. For example, I observed that:

That is four different behaviors for handling non-strings, all discovered within the same module. Except for blank? mentioning that it returns true when given nil (it doesn't mention false), none of the docs for clojure.string functions describe how they handle non-strings. They don't even bother to say something like, "The behavior of this function is unspecified when s is not a string". There's just no mention at all. Short of tracking down whoever wrote these functions and asking, I have no way to know whether these are intentional behaviors decided upon for a good reason, or incidental quirks of implementation that may change in the future, or bugs that should be reported.

You may be thinking: "These are clojure.string functions. You're obviously only supposed to use them with strings. It's right there in the name! Why should they bother to document what will happen if you use them with non-strings?"

This is why. Because if you don't document it, your API will accidentally end up with multiple, inconsistent ways of handling the same edge case. Now, Clojure can't resolve those inconsistencies without possibly breaking existing software that relies on the current behavior. And you can't blame the existing software's authors for programming based on observed behavior instead of relying on the docs, because the docs are not reliable. They simply do not provide enough information for a user to know how most Clojure features are supposed to behave. That is a sign that the docs are failing.

Imagine if a stable version of Clojure was released with hundreds of failing unit tests. It would be a huge embarrassment, and the stewards would give the highest priority to fixing it ASAP. Then they would (or should) conduct a review of the development and release processes to discover how it happened, and make sure it would never happen again.

Yet, stable versions of Clojure are being released with hundreds of failing docs, many of which have been failing for years, and the stewards do nothing about it. "Meh. It's only the docs, not anything important. Someone will probably submit some patches eventually?" Shrug emoji

Anyway, for Cloje 0.1, I deliberately attempted to emulate all observed behaviors as faithfully as possible, including the quirks and maybe-bugs. Often, that involved writing significantly more code to make a feature significantly less useful and less internally consistent. All in the name of faithfully emulating Clojure's current observable behavior.

That's not much fun, but it might end up being worth the sweat and tears, if existing Clojure software would be able to run on Cloje.

Except, as I soon realized, it wouldn't.

You Can't Do That In Clojure

The idea that Cloje might some day be able to run existing Clojure code, began to fall apart when I noticed how many gaps Clojure relies on Java interop to fill.

Screenshot of a tweet. Text: [Clojure and Java cuddling in bed] C: [whispers] You complete me. J: Aww you're so sweet! C: No, like, I don't even have a sqrt function. For example, Clojure lacks many common math functions. A program written in "pure Clojure" cannot calculate a square root, sine, or logarithm, among many other things. Instead, you are expected to use Java interop, calling Math/sqrt, Math/sin, Math/log, and so on.

Nor does Clojure have a function to determine the location of one string within another string; instead, you are expected to use Java interop, calling .indexOf. This is particularly curious, because .indexOf and four other string operations (.lastIndexOf, .startsWith, .endsWith, and .contains) are apparently so often needed by Clojurists that the Clojure cheatsheet lists how to do them in Java. Yet in all these years, Clojure's stewards apparently didn't see this as a sign that it would be worth spending a few hours to add functions to do these things in Clojure. ("A few hours" is a high estimate, to allow time for planning, unit tests, and docs.)

Perhaps it makes little practical difference to a typical Clojure user whether you write (.indexOf s1 s2) versus (index-of s1 s2), assuming you only care about your code running on the Java VM. But as someone trying to create a Clojure clone not based on Java, it has some troubling implications:

  1. There are many fundamental programming tasks that cannot be accomplished in Clojure without using Java interop.
  2. Therefore, a lot of existing Clojure software must be using Java interop.
  3. Therefore, even if I perfectly cloned all of Clojure itself, a lot of existing Clojure software would not run.
  4. Therefore, if my goal is to support existing Clojure software, not only would I have to clone Clojure, I would also have to clone Java.

Clojure by itself already has so many features that it makes my head swim just to think about them all. The realization that I would also need to implement Java, or at least a non-trivial subset of it, was rather discouraging, to say the least.

A Better Goal

To summarize, the goal of faithfully emulating Clojure:

Those sound like pretty good reasons to re-evaluate a goal, I'd say!

So, I took a step back. What did I want to accomplish by starting Cloje in the first place? After some contemplation, I realized my motivations were/are:

  1. To be able to use the cool features of Clojure, without the baggage (especially Java).
  2. To challenge and grow my technical and project management skills.
  3. To keep my mind active with something fun and creative.

So the original plan of faithfully emulating Clojure, besides being impractical, was not even well-aligned with my motivations. I was spending a lot of time painstakingly recreating some of Clojure's baggage, and doing so was neither challenging nor fun, it was merely tedious, frustrating, and discouraging.

So, what would be a better goal, one more practical and better aligned with my motivations?

I decided that instead of trying to support existing Clojure code, a better goal would be to create an API that borrows the best parts of Clojure, and is similar enough that it would not be difficult to port code between the two. But, I would address the quirks and inconsistencies instead of perpetuating them, and fill in Clojure's gaps in a Clojurish way instead of by emulating Java.

Going hand in hand with that goal, would be the need for:

Not only would this approach be much more enjoyable and result in a better API for Cloje, it might also make life a little bit better for any Clojurists who care about the gaps and inconsistencies. And any Clojurists who don't like my so-called improvements can just ignore them. Everybody wins! \o/

So, that's the story behind the change of goal between Cloje 0.1 and 0.2.

Host language differences

Clojure's shortcomings aren't the only challenges I have faced while developing Cloje. I also had to deal with the differences between the two currently targeted host languages, CHICKEN and Racket.

CHICKEN and Racket are both implementations of Scheme, but that doesn't mean code written for one will necessarily run in the other. For one thing, CHICKEN is an implementation of Revised5 (five) Scheme (R5RS), whereas Racket is an implementation of Revised6 (six) Scheme (R6RS). Plus, each of them adds many extra features beyond what is specified in their respective standards. Ideally those extra features are cross-compatible SRFIs that both languages implement, but often they are implementation-specific additions that are not cross-compatible.

The strategy I have used to deal with this in Cloje is:

The advantages of this are that I only had to write most code (including tests) once, which means less work for me, and less surface area for bugs to appear. And using shared tests helps ensure that Cloje has the same features and behavior on all host languages.

This approach will also make it less work and less stress to add support for additional host languages in the future. Other Scheme implementations would be particularly easy, as 90% of the code and 99% of the tests are already written. I'd just need to implement a few internal APIs and wrap it up in a module!

In fact, most of the code is written in Clojure-style special forms (defn, let, cond, etc.), so even for a potential non-Scheme host language like Common Lisp, probably 40% of the code and 80% of the tests are already written, once you implement a few Clojure special forms. (If you'd like to volunteer to help add Common Lisp support to Cloje, get in touch.)

Indeed, one of the cool side benefits of Cloje, is that it would (potentially) allow you to write code that would run on Clojure, ClojureScript, CHICKEN, Racket, and any other host language (e.g. Common Lisp) that may be added in the future.

But, although there is a lot that I can do to smooth over the languages differences after Cloje has been imported, this dream is somewhat foiled by the fact that the act of importing libraries is different on each language. On CHICKEN you do (use cloje), and on Racket you do (require cloje).

This difference is particularly unfortunate if you want to write a portable script in Cloje. You either have to make a separate file for each host language's entry point, or require users to pass some weird command line flags.

Alas, I don't think there's anything I can do in Cloje to work around this. It would require changes in the host languages. One solution would be for them all to have a uniform way of importing libraries, but this is not a likely solution, and it only addresses the immediate surface problem. A deeper solution would be for them all to have a uniform way of executing different code depending on the platform — something like SRFI-0 (which CHICKEN supports, but Racket doesn't), or SRFI-7 (which Racket supports, but CHICKEN doesn't), or #+foo syntax (which CHICKEN and Common Lisp support, but Racket doesn't).

Of the existing solutions, cond-expand from SRFI-0 is the simplest to implement, least invasive, and (from what I can see) most widely supported among Scheme implementations already. So the simplest way to achieve this unification would be to add support for cond-expand to whichever Scheme implementations don't already have it, to the various Common Lisp implementations, and to Clojure and ClojureScript.

In Racket and other Scheme implementations that implement SRFI-7, it would be trivial to define a cond-expand macro that expands to a use of SRFI-7's program. (I see no good reason why any implementation which already implements SRFI-7 should refuse to implement SRFI-0. That's just being stubborn at the expense of your users.) If there are any Scheme implementations which have neither SRFI-0 nor SRFI-7, the SRFI-0 document provides a simple example implementation using only syntax-rules.

In Common Lisp, I suspect it would be not too hard to implement cond-expand in terms of the *features* list. Good luck convincing the maintainers of the various CL implementation to add a feature that comes from Scheme, though. I'm tempted to write the code myself and offer it on a silver platter, just to see who would turn down free code, and what excuses they would invent.

Clojure and ClojureScript have recently implemented their own version of this concept, reader conditionals. They decided to implement it as a reader macro, which was a vastly more complex and invasive solution than was necessary, but gives users a modicum more flexibility. (You can use it anywhere inside arbitrary macro calls, and you can do splicing. Those are the only advantages that I can see over cond-expand, which can be implemented in less than a day and requires no changes to the reader. But they're the experts on Simplicity™, so I'm sure they know what's best.) I'm pretty sure it would be impossible to implement cond-expand in terms of the new reader macros (read time comes before macro-expansion time), but it would still be very simple to implement cond-expand as a separate, normal macro.

Of course, just because all this would be simple technically, doesn't mean it would be easy politically. There would need to be a really compelling reason to implement cond-expand, especially for popular implementations/languages, who have the resources and leverage to do things their own way, without needing to cooperate with anyone else.

syntax-rules

Scheme's system for defining hygienic macros, syntax-rules, has been a very useful tool in the implementation of Cloje. It has also occasionally been a source of challenges. Thankfully they have so far been interesting challenges, rather than frustrating challenges.

A macro, for those not familiar, is code that undergoes a transformation before it is compiled or executed. Macros allow you to, for example, write simple code that is transformed into more complex code when compiled — thus sparing you from having to write out that complex code by hand yourself.

Cloje contains quite a few macros that transform Clojure code into functionally equivalent Scheme code. For example, I wrote a cond macro that transforms Clojure code like this:

(cond
 (< x y) -1
 (> x y)  1
 'else    0)

... into Scheme code like this (which also happens to be valid Clojure code):

(if (< x y)
  -1
  (if (> x y)
    1
    (if 'else
      0
      nil)))

Indeed, I even created an if macro for Cloje, to account for the fact that Clojure has nil, while Scheme does not.

(By the way, I consider Clojure's version of cond to be a step backwards in terms of readability compared to Scheme/Lisp's, especially when any of the test forms are longer than about 30 characters, which is not uncommon in practice. Fortunately, Cloje users can easily use the host language's version of cond, case, etc. if they prefer.)

syntax-rules is based around pattern matching and templates, which makes it very well-suited for these sort of simple transformative macros. You give it a pattern (the "shape" of code to match), and a template (the "shape" of code to transform into), and it creates a macro that will take any code matching the pattern and transform it into code like the template. (Actually you can give it more than one pattern/template, and it will use the first pattern that matches.)

For example, here's a simple definition of a when macro using syntax-rules:

(define-syntax when
  (syntax-rules ()
    ((when condition expr ...)
     (if condition (do expr ...)))))

This macro says that when it sees code shaped like (when condition expr ...), that code will be transformed into code shaped like (if condition (do expr ...)). In this macro, condition and expr are used as placeholders; e.g. whatever code was found in the condition place when matching the pattern, will be put in the condition place in the template when expanding the macro. The ellipsis (...) is one of the coolest parts of syntax-rules; it allows you to match multiple forms, and reassemble them in many useful ways. This example barely scratches the surface of what you can do.

Clojure has a macro system too, but it's not based on patterns and templates. Instead, you manually write code that takes apart the old code piece by piece and transforms it into the new code. Thankfully, backquote-splicing makes that mostly tolerable. Clojure's macro system is not hygienic, so you have to worry about using gensyms, lest you accidentally clobber any variables used elsewhere. You get more power, but also more responsibility (and more potential for bugs). Clojure's macro system is very similar to Common Lisp's macro system in these ways.

syntax-rules is very useful, but there were times where I wished there was a way to write low-level macros in R5RS Scheme. Both CHICKEN and Racket have their own ways to do it, but they are not cross-compatible. I sometimes had to stretch my brain to come up with solutions using only syntax-rules.

For example, one conundrum I encountered is that cond in Clojure throws an error if you give it an odd number of forms. With a low-level macro, you could just do something like (when (odd? (count forms)) (throw-an-error)). But, syntax-rules doesn't allow you to run arbitrary code during transformation.

Instead, I had to be a little more clever. I wrote a utility macro that expands to one of two expressions depending on whether its first argument contains an even or odd number of forms:

(define-syntax _even-form-guard
  (syntax-rules ()
    ;; pattern A: matches if there are two or more forms
    ((_even-form-guard (form1 form2 more ...) pass-expr fail-expr)
     ;; discard form1 and form2, and repeat
     (_even-form-guard (more ...) pass-expr fail-expr))

    ;; pattern B: matches if there is one form left over
    ((_even-form-guard (form1) pass-expr fail-expr)
     fail-expr)

    ;; pattern C: matches if there are zero forms left over
    ((_even-form-guard () pass-expr fail-expr)
     pass-expr)))

It works by repeatedly "peeling away" the forms two at a time, until either one or zero forms are left, which indicates whether there was originally an odd number of forms or an even number of forms. With that helper macro in hand, writing cond wasn't hard (although I did have to split it up into two parts):

(define-syntax cond
  (syntax-rules ()
    ((cond form ...)
     (_even-form-guard (form ...)
      (_cond form ...)
      (_raise-syntax-error
       'cond "must have an even number of forms")))))

(define-syntax _cond
  (syntax-rules ()
    ((_cond) nil)
    ((_cond x y more ...)
     (if x y (_cond more ...)))))

Note that _even-form-guard and _cond are both recursive macros: they sometimes expand into another use of the same macro (with different arguments), which then gets transformed, and so on. Schemers love recursion, you know. :P

Anyway, although this approach was a bit more circuitous, it was also a fun puzzle to solve, and overall syntax-rules has been a huge help in terms of productivity and clarity of code.

Eventually I will find a situation where I absolutely must resort to using CHICKEN's and Racket's low-level macro systems — for example, if/when I reimplement Clojure's macro system, which I am somewhat dreading. But so far syntax-rules has been serving nicely.

Project hosting

For years, my default place to host new projects was GitHub. Because GitHub is so popular, hosting Cloje there would have the obvious benefit of lowering the barrier to entry for issue reporters and contributors. So why isn't Cloje hosted there?

You may be aware of the revelations last year of sexism and intimidation by one of GitHub's founders, and of a workplace culture that was hostile to women. Although Tom Preston-Warner (co-founder and former president of GitHub, against whom some of the most serious allegations were made) resigned to avoid further scandal, I am still wary of hosting projects on GitHub.

Even if you're not paying a subscription, hosting a project on GitHub increases their influence and market reach, which allows them to attract more paying subscribers and enterprise customers. Furthermore, I am trying to make Cloje an inclusive and welcoming project. Hosting on a service with a recent history of toxicity and hostility to female developers, would send the message that I consider that behavior "no big deal".

Perhaps in another year, if GitHub has shown visible signs that its culture has become substantially healthier, I might reconsider hosting new projects with them. But in the meantime, it is not something I'm comfortable doing. (I do have a number of old, inactive projects still lingering on GitHub. I made the judgement call that it was not worth it to relocate them all, although I did remove my private repositories and cancel my subscription.)

What about the alternatives? Bitbucket is pretty popular, and it has support for Git nowadays. But the company behind Bitbucket, Atlassian, doesn't exactly have a spotless past when it comes to sexism, either. The company apologized, but like GitHub, I'm keeping my eye on them to see what they'll learn from the incident.

I could host my own project infrastructure, of course. But, that requires significant ongoing maintenance work that I would prefer to avoid for the time being. It also presents a higher barrier to collaboration, compared with hosting on a "social coding" site. I'll probably make the investment if Cloje starts to gain traction, but not yet.

I eventually decided to host Cloje on GitLab, because:

P.S. I am aware of the routine verbal abuse and toxic behavior of Linus Torvalds, the creator of Git itself (oh, and Linux). Alas, I have to use some form of version control, and I find the other VCSes unbearably frustrating to work with, so unfortunately I'm stuck with Git. Sigh.

Racket's docs

Let's end on a positive note.

Racket's docs are superb. I don't just mean in comparison to Clojure's. Racket has the most thorough and helpful documentation of any language I have ever seen, with the possible exception/tie of Python. (It's no coincidence that both Racket and Python have excellent documentation and are popular in CS education.)

Not only does Racket have thorough reference documentation for the language and standard libraries, but it also has an excellent prose guide, numerous topic-oriented tutorials targetting many different levels of experience, and manuals for various companion programs. It is plain to see that the Racket folks care about their users, both new and established.

Kudos to everyone involved in Racket's documentation!

P.S. While we're talking about excellent docs, the docs for Rust are also superb for a language so young. The Rust project and community just generally seem to have their shit together, and it's refreshing and inspiring to watch. Major kudos to the Rust stewards for such great leadership.

Final Thoughts

Well, this post turned out a bit longish, didn't it? Whew. Hopefully there were at least one or two parts worth reading. (If nothing else, it'll be a nice time capsule that I can come back and read in a few years and chuckle to myself.)

I'm looking forward to pursuing this new goal/approach in Cloje 0.2. It feel much more positive, enjoyable, feasible, and useful.

I'm aiming for a ~6 week release cycle. Sometimes life gets in the way, so no promises, but if all goes well Cloje 0.2 should be out in mid to late June.

[Update (May 9): Bonus fun fact for the history books: Cloje was originally going to be named Clojeme (Clojure + Scheme), but that was discovered to be too similar to a lewd slang phrase in Mexican Spanish.]

Previous post: Announcing Cloje Next post: Implementing Metadata in Cloje