A Rubyist's Impressions of Common Lisp

It has been nearly 6 months since I dove into Common Lisp. I have been studying and/or using Common Lisp almost every day, working on Ambienome and related code. So, I wanted to take some time to document my observations, feelings, and impressions of Common Lisp at this time. Be advised that this is a fairly long post, even though I have trimmed out a lot of details and examples to keep it from growing even longer. (Apparently I have a lot of impressions!)

I have approached CL from nearly 8 years of programming in Ruby. I mention this because my experience with Ruby has certainly colored my impressions of CL, and many of my observations are about how CL compares to Ruby.

Different Yet Similar

Overall, I'd rate Ruby and CL as being about equal, on my personal, subjective scale of programming language goodness. They differ in many ways, both good and bad, but it feels to me like the differences balance out. So, the two languages are different, but neither one is clearly superior.

On a conceptual level, the two languages are actually fairly similar. They are certainly more similar to each other than either one is to a language like C, for example. They both have an interactive prompt, automatic memory management and garbage collection, first-class functions, anonymous functions and closures, arrays and hash tables, mutable strings, arbitrarily large integers, namespaces (modules in Ruby, packages in CL), and class-based OOP systems that don't force you into an OOP style. There are probably many other smaller similarities that I could think of if I took the time. (A dedicated Lisper would point out that many of these concepts originated in some Lisp dialect or another.)

Naturally, the details of many of these things are different. For example, CL's packages work very differently than Ruby's modules, despite fulfilling the same basic purpose. But in general, the concepts in each language are similar enough that knowing one language will make it significantly easier to learn the other.

Strengths (In Brief)

There are several aspects of Ruby that I feel are especially strong relative to Common Lisp. Stated briefly, they are:

• Community (active, supportive, sharing)
• Approachability (easy for someone new to get started)
• Convenience (many useful constructs and patterns are built in)
• Consistency of the object model (e.g. everything has a class)
• Wide variety of available libraries (both standard and installable)

Of course, Common Lisp has several areas where I feel it has the advantage:

• Flexibility (macros and read macros)
• Generic functions system
• Conditions and restarts
• IDEs, built-in documentation
• Optimization hints

I describe these in more detail near the end of this post.

Weaknesses

I'd like to be able to make a list of the areas each language is weak, but I'm too close to Ruby because of my years of use, yet also distant from it because I haven't used it actively for perhaps a year. Both of those situations affect my perspective and make it hard to see Ruby's real flaws.

Of course, Ruby has some weird bits of syntax (like {|args| ... } code blocks) and other quirks, but no more so than most languages, including CL. Ruby's performance is traditionally a bit weak, but that varies across implementations, and is improving in leaps and bounds. The community is generally pretty great, although I find the “brogrammer” trend in some parts of the community to be distasteful and regressive.

My relationship to Common Lisp is a different situation. My eyes are still fresh and I'm actively using it, so I can see and describe Common Lisp's shortcomings pretty clearly. But, I want to emphasize that just because I can still see the flaws in Common Lisp, that doesn't mean there are no flaws in Ruby. I'm sure someone else could point out just as many problems in Ruby as I can in Common Lisp.

So, with that caveat, I'll describe what I consider to be Common Lisp's problem areas.

Minor Problem: Batteries Not Included

Ruby, Python, and other “batteries included” languages come with all kinds of nifty features (e.g. networking, regular expressions, XML/JSON/etc. support) right out of the box. A programmer coming to CL from one of those languages might expect CL to be the same way, but it's not. Why?

The main reason is that CL has a formal ANSI standard, designed in the 1980s and early 1990s as a way to consolidate several different Lisp dialects. It took 10 years, countless man hours, and hundreds of thousands of dollars in direct expenses to finish the standard. Every detail was weighed and debated to make sure it was acceptable to the many people and groups with a vested interest in the process, such as the vendors who sold implementations of the old dialects.

So basically, only features that were acceptable to everybody were put in the standard. Things that they disagreed about, or that would put too much burden on the implementors, or that didn't even exist back then, are not in the standard. Adding a new feature to the standard now would probably take many more years of debate and nitpicking, and would only succeed if practically everybody agreed that it was such a great feature that it is worth the effort. In other words, it would take a miracle and a half.

That is a very different scenario than you have with Ruby/Python/etc., where someone can submit a new feature to the most popular implementation, a core dev likes the features and commits it, and it's released in the next version.

So, why is this not a huge deal in Common Lisp?

Well, most features can be written as libraries that you can install when you need them. CL is so flexible that even major changes to the language syntax can be installed as libraries! And with the advent of Quicklisp, installing libraries in CL is just as easy as installing gems in Ruby, or eggs in Python.

As for the features that can't be written as libraries, individual CL implementations can provide them as extensions. For example, the implementation I use comes with a bunch of goodies that aren't in the ANSI standard, such as threading, networking, code coverage and profiling tools, and a foreign function interface (FFI) for wrapping C libraries (such as OpenGL, as I'm using for Ambienome). If a feature is popular and many implementations provide it, someone will usually create a wrapper library that smooths out the differences between implementations, so that programmers can use the feature without being tied to one particular implementation.

(That's an idealized scenario, of course. It doesn't always work out so smoothly. Some implementations don't add new features very often, if ever. Sometimes an implementation adds a feature in a way that isn't compatible with other implementations, making it difficult or impossible to create a portable wrapper library. In such cases, if a programmer really needs the feature, they might target a specific implementation, rather than trying to keep their code portable. If they really need their code to be portable too, they can write code that acts differently for each implementation.)

That process of adding a feature to an implementation, letting other implementations copy it, and writing a wrapper library on top, is far from perfect. But, it's faster, less onerous, and more flexible than trying to update the ANSI standard, and the results are usually satisfactory.

So, once you realize the implications of Common Lisp being formally standardized, and find out that there's an easy way to install libraries, it's not such a big deal that Common Lisp does not come with “batteries included”.

But, it's still a minor problem that affects new Lispers coming from other languages. It might be possible to alleviate the problem through expectations management. If new Lispers understood from the start that many features are not part of the standard, but nevertheless are readily available as libraries, they wouldn't be so surprised and disappointed.

I'm not sure who would be most effective at this sort of expectations management, though. Implementations? Book authors? Teachers? Bloggers? Wiki editors? IRC and newsgroup participants? I suspect all of the above are already doing it to some degree, yet many newcomers are still surprised. This is one area where having an “official website” for Common Lisp might make things easier.

Moderate Problem: Historical Baggage

Common Lisp has a lot of historical baggage: things that are done a certain way just because that's how they were done in the past, even if they don't make much sense nowadays. Perhaps that's only natural for a programming language with such a long history. CL itself is nearly 20 years old (or 30, if you count from the first edition of Common Lisp the Language), and it inherited baggage from several older Lisp dialects, going all the way back to the original Lisp over 60 years ago.

That's not to say that other languages don't have some historical baggage of their own. For example, Ruby and many other languages inherit the bizarre old syntax of putting a 0 (zero) in front of a number to interpret it as octal form. (E.g. 10 means ten, yet 010 means eight. Surprise!) CL seems to have an especially large amount of baggage, though, and many Common Lispers seem to cling to that baggage rather tightly.

Historical baggage manifests itself in CL in many ways. One way is as individual quirks, like the setq operator. Setq was, I hear, originally invented as shorthand, so you could write (setq x 3) instead of (set (quote x) 3). But that was before the read macros were available; these days you can type (set 'x 3), which also means (set (quote x) 3) and is not any more characters than using setq. (Although, setq works a bit differently in CL than in earlier Lisps, so (setq x 3) isn't quite the same as either (set (quote x) 3) or (set 'x 3) anymore.) Furthermore, CL has the setf operator, which can do everything setq can do and more, so there's not really any reason to retain setq. It's just historical baggage, kept around because that's what they used in the old days. (Setq is still quite widely used today, usually with the rationale that it expresses the programmer's intentions more clearly because it can do fewer things than setf.)

Historical baggage also manifests as old idioms that affect many parts of the language. Take for example the idiom of adding “p” to the names of predicate functions, functions that check something and return true or false (actually t or nil, another bit of historical baggage). For example, (evenp x) returns true if the value of x is an even number. It could have been (even? x), which I would argue is significantly clearer. But, Lispers used “p” in the old days, so (most, but not all) predicate functions in the CL standard use “p”, and therefore most Common Lispers still use “p” when they write their own predicate functions today.

Historical baggage also manifests itself as strange inconsistencies and curiosities in the underlying architecture. For example, Common Lisp has types, which it inherited from older Lisp dialects. It also has classes, which were added fairly late in the ANSI standardization process. Types and classes are very similar in many ways, but just different enough that they can't be unified without breaking tons of old code. As a result, CL has both systems existing simultaneously, where each class has a corresponding type, but not all types have a corresponding class. Some features use types (like function and variable type declarations), while other features use classes (like generic function dispatching). I suppose the standards committee made the right choice given the circumstances back then, to avoid breaking all that old code. But, it is nevertheless historical baggage that all Common Lispers still carry nearly 20 years later.

Finally, historical baggage manifests itself as concepts and features that were useful many decades ago, but are pretty much obsolete today (and in some cases were already obsolete when the standard was written). For example, every symbol in Common Lisp has “properties”, a list of data that the symbol carries around in its guts. But nowadays, it would often be just as efficient to store that data in hash tables with the symbol as the key, and doing so would entirely avoid the possibility of two unrelated pieces of code coincidentally using the same property names. But, Lisp has had symbol properties since the very beginning, so Common Lisp has symbol properties too, along with the half-dozen functions used to manipulate them. (Symbol properties don't seem to be used much anymore.)

These kinds of historical baggage aren't a serious problem, because for the most part you can ignore the obsolete features, and either cover up or learn to live with the weird idioms and inconsistencies. But it is a moderate problem, because this historical baggage adds to the mental burden of every Common Lisp programmer in the world. (There are many, many other examples of historical baggage that I have omitted for the sake of brevity.)

It's especially burdensome for new Lispers. Year after year, new Lispers have to go through a kind of rite of passage, learning the obscure idioms of years gone by, separating the useful features from the cruft, and trying to remember function names that seem arbitrary and inconsistent. Many of them give up and leave for other languages because of needless obstacles (this being just one of many they face). Some of them could have contributed a lot to the community over time, if only “the wall” hadn't been built so high.

Serious Problem: The Community Atmosphere

That brings me to the most serious problem Common Lisp has: the community's atmosphere, its prevailing moods and attitudes. This is such an important topic that it deserves its own post, but I'll summarize the problem here.

I'll admit that I started learning CL with the knowledge that many people (usually people who tried to join the community but were repelled) consider the community to be antagonistic, especially towards newcomers. So, I may be exhibiting some confirmation bias: seeing what I expected to see, and tending to ignore evidence to the contrary. But with issues like this, the widespread perception of a problem can be just as damaging as the reality of the problem itself.

I suspect that most people who use Common Lisp, as with any language, are probably decent folk who just want to write nifty code without a lot of fuss or drama. But these people are not very visible or active in community discussion venues (e.g. the comp.lang.lisp newsgroup or #lisp IRC channel). They're off somewhere else, writing their nifty code in peace.

Alas, many of the people who are highly visible and active can best be characterized as “toxic”. These are people who, because of the nature of their personalities and attitudes, have a consistently negative emotional effect on the people they interact with. Without necessarily intending to do so, they have created a constant miasma of disrespect, nitpicking, defensiveness, discouragement, and intolerance. This toxic atmosphere permeates the entire culture, gradually driving less toxic people into seclusion or to other languages, or souring their moods such that they become toxic as well. This leaves an even higher concentration of toxicity, affecting even more people, on and on.

Is it possible to reverse this trend? I don't know. It may be too late. If it can be reversed, I suspect the way to do it is for the quiet, decent folk to put on emotional hazmat suits and start being more active in the community. Participate in discussions, help new Lispers, and discourage toxic behavior by privately and tactfully informing people about the effect they have. Reducing the toxicity of the community atmosphere would be a major culture shift, but with a sustained effort it might be possible to create a more healthy and positive atmosphere.

I can't help but compare this to the Ruby community's notion of MINSWAN (Matz Is Nice So We Are Nice), and wish there were more positive role models in the CL community to set a good example. Most of the people who are revered in the CL community are quite intelligent and knowledgeable about CL and computer science, but are also very toxic. The motto for this community would be something along the lines of NaWTSWAT: Naggum Was Toxic So We Are Toxic. The community discussion venues have become an echo chamber, perpetuating and reinforcing the notion that it is okay to be derogatory and inflammatory, as long as you are intelligent or know a lot about CL. The people who disagree with this attitude tend to give up in disgust and either leave or stop talking, so you won't hear many dissenting opinions about that from the people who remain.

This may seem like a gloomy assessment, but the situation is not all bad. There are some awesome people making cool things with Common Lisp, and many people who will try to help as best they can when you have a problem. It's just a shame that the predominant attitude is so negative.

It's Not All Bad!

I have spent many more words so far describing the weaknesses and problems of Common Lisp, than describing its strengths and interesting features. But despite its quirks and baggage and toxic people, Common Lisp is an incredibly flexible and powerful tool for creating software. Ruby may offer a more polished baseline experience, but you can't take it as far as you can take CL.

The ANSI Common Lisp standard may stand still, but Common Lisp does not. It is constantly growing and evolving on top of the stable (albeit lumpy and uneven) foundation provided by the standard. For example, ASDF (which is a bit like Rake or Make) was created in 2001, and revolutionized the way CL libraries are defined and loaded. ASDF laid the groundwork for Quicklisp (CL's analog to RubyGems), created in 2010, to revolutionize the way CL libraries are downloaded and installed. That in turn lays the groundwork for further development and progress.

Flexibility (Macros and Read Macros)

One reason CL can keep evolving without needing to change the standard, is the flexibility provided by features like macros and read macros. In addition to the usual small utility macros, I've created macros for Ambienome to implement a limited form of prototypal inheritance, and extensible object properties. They are somewhat longer than the average macro, but fairly mundane; they merely expand into a few formulaic functions with the details filled in. There are much more complex and interesting things you can do with macros, like the famous (or infamous) loop macro, which implements a specialized mini-language within CL, dedicated to making it easy to write fairly sophisticated code loops. Even the Common Lisp Object System (CLOS), which provides CL's class-based OOP system, is largely implemented via some very complex macros. CLOS and loop are both defined in the standard, but they probably could have been implemented as separate libraries. (I'm guessing that having those things standardized enables implementations to optimize their performance. Or maybe they were just considered fundamental to the language.)

Read macros take flexibility and extensibility even further. They let you write code in CL that reprograms the way CL parses the text of your source code, potentially altering the language in radical ways. One fairly common and not-so-radical use for read macros is adding syntax sugar. For example, the CL standard doesn't have a literal syntax for reading or printing hash tables, but thanks to read macros you can add Ruby's hash table syntax to CL in 40 lines of code (or less if you don't need the hashrockets and commas). Similarly, you can add literal syntax for regular expressions, even though regular expressions are provided by libraries like CL-PPCRE instead of being defined in the standard. I don't know of any other language where you can think of some syntax that would make the language more expressive or powerful, then add it with just a couple of afternoon hacking sessions.

Generic Functions

Next to CL's flexibility from macros and read macros, the thing I find most interesting in CL is the generic functions system. Classes in CL don't (usually) have methods in the same sense that classes in Ruby do. Instead, CL has generic functions, and each method implements the behavior as it relates to instances of a certain class (or combination of classes). It's quite different from Ruby or any other object-oriented language I've used, but very flexible and powerful. I'd like to write more about the CLOS object model and how it compares to Ruby's, but this post is already rather long.

Conditions and Restarts

Another interesting feature of CL is its condition system. Conditions are analogous to Ruby's exceptions, but much more flexible and powerful. (That seems to be a recurring theme.) Besides raising errors and warnings, you can use conditions to send any kind of signal up the call chain, with the lower-level code possibly providing some “restarts”, ways to proceed. Depending on how you set up the restarts, your higher-level code might interrupt the lower-level code like in Ruby, or ignore the signal and resume the lower-level code where it left off, or modify something and then resume, or pretty much anything else you can imagine. I haven't had much opportunity to use conditions in a substantial way yet, but I'm sure they will prove useful as my code matures.

Integrated Development Environments

CL also has some nice IDEs. I use Emacs with SLIME, which is a big step up from my Ruby workflow. It offers the usual IDE amenities like tab-completion, parameter hints, jumping to function definitions, and looking up documentation (which is stored in each function/macro/etc., not derived from scanning comments like RDoc and YARD do). It also lets you inspect and modify the guts of most objects, which is pretty handy for debugging. And apparently even SLIME pales in comparison to the fancy IDEs provided by some commercial CL implementations.

Optimization Hints

Finally, CL has a system for declaring optimization hints to tweak the way the implementation compiles or interprets your code. (These are hints, so the implementation decides what to actually do. Some implementations might just ignore your hints.) Besides optimizing for computational speed, you can optimize for space (compiled code size, and runtime memory use), safety (run-time error checks), compilation speed, and/or ease of debugging. You can also optionally declare the argument and/or return types for functions, which can help the implementation generate even more efficient code. And finally, you can declare that a function can be compiled inline into other code that calls it, which can reduce or eliminate the overhead from function dispatch (this works well for short utility functions). I have read that with the right optimization hints, some implementations can compile number-crunching functions into code that is nearly as performant as C code. I'm pretty relaxed about performance (I used Ruby for over 7 years, after all!), but it's nice to have extra tools available for dealing with bottlenecks.

Final Thoughts

So, those are my impressions of Common Lisp after 6 months of focused study and use. There are a lot of good things about CL, but also some really bad. It's a pretty amazing language, but has some warts and flaws. The community is mostly decent people, but anyone who tries to participate in discussion ends up choking in the toxic atmosphere. The warts and flaws can be covered up or worked around pretty easily, but creating a more healthy community atmosphere seems nearly impossible. But I hope it can happen somehow, because it's a shame to see the language being held back like this.

(setq common-lisp-hyperspec-root
      "file:/usr/local/doc/HyperSpec/")