A Personal Lisp Crisis

Over the past 7 months or so, in addition to learning and using Common Lisp, I have been giving thought to the social issues affecting Common Lisp and its community. Over the past two months, I have been working on two projects, described below, to try to address the issues that most affect newcomers.

To gain perspective on the causes and social context surrounding these issues, I have been studying Common Lisp's history, reading (among other things) the historical accounts of Richard Gabriel, Guy Steele, Kent Pitman, and others. (I can recommend, for example, The Evolution of Lisp, A Pattern of Language Evolution, A Critique of Common Lisp, Common Lisp: The Untold Story, and of course Lisp: Good News, Bad News, How to Win Big.) I have also been reading Common Lisp the Language, 2nd Edition and the Common Lisp HyperSpec, scrutinizing every inch of the language itself.

My research has given me a broader and more informed perspective than I had before. Before, I could see that the social climate surrounding CL was slowing its adoption rate and holding back progress, but I assumed its roots were healthy and strong. But now, it seems that CL is subsisting mainly on a fleeting aura of mystique, while other languages are surpassing it both technically and socially. That situation is untenable. Unless something changes, I don't see how CL can have much of a future, except as an amusing curio for fans of antique computer technology.

If that was the only thing I had realized, it would have just strengthened my resolve to address the issues head-on. But, I also realized than CL is not the culmination of Lisp's evolution, it is merely one box in a vast flowchart. It is a long-lived and popular dialect, and its creation was a significant milestone in Lisp history, but aside from historical interest CL is not an especially remarkable Lisp. It is a middle-of-the-road dialect, just as you would expect from something designed to be a compromise among numerous competing dialects.

CL is a fairly solid practical language, but it could have been much better. Inter-dialect politics, and the need to appease established interest groups, prevented the language from reaching its potential when it was created. Similar attitudes have prevented the core language from being cleaned up in the past 20 years, and likely will continue to do so indefinitely. There has been some activity building improved languages on top of CL, but there are better languages (particularly Scheme) to be foundations for that.

These realizations prompted a crisis for me. A month ago, I was ready and willing to dedicate myself to building a place for CL among a new generation of programmers. But now I'm not so sure. On the one hand, I have invested a lot of time into CL already, and it's not a bad language, just not very exciting or inspiring. I could learn to live with it, and even smooth out most of its warts. On the other hand, there are certain attitudes that I find extremely demotivating and frustrating, and those attitudes seem to be more prevalent in the CL community than in many other languages' communities.

To put it simply, the CL language and community both fall far below their potentials. It might be possible to shift the culture in a more positive direction, but that would require a sustained effort by many people, and would probably involve frequent stressful confrontrations with the “worst offenders” in the community. There are decades of inertia and ingrained attitudes that would not be easily changed.

The reward for all that effort would merely be preserving a lackluster language, a “fixer-upper” that would require further effort to turn it into something really worth using. And even if the core language were changed, CL would lose its mystique and become “just another Lisp”, unable to use longevity or historical significance as a crutch. It would have to compete with other languages solely on its actual merits, which nowadays are not as strong as CL proponents like to believe.

I enjoy a good challenge, but this seems more like a fool's errand. There are other languages with healthier communities, more momentum, cleaner cores, and features on par with CL. So I have to ask myself, why bother with CL?

I would love to discover some factor I have overlooked, some redeeming quality that would make CL worth fighting for. But the harder I look, the less likely that seems.

The Next Generation of Common Lispers?

There are numerous issues affecting CL, but one that I thought I might be able to make an impact on was its low adoption rate. CL has a smaller user base than many other languages, as well as a smaller influx of new users, and (I suspect) a lower retention rate among those new users. This issue has many underlying causes, most of them social. There are a few technical issues, but they would be easily solved if the social issues did not prevent it.

(Some CL proponents insist that a low adoption rate is not a problem. The usual argument is that it is better to have a small number of highly skilled and intelligent users than a large number of mediocre users. But that is a false dichotomy, a cop-out to avoid actually addressing the problem. And even if it were logically valid, it is based on the assumption that CL is attracting and retaining users based on skill and intelligence. I would argue that the social climate surrounding CL merely favors people with certain personality traits, regardless of skill or intelligence.)

Many people in the CL community have long had an attitude of, “We do not need to make an effort to attract the next generation of programmers, or to make things easier for them. CL is so amazing that the best programmers will naturally seek it out and devote themselves to learning and using it.” That attitude may have been justified at some point in the past, but it is certainly not anymore.

These days, there are numerous languages whose technical merits are on par with CL, and many of them have more positive social climates, more forward-looking attitudes, and more effective ways of welcoming, supporting, and educating new users. CL is barely competitive from a technical standpoint, and significantly outclassed from a social standpoint.

Aside from a few unusual features, the only significant attraction CL seems to have left is the exaggerated mystique created by the writings of Paul Graham and other vocal Lisp advocates. Even that won't draw new programmers to CL for much longer, especially with languages like Clojure and Racket offering the “Lisp Experience” with more useful and relevant built-in features, better community management, and more momentum.

“Handling Newbies Effectively”

The first project I planned to release was a paper highlighting and analyzing the various social issues affecting CL newbies, and offering practical advice for addressing those issues. These issues include not only the apathy (and occasionally outright hostility) that many oldbies show toward newbies, but also the inadequate social framework the community has in place for welcoming, orienting, supporting, and educating newbies.

For example, CL has no central location where newbies can find reliable information, documentation, or other learning resources. Instead, information is scattered across the web, with no way for newbies to know if it is up to date and reliable. (I'm sure many CL proponents are now itching to “correct my ignorance” and explain that CL is just a language standard with many different implementations, which is why there is no central website. Well, guess what? Newbies don't know or care about that. They want reliable and helpful information, not excuses for why the community is so disorganized.)

There are also not many good places for CL newbies to get support from other Lispers. The only well-established venue I know of that caters to newbies is LispForum.com, and it does not seem to get much promotion or attention from the broader community. So instead of newbies finding someplace where they are welcome, they usually end up someplace where they are barely tolerated, such as the #lisp IRC channel or comp.lang.lisp newsgroup. Those are probably the two least hospitable places for newbies, yet they are the two most popular and widely promoted venues, so many newbies end up there anyway (much to the vexation of some oldbies). Granted, #lisp and comp.lang.lisp can be great if you need answers to specific technical questions, but they are really lousy if you need orientation, advice, encouragement, or other forms of “hand holding”.

I have also observed many negative social attitudes which, while certainly not universal among CL programmers (and not unique to the CL community), are prevalent enough that they affect the entire community, and make the social climate less hospitable to newbies. A few of the most common and most harmful attitudes are defensiveness, nitpicking, zealotry, arrogance, lack of empathy, and social dominance based on a narrow set of values. Social dominance is an especially significant factor, because it means that the “alphas” in the community cannot be censured for bad behavior, unless those behaviors are truly egregious. (For example, if an alpha were to provide a newbie with “sample code” that would in fact delete all files on the newbie's computer, the worst reprimand the alpha can expect is a few people saying, “Tsk tsk, you know you shouldn't do that”. But just as many people would consider it an amusing prank, albeit cliché. They might even congratulate the alpha on teaching the newbie a lesson, thereby negating any reprimand the alpha received.)

The paper I've been working on goes into much greater detail, and gives more examples of challenges newbies face. But, I have started to doubt that publishing it would help the CL community at all. Many of the most serious social issues that affect the CL community are older than CL itself. Many people over the years have pointed them out, yet the same issues remain. So what good would it do to point them out again? The people whose attitudes and behaviors are most harmful, are the least likely to listen. The people who would listen, have probably heard it before. Why bother?

Yet, it's obvious that the situation cannot possibly improve if no one speaks out. Sweeping things under the rug doesn't make them go away. As uncomfortable as it may be, the only hope for addressing the issues is to start discussing them openly. Indeed, perhaps the most important social issue affecting the CL community is widespread complacency and apathy, which prevent the other issues from being addressed.

I will probably publish my paper eventually, because the same issues affect many tech-oriented communities to some degree. The CL community may be too far gone, but my paper might do some good elsewhere.

CLUEL

The other project I have been working on is CLUEL, the Common Lisp Usability Enhancement Layer. Whereas my paper would highlight the social issues and challenges newbies face while learning Common Lisp, CLUEL would directly tackle the problems that make the language itself unnecessarily difficult to learn and use.

About two months ago, I described some of the historical baggage that makes CL harder to learn, and adds to the mental burden on all CL programmers without adding any real value to the language. Since that time, I have spent over 100 hours poring over the CL language (not including the time spent researching CL's history, writing about social issues, or planning CLUEL's project infrastructure). I have personally looked up the documentation for every one of the 987 symbols in CL. I certainly can't remember every detail, but I can truthfully say that I have some knowledge of the purpose of every single function, macro, special operator, global variable, and constant in standard CL.

In the same way that researching Lisp history has given me a better understanding of CL's origins, scrutinizing the language itself has given me a better understanding of how inconsistent and crufty it is. Depending on how forgiving you feel, there are between a few dozen and a few hundred examples of features whose names are inconsistent, obscure, misleading, or otherwise poorly chosen. There are numerous features that are redundant, obsolete, or rarely needed, yet they are given the same prominence as the most essential parts of the language, because everything is lumped together in a single package.

Frankly, CL is just not a well designed language. That fact is not a reflection of the people involved, who as individuals were quite intelligent and capable. It merely reflects the process used to design the language. Considering that CL was debated in committee for 10 years, with numerous competing interest groups each tugging in a different direction, I'm impressed that it didn't turn out even worse than it did. (I'm even more impressed that the whole standardization process didn't fall apart entirely. Although I sometimes wonder if that might have been preferable in the long run.) However, there is a big difference between a language being “not bad, given the circumstances” and it being “well designed”.

The most obvious solution to the problem of it not being well designed, would be to revise the language standard to make it better. But that would be a futile and misguided effort. Even if you could get enough people in the CL community to agree that revising the standard was worthwhile, there would be just as much debating, bickering, and nitpicking as there was during the original standardization process. A committee is just not a good mechanism for designing a coherent language (or much of anything else, for that matter).

A better approach, more practical and more likely to yield real improvements, would be to create a new package (or multiple packages) that can be used as a test bed for experimenting with the language. It's easy to start, you don't have to create widespread consensus, and it doesn't disrupt anyone who doesn't want to use your package. This approach has been recommended by quite a few knowledgeable people, including Peter Seibel at ILC 2010 (near the end of his talk). This also harkens back to the tradition of prototyping new dialects on top of existing dialects (although I do not consider CLUEL to be a new dialect).

That approach is exactly what I had planned for CLUEL. The process I used for creating the proof of concept was something like this, although with lots of jumping back and forth between steps:

1. Partition all standard CL symbols into packages based on a theme or concept. At last count, I had tentatively settled on 22 packages: core, general, types, cons, sequence, array, string, stream, file, read, write, symbol, package, clos, structure, hash-table, math, bitwise, condition, debug, host, and time. Some symbols belong to more than one package.
2. For each package, partition the symbols into layers based on how useful they are:
   • Symbols that are “often useful” stay in the main packages, which are available by default when using CLUEL.
   • Symbols that are “sometimes useful” are moved to the extra packages, which can be included when needed.
   • Symbols that are “rarely useful” are omitted from CLUEL entirely, but could still be accessed via the standard CL package if desired.

3. For each symbol in CLUEL, decide whether its name is sufficiently meaningful, memorable, and consistent with the rest of the language. If it is not, give it a new, better name. (I developed a way to easily create aliases for existing functions, macros, special operators, global variables, and constants.)

   This involved a lot of jumping back and forth, because renaming one symbol might affect the consistency of other symbols. Or, I might discover some pattern or naming convention that affects symbols I had already processed, so I would have to go back and rename them.

   This step also involved a lot of consulting the documentation to make sure I understood exactly what an operator did. That led to a few surprises. For example, I didn't realize that ldiff and tailp are so closely related. Indeed, their names suggest that they are entirely unrelated, which is a sign that they are poor names (or at least, ldiff is). Those functions were likely invented independently, and they make sense individually, but when you put them together their names (and argument orders) are inconsistent.

4. Define a handful of new functions and macros to improve consistency, “fill in the gaps”, or improve usability in other ways. For example, there is no non-destructive sort function in standard CL, so I added a simple one. It may seem counterintuitive, but adding a new function in cases like this can reduce the cognitive load for users, because it makes the rules and patterns more consistent, with fewer exceptions to learn and remember.

   Another way to improve consistency and reduce cognitive load is to consolidate many similar functions into one generic function. For the proof of concept, I experimented with this for accessing containers (consolidating nth, elt, aref, svref, char, schar, and gethash) and counting their contents (consolidating length, array-total-size, and hash-table-count).

If that process seems to involve many subjective judgement calls, that's because it did. But this was merely the proof of concept, to demonstrate that this sort of endeavor is feasible. In the long run, the initial design of CLUEL is irrelevant, because it was intended to be changed soon, with the involvement of more people. The more important aspects of CLUEL, its cultural values, emerged as I was planning the project and working on the proof of concept:

• A focus on the needs of programmers of many different experience levels. That means, among other things, going out of our way to get feedback from newbies. Contrast that with Common Lisp, which was designed mainly to satisfy the needs of the kinds of people on the committee: expert Lisp hackers with code bases written in earlier Lisp dialects, and the implementators of those dialects (who were also expert Lisp hackers). A well-designed programming language (or nearly any other kind of tool) can be suitable for newbies and experts alike. It is not necessary to “dumb it down” to make it approachable for newbies.
• An emphasis on user-centered design. Many of the same principles and techniques that are used to design doorknobs, toasters, electronic gadgets, and software user interfaces are just as applicable to designing programming languages. Some established usability testing techniques would be difficult to run, because of the time investment involved in learning a programming language, but other testing methods are applicable, and new testing methods can be invented to suit the new domain.
• An emphasis on iterative design and change. Good user-centered design requires the ability to field-test a design, gather feedback, and incorporate that feedback as changes into the next round of design. With most programming languages, the cost of changing the language (once it is out “in the wild”) are very high, so design iterations are usually measured in years or decades, if they occur at all. I planned CLUEL to support multiple API editions, so that each new API edition could potentially be quite different from past editions, without breaking any software that depended on past editions. (Once an API edition is released, it would never be changed in a way that breaks compatibility, only to fix bugs or improve performance.) And, I extended defpackage to make it very easy to define new editions based on older ones, so the costs to iterate are very low.
• A culture of collaboration, support, and respect. Beyond addressing purely technical issues, CLUEL would be a chance to lead by example in addressing social issues. Besides being a good thing on general principle, having a positive community atmosphere would also be a practical necessity for CLUEL. It is impossible to perform user-centered design in an atmosphere of self-centeredness, defensiveness, or resistance to change. You must be willing to look beyond your own needs and preferences, accept feedback and criticism of your design, and change whatever needs to be changed. This naturally means you must embrace diversity, respect other people's viewpoints, and earnestly consider even ideas that you don't yourself like. And, because newbies are included in the target demographic for CLUEL, you must give attention and effort to supporting them, mentoring them, and soliciting their feedback.

It's important to point out that CLUEL was not designed to be a new dialect, or to ever exist separate from standard CL. Like it says in the name, CLUEL was a “usability enhancement layer”, a more usable and well-designed language interface for writing Common Lisp software. CLUEL would not add any new semantics or functionality to CL, just present (a subset of) the existing functionality in a better way.

A key feature of CLUEL was that code written in CLUEL would be fully compatible with code written in standard CL. So, for example, an application written in CLUEL could use libraries written in standard CL, and vice versa. (Unfortunately, that makes it impractical to address certain fundamental issues, such as the messy semantics of nil.)

I really poured myself into CLUEL. I estimate that over the past 2 months, I've spent over 100 hours studying the CL language and considering how to make it more usable, another 100 hours researching Lisp history and other dialects to see what has been tried before, 60 hours planning project organization and policies, and 20 hours developing infrastructure for packages, aliases, etc.

Indeed, the code for the first iteration of CLUEL, the proof of concept, is about 80% done, and the project planning is about 60% done. But I don't plan on finishing it, or even releasing the in-progress code. It hurts to throw something away after spending so much time and effort on it, but at this point that seems to be the best course of action.

Why? Because CL doesn't seem worth it anymore. I certainly don't want to potentially spend a few years maintaining CLUEL and building up its community, shackling myself to a language I don't really care about. Worse, I suspect that even handing off CLUEL for someone else to run it would be a mistake, because it might prolong CL's decline, and delay a better Lisp dialect with a healthier community from rising to prominence.

I'm not sure what dialect that might be, but the two main contenders, Scheme and Clojure, are both promising. They both have cleaner core languages, more momentum, and (from my initial impressions, at least) healthier communities than CL. I don't know yet how prevalent the negative attitudes I've observed in the CL community are in the Scheme and Clojure communities, but at least they do not seem to be so inhospitable to newbies, which is a good sign.

(I have heard some CL proponents insist that Scheme and Clojure are not really Lisp dialects, or at least are not as Lispy as CL. I've even given that impression myself on at least one occasion. But that is just ideology, with no grounding in reality. There have been many different Lisp dialects over the past 60 years, on many different architectures, with many different features and syntaxes. There is no rational basis for claiming that CL is a more “true”, “pure”, or “faithful” Lisp dialect than Scheme or Clojure.)

I don't currently have any plans to create something like CLUEL for Scheme or Clojure, for the simple reason that they don't seem to need it. Both languages are much more well-designed, consistent, and usable than CL is. But they are certainly flexible enough to implement a similar “usability enhancement layer”, should I happen to discover that implementing one would be beneficial.

What Now?

Despite my concerns about Common Lisp, I haven't completely ruled it out as a language choice for Ambienome or future projects. I'm keeping my mind open to the possibility that CL has some redeeming characteristic or feature that makes it worth using, maybe even worth fighting to preserve.

Lately, I have been investigating other languages, including Clojure, Racket (a R6 Scheme implemention), and Chicken (a R5 Scheme implementation). Clojure is quite interesting, and there are a lot of cool things being created with it, but I'm concerned the emphasis on functional programming and immutable data structures might make an already ambitious game project even more challenging. Racket is very polished and featureful, with great documentation, but it is licensed under the LGPL. Ambienome would be a commercial game, and although the LGPL probably wouldn't be a problem, there are certain ambiguities that make me hesitate to use it. Chicken is not as polished or featureful as Racket, but the community is quite friendly and welcoming, and its close integration with C might make it easier to use various game libraries or interface with the OS.

I am also considering non-Lisp languages, like CoffeeScript with Three.js (to make a browser game), or even going back to Ruby (perhaps JRuby with LWJGL). If possible, though, I'd like to keep using some Lisp dialect, because I have found Lisp to be conceptually powerful and enlightening.

Maybe, just maybe, I will start to miss something from Common Lisp, something that cannot be easily recreated in those other languages. But even if I don't end up using CL, I don't regret learning it. The experience has been very educational.