Lisp (EN)

I have just returned home from the European Lisp Symposium of 2008, held in Bordeaux, and I shall now inflict upon you my experiences from the event.

I'm not a very experienced traveller - indeed, this was my first time travelling by plane by myself, so I was slightly worried about getting lost along the way from Copenhagen to Bordeaux. My trip was fairly uneventful, though I was a bit mystified by travelling via a KLM-owned Fokker 100, a plane manufactured by a company that left the business quite a while ago. I was not less mystified when the rest of the flight was done in a plane that the safety guide claimed was a Fokker 50, but which most assuredly did not have turboprop engines.

For my part, the stereotype about French people not understanding English turned out to be fairly true. Even more, the tram ticket machine did not understand English either, the clearly labelled button for changing the interface language seemed totally nonfunctional, as if it was literally just painted on the machine. I was told that the attitude towards English had changed in France - I can only imagine it has changed from open disdain to a feeling of irrelevance.

Anyway, on to the interesting part: the actual symposium, which spanned over two days. The first day contained confusingly named "Birds of a feather" sessions (also known as workshops/discussion sessions). They were quite interesting, though I admit I had too little interest in parallel programming to be that aware during the first session. The CLIM/ McCLIM session was interesting, though my own part in it was pretty much a disaster due to my SBCL session getting totally messed up in some way. Scott McKay (the original designer of CLIM) managed to inject some interesting comments though. Amusingly enough, it did not seem like he had the same vision for CLIM as I have now, and rather took great pride in how the spiritual successor (DUIM for Dylan) could perfectly resemble a mainstream, native user interface. The informal session about the eventual successor to ASDF was perhaps the most interesting, some actual and involved discussion ensued and Nikodemus came up with the idea for a clever/horrible reader macro that inspired the title of this blog post. It seems like the ASDF successor will be motivated by two main things: facilitating a sane ASDF-Install replacement and elegantly handling weird compilation options. Exactly what I need, so I won't complain. I may even contribute code if it comes to that.

The presentations were decent, but I found that I learnt more interesting stuff by talking with the authors than by listening to their talks, so I'll elegantly skip to perhaps the most important thing about the meeting: talking to other Lisp programmers in real life and finally meeting people I've talked to online for years. It was very educative and entertaining; from listening to Scott McKays view on CLIM, to discussing more general programming with Christophe Rhodes and Robert Strandh, to getting a primer on Finnish politics by Nikodemus Siivola. I obviously can't give a comprehensive description of every topic touched by these conversations, but I can encourage anyone who has never been to one of these events to attend the next possible one, since actually talking face to face allows some discussions to work much smoother than they do online.

Unless practical problems get in the way, I'll definitely be attending the next European Lisp Symposium. It was certainly a very rewarding investment of my time.

McCLIM 0.9.6 "St. George's day" has just been released (Andy Hefners suggestion of "Discovery of the AIDS virus day" was not used), with quite a number of features. It's been over half a year since the last release (which is unfortunately pretty typical for McCLIM), so here's a list of the major improvements:

• Runs on SBCL: shortly after the McCLIM 0.9.5 release, the SBCL hackers cruelly removed an unexported, internal symbol that we relied on for some reason, preventing McCLIm 0.9.5 from running on anything but an oldish SBCL. ASDF-Installable McCLIM should therefore now run on SBCL again!
• Functions make-pattern-from-bitmap-file and read-bitmap-file added. These are from CLIM 2.2 and permit relatively simple reading of various bitmap image formats (currently supporting JPEG, GIF and XPM).
• Editor redisplay is much faster: I wrote a new redisplay engine for the Drei editor substrate a while back, it has vastly better performance and a few extra features. It also improves Climacs a lot.
• Fully in-Lisp Truetype font rendering. McCLIM has used an FFI to access Freetype for a while, but Andy Hefner implemented a Truetype renderer in Common Lisp, meaning that McCLIM applications can now have pretty fonts without having to degrade themselves by asking C for help. Load the mcclim-truetype system (as opposed to mcclim-freetype) to activate it.
• Tons of bugfixes. As usual, this new release is the most stable McCLIM release ever! A lot of annoyances have been fixed, so McCLIM should now behave more predictably, though there is still some way to go before we reach perfection.

To celebrate this release, I hacked together a simple Logo interpreter. The cool thing about this interpreter is that it abuses accept-from-string to implement a pseudo-Logo programming language. A program can look like this:

Pen Up
Lock
To "Draw Left Half-Circle"
Repeat 18
Forward 10
Left 10
Next
End
To "Draw Circle"
Invoke "Draw Left Half-Circle"
Invoke "Draw Left Half-Circle"
End
Unlock 
Pen Down
Center
Repeat 36
Invoke "Draw Circle"
Forward 20
Next

I'm not going to tell you what the output looks like, instead you can download McCLIM 0.9.6, load tortoise.lisp and use Load Commands to load a file containing the above program (or just enter it manually, if you want) and try it yourself!

Method combinations are one of the really cool, yet often overlooked, parts of CLOS. They allow you to elegantly and succinctly specify protocols that would otherwise require wrapper functions, ugly calls to call-next-method and a fair amount of similar logic in every defined method. Take for instance Climacs, where we have to generate a list of the available commands. This list depends on many things, in particular the current view, syntax and active modes, and so methods must be defined for all these. This practically screams for an append method combination, wherein every part of an object (that is, all subclasses) get a chance to add their own contribution to the resulting list, without any method having to implement any kind of list-appending logic. Without method combinations, this would have required explicit calls to append and call-next-method; rote code that is not essential to the purpose of producing a list of applicable commands. CLOS permits me to write methods that are solely focused on the actual task they are supposed to do, while the system takes care of the tedious task of assembling the various results into the final value. It also makes it trivial to add further methods later, as they are automatically picked up by the method combination. And compared to use of call-next-method, I don't have to worry about whether there is a next-method, CLOS takes care of that for me.

This kind of functionality is, I think, what makes Lisp a powerful language. Many other languages have syntactical shortcuts and convenient functions for everything, enabling very short programs that do interesting things (Ruby is particularly good at this). Common Lisp is not well suited for one-liners and very short programs, the syntax and provided facilities are usually too general. On the other hand, Common Lisp shows its power once the program grows - the language facilities are well suited for splitting up complex code and defining small ad-hoc frameworks (like generic functions with unusual method combinations) that would require a fair bit of code in other languages. Thus, I believe that while Common Lisp may look overly verbose for short programs, it makes it possible to implement long programs in a smaller amount of code than it would take in many other languages, and the resulting code can be flexible and extensible, even for complex problems.

A few days ago, the great topic of #lisp was type conversion functions, and in particular the notation that should be used for them. Some advocated fromtype-to-totype, others totype-from-fromtype, and of course perhaps the most common notation, fromtype->totype. I declared that I preferred a variant of the latter, namely totype<-fromtype, due to the fact that this permits clean and readable nesting, but Nikodemus of SBCL fame pointed out that this could be turned into a more general (<- (totype middletype fromtype) object)-facility. And he was right, the Power of Lisp macros, as commonly pointed out by the sublime Paul Graham, permits all sorts of notational conveniences. I therefore set out to write the most intuitive and clear general type conversion facility I could think of, so that the current common practise of writing Lisp programs as large type transformations will yield more readable and consistent programs in the future.

As my focus was on ease-of-use, an intuitive interface and predictable semantics, I decided that the overall pattern of (<- (totype middletype fromtype) object) was worth keeping. The <- macro will rewrite this into calls like (totype<-middletype (middletype<-fromtype (the fromtype object))) (note that the is used as an optimisation - this is the kind of thing macros allow us to do!). This means that you just have to define a function with the name totype<-middletype, and then it will automatically be used by the macro - you don't even have to export the symbol! Truly, Paul Graham was inspired when he invented macros (and they certainly paid off). Now you could convert types like this:

(<- (integer string) "123") => 123
(<- (integer string integer) 123) => 123

However, I didn't feel like it was "fully there" yet, as you might want to define a type conversion function like this:

(defun integer<-string (string &optional
                        (start 0) (end (length string))
                        (radix 10))
  (values (parse-integer string :start start :end end :radix radix)))

I think you should be able to provide values for those optional arguments. And since I'm writing a type converter, it would be outright embarrassing if you had to provide those values directly, and couldn't convert them from something else - and why not integrate that directly into the single type-list (the first argument to the macro)? That is good program design, as it clearly puts all type-specifiers in a single place. This is now possible:

(<- (integer (string integer)) 1 "123") => 23
(<- (integer (string integer integer)) 1 2 "123") => 2
(<- (integer (string integer integer)) 1 3 8 "123") => 19

As you may note, I decided that the arguments to the type conversion function should go before the actual object to be converted. This is good program design, as the overall macro works by first saying how to convert something, then providing the actual value(s). I think the same rule should count for the values. As an added bonus, it is now possible for machines and humans to perform sophisticated reasoning about the macro without looking too much at the type list, as you know things like "the second through to the second-last argument contains option values" and "the last argument is the value to be converted". You may note that the "real" order of types is preserved in the type list, that's because it's more logical that way. Also, you may argue that Lisp is about freedom, and my requirement to put parameter values first in the value-list conflicts with that ideal; but while Lisp is indeed about freedom, putting the value to be converted first is so plainly wrong that no-one would ever do it, or expect it to work. Is it really loss of freedom if you weren't going to (or weren't supposed to) do it anyway?

But wait, there's more. I mentioned it would be embarrassing if you couldn't easily type-convert the optional arguments, so of course that was implemented too. When the type-list turns into an actual tree, the objects to be used are retrieved from the object-list as types are encountered in the tree during a depth-first search (and, of course, with the above-mentioned rule about the first type in a list, or the "target type", being selected last). So you can now write expressions like:

(<- (integer (string (integer string))) "1" "123") => 23
(<- (integer (string (integer string)) integer) "1" 123) => 23
(<- (integer
     (string (integer string)
             (integer (string integer integer integer)
                      integer))
     integer)
    "0" 0 2 2 10 123) => 12

The final example above truly shows the power of the <- macro. We have (are you ready for this?) an integer, 123, that is converted to a string. This string-conversion takes arguments, namely an integer that is converted from a string ("0"), and also (this is the big one) an integer 10, that is converted to a string, and then back into an integer starting from string offset 0, ending at 2, and in base 2 (the result is the integer 2, of course). Finally, the string (which is now "12") is converted to an integer, and the final result is thus the integer value 12.

But of course, if you have nothing but an elegant and succinct notation, you end up with theoretical mathematics, and I'd like to take this type conversion utility a bit further than that. Thus it is useful to predefine a set of type conversion functions. I'd like to talk a bit about some of the more interesting of these.

A very common task in modern programming is converting a bit-string into an integer, this is done via the bitstring type (note that this is not defined as a type anywhere - it exists purely for the function of conversion). The conversion function from bitstring reuses the integer<-string function, of course - code reuse is a great thing, and that it is possible is a sure-fire symptom of excellent design:

(defun integer<-binarystring (binarystring)
  (integer<-string binarystring 0 (length binarystring) 2))

Another useful conversion is integers to strings - in many other languages the null string, the string "0", the string "false" and others are all equivalent to the false value, while "1", "true" and the like are the true boolean value. I don't know why this feature isn't part of ANSI CL, but I suspect it may have to do with lack of time and resources on part of the standardisation group. But never mind that - while Lisp was not the first language to get an object system, it got the best one by far when it catched up. I was looking for something similar with this conversion function. There is a list of strings that are true ("true", "1", "correct", "proper", etc) and a list of strings that are false ("false", "0", "bad", "deficient", etc), but this alone is just slightly more comprehensive than most other languages, not in an entirely different league, as CLOS is when compared to other object systems. Lisp used to be an AI language, and therefore I felt it would be highly appropriate if this conversion function used intelligent, language-aware semantics as well as the plain list, and thus, it handles "in" and "not "-prefixed strings, meaning that the following useful cases work as one would intuitively expect:

(<- (boolean string) "true") => T
(<- (boolean string) "not true") => NIL
(<- (boolean string) "not false") => T
(<- (boolean string) "correct") => T
(<- (boolean string) "incorrect") => NIL
(<- (boolean string) "bad") => NIL
(<- (boolean string) "inbad") => T

Finally, I'd like to describe the "integer to boolean" conversion. Conformity with prior experiences, and the expectations that arise from these, are important when designing intuitive semantics, so I looked to what boolean values other languages attribute to integers. The generally accepted pattern seemed to be 0=false, 1=true, ..., so that is what I implemented:

(defun boolean<-integer (integer)
  (not (zerop (mod integer 2))))

In conclusion, I think I fulfilled my initial design goals of intuitive semantics and a simple interface for the type conversion utility library. The source code can be retrieved here, but I'm going to submit it for inclusion into Alexandria in a few days, so you hopefully won't need to manually include the file in all your projects in the future.

Hello, reddit! There's a link to a screenshot near the bottom, and as an extra treat, this is from my test of the new JPEG reading facilities in McCLIM: jpegsinclimacs.png

As some of you may know, Climacs is an attempt at writing an Emacs-like editor in Common Lisp. This is obviously a hard problem that requires a lot of work - I have often been amazed at just how many exotic cases and conditions GNU Emacs handles in even simple editing functions. It is a very mature program, and it shows, while Climacs is still far behind Emacs in terms of functionality and stability.

But while clever editing commands and similar things aren't that hard to implement - they mostly require time - most peoples initial impression of Climacs has been one of very bad performance, which is another class of problem entirely. Climacs was quite sluggish, even on fast machines, and it's quite understandable that most people didn't get a very good impression of a thin-featured program that was several orders of magnitude slower than Emacs. To rectify this, I have spent the last week and a half writing a new redisplay engine for Climacs (specifically, the editing substrate Drei). Performance is not yet as good as that of Emacs, nor as good as I would like it to be, but it's much faster than the old engine, and even more important, it does not get (significantly) slower as the size of the buffer increases, only the size of the actual displayed buffer region affects redisplay linearly. While doing research on efficient redisplay in text editors, I discovered that there isn't a lot of writing on doing this on a bit-mapped display with colours - I used an article on the Andrew text editor as initial inspiration, but it didn't cover everything I needed to do. Therefore, I'm going to write a bit about Drei's redisplay algorithm.

The redisplay engine was developed based on a key assumption: output records are nice, flexible and useful, but they are too slow and heavy. Hence, the redisplay engine does not use output records, except for handling cursors and some other exotic situations. Instead, it divides the visible region of the buffer into strokes. A stroke is a unique buffer region (does not overlap with other strokes) that can be drawn in a single operation with a single set of drawing options, and that does not cross lines. Put another way, a stroke is a sequence of characters in a line with the same colour and font (strokes can also cover non-characters, but let's ignore that for now). The new redisplay algorithm thus works by fetching strokes from the buffer, starting at the top of the display, and drawing them to the screen until we reach either the end of the buffer or the bottom of the visible part of the output sheet. When strokes are drawn, we remember the location and size of their output.

Due to the constraint that strokes cannot cross lines, we can trivially organise strokes into lines and just check for whether a stroke directly precedes a #\Newline character to figure out when we should go to the next line, and when we do this, look at the strokes of the line and figure out the dimension of the line. Additionally, we get the nice property, that when redisplay is finished, no two strokes overlap.

Stroke objects are kept across display, and are mutated by the mechanism that retrieves strokes from the buffer (the stroke pump, see below), so we can easily check whether a stroke has changed (we mark it as "dirty" and "modified") since the last redisplay, simply by having the pump check whether it is going store already-existing data in the stroke object. If a stroke is obscured for some reason (for example due to a moving window, or part of the cursor being drawn over it), we also mark it as dirty. Taken together, this enables incremental redisplay, as we only redraw strokes that are dirty, and only recalculate their size if they are modified.

The interesting problem is now how to generate strokes for the redisplay engine. This is done through two generic functions, pump-state-for-offset and stroke-pump, that are used to "pump" stroke data into existing stroke objects. This is both to implement incremental redisplay, as mentioned above, and to avoid consing (if you read the code, you'll notice that I've been obsessed with minimising consing in general, perhaps this was not always necessary). For the most common case, these functions just relay to the syntax of the view, which results in either the simple stroke pumping defined for Fundamental Syntax, or a general parse-tree walker defined for LR syntax that can select drawing colours based on the actual parse symbols (wrapped in some macrology, syntax highlighting rules can then be defined like this). There is nothing special about stroke pumping, except that it has to be really fast, as it is completely non-cached and done in full for every redisplay. All the clever caching and "only handle that which has actually changed"-stuff is done at the higher level by looking at the dirtiness of strokes, and at the lower level by the incremental syntax parsers. The stroke pump just has to be fast, something that is most efficiently done by connecting it closely to the parse information. When the view is a pure buffer-view (that is, has no syntax) there is a simple fallback pump defined that just turns each line into a stroke (possibly chopping it up if it's very long), it's not fast, but it's simple, and you can look at it to get a general idea of how it works, if you're curious. It's defined in drei-redisplay.lisp

As an interesting special case, and as hinted at by the title of this post, strokes do not have to cover sub-strings of the buffer. For each stroke we have an associated set of drawing options, and in addition to holding colour and font information, it also contains the function to be used for drawing the stroke. Commonly, this is the default function, which copies a string out of the buffer and prints it with the function draw-text*, but if a sufficient function is provided, any Lisp object can be drawn. Indeed, Fundamental and Lisp syntax recognise non-characters in the buffer, and create single-object strokes, with a special drawing function, for these. This function presents the object, via CLIM, and draws the resulting output record. This means that any Lisp object can at least be displayed as if by print-object, or if a CLIM presentation method has been defined, as pretty much anything. This screenshot demonstrates the feature. Of course, stuffing the buffer full of arbitrary objects can still fail badly in other ways, as not all editing commands know how to deal with it.

In conclusion, if you were previously put off by the poor performance of Climacs, you should give it another try. It's not near Emacs speed yet, but it's quite usable on modern machines, and should improve further in the future (compared to the old redisplay engine, the new can actually be further developed and optimised in a straightforward way). There's also a pretty low barrier-to-entry for hacking, and lots of low-hanging fruit still.

For some time now, I have been using Paredit for editing Common Lisp code in Emacs (if you don't already, you should start too!). It's a highly useful approximation of an actual structure editor, with the important detail that it modifies the files just as a sequence of characters, putting intelligence into the editing commands rather than the data structure, and so works perfectly with all the tools that expect flat character files. While I can't claim to write much code with Climacs, whenever I do, I am annoyed by the lack of Paredit. Thus, I spent some time implementing a preliminary version of a Paredit clone for Climacs (named Structedit, as I want to emphasise the similarity to proper structure editors). It is not a perfect clone, apart from being far less full-featured and tested, it implements a few facilities slightly differently from Paredit. The base is (or should be) the same, however

While hacking out the code, I performed a number of observations:

• Having a full-fledged parse tree at your fingertips is highly useful for writing this kind of code, but it does not free you from handling special-cases when it comes to actually modifying the buffer. Also, interrogating the parse tree for information is tricky and requires in-depth knowledge of how it looks, meaning that a large number of utility functions are probably needed if it is to be possible to write syntax-functions without spending a lot of time studying the syntax implementation first.
• Climacs needs toggle-able "modes," like minor modes in Emacs. Currently, Structedit just overwrites the default key-bindings of Lisp syntax in Climacs.
• Generic functions should be used more often to implement user-level functionality, and these must dispatch on something that can be modified by "modes." For example, Structedit introduces new commands for C-d and Backspace, but if the user has bound some other key globally to, say, Delete Object, the default Delete Object will be used whenever he presses it, instead of the structurally aware variant of Delete Object provided by Structedit. As far as I know, Emacs suffers a bit from this as well.

Structedit requires the newest CVS-version of McCLIM, and its commands are not yet as comprehensive as those of Paredit, so use at own risk. If you're feeling adventurous, try hacking in some missing functionality (C-k is a bit bare-bones, if you need a starting point); it shouldn't be too hard.

Some time ago, Robert Strandh proposed to change Climacs by moving away from the classic Emacs concept of operating on buffers, and insert a middle layer of objects called "views". I just spent the two weeks implementing his ideas in Drei and Climacs itself, prompting me to write this blog post.

So, presuming you care about Climacs, why should you care about views? What problem do they solve. To put it in somewhat simplified terms, the problem with the Emacs buffer/mode-concept is that a single buffer can only be in a single major mode (called "syntax" in Climacs parlance) at a time, and any enabled modes is an intrinsic quality of the buffer itself. This prevents some neat programs from being written (more on this below), and prevents the user from having a buffer displayed in two separate windows with two different rules for syntax highlighting.

Of course, this is a rather dubious usage scenario, so let's grab hold of a concrete problem views solve for Drei (the editing substrate behind Climacs). Essentially, Climacs had a very ugly problem, wherein commands making multiple operations on a buffer had to call the function update-syntax to make sure the syntax parse tree was up to date with the buffer contents - failure to do so could cause commands to behave strangely (as with the Indent Region-command before it updated the syntax after indenting each line) or crash outright, as with some of the more complex commands in Lisp syntax. So, easy solution: make syntax functions call update-syntax themselves to make sure they always operate on, or return, up-to-date information. Well, not so easy as it seems, as the only place a description of buffer-changes, since the last syntax-update was performed, was stored, was in the buffer object itself, and update-syntax would overwrite this information with a "no changes" flag, which might not be expected by other pieces of code interested in when the buffer changes. To be fair, some Emacs-style hacks could have been implemented, buffers could have been tied inseparately to their chosen syntax, and we'd end up with something usable. But the pain this would bring if we want to do more complicated things with buffers and syntaxes (like having two windows displaying the same buffer of HTML code, one showing syntax-highlighted raw code, and the other showing a rendered WYSIWYG version) made this seem like a really bad idea.

With views, the syntax is not an intrinsic property of a buffer, rather it is a property of a specific view of a buffer, and a buffer is not specific to anything, except the Climacs instance itself. The new buffer-modification-protocol enables views to register themselves as observers of a buffer object and be notified whenever changes happen, with a description of what part of the buffer was modified by the changes. The view can then maintain a clear picture of how much of the buffer changed since the views internal data structures were last synchronised to the actual buffer contents (for a typical view, this will be a parse tree based on some grammar). Whenever a view is asked to do something (like, redisplay itself on a pane) it will synchronise itself, using its stored information about buffer changes to figure out which parts of its data is outdated and needs to be brought up to date. In Climacs, views are dealt with in approximately the same way as buffers - you use C-x C-b to change between them, C-x k to kill them, etc, but there are some minor differences, in particular because a view cannot be displayed in more than one window at a time (in contrast to Emacs buffers).

An interesting consequence of views is that there is no rule for how a view of a given buffer should be presented - the obvious way is of course to show the contents of the buffer coloured to fit some provided syntax scheme, but an interesting alternative is to show a more high-level interpretation of the buffer contents. For instance, this file defines a view displaying the definitions of a Lisp buffer of some other view (screenshot of incestuous use here), with each definition being a clickable hyperlink to its location in the buffer (this is CLIM, after all). The view is automatically updated whenever the buffer contents change, removing a lot of the tedium of writing somewhat interesting views. Of course, there's still problems: the Climacs UI is not really geared to deal with so unusual views yet, and the two views both maintain their own parse trees of the buffer, despite the fact that they're equivalent. Some optimisation is probably in order here. Of course, Drei, Climacs and McCLIM has slightly more pressing needs than the implementation of whimsy views of dubious utility, so it is unlikely this particular situation will improve significantly any time soon.

During a semi-recent reading of CLtL2, I came upon a feature of macro lambda lists I had not seen before - namely, the ability to create destructuring optional and keyword parameters in addition to the common positional destructuring parameters. I wondered why I had never seen this feature used in any of the code I had read, and came to the conclusion that it had to be SLIME's poor support for destructuring parameters in general. I decided to spend some time implementing this critical feature in Climacs, and on the way created a protocol for handling lambda lists that combines the ease of use of pathnames with the performance of CLIM incremental redisplay. This is a screenshot and this is too.

Now, what's next? Ironing out the last Drei input-editing bugs is probably in order, and adding more indentation rules (like :method-options to defgeneric forms, something SLIME doesn't handle well either) and restoring Climacs' "group" feature (a facility for performing commands, such as search/replace, across several buffers or files) is probably also due. Now, the elephant in the china store is still redisplay, of course. I suspect anyone who has tried out Climacs has first and foremost experienced the very poor redisplay performance, compared to just about every other editor. The problem here is that there doesn't seem to be any consensus on why McCLIM's incremental redisplay is so slow, or if it's even supposed to be usable for real-time redrawing as in an editor, and in that case, what to replace it with. I guess a first try would be to examine other editors redisplay engines, copy what they do, and try to fit it into the CLIM framework. One potential source of hair is that the Climacs display contains full-fledged CLIM presentations, meaning that they are automatically click-sensitive when applicable, and associated with some Lisp object in the CLIM way, and it'd be nice to retain this functionality even if Climacs (actually, its editor substrate, Drei) ends up doing its own drawing.

Anyway, the nice thing is that no matter what the redisplay engine ens up like, the improvements based on the syntax and buffer representations will stay usable without further work, so when redisplay performance becomes good enough, Climacs should already be a fairly capable editor.

McCLIM and Climacs development has been somewhat silent for the last few weeks, but a few days ago, John Q. Splittist created that thing that most Climacs hackers have always agreed would be nice to have, but nobody really implemented: a C syntax module. At the same time, he also factored the general LR parsing stuff out of the Lisp syntax module in order to make it somewhat easier to make relatively fast parsers for Drei, the editing substrate of Climacs. The C syntax module itself lives in the Climacs repository, but since Drei syntaxes are global, all Drei-using programs can make use of it. And also, since Drei is meant to be polymorphic, the C syntax module is automagically available in the input-editor version as well, which can lead to interesting (to me!) hacks: screenshot!

(I should note that the presentation type definition on that screenshot has a bug: it really should have :inherit-from 'string, or CLIM will complain that it cannot convert the typed input to a Lisp object.)

So, now that the hard part of the job is done, who's going to write that C Listener?

Based on Christophe Rhodes' earlier work, I have created some standalone binaries of CLIM-Desktop using SBCL's simple facility for creating executable images. They were built with SBCL 1.0 on GNU/Linux on a standard 32bit x86 machine, and have been thoroughly tested (by McCLIM standards) on other machines. When run, the CLIM Launcher program will start and present a short menu of available CLIM applications, with the exceptions of the Listener and the Inspector, and the image will terminate when the launcher application is stopped (and you won't be queried for saving work you have open in, for example, the editor, so be careful). Apart from the usual bugs you might find in McCLIM, CLIM-Desktop undoubtedly introduces a bunch of new ones, but the amount of total Lisp system corrupting bugs should be fairly low, so at worst, you should just get dumped into the CLIM debugger. Of course, if you mess up the connection to the X server, the debugger won't run, and SBCL can get mighty surly.

The applications and libraries included are mostly CVS versions, with a few patches here and there to work around the most common bugs and display a little bit of extra demo-stuff. These are the ASDF systems loaded and willing to provide a version number: usocket 0.2.4, cl-irc 0.8-dev, cl-ppcre 1.2.12, cl-fad 0.5.0, bordeaux-threads 0.0.1, trivial-sockets 0.2, flexi-streams 0.5.7, skippy 1.3, clim-desktop 0.2, sb-bsd-sockets 0.58, sb-grovel 0.01, clx 0.7.2, flexichain 1.1, mcclim 0.9.4, split-sequence 20011114.1 and asdf-binary-locations 0.1. The real list is significantly larger, so it might be hard to convince someone to set up all of that just to try out CLIM-Desktop, but hopefully, this release will make people more receptive, once they've tried it and discovered that it isn't solely hot air. The only thing I can immediately think of that won't work is the Meta-click functionality, where you can hold Meta and click on certain things (symbols, classes, functions, commands) and be taken to their definition in the editor, because the source code isn't available. You need the full version for that! Also added is the use of the M-Tab gesture for general completion during input-editing, because the default (Mouse-2) is annoying. The program used to build the CLIM Desktop binary is available here.

There are two versions available, if one doesn't work, try the other: CLIM-Desktop 0.2 with Freetype, CLIM-Desktop 0.2 without Freetype.

Today, version 0.9.4 of McCLIM was released. McCLIM is a free and increasingly-complete implementation of the Common Lisp Interface Manager 2.0 specification. Version 0.9.4 has some nifty new features that make McCLIM nicer to use - for one, the old editor substrate, Goatee, has been replaced with a new one based on Climacs code - Drei. In general, Drei has a lot of nifty features that Goatee lacks, such as syntax-highlighting and an official way of defining new commands. When using the CLIM Listener, you now get proper syntax-highlighting and symbol-completion, bringing it closer to being a usable tool. Drei has some problems, though, some of which it has inherited from Climacs. For one, it has quite slow redisplay. Secondly, it's quite complex and offers far more nooks and crannies for bugs to hide in. Thirdly, it attempts to do some things (such as literal Lisp objects in the buffer and variable-width fonts) that prevent us from handling some things easily. If you hit a bug in Drei that totally prevents your favorite CLIM application from working, there is a not-so-secret internal variable, clim-internals::*use-goatee* that you can set to true in order to disable Drei and use the old editor substrate instead.

One of the main points of Drei was to provide the ability to define editing commands that apply in every text-editing context across CLIM applications, and I think it's become as easy as possible. Here's an example (adapted from the under-development McCLIM User's Manual).

A common text editing task is to repeat the word at point, but for some reason, Drei does not come with a command to do this, so we need to write our own. Fortunately, Drei is extensible software, and to that end, a DREI-USER package is provided that is intended for user customizations. We're going to create a standard CLIM command named com-repeat-word in the command table editing-table. The implementation consists of cloning the current point, move it a word backward, and insert into the buffer the sequence delimited by point and our moved mark. Our command takes no arguments.

(define-command (com-repeat-word :name t
                                 :command-table editing-table)
    ()
  (let ((mark (clone-mark *current-point*)))
    (backward-word mark *current-syntax* 1)
    (insert-sequence mark (region-to-sequence mark *current-point*))))

*current-point* and *current-syntax* are two of a number of special variables that provide access to editor state during command invocation.

This command facilitates the single repeat of a word, but that's it. This is not very useful - instead, we would like a command that could repeat a word an arbitrary (user-supplied) number of times. The primary way for a CLIM command to ask for user-supplied values is to use command arguments. We define a new command that takes an integer argument specifying the number of times to repeat the word at point.

(define-command (com-repeat-word :name t
                                 :command-table editing-table)
    ((count 'integer :prompt "Number of repeats"))
  (let ((mark (clone-mark *current-point*)))
    (backward-word mark *current-syntax* 1)
    (let ((word (region-to-sequence mark *current-point*)))
      (dotimes (i count)
        (insert-sequence mark word)))))

Great - our command is now pretty full-featured. But with an editing operation as common as this, we really want it to be quickly accessible via some intuitive keystroke. We choose M-C-r. Also, it'd be nice if, instead of interactively querying us for commands, the command would just use the value of the numeric argument as the number of times to repeat. There's currently no way to do this with a named command (SE. when you run the command with M-x), but it's quite easy to do in a keybinding. We use the set-key function from ESA:

(set-key `(com-repeat-word ,*numeric-argument-marker*)
         'editing-table
         '((#\r :control :meta)))

Now, pressing M-C-r will result in the com-repeat-word command being run with the first argument substituted for the value of the numeric argument. Since the numeric argument will be 1 if nothing else has been specified by the user, we are guaranteed that the first argument is always an integer, and we are guaranteed that the count argument will have a sensible default, even if the user does not explicitly provide a numeric argument.

There are many other improvements, the most majors ones probably the improvements to Gtkairo, but since I do not work on Gtkairo, so I do not feel qualified to talk much about them (but you can look at some screenshots). There is also still much work to be done. I don't have a TODO-list, as that suggests some kind of discipline, but I do have a NICE-TO-DO list:

• Make Drei redisplay less slow and structure the output records so variable-width fonts and variable-width lines can be implemented sanely and efficiently. Perhaps by improving the performance of McCLIM incremental redisplay?
• I suspect that it should be possible to call presentation generic functions for objects that have defined a method on presentation-type-specifier-p that returns T. Current McCLIM has some internal function try to find the metaclass for a candidate object, and error out if it can't find it.
• Presentation histories still do not behave exactly as people might expect, as they do not actually store the textual input. Perhaps they should? Views would have to be handled somehow.
• Gadget views (picture)!
• Maybe revise the names of the Drei special variables. Why not just *buffer* instead of *current-buffer*? And *current-window* should probably be *editor*, since it's not really a window when the command is running for an input-editor.
• Implement an output-record based minibuffer for the input-editor so we can have M-x extended commands.
• Fix the layout managing of certain composite classes. Many of them just ignore the max-width and max-height attributes of they child panes, leading to visual distortion or just having your space requirements plain ignored, which can be very frustrating.

If you have never tried McCLIM before, now's a good opportunity to try it out. This release points out two significant things - that McCLIM doesn't have to be ugly (see Gtkairo), and that non-Emacs text editors don't have to be weak. I'm probably biased, but seeing the glimpses of Drei's potential has made me far more certain that the whole "put Emacs in everything" idea (as opposed to "put everything in Emacs") is going to work out fine.

By a twist of fate, I have been introduced to two different test frameworks these past two weeks - JUnit for Java and FiveAM for Common Lisp. This has given me an interesting opportunity to look at how tool design differs between these two languages - in JUnit, I had to define a class (inherited from a JUnit TestCase class) and define methods on it. These methods could contain calls to handy assertion methods, such as AssertTrue and AssertEqual, that were used to actually perform tests. Then I have to create a test suite, add my classes to it, provide a TestResult object for the suite, wherein the test results will be dumped; after that, I can enumerate through the failures (again objects) and figure out what to do. This is quite a bit of overhead for simply running tests, in my opinion, and it doesn't seem like much is done on improving it, since most Java IDE's make this process way less painful. (In general, you just point at a file containing a test case class definition and press "Run test", but that doesn't help me that much when I prefer to use Emacs.)

FiveAM is far simpler, at least for a Lisper. You just define your tests with the test macro, use the simple is macro for most of the assertions, and use the run! function to run a test and print the result to standard output. If you want more organization, and you probably do for large sets of tests, you use def-suite to define a test suite, and in-suite with a test suite argument to activate it - thereafter, the test macro will put the test in the specified test suite. It's simple, easy and convenient. Also, due to the use of macros, testing for such things as whether a given expression signals a condition/throws an exception can be expressed in a much more elegant way than in JUnit.

One thing one has to be aware of, though, is that FiveAM doesn't wrap the body of the test macro in a lambda form and stores it in the test suite - it stores the literal code and (re)compiles it every time the test is run. This has at least one good point, as well as some bad ones - on the upside, it means that if you redefine a macro, you don't have to redefine the test to apply the changes - they're automatically picked up when the macro is expanded at the compilation step when running the test. On the downside, running tests can take a very long time due to the compilation, you're not informed of compiler warnings when you define the test and, worst of all, the tests are compiled and evaluated in the null lexical environment, which means you can't do things such as (with-neat-test-function (test my-test-1 (neat-test ...) ...) ... (test my-test-n (neat-test ...) ...)).

All in all, I'm highly satisfied with FiveAM - it's simple to use (and I really don't see a good reason for why a test framework should be complex) and does what it's supposed to do. It's a really nice tool, and I recommend it to every Lisp programmer who is trying to decide upon a test framework.

For me, one of the goals of the McCLIM project is to facilitate the creation of an environment where everything is written in Lisp. The difference from the other failed (or stagnant) LispOS projects is that this is a top-down effort where we start with writing directly-usable (well, somewhat) applications instead of hacking on device drivers and that sort of thing, so we get useful results fairly quickly. If you pair McCLIM with Eclipse, you get an environment that is Lisp all the way down to the socket talking to the X server. (The fonts in the screenshots aren't Lisp, though, and the optional interface to Freetype is based on FFI. Also, I use a C program to take the screenshots). This morning, I took a few screenshots of what this looks like:

• Here we see that Closure, the CLIM webbrowser written by Gilbert Baumann, is capable of displaying all websites of significance, though it seems to dislike some of the characters in the entry titles. (Closure needs quite some love, so if you feel like hacking on a somewhat-crufty CLIM program, here is a great opportunity!)
• Beirc, the premier CLIM IRC client works with Closure, as seen here, if you use CLIM-Desktop. Beirc is actually a pretty nifty application in its own right, probably one of the most directly usable CLIM applications I know of.
• As seen on this screenshot, I got tired of using the Listeners Run command explicitly, so I wrote a little function to do it for me (twice, because I got it wrong the first time, which cause the also-Lisp CLIM Debugger to pop up). If you use Drei, the Climacs-based editor substrate, you get nice things such as syntax-highlighting and symbol-completion in the Listener. Parameter hinting doesn't work that well, because the Listener doesn't have a minibuffer, so we have to attempt to put the text in the pointer-documentation-pane (the black bar), but that approach is unreliable (the pane is refreshed very quickly by McCLIM itself). Also, the rectangular border around the (shot "3.png") is there because I have just clicked on the expression to re-evaluate it, but the screnshot was apparently taken before the presentation was inserted at the input area.
• This screnshot is supposed to demonstrate CLIM's "presentation history" feature. Basically, I navigated backwards in the list of Beirc input. Most IRC-clients and command shells support this, but CLIM generalises it to be presentation-specific and relevant in every text-input context. So, for example, whenever an application asks me for a number, I can browse previous inputs of numbers (for that frame only, though). This doesn't yet work reliably in practice, but it's pretty nice when it works.
• On this screenshot, I have just run the Listener command Show Class Subclasses with clim-stream-pane as the argument. After that, I M-clicked on the DREI-GADGET-PANE output and had the definition for it show up in the already-running editor. You can also M-click on commands and be taken to their definition, something that's actually pretty useful when hacking on CLIM applications.
• This is what happens when you push your luck. I accidentally triggered a nasty X-LENGTH issue that (I think) has something to do with using McCLIM-Freetype, which caused McCLIM's connection to the X server to corrupt. It might have been recoverable, but I wasn't dropped into a debugger - probably because the debugger itself is a CLIM application. Interestingly, since the Eclipse window manager didn't use the same connection to the X server as McCLIM, I was able to conjure up an xterm and take the screenshot.

McCLIM does some cool things already, but it needs work - patches are, as always, welcome.

I often get the impression that many in the Lisp community dislike library dependencies, and prefer to build as much as possible out of standard Common Lisp. I can understand why completely unrestrained addition of library dependencies can be a bad thing - the libraries may be buggy, unportable, hard to install or hardly accessible. The user might be annoyed at having to download and install a large amount of libraries in order to run some application. I don't mind Lisp systems that depend on portable, high-quality libraries (such as most of what Edi Weitz writes), nor do I mind adding a dependency on those libraries to my code. I'd much rather use something like FLEXI-STREAMS instead of hand-rolling my own encoding-handling code, even if I only end up using a fraction of the functionality of FLEXI-STREAMS. I also see parts of the functionality of SPLIT-SEQUENCE and CL-PPCRE duplicated when needed, despite the fact that both of these libraries are of high quality and have good documentation. The performance difference between the use of one of these libraries and a hand-rolled piece of code is likely to be negligible, and a call to split-sequence:split-sequence or using a regular expression with CL-PPCRE is also likely to be much easier to figure out for the reader of the code, as opposed to making the reader figure out some complex-looking do or loop form.

Even if using a library might technically be the right decision, it might not be in the context of the users of your application. A Lisp system that depends on a large amount of hard-to-install libraries, perhaps even some that must be retrieved from well-hidden source repositories, can be highly painful to install, and even a seasoned hacker might get annoyed with the whole thing and give up. Especially if one of the dependencies is a forked-CVS-version of a library that is not compatible with the library already installed (and used) by the user. In most modern operating systems, dependencies are no longer a serious problem - any reasonable package manager is capable of automatically downloading all the dependencies of a package, and installing them before installing the package itself. We have such a system in the Lisp world as well - ASDF-Install. While ASDF-Install certainly has its problems, it has worked very well for me when I just wanted to check out some cool application, and didn't want to bother with installing stuff manually. A good package system seems to be a core part of a modern software infrastructure, and it seems to be just the thing that makes usage of large quantities of libraries realistic, which is why I believe it is worth fixing the imperfections of ASDF-Install, or develop a replacement if the current design is insufficient (the use of Cliki as the database might not be such as good idea - perhaps something common-lisp.net or cl-user.net based would be better?).

While I do not claim to be an expert on every software development ecology in the world, this dislike of libraries is something I have mostly encountered in the Lisp world. The programmers of other languages seem to embrace a large number of highly specialized utility libraries - perhaps because they have a well-working package management infrastructure in place (Perl CPAN or Ruby Gems, for example)? As a horribly meaningless data point, GNOME's standard editor, gedit, is dynamically linked with 63 libraries on my system, but because my package system has taken care of the dependencies, I do not really care - as a user, what matters is that gedit has functionality, and if I were to hack on it, I would appreciate that I could use my knowledge of the standard libraries it uses, instead of having to figure out the implementation details of yet another general, yet custom-rolled, piece of functionality. Fortunately, not all Common Lisp projects seem to shy away from the use of libraries - UnCommon Web, for example, seems to be created mostly as a shortcut for loading everything on common-lisp.net. I like using libraries - I want to write new stuff, not rewrite stuff written by thousands of programmers before me. I hope that I can find out why some Common Lisp hackers dislike dependencies on larger sets of libraries, and either fix the problems, or be enlightened as to why dependencies are a Bad Thing.

Thanks to tireless Release Engineer Andreas Fuchs, McCLIM version 0.9.3 has been released! I was about to write a blog entry about how this was a release short on shiny features, and that it mostly had solid bugfixes and changes of that sort. But as I have already used up my quota of unsubstantiated claims for this week, I decided to check the list of changes, just to be sure, which resulted in me being reminded that this release actually has some pretty major additions. The most shiny new feature is probably Gtkairo - a new backend using GTK widgets and Cairo for drawing. It's not yet as functional as the default CLX backend, but it presents the interesting future prospect of McCLIM applications not being as ugly as they are now. Also, using high-level Cairo drawing primitives easily beats raw X drawing for convenience. Personally, the McCLIM change I find most interesting is that Andreas Fuchs removed a nasty memory leak in the incremental redisplay code, that would eventually cause applications using incremental redisplay to gobble up all available memory. This means that Climacs will now run for more than ten minutes before getting unusably slow due to constant large-scale garbage collections! (It's still slow, but at least it no longer degrades even further with use.)

Also, making the freeze period so short (McCLIM was frozen last saturday) seems to work much better than long freeze periods that eventually just fizzle out. I doubt freeze periods cause the McCLIM hackers to start testing obsessively anyway, so the primary motivation and goal for the freeze period should be to make sure that no-one will commit any experimental buggy code just before a release.

McCLIM 0.9.3 probably works best on a threaded SBCL, and doesn't work on CMUCL and CLISP. The reason for the CMUCL breakage is the lack of proper support for forward-referenced classes in CMUCL's CLOS implementation, but that has been fixed in very-recent CMUCLs, so you might be able to find a patch so you can run McCLIM (it's fixed in 19d patch1 I'm told). The reason for the CLISP breakage is somewhat similar - CLISP chokes when compiling certain files that contain function definitions in whose body can be found a form resembling (typep x 'foo), where foo is the name of a class that has not yet been defined. I have not yet been able to create a small test case for demonstrating this, but CLISP hackers are most welcome to look at the issue by compiling McCLIM. I had an idea about how to create a hack that defined stub classes in advance, but for some reason CLISP suddenly compiles McCLIM without problems on my system, so I'm unable to get a list of the affected classes. You might be just as lucky. No matter what, remember that your CLISP will need to use MIT-CLX, NEW-CLX is known not to work.

... my project to create a powerful CLIM-integrated editing component based on Climacs seem to be nearning success. I now have a functional input editor and text editor gadget implementation (screenshot), and I even have it completely integrated with the McCLIM build system (harder than it sounds, because the editing component relies on almost all of CLIM in order to compile, while McCLIM expects the input editor to only use a minimal subset of CLIM).

It will be interesting to see how an alternative "put Emacs in everything" design will fare against the traditional "put everything in Emacs" principle. Anyway, I finally begin to see some results of two and a half weeks of hacking, which is nice.

Amusingly, ever since I started attending university, the ratio between my programming productivity and my free time has increased. Perhaps the sense of urgency I get from not being able to whatever I want all the time has helped me focus better? I know that I did (and still do) procrastinate quite a bit during times I actually had picked out for hacking. Fortunately, this means that I have done some, in my opinion, interesting work on Climacs, the premier CLIM-based reimplementation of Emacs in Common Lisp. One thing I am very happy to have solved is a stupid bug that effectively made Climacs unable to handle files with lines longer than the width of the display - a rather aggravating deficiency, don't you think? And just as embarrassing as the bug was, just as easily was it fixed - the redisplay code very plainly "forgot" to tell the scroll bar mechanism of the text width, something that was fixed in a few lines. After a dozen more lines or so, Climacs also sported automatic scrolling based on the horizontal position of point in the buffer. A lot of this has to be done manually by Climacs, which is the price of circumventing the usual CLIM text editor facilities. I'm still not entirely sure that McCLIM's failure to pick up the output width is the right behavior, though.

Another, and, perhaps, more interesting addition is the creation of commands for rectangle editing - an interesting and powerful concept that I have not before seen outside of Emacs (Edit: it seems that many editors actually support this. Doesn't make it less cool, though.), that permits one to do text manipulation of rectangles that ignore line bounds, as opposed to traditional editing commands, that are limited to perceiving the buffer as a linear stream of characters. Useful stuff. Anyway, during my research on how to properly implement this feature, I, of course, read how GNU Emacs does it. My experience reading the Emacs Lisp-code once again assured me that Climacs Really Is Necessary, because... well, let's just say that GNU Emacs' implementation of rectangle editing is not very clean, and quite a bit more verbose than it needs to be. Well, really, you can take a look for yourself: Climacs' rectangle.lisp versus GNU Emacs' rect.el. In all fairness, however, it should be noted that GNU Emacs' implementation is more complete in the "nice-to-have" department, with better history for commands and better handling of the difference between spaces and tabs. Still, I generally find Climacs' Common Lisp code far easier to read and modify than Emacs' Emacs Lisp code - but that should not really come as a surprise to anyone. Emacs Lisp is quite old-fashioned. It works, however, and it has permitted the implementation of one of the greatest pieces of software ever, so it's not all bad.

From reading old Hemlock documentation I got an idea for implementing a concept called groups in Climacs. Basically, a group is a set of buffers or filenames, and when a group is selected, search- and replace-commands will act across all buffers or files in the selected group, instead of just the current buffer, as it most commonly is now. This is actually something I am not aware exists in out-of-the-box GNU Emacs (though Dired has something like it). I also figured that there was no reason to limit groups to being static lists of files or buffers, so I also implemented groups that calculate their file-list on-the-fly when they are needed - permitting abstract groups that designate sets such as "all modules of a specific ASDF system". This could be the start of proper refactoring facilities in Climacs. Interestingly, John Q. Splittist informed me, that a similar facility, called possibilities buffers, existed in ancient Zmacs, the Emacs of the Lisp Machines! Since Zmacs is arguably the Emacs we're trying to imitate in Climacs, the fact that we have ended up duplicating features, that we were not even aware of existed, is a good sign.

As usual, hacking on Climacs has been a very enjoyable process - apart from everything being in Common Lisp, making development very rapid, Climacs' functionality as a full-fledged, introspective Common Lisp editor makes it very easy to test Climacs from within itself. It's certainly more fun than writing extensions for GNU Emacs, so if you're a bored Common Lisp hacker looking to pass some time, writing more commands for Climacs is really not that hard, but very rewarding. Instant gratification makes this kind of hacking quite fun.

Swine has finally been moved to Climacs - this means that Climacs (a reimplementation of Emacs in Common Lisp) is now a reasonable editor for hacking Common Lisp code, out-of-the-box or fresh-from-CVS. All it requires is that Swank is either loaded (detected by the presence of a :swank package) or available through ASDF.

Also, I hacked up a defclass prettifier to neat up slot definitions: clicky. It's not yet very complete (or bug-free I guess), but I've been thinking about creating a more general code prettifying facility for Climacs that can prettify more than just defclass forms (and more intelligently too, I hope). Also, this little project has convinced me that writing text manipulation code in Common Lisp for Climacs by moving around marks and calculating offsets, sucks just as much as it does in Emacs Lisp for Emacs. Climacs has a really neat and elegant syntax facility for interrogating the buffer contents, but when you actually have to modify the contents of a buffer, you still have to deal with a simple array of objects. Some heavily macroified interface will have to be developed if I'm going to write much more code of this kind, but I have not yet thought enough about this.