SIGKILL

LinkedIn phishing vulnerability

The popular social networking site LinkedIn suffers from a phishing vulnerability that makes it possible to take control of another persons account. In particular, it is possible to silently attach an additional email address to another persons account, then use that email address to reset the password and obtain full control of the account. This requires nothing more than the owner of the other account agreeing to establish a connection. The procedure is as follows:

On the Add Connections page, send an invitation to an email account that you control.
Send the invitation email onwards to your intended victim, forging headers as appropriate to make it seem as if the mail came from LinkedIn (the forgery does not have to be perfect, most email clients will not display the defects). Notably, do not change the contact link within the message.
If the victim clicks on the link and accepts the contact, the original email address (which you control) will be silently attached to the victims account, and you can then request a password reset link be sent to it.

The vulnerability lies in step 3, in that the victim is never informed that a hidden piece of information (the email provided in step 1) will be attached to their profile. While not strictly a technical problem, the phishing vulnerability is based on an implementation that does not make the consequences of user actions explicit, and which embeds potentially harmful information in a seemingly innocent link. Problems such as these are every bit as important as a buffer overflow or XSS-vulnerability, and should be guarded against just as vigilantly.

This vulnerability was discovered (and tested!) on a grey afternoon by Martin Dybdal, Ulrik Rasmussen, and myself.

Den Datamatiske Devolution

I datamaternes tidligste tid interagerede man med dem ved at ændre dem direkte: Ved at indstille kontakterne på deres kontrolpanel kunne man kommunikere med dem i deres eget sprog, i den materielle struktur som udgjorde deres egentlige eksistens.

Så fik man præcise og veldefinerede skriftsprog, hvor alt havde en veldefineret og kontrolleret mening: Der var yderst få tvetydigheder, meningen blev bestemt og veldokumenteret af centrale administrationer, og det var muligt at kommuniker effektivt og præcist med datamaten, så snart man havde investeret en moderat mængde tid i at sætte sig ind i konventionerne.

Med tiden steg mængden af disse skriftsprog eksplosivt: Hver datamat havde sine egne kommunikationsregler, ja visse enkelte installationer afviger endda drastisk fra deres naboer, og det kan være utroligt svært at vide hvordan kommunikationen skal foregå, med mindre man kan spørge en lokalkendt om råd.

I 80'erne og 90'erne udviklede man for alvor musen og de grafiske brugergrænseflader, hvor man nu måtte udtrykke sin mening med mere eller mindre vagt kropssprog. Nu hvor talegenkendelse er ved at slå igennem er vi endelig nået mod slutpunktet for devolutionen indenfor grænseflader: Som jeg har illustreret ovenfor er vi indtil videre gået modsat evolutionen for menneskeligt talesprog, og derfor er vi snart reduceret til at pege og grynte ad datamaten for at bibringe vores intentioner.

Hvis idéen med passive, allestedsværende, indlejrede enheder ("pervasive computing") virkelig slår igennem vil vi endda afvikle os til et punkt hvor vi slet ikke kommunikerer med datamaten længere. Det tog os millioner af år at bevæge os den distance med menneske-til-menneske-kommunikation! (Omend det naturligvis var i den anden retning.)

Skønlitteratur

I de seneste par uger har jeg genopdaget en interesse som jeg har forsømt i alt for mange år - læsning af skønlitteratur. Derfor er jeg gået i gang med at indkøbe og læse en serie af fantasy-romaner skrevet af min yndlings fantasy-forfatter, Raymond E. Feist. Hans historier, som tager deres begyndelse med Riftwar-serien, er især værd at læse fordi de forstår at kombinere det fantastiske med det hverdagsagtige, på en måde så det fantastiske ikke bliver dagligdags og derved uinteressant. Tolkiens bøger er baseret på samme koncept, men sætter dog også et relativt lavt loft over omfanget af supernaturlige fænomener, hvor Feist nærmest har gjort en sport ud af at gøre den lejlighedsmæssige magiske indblanding mere og mere ekstrem. En anden ting, der gør Feists bøger interessante, er at de tilsammen dækker en meget lang tidsperiode (over 100 år) i hans fiktive verden; man følger derfor hovedpersoner vokse op, ældes, få børn og dø af naturlige (eller unaturlige) årsager, hvorefter deres børn tager over. Det giver verdenen en atmosfære af realisme at de mange store begivenheder således kan spredes over en længere periode, i modsætning til visse andre langvarige fantasy-serier som lader til at have potentielt altødelæggende katastrofer hvert andet år.

Haskell og Debian

Tilbage i 2007 var mit nytårsforsæt at lære Haskell. Eftersom jeg skriver om det nu, må det vel antages at det ikke blev opfyldt, hvilket er korrekt. Dog er jeg nu begyndt at rette op på det ved at læse lærebogen Programming in Haskell og løse Euler-problemer. Lærebogen er ganske udemærket, dens eneste brist er at den ikke forudsætter nogen form for forgående programmeringserfaring og således bruger en del sider på at beskrive grundlæggende programmeringskoncepter (og bogen er ikke for tyk til at begynde med). Det lader til at være et generelt problem med mange af de funktionsorienterede sprog: Stort set alle tilgængelige bøger er lærebøger beregnet til helt nye programmører, en situation hvor Common Lisp, der har mesterværker som Art of the Metaobject Protocol, skiller sig noget ud.

For at opnå erfaring med brug af Haskell i rigtige projekter er jeg gået over til en ny window manager, xmonad, baseret på den tese at jeg vil blive så irriteret over diverse ubekvemmeligheder at jeg bliver motiveret til at ændre i koden. Der er allerede adskillige ting der irriterer mig (først og fremmest de elendige genvejstaster, et punkt som ratpoison har forvænt mig med).

Jeg er også begyndt at overveje at droppe Debian som styresystem til fordel for Ubuntu. Min oprindelige argumentation for at bruge Debian var at det ville være lettere at vedligeholde Debian end at tilpasse Ubuntu til mine til tider usædvanlige præferencer. Efterhånden er jeg begyndt at tvivle på dette, især efter mit system er begyndt at nægte at vise output på både laptop-skærmen og den tilsluttede VGA-skærm på samme tid.

Terroristerne kommer!

Som de fleste vel efterhånden er klar over var der for et par dage siden et selvmordsangreb på en dansk ambassade i Pakistan. Siden da har medierne været fyldt med artikler og ledere om at Danmark har overstrukket sig selv og nu er blevet et terrormål helt på højde med Storbritannien, USA og Israel. Lad os betragte den handling der affødte dette hysteri. Det var et terrorangreb, jovist, men det fandt sted i udlandet, ingen danskere kom til skade og det skete adskillige år efter den gerning, som det angiveligt skulle hævne. Forstå mig ret, jeg mener ikke at terrorangreb bliver mindre afskyelige af at der ikke er dødsfald i min egen klub, men det virker nærmest respektløst at sammenligne dette med de voldsomme angreb som er sket på Storbritannien, USA og Israel.

På sin vis minder det om borgerkrigen i Afghanistan, hvor vi hver eneste gang en dansk soldat mister livet, får at vide at vi er ved at tabe krigen og ikke er i stand til at føre den. Det synes jeg igen er en forvanskning af sandheden. Betragt krigs--statistikkerne: De danske tropper og deres allierede mister voldsomt færre mænd end deres fjender, og i forhold til tidligere krige er den materielle og personalemæssige investering minimal. Selv Danmark har personligt ført væsentligt større krige i vor fortid, og dengang var vi langt fattigere og fåtallige end nu. På nuværende tidspunkt er vores indsats i Afghanistan, samt terror-"truslen", langt mere baseret på politisk vilje end egentlige militære og sikkerhedsmæssige overvejelser. Det virker mest af alt som et ønske om at slutte sig til de seje lande som virkelig er i fare for terrorangreb, men i sidste ende er det bare patetisk.

#:els:2008

I have just returned home from the European Lisp Symposium of 2008, held in Bordeaux, and I shall now inflict upon you my experiences from the event.

I'm not a very experienced traveller - indeed, this was my first time travelling by plane by myself, so I was slightly worried about getting lost along the way from Copenhagen to Bordeaux. My trip was fairly uneventful, though I was a bit mystified by travelling via a KLM-owned Fokker 100, a plane manufactured by a company that left the business quite a while ago. I was not less mystified when the rest of the flight was done in a plane that the safety guide claimed was a Fokker 50, but which most assuredly did not have turboprop engines.

For my part, the stereotype about French people not understanding English turned out to be fairly true. Even more, the tram ticket machine did not understand English either, the clearly labelled button for changing the interface language seemed totally nonfunctional, as if it was literally just painted on the machine. I was told that the attitude towards English had changed in France - I can only imagine it has changed from open disdain to a feeling of irrelevance.

Anyway, on to the interesting part: the actual symposium, which spanned over two days. The first day contained confusingly named "Birds of a feather" sessions (also known as workshops/discussion sessions). They were quite interesting, though I admit I had too little interest in parallel programming to be that aware during the first session. The CLIM/ McCLIM session was interesting, though my own part in it was pretty much a disaster due to my SBCL session getting totally messed up in some way. Scott McKay (the original designer of CLIM) managed to inject some interesting comments though. Amusingly enough, it did not seem like he had the same vision for CLIM as I have now, and rather took great pride in how the spiritual successor (DUIM for Dylan) could perfectly resemble a mainstream, native user interface. The informal session about the eventual successor to ASDF was perhaps the most interesting, some actual and involved discussion ensued and Nikodemus came up with the idea for a clever/horrible reader macro that inspired the title of this blog post. It seems like the ASDF successor will be motivated by two main things: facilitating a sane ASDF-Install replacement and elegantly handling weird compilation options. Exactly what I need, so I won't complain. I may even contribute code if it comes to that.

The presentations were decent, but I found that I learnt more interesting stuff by talking with the authors than by listening to their talks, so I'll elegantly skip to perhaps the most important thing about the meeting: talking to other Lisp programmers in real life and finally meeting people I've talked to online for years. It was very educative and entertaining; from listening to Scott McKays view on CLIM, to discussing more general programming with Christophe Rhodes and Robert Strandh, to getting a primer on Finnish politics by Nikodemus Siivola. I obviously can't give a comprehensive description of every topic touched by these conversations, but I can encourage anyone who has never been to one of these events to attend the next possible one, since actually talking face to face allows some discussions to work much smoother than they do online.

Unless practical problems get in the way, I'll definitely be attending the next European Lisp Symposium. It was certainly a very rewarding investment of my time.

Voxels

Tænk tilbage på de tidlige halvfemsere, tilbage til 3D-computerspillenes barndom. Dengang var 3D ikke nødvendigvis lig med polygoner, sådan som de er nu, i stedet var voxels (3D-pixels) en yderst realistisk mulighed. Som alle ved vandt polygonbaserede spilmotorer stort, hjulpet godt på vej af hardwareacceleration, men voxels har stadigvæk en fordel i kraft af deres fleksibilitet og gode understøttelse for kurvede overflader. Et blik på hvordan 3D i stedet kunne have udviklet sig kan man få ved at prøve Voxelstein 3D, et kort skydespil der bruger en voxelbaseret grafikmotor. Den vigtigste voxel-egenskab dette spil demonstrerer er hvor relativt nemt det er at lave spilområder som kan ødelægges - hvad der er yderst kompliceret med polygoner er trivielt med voxels. Jeg synes selv at det er interessant, men det er næppe overraskende at jeg er interesseret i halvdøde og obskure teknologier der påstår at kunne gøre alting bedre end det, der nu er mainstream.

McCLIM 0.9.6, day of the lizardslayer

McCLIM 0.9.6 "St. George's day" has just been released (Andy Hefners suggestion of "Discovery of the AIDS virus day" was not used), with quite a number of features. It's been over half a year since the last release (which is unfortunately pretty typical for McCLIM), so here's a list of the major improvements:

Runs on SBCL: shortly after the McCLIM 0.9.5 release, the SBCL hackers cruelly removed an unexported, internal symbol that we relied on for some reason, preventing McCLIm 0.9.5 from running on anything but an oldish SBCL. ASDF-Installable McCLIM should therefore now run on SBCL again!
Functions make-pattern-from-bitmap-file and read-bitmap-file added. These are from CLIM 2.2 and permit relatively simple reading of various bitmap image formats (currently supporting JPEG, GIF and XPM).
Editor redisplay is much faster: I wrote a new redisplay engine for the Drei editor substrate a while back, it has vastly better performance and a few extra features. It also improves Climacs a lot.
Fully in-Lisp Truetype font rendering. McCLIM has used an FFI to access Freetype for a while, but Andy Hefner implemented a Truetype renderer in Common Lisp, meaning that McCLIM applications can now have pretty fonts without having to degrade themselves by asking C for help. Load the mcclim-truetype system (as opposed to mcclim-freetype) to activate it.
Tons of bugfixes. As usual, this new release is the most stable McCLIM release ever! A lot of annoyances have been fixed, so McCLIM should now behave more predictably, though there is still some way to go before we reach perfection.

To celebrate this release, I hacked together a simple Logo interpreter. The cool thing about this interpreter is that it abuses accept-from-string to implement a pseudo-Logo programming language. A program can look like this:

Pen Up
Lock
To "Draw Left Half-Circle"
Repeat 18
Forward 10
Left 10
Next
End
To "Draw Circle"
Invoke "Draw Left Half-Circle"
Invoke "Draw Left Half-Circle"
End
Unlock 
Pen Down
Center
Repeat 36
Invoke "Draw Circle"
Forward 20
Next

I'm not going to tell you what the output looks like, instead you can download McCLIM 0.9.6, load tortoise.lisp and use Load Commands to load a file containing the above program (or just enter it manually, if you want) and try it yourself!

Den moderne legende

Danmark har endelig fået sig en rigtig helt, en ambassadør for landet, en personificering af den danske ånd og kultur, som kan bidrage til at fremme vores respekt i udlandet. Jeg taler naturligvis om den fænomenale Martin, som jeg har fulgt på kanten af sædet i mange uger under det forløb hvor han udviklede sig fra en naiv, ung mand til den bastion af selvsikkerhed og karisma han udgør i dag.

Det er måske ikke tydeligt ved første øjekast, men han er faktisk en historisk betydningsfuld faktor i dansk kulturhistorie - man opfatter ofte komponister som Beethoven og Mozart som altid at have fokuseret på den aristokratiske elite, men sandheden er at "klassiske" komponister blot var den tids pop-musikere - sandelig, det trak dem mere ned end løftede dem op, at den aristokratiske overklasse forsøgte at gøre dem til sine egne. I modsætning hertil er Martin nærmest demokratisk valgt, fælles for hele befolkningen, og begunstiget med et guddommeligt talent. Han har ingen af de klassiske komponisters svagheder, men alle deres styrker, så jeg forudser at man om århundreder vil se tilbage på 2008 som gennembrudsåret for den største danske musiker - nogensinde.

Se South Park online!

Måske er jeg sløv som ind i helvede, men jeg har just erfaret at man kan se alle South Park-afsnit online. Idet South Park, især de tidlige sæsoner, er en af de bedste serier nogensinde, har jeg allokeret de næste par dage til at genopleve den del af min opvækst, og det samme bør du.

Dobbelt så godt!

Der har været stille på denne blog i lang tid - nu vil jeg starte op igen med hvad jeg gør bedst, nemlig at brokke mig. I dette indlæg vil jeg kritisere vores kære statsminister, nærmere bestemt hans seneste nærmest tragikomiske besøg hos verdens cowboy, George W. Bush.

For det første lader det til at Fogh opfatter Bush som en god leder af den årsag at han er en "flink fyr" som man "gerne vil drikke en bajer med nede på værtshuset" - en klassisk beskrivelse der åbenbart ofte bruges af den menige amerikaner. Jeg ved ikke om Bush er en sådan flink fyr, jeg ved ikke om han er god at tage på druk med - men jeg ved at det er totalt ligegyldigt for om han er en god statsleder, og jeg forstår ikke hvorfor det overhovedet er relevant at snakke om den slags ting. Jeg vil klart foretrække en usympatisk, ukarismatisk stivstikker af en politikernørd, såfremt han er kompetent, frem for et flinkt, men inkompetent, fjols. Det er grunden til at jeg normalt godt kan tolerere Fogh som statsminister, til trods for mine politiske uenigheder med ham.

Hvad siger Fogh ellers? Vi har bl.a verdens demokratier og verdens frie samfund må stå sammen om at fremme demokrati og sikre menneskerettigheder, samt at han åbenbart deler ideologisk idégrundlag med Bush. Afhvad? Jeg troede Fogh var liberal? Hvad er der liberalt ved at afskaffe habeas corpus? Eller hvad med det fantastiske juridiske trick med at indespærre folk på Guantanamo-basen i årevis uden at stille dem for en dommer - et sted der hverken er rigtig amerikansk eller cubansk jord, og hvor ingen forfatningsgaranterede rettigheder gælder? Vi sikrer menneskerettighederne ved at lære folk at sætte pris på dem! Man værdsætter ikke hvad man har før man mister det!

Jeg mener, jeg er ikke engang synderligt liberal, og alligevel synes jeg den slags tricks er totalt vanvittige. I Danmark er det ikke gået helt så galt (endnu?) - vi er stadigvæk på det punkt hvor vi blot bruger tonsvis af penge på overvågning uden beviselig effekt, projekter der satser mere på at skabe en følelse af tryghed end egentlig sikkerhed. Bah, findes der overhovedet nogen lande i vesten der ikke er blevet grebet af denne tryghedssyge? Hvorfor er de ikke i Skandinavien? Vi skulle forestille at være et folk drevet af fornuft, rationalitet og gensig tillid...

Fakta

Stallmans kode har ikke fejl. Den definerer korrekthed.
Stallman kan afgøre om et vilkårligt program terminerer.
Stallman bruger ikke netværksprogrammer, han læser output fra tcpdump. Stallman foretrækker big-endian.
Stallman kender antallet af decimaler i π
Stallman skriver ikke sine programmer. Han angiver blot et indeks i et transcendent tal.
Stallman kan skrive en kontekst-fri grammatik til C.
Stallmans computer har blot to knapper. Den ene er til gæster.
Stallman kender til en urettet fejl i TeX.
... og i MetaFont.
Forskellen på Gud og Richard Stallman er at Gud ikke er Richard Stallman.

Unusual method combinations are your friend!

Method combinations are one of the really cool, yet often overlooked, parts of CLOS. They allow you to elegantly and succinctly specify protocols that would otherwise require wrapper functions, ugly calls to call-next-method and a fair amount of similar logic in every defined method. Take for instance Climacs, where we have to generate a list of the available commands. This list depends on many things, in particular the current view, syntax and active modes, and so methods must be defined for all these. This practically screams for an append method combination, wherein every part of an object (that is, all subclasses) get a chance to add their own contribution to the resulting list, without any method having to implement any kind of list-appending logic. Without method combinations, this would have required explicit calls to append and call-next-method; rote code that is not essential to the purpose of producing a list of applicable commands. CLOS permits me to write methods that are solely focused on the actual task they are supposed to do, while the system takes care of the tedious task of assembling the various results into the final value. It also makes it trivial to add further methods later, as they are automatically picked up by the method combination. And compared to use of call-next-method, I don't have to worry about whether there is a next-method, CLOS takes care of that for me.

This kind of functionality is, I think, what makes Lisp a powerful language. Many other languages have syntactical shortcuts and convenient functions for everything, enabling very short programs that do interesting things (Ruby is particularly good at this). Common Lisp is not well suited for one-liners and very short programs, the syntax and provided facilities are usually too general. On the other hand, Common Lisp shows its power once the program grows - the language facilities are well suited for splitting up complex code and defining small ad-hoc frameworks (like generic functions with unusual method combinations) that would require a fair bit of code in other languages. Thus, I believe that while Common Lisp may look overly verbose for short programs, it makes it possible to implement long programs in a smaller amount of code than it would take in many other languages, and the resulting code can be flexible and extensible, even for complex problems.

A uniform macro for converting objects to other types

A few days ago, the great topic of #lisp was type conversion functions, and in particular the notation that should be used for them. Some advocated fromtype-to-totype, others totype-from-fromtype, and of course perhaps the most common notation, fromtype->totype. I declared that I preferred a variant of the latter, namely totype<-fromtype, due to the fact that this permits clean and readable nesting, but Nikodemus of SBCL fame pointed out that this could be turned into a more general (<- (totype middletype fromtype) object)-facility. And he was right, the Power of Lisp macros, as commonly pointed out by the sublime Paul Graham, permits all sorts of notational conveniences. I therefore set out to write the most intuitive and clear general type conversion facility I could think of, so that the current common practise of writing Lisp programs as large type transformations will yield more readable and consistent programs in the future.

As my focus was on ease-of-use, an intuitive interface and predictable semantics, I decided that the overall pattern of (<- (totype middletype fromtype) object) was worth keeping. The <- macro will rewrite this into calls like (totype<-middletype (middletype<-fromtype (the fromtype object))) (note that the is used as an optimisation - this is the kind of thing macros allow us to do!). This means that you just have to define a function with the name totype<-middletype, and then it will automatically be used by the macro - you don't even have to export the symbol! Truly, Paul Graham was inspired when he invented macros (and they certainly paid off). Now you could convert types like this:

(<- (integer string) "123") => 123
(<- (integer string integer) 123) => 123

However, I didn't feel like it was "fully there" yet, as you might want to define a type conversion function like this:

(defun integer<-string (string &optional
                        (start 0) (end (length string))
                        (radix 10))
  (values (parse-integer string :start start :end end :radix radix)))

I think you should be able to provide values for those optional arguments. And since I'm writing a type converter, it would be outright embarrassing if you had to provide those values directly, and couldn't convert them from something else - and why not integrate that directly into the single type-list (the first argument to the macro)? That is good program design, as it clearly puts all type-specifiers in a single place. This is now possible:

(<- (integer (string integer)) 1 "123") => 23
(<- (integer (string integer integer)) 1 2 "123") => 2
(<- (integer (string integer integer)) 1 3 8 "123") => 19

As you may note, I decided that the arguments to the type conversion function should go before the actual object to be converted. This is good program design, as the overall macro works by first saying how to convert something, then providing the actual value(s). I think the same rule should count for the values. As an added bonus, it is now possible for machines and humans to perform sophisticated reasoning about the macro without looking too much at the type list, as you know things like "the second through to the second-last argument contains option values" and "the last argument is the value to be converted". You may note that the "real" order of types is preserved in the type list, that's because it's more logical that way. Also, you may argue that Lisp is about freedom, and my requirement to put parameter values first in the value-list conflicts with that ideal; but while Lisp is indeed about freedom, putting the value to be converted first is so plainly wrong that no-one would ever do it, or expect it to work. Is it really loss of freedom if you weren't going to (or weren't supposed to) do it anyway?

But wait, there's more. I mentioned it would be embarrassing if you couldn't easily type-convert the optional arguments, so of course that was implemented too. When the type-list turns into an actual tree, the objects to be used are retrieved from the object-list as types are encountered in the tree during a depth-first search (and, of course, with the above-mentioned rule about the first type in a list, or the "target type", being selected last). So you can now write expressions like:

(<- (integer (string (integer string))) "1" "123") => 23
(<- (integer (string (integer string)) integer) "1" 123) => 23
(<- (integer
     (string (integer string)
             (integer (string integer integer integer)
                      integer))
     integer)
    "0" 0 2 2 10 123) => 12

The final example above truly shows the power of the <- macro. We have (are you ready for this?) an integer, 123, that is converted to a string. This string-conversion takes arguments, namely an integer that is converted from a string ("0"), and also (this is the big one) an integer 10, that is converted to a string, and then back into an integer starting from string offset 0, ending at 2, and in base 2 (the result is the integer 2, of course). Finally, the string (which is now "12") is converted to an integer, and the final result is thus the integer value 12.

But of course, if you have nothing but an elegant and succinct notation, you end up with theoretical mathematics, and I'd like to take this type conversion utility a bit further than that. Thus it is useful to predefine a set of type conversion functions. I'd like to talk a bit about some of the more interesting of these.

A very common task in modern programming is converting a bit-string into an integer, this is done via the bitstring type (note that this is not defined as a type anywhere - it exists purely for the function of conversion). The conversion function from bitstring reuses the integer<-string function, of course - code reuse is a great thing, and that it is possible is a sure-fire symptom of excellent design:

(defun integer<-binarystring (binarystring)
  (integer<-string binarystring 0 (length binarystring) 2))

Another useful conversion is integers to strings - in many other languages the null string, the string "0", the string "false" and others are all equivalent to the false value, while "1", "true" and the like are the true boolean value. I don't know why this feature isn't part of ANSI CL, but I suspect it may have to do with lack of time and resources on part of the standardisation group. But never mind that - while Lisp was not the first language to get an object system, it got the best one by far when it catched up. I was looking for something similar with this conversion function. There is a list of strings that are true ("true", "1", "correct", "proper", etc) and a list of strings that are false ("false", "0", "bad", "deficient", etc), but this alone is just slightly more comprehensive than most other languages, not in an entirely different league, as CLOS is when compared to other object systems. Lisp used to be an AI language, and therefore I felt it would be highly appropriate if this conversion function used intelligent, language-aware semantics as well as the plain list, and thus, it handles "in" and "not "-prefixed strings, meaning that the following useful cases work as one would intuitively expect:

(<- (boolean string) "true") => T
(<- (boolean string) "not true") => NIL
(<- (boolean string) "not false") => T
(<- (boolean string) "correct") => T
(<- (boolean string) "incorrect") => NIL
(<- (boolean string) "bad") => NIL
(<- (boolean string) "inbad") => T

Finally, I'd like to describe the "integer to boolean" conversion. Conformity with prior experiences, and the expectations that arise from these, are important when designing intuitive semantics, so I looked to what boolean values other languages attribute to integers. The generally accepted pattern seemed to be 0=false, 1=true, ..., so that is what I implemented:

(defun boolean<-integer (integer)
  (not (zerop (mod integer 2))))

In conclusion, I think I fulfilled my initial design goals of intuitive semantics and a simple interface for the type conversion utility library. The source code can be retrieved here, but I'm going to submit it for inclusion into Alexandria in a few days, so you hopefully won't need to manually include the file in all your projects in the future.

I can put what in my buffer?

Hello, reddit! There's a link to a screenshot near the bottom, and as an extra treat, this is from my test of the new JPEG reading facilities in McCLIM: jpegsinclimacs.png

As some of you may know, Climacs is an attempt at writing an Emacs-like editor in Common Lisp. This is obviously a hard problem that requires a lot of work - I have often been amazed at just how many exotic cases and conditions GNU Emacs handles in even simple editing functions. It is a very mature program, and it shows, while Climacs is still far behind Emacs in terms of functionality and stability.

But while clever editing commands and similar things aren't that hard to implement - they mostly require time - most peoples initial impression of Climacs has been one of very bad performance, which is another class of problem entirely. Climacs was quite sluggish, even on fast machines, and it's quite understandable that most people didn't get a very good impression of a thin-featured program that was several orders of magnitude slower than Emacs. To rectify this, I have spent the last week and a half writing a new redisplay engine for Climacs (specifically, the editing substrate Drei). Performance is not yet as good as that of Emacs, nor as good as I would like it to be, but it's much faster than the old engine, and even more important, it does not get (significantly) slower as the size of the buffer increases, only the size of the actual displayed buffer region affects redisplay linearly. While doing research on efficient redisplay in text editors, I discovered that there isn't a lot of writing on doing this on a bit-mapped display with colours - I used an article on the Andrew text editor as initial inspiration, but it didn't cover everything I needed to do. Therefore, I'm going to write a bit about Drei's redisplay algorithm.

The redisplay engine was developed based on a key assumption: output records are nice, flexible and useful, but they are too slow and heavy. Hence, the redisplay engine does not use output records, except for handling cursors and some other exotic situations. Instead, it divides the visible region of the buffer into strokes. A stroke is a unique buffer region (does not overlap with other strokes) that can be drawn in a single operation with a single set of drawing options, and that does not cross lines. Put another way, a stroke is a sequence of characters in a line with the same colour and font (strokes can also cover non-characters, but let's ignore that for now). The new redisplay algorithm thus works by fetching strokes from the buffer, starting at the top of the display, and drawing them to the screen until we reach either the end of the buffer or the bottom of the visible part of the output sheet. When strokes are drawn, we remember the location and size of their output.

Due to the constraint that strokes cannot cross lines, we can trivially organise strokes into lines and just check for whether a stroke directly precedes a #\Newline character to figure out when we should go to the next line, and when we do this, look at the strokes of the line and figure out the dimension of the line. Additionally, we get the nice property, that when redisplay is finished, no two strokes overlap.

Stroke objects are kept across display, and are mutated by the mechanism that retrieves strokes from the buffer (the stroke pump, see below), so we can easily check whether a stroke has changed (we mark it as "dirty" and "modified") since the last redisplay, simply by having the pump check whether it is going store already-existing data in the stroke object. If a stroke is obscured for some reason (for example due to a moving window, or part of the cursor being drawn over it), we also mark it as dirty. Taken together, this enables incremental redisplay, as we only redraw strokes that are dirty, and only recalculate their size if they are modified.

The interesting problem is now how to generate strokes for the redisplay engine. This is done through two generic functions, pump-state-for-offset and stroke-pump, that are used to "pump" stroke data into existing stroke objects. This is both to implement incremental redisplay, as mentioned above, and to avoid consing (if you read the code, you'll notice that I've been obsessed with minimising consing in general, perhaps this was not always necessary). For the most common case, these functions just relay to the syntax of the view, which results in either the simple stroke pumping defined for Fundamental Syntax, or a general parse-tree walker defined for LR syntax that can select drawing colours based on the actual parse symbols (wrapped in some macrology, syntax highlighting rules can then be defined like this). There is nothing special about stroke pumping, except that it has to be really fast, as it is completely non-cached and done in full for every redisplay. All the clever caching and "only handle that which has actually changed"-stuff is done at the higher level by looking at the dirtiness of strokes, and at the lower level by the incremental syntax parsers. The stroke pump just has to be fast, something that is most efficiently done by connecting it closely to the parse information. When the view is a pure buffer-view (that is, has no syntax) there is a simple fallback pump defined that just turns each line into a stroke (possibly chopping it up if it's very long), it's not fast, but it's simple, and you can look at it to get a general idea of how it works, if you're curious. It's defined in drei-redisplay.lisp

As an interesting special case, and as hinted at by the title of this post, strokes do not have to cover sub-strings of the buffer. For each stroke we have an associated set of drawing options, and in addition to holding colour and font information, it also contains the function to be used for drawing the stroke. Commonly, this is the default function, which copies a string out of the buffer and prints it with the function draw-text*, but if a sufficient function is provided, any Lisp object can be drawn. Indeed, Fundamental and Lisp syntax recognise non-characters in the buffer, and create single-object strokes, with a special drawing function, for these. This function presents the object, via CLIM, and draws the resulting output record. This means that any Lisp object can at least be displayed as if by print-object, or if a CLIM presentation method has been defined, as pretty much anything. This screenshot demonstrates the feature. Of course, stuffing the buffer full of arbitrary objects can still fail badly in other ways, as not all editing commands know how to deal with it.

In conclusion, if you were previously put off by the poor performance of Climacs, you should give it another try. It's not near Emacs speed yet, but it's quite usable on modern machines, and should improve further in the future (compared to the old redisplay engine, the new can actually be further developed and optimised in a straightforward way). There's also a pretty low barrier-to-entry for hacking, and lots of low-hanging fruit still.