I have been looking at shipping lisp code this week, some parts of this are very cool, some are depressingly still issues.

shipping is essentially pretty easy, you load all the code you want and then at some point in your program you save-and-die, this takes the current state of your program and saves it to disk. You have to name a function that will be used as your 'main' when it is run again. load times are then incredibly fast but the binary itself is massive as it includes everything that was in the program, which for lisp means the entire compiler project loading mechanism etc etc. Luckily there is a compress option that trades of a tiny bit of load time for a much smaller exe, not perfect but ok for now.

The next problem is that my programs are not just lisp, I use C libraries in most my projects (usually at least SDL2, SDL2 Mixer & libSOIL) this makes things interesting as when the lisp binary loads it's going to look for those shared-objects again, and it's going to look in the last place it found them. This is cool when I'm running the project on my machine but not when the binary is on someone else's.

The fix for this is to, just before calling save-and-die, close all the shared-objects, and then make sure I reload them again when the binary starts. This is cool as I can then make local copies of those shared-object files and load them instead.

But this is programming, we don't want to have to go copy those libraries ourselves, our code should do that. Luckily, as there are no strict access modifiers in lisp, we can go looking for this data. The ffi has a method called #'list-foreign-libraires, and we then inspect the paths it used to load the library. We copy the files to the local folder and boom we should be done.

Of course we arent though.. because even with a cross platform language, cross platform coding is a bitch. OSX has this concept called run-path dependent libraries, feel free to go read the details but the long and the short is that we can't just copy the files as they have information baked in that means it will look for dependent libraries outside of our tidy little folder.

GRR. Luckily, I often chat with a wicked smart dude called shinmera, he has been working on the QT integration for common lisp and has done an amazing job. Basically everything I was running into (and plenty more) he had dealt with ages ago. So he has given me some tips on how to use otool and install_name_tool. I've also been in touch with one of the guys who works on the main FFI library for lisp and we agree at least some of these features should make their way into the ffi library.

To experiment with this stuff I started writing shipshape which let's me write (ship-it :project-name) and it will:

• compile my code to a binary
• pull in whatever media I specify in a 'manifest'
• copy over all the C libraries the project needed and do the magic to make them work.

This library works on linux but not OSX for the reasons listed a couple of paragraphs up. However it is my playpen for this topic.

In the process of doing this I had a facepalm moment when I realized I hadn't looked to see what was already available for packaging lisp code. This is so dumb as this language is much older than I am and it's only natural there would have been efforts to do this. And doooown the rabbit-hole I went!

So there was a great cross-implementation version of save-and-die available.. In a library I was already using. I felt so dumb that I hadnt noticed this so I resolved to thoroughly read up on the project. The project in question was ASDF the defacto-standard build system for common lisp, this project has a long history that I won't tell here, but its lineage stretches back to the 60's and it can still be used on the old lisp machines today (madness). Anyhoo I spent a lot of time reading up on this stuff and one area in particular filepaths.

File paths, or pathnames in lisp speak are meant to be an abstraction over paths, however back when lisp was young there we a lot of competing filesystems with no clear winner. Here is an example of what they were facing from one of the guys who worked on the common-lisp spec:

The dominant file systems at the time the design [of Common Lisp] was done were TOPS-10, TENEX, TOPS-20, VAX VMS, AT&T Unix, MIT Multics, MIT ITS, not to mention a bunch of mainframe [OSs]. Some were uppercase only, some mixed, some were case-sensitive but case- translating (like CL). Some had dirs as files, some not. Some had quote chars for funny file chars, some not. Some had wildcards, some didn't. Some had :up in relative pathnames, some didn't. Some had namable root dirs, some didn't. There were file systems with no directories, file systems with non-hierarchical directories, file systems with no file types, file systems with no versions, file systems with no devices, and so on.

Consider how fucking around we do just for windows and unix paths, abstracting the mess above was a heck of a task. Unfortuately the resulting abstraction was massively underspec'd which resulted in very implementation specific behavior. This is turn meant that people had to stick a very limiting subset of what pathnames were meant to do and even then you still could get bitten. ASDF has easily the best thorough fixes for the above, but even it concedes that there are quite simply some things that cannot be resolved satisfactorily.

So of course this makes a programmer's brain start churning and I have been coming up with the most minimal reliable system I can, it's not nearly ready to show yet but the development is happening here. The goal is simply to be able to define and create kinds of path. I will then use it to define unix and ntfs paths. The library will not include anything about the OS, so then I will look at either writing a little library to use this with some existing OS layer or make my own very minimal one.

So yeah, lisp has some issues, but all look fixable. [0]

One thing that I am feeling really good about is that twice in the last week I have sat down to design APIs and it has been totally fluid. I was able to visualize the entire feature before starting and during the implementation very little changed. This is a big personal win for me as I don't often feel this in any language.

Wow, that was way longer than I expected. I'm done now. Seeya!

[0] To be honest there is a library for working with the OS already including (amongst other things) handling paths, but it uses a C compiler at lisp build time to generate a needed library. This feels bad to me as I want my uses to feel like they are just dealing with lisp, so taking them out of that is a no go for me.