Cakelisp cleanup and File Helper work

By Macoy Madson. Published on .

The following post was submitted to Handmade Network and is mirrored here.

There were several cleanup and features added to Cakelisp since the last post:

Precompiled headers

Precompiled headers for comptime compilation is now merged to master. I put in a lot of work to get them working on Windows as well, though I didn't see the performance improvements I was hoping for on that platform.

tokenize-push was rewritten

This generator is the primary way to populate new Token arrays, and is the foundation of all macros in Cakelisp. For example:

This example macro uses tokenize-push to add an array-size function to the output array. This is approximately like the C preprocessor except that tokenize-push works on the token level rather than the individual character level.

My first implementation of tokenize-push was dreadful, but got the job done at the time. It would generate code to create the tokens which looked like this:

As you can see, this calls the tokenizer on a static string passed in as an argument, which means the tokens that were already parsed when the macro definition was originally evaluated must be re-tokenized and allocated.

This rewrite accomplished many goals by solving all the limitations of tokenize-push. The new version reuses the tokens already in memory to output tokens, which gives the following benefits:

It works by saving a pointer to the tokens in the ObjectDefinition. This pointer is later retrieved by the macro at runtime via the definition name and tokens CRC. I decided to use a CRC because it was the only stable identifier I could think of. The following were considered and rejected as identifiers:

While this system is better in every other way, it does require all macros be evaluated each time cakelisp is run in order to have the tokens loaded. This may be an issue if incremental compilation is ever introduced, because now the macro's tokens need to be loaded by some separate system. The old tokenize-push generated source files were completely self-contained.

Here's what the generated code looks like now:

TokenizePushExecute() traverses the tokenize-push token list stored under the CRC 1677447134 on the "array-size" definition and pops expressions from the spliceContext based on the interpretation.

Cakelisp within Cakelisp

Cakelisp can now be executed within another Cakelisp's comptime.

This was a feature I picked up from Jonathan Blow's language, specifically this video, if I recall correctly. Previously, my "RunTests" file, which tests various features of the language, would run Cakelisp in separate subprocesses. This had the drawback that I couldn't easily attach a debugger or run valgrind on individual tests, nor test the overall memory of the system.

Cakelisp is already being exposed to the comptime functions, so I moved some functions around so that creating a new sub-environment and evaluating Cakelisp within that environment was possible.

File Helper

The project I'm working on next using Cakelisp is a file manager application with an interface based on Emacs' find-file. I focused on getting the directory browsing feeling fast first, and planned the next versions. I also did some research on the important file-size visualization techniques which will be File Helper's X-factor.

Unexpected benefits of Cakelisp

I included Cakelisp prominently on my résumé, which got a lot of interest and questions during a recent job interview I did. I think it was a great project to show my skills and interest in programming. It helps my résumé stand out from the rest.