Update: See the Hacker News thread, /r/programming, /r/ProgrammingLanguages, /r/gamedev, and /r/lisp posts for discussions on this article and Cakelisp.

I have been working on a new programming language since the end of August 2020. It is hosted on my site.

If you want to see a working example first, VocalGame.cake is a simple audio looper with Ogre 3D graphics and SDL for windowing, input, and audio. This demo supports code hot-reloading and doesn't require an external build system, only Cakelisp. You can also check Macros.cake, which demonstrates some use-cases for compile-time code.

I figured showing non-trivial examples would be much more interesting. It also proves that Cakelisp is working.

Who is it for?

Cakelisp is built for me first, but it should appeal to fellow programmers who know what they're doing and want to try a more powerful language.

Cakelisp might be for you if you want…

Uncompromised performance
Trust in you, the programmer
Powerful code generation

If any of the things in that list don't make sense to you, or you think you're already getting them in language X, then Cakelisp isn't for you, and that's okay! We have different domains and different problems, so it makes some sense to use different languages and methodologies.

While many languages have these features, few have the combination of all three. Lisp has extremely powerful code generation, but makes serious performance compromises. C is great for performance but can require extremely repetitive code writing to accomplish tasks a simple code generator could handle. Rust is fast (well, apart from compilation, which is very important for iterative development to be productive), but doesn't trust the programmer.

Goals

My goal is to "have my cake and eat it too", meaning all three of these features in one coherent package. Importantly, there isn't one dominating principle in Cakelisp (no Big Idea). I've found that the small things like removing the need for header files, no longer dealing with external build systems, or being able to run Cakelisp files like scripts, end up making a big difference when combined in one package.

It is useful to go over the goals in detail so you can understand my decisions.

Uncompromised performance

This means no garbage collection, no type boxing/unboxing, etc. Fewer abstractions (besides the ones you create) between you and what the computer is actually doing. Idiomatic usage of the language should result in performance comparable with C (in most cases, it should be identical, because it's only a thin layer on C).

Trust in you, the programmer

While languages like Rust offer benefits in terms of security and stability, they cost programmers in terms of productivity. It makes sense to value safety so highly if your code is safety-critical (operating systems, aerospace, automotive, etc.), but it's much less valuable when safety isn't as important (e.g. in games).

In a perfect world all programs would be as robust as space flight software, but in reality, that level of robustness is unnecessary for most programs. It's important to realize that the safety focus is just one way of doing things, not the One True Way or anything.

Powerful code generation

In my opinion, most languages offer far too little opportunity for the programmer to automate the actual writing of code. This power also relates to trusting in the programmer, because gone wild, the code can become incomprehensible.

The company I work for has what I consider to be a state-of-the-art code generator built for the company's use-case: multi-platform MMOs. It's used very effectively on serialization, RPC, automatic commands, monitoring, automatic documentation, and more.

To give more credence to the use of code generation in games, Unreal and Naughty Dog also rely on code generation.

Simplify project setup and management

I want to dramatically reduce time wasted on C++ project set-up and "code logistics". This includes setting up build systems, creating header files, adding and managing new C/C++ 3rd party libraries, and other things of that ilk.

Gain more power!

Every language has limitations. The lack of straightforward, all-powerful code generation was my primary gripe with C++.

For example, automatically creating function and structure bindings using C++ template metaprogramming is very complex. These are two very useful tools in game development: function bindings for commands, scripting languages, and RPC; structure bindings for serialization or game monitors.

I also wanted features like hot-reloading (being able to load new versions of the code without restarting the program/losing runtime state). Cakelisp made it possible to implement hot-reloading entirely in "user-space", thanks to code modification.

"Have my cake and eat it too."

By this I mean lose little-to-nothing on metrics I care about, which include build time, runtime performance, overall complexity, and various other things. I looked into several languages in my LanguageTests experiment and found they all had major drawbacks I couldn't accept.

Unexpected freedom

I did not realize when I started Cakelisp how freeing it felt. All of the sudden, I got to decide what made sense to me, not what made sense to previous language designers.

Freedom in syntax

A simple example is type declarations. In C:

const char* myString = "Blah";

The same variable, in Cakelisp:

(var my-string (* (const char)) "Blah")

In my opinion C type declarations are much harder to parse than my explicit type declarations. You need to work backwards from the name to properly interpret the type. The parentheses do add more typing, but they're more clear, machine-parseable, and can be read naturally (e.g. read left to right "pointer to constant character" vs. C's "constant character pointer", which seems worse in my mind).

My form also handles arrays as part of the type: (var my-array ([] 5 int)) rather than int myArray[5];, another way it is more consistent, readable, and parsable.

I chose to swap the order of name and type because it places more emphasis on the name. A well-written program will convey more useful information in the name than in the type, so it makes sense to me to have it come first for the reader.

Freedom in process

I also found that having an executable which preprocesses my code exactly how I want it opens the door to a huge amount of awesome features:

Compile-time code execution. "Macros" and "Generators" are defined in-line with the rest of your code, making them feel like a natural part of your code. Defining them in-line makes it acceptable to add one-off macros, whereas adding such a thing to an external code generator would quickly become unmaintainable
Build optimization. A recent idea I discovered is automatically creating precompiled headers for large batches of 3rd-party headers. This would be a complex task that would need to be integrated in whatever build system you use, whereas Cakelisp can have it built-in
Other data processing. Compile-time code execution means you can do things like prepare assets, download 3rd-party code, run tests, etc. without having to set up all these additional tools

Influences

I was inspired by Naughty Dog's use of Game Oriented Assembly Lisp, GOOL, and Racket/Scheme (on their modern titles). I've also taken several ideas from Jonathan Blow's talks on Jai.

I'm a software engineer in the game industry. I've been working since July 2015 at a studio that makes cross-platform MMOs. The company has a custom engine written in C (with some C++).

I experimented with other languages before deciding I needed to write my own.

Implementation

Now that my goals are clear, I will show you how I approached achieving them.

Notation

Cakelisp uses an S-expression-style notation. Here is some Cakelisp code from VocalGame:

(defun-local audio-dump-recorded-buffer (output-filename (* (const char))
                                         buffer (* Uint8)
                                         buffer-size int)
  (var dest-file (* FILE) (fopen output-filename "w"))
  (unless dest-file
    (printf "Could not open file to write data\n")
    (return))

  (var i int 0)
  (while (< i buffer-size)
    (fprintf dest-file "%d %d\n" i (at i buffer))
    (incr i))
  (fclose dest-file))

There are a few things you can notice from reading this code:

Types. Cakelisp is strongly- and explicitly-typed. I prefer reading code with explicit types because I can better imagine what the computer is actually doing, and what possibilities I have with each variable
Name-type order. I talked about this in a previous section. I wanted to emphasize the name of a variable for conveying meaning, especially when you may have many variables of the same type
Explicit return. I find I prefer code where return points are made explicit. Lisp will implicitly return the result of the last evaluation
Lisp-y style. The parentheses, plus keywords like unless, defun, var, at, and incr. I matched Lisp only when I didn't have strong opinions for a better notation. I am not trying to create something which is compatible with existing Lisps
C types and function calls. Cakelisp has seamless C interop, which means Cakelisp's "standard library" is C's standard library. No bindings had to be written to use the C types or make the function calls

You can read more Cakelisp code in Gamelib.

You should think of Cakelisp more as "C in S-expressions" rather than "Lisp with C performance". If you know C, you'll have a relatively smooth transition to Cakelisp. If you only know Lisp, you're going to have a rougher time.

Why S-expressions?

When I set out to make Cakelisp, I decided on S-expressions syntax for several reasons:

Parsability. S-expressions shift the burden of creating a syntax tree onto the programmer. This does result on more work for the human, but I value its extremely explicit nature. It also facilitates simpler tokenization, domain-specific-language implementation, and external tool support
Consistency. There are only four types of tokens in Cakelisp: open and close parenthesis, symbol, and string. The consistency is admittedly limiting, so things like paths (myThing->member.member) become much more verbose to type, unfortunately. However, this limitation keeps Cakelisp code parsable, and has an elegant feel that I appreciate

I don't believe there is one notation to rule them all, especially after I've encountered the disadvantages of using S-exprs. I'm still happy with the decision though, and it does give Cakelisp a novel and distinguishing characteristic from the many C-style languages being made.

C++ output

I chose to implement Cakelisp by making it a transpiler, meaning it does not output machine code. It outputs C++ (though I also plan to support pure C output).

This comes with several advantages and disadvantages. The advantages are:

Vastly accelerated implementation. I would be years off from a usable implementation had I implemented a "proper", machine-code-generating compiler. I don't have the time nor the interest to focus on that level yet
Good platform support. Cakelisp has no runtime, which means the code it generates can run wherever C++ can run, which is damn near everywhere. I would have to output to LLVM or something to get this kind of cross-platform support, which would take a lot more time than I was willing to spend
Exit opportunity. If for some reason Cakelisp crashes and burns (or more likely, interest wanes), I can output the C++ one last time and keep that valuable code in a useful form for future projects. C especially is a useful form to increase code longetivity, because every language worth its salt supports calling C code
Seamless interaction with C/C++. An important goal with Cakelisp was to never require tedious binding writing (and avoid buggy binding auto-generation). In order to support this, Cakelisp has a fairly limited knowledge of types and functions. There are definitely some features that are harder to implement because of this (for example, more advanced type systems or memory type annotations), but having natural and complete access to the vast wealth of existing C/C++ code is more valuable to me. This is especially because games are heavily reliant on libraries written in C/C++, and will be for the foreseeable future (this includes consoles, middleware, APIs like OpenGL/DirectX, and direct operating system interaction)
Personal interest. I am more interested in the high-level feel of the language, the interface, rather than the nitty gritty details of assembly generation. Outputting C gives me the ability to stay at a level I'm happy with without losing too much on performance (though C/C++ compilers take significantly longer to compile than I would like, especially after seeing how fast Jai is)

Compile-time code execution

There are multiple ways to influence the final output code, as well as opportunities to do arbitrary code execution.

Cakelisp automatically determines dependencies for compile-time code and lazily builds their definitions. Note that Cakelisp does not have an interpreter - all code is compiled to machine code before execution (which admittedly makes the first build longer than if it were interpreted).

Macros

When I say macros, I mean Lisp-style macros, not C/C++ preprocessor macros. The difference is that Lisp-style macros are actual functions which can perform arbitrary computation, whereas preprocessor macros are essentially a template system that can only insert text.

Macros take an array of Cakelisp tokens as input and can output arbitrary tokens in place of the macro's invocation.

Macros facilitate creating domain-specific languages, where the entirety of the syntax is defined by the programmer within the language. They can also be used for simple substitution like C preprocessor macros, with the option to add validation of scope or variable arguments.

Generators

Generators take Cakelisp code and output C/C++ text.

While I have not implemented all features of C or C++, generators allow the programmer to access more features of those languages without having to modify Cakelisp itself.

Generators form the foundation of Cakelisp. Via option --list-built-ins, the user can see all built-in generators. Crucially, the generator defgenerator facilitates adding new generators.

Hooks

Hooks at various stages of compilation and building allow the programmer to greatly influence Cakelisp's behavior. Hooks are lists of functions, so multiple hooks can be added.

For example, the pre-build hook could be set to check 3rd-party libraries and build them if necessary. A post-build hook could copy or generate configuration files which the executable requires to run, or perform system install steps.

I also plan on providing a way to completely override the build stage, which will be useful if you need to output multiple different executables, libraries, etc. One could use this hook to more easily generate configuration files for external build systems, if that integration is important.

Code modification

Code generation creates code; modification changes existing code. This opens the door to unique features which would be unreasonable or impossible to do with generation alone. See this Jai demonstration for examples of how it can be used.

Using the correct hook, arbitrary code modification can be performed. I have used this feature twice so far to good effect:

AutoTest.cake will search for functions prefixed test-- and call them from a main() function. This 68-line program provides a simple way to do high-level tests of all your modules, with minimal intrusion
Hot-reloading automatically converts global and module-local variables to use the heap, then automatically inserts dereferences whenever they are accessed. This allows the programmer to make changes to functions and dynamically reload them without losing state. This feature is excellent for facilitating iterative, low-latency development. Add that to the build system, and you have a fast, simple, and convenient development environment

Build system

I originally thought I would rely on 3rd-party build systems to assemble Cakelisp projects. I found that doing away with manual header files and using a Python-inspired module system instead made that unfeasible. Cakelisp now has a build system built in. This ended up having some huge advantages:

Strong, simple integration with 3rd-party libraries

Cakelisp modules can declare which libraries they link to, where they are, and even build arbitrary C/C++ files. These declarations are all in-line with the rest of the module code.

This makes it trivial to include new 3rd party modules in a project, because the build configuration stays within the module. Simply e.g. (import "SDL.cake") and the build system will handle adding all the compile and link flags to make it happen.

With a multi-project perspective, this prevents the complexity and tedium of copying build flags from spreading outside the module itself. The more projects you make, the more it pays off.

Opportunities for build optimization

The build system being closely tied to the language it is building means you can add hints and build configurations inline.

For example, I plan to make an opt-in precompiled headers feature which can drastically speed up compiling files with large amounts of included headers. Simply add the &precompile tag to the list of includes, and Cakelisp will handle bucketing them all into a single header and will keep that header up-to-date.

Debuggability

You can use the same debugger for compile-time, runtime, and build system debugging (any C++ debugger), in case you run into tricky build configuration problems.

Usability

By including the build system, the user interface becomes simpler. Cakelisp has a --execute flag which will build the given .cake files and then execute the binary which is output. This is a simple add which makes running C/C++-quality programs as easy as scripts, while still retaining performance and keeping them up-to-date.

I've also made the decision that command-line arguments should never change the behavior of the compiler itself. They may only change verbosity, perform additional actions like --execute, or facilitate clean builds (--ignore-cache). This constraint causes configurations which are important to successfully build a project to reside within the project's code. The use of external build systems and .sh scripts diminishes when all the options are "built-in".

This decision also encourages creating composable build configurations, where a using a different build configuration involves importing a different .cake module, or collection of modules. Build configuration labels are composed together, e.g.:

cakelisp Config_Debug.cake Config_MakeHotReloadable.cake MyGame.cake

…would build with a configuration label Debug-HotReloadable. Making new configurations is a one-line change:

(add-build-config-label "HotReloadable")

This allows you to easily do complex configurations, or try things like A/B tests without having to manage all the new build artifacts.

Programmability

Sufficiently complex projects require full programming languages to build. The integrated build system enables the programmer to do whatever they need to build without requiring the programmer to learn a different, often inferior language just for build configuration. Note that arbitrary code execution during compile/build time means you can do much more than just build the project, if you so desire (e.g. push artifacts to a server, download code, tweet, whatever).

To see the build system in practice, see Gamelib.

Potential

Cakelisp's features open the door to many exciting possibilities.

Naughty Dog co-founder Andy Gavin talks about how C is poor when implementing game object state machines, which is one reason why he made GOOL. I would like to explore ideas like this - new ways of more efficiently implementing games and other programs.

For example, I have a hunch that retained-mode UI could have an immediate-mode feel like Dear ImGui (which is quickly gaining popularity, especially in game development) when combined with a domain-specific language. It wouldn't be separate from the actual UI implementation, like UI systems which require models and bindings to update. At compile-time, a custom macro would discover which UI widgets need to be created and generate the code appropriately.

Data-oriented design is compelling, but often requires jumping through hoops during implementation. I would like to explore ways to implement things that are cache-friendly while still being programmer-friendly (see e.g. Jai's SoA vs. AoS).

Computers are increasingly getting more cores, but writing multi-threaded applications is as hard as ever. Gameplay programming is especially challenging because of how many systems a single game behavior can involve (animation, audio, AI, entity-entity interaction, spatial queries, physics…). Similar to GOOL's defgstate from the above Andy Gavin article, a more human-friendly interface to multithreaded development would be compelling to work towards.

Future

There are of course new features to be implemented and new lessons to be learned.

I don't intend Cakelisp to be a "toy language". I built it to replace my usage of C++ for my projects going forwards.

In terms of adoption, I don't expect everyone to love Cakelisp and loudly evangelize it. I have a relatively specific use-case that Cakelisp is perfect for me for. I don't believe it will be perfect for a lot of other people.

My main hope in announcing it is to expose the ideas I've found really valuable that are core to Cakelisp:

Full-power code generation
Code modification
Better, simpler project dependency management
Performance doesn't have to be compromised when adding more power
Optimize for the developer's time. Trust them to decide how to best create their programs, and get out of their way

Zig and Jai are promising languages in a similar domain, but I think there should be even more languages which explore the performance-and-programmer-trust domain.

More info

For more information on Cakelisp, and comparisons with other similar languages, see Cakelisp on my site.

For a project which uses Cakelisp, see Gamelib. You'll see that Cakelisp is already quite usable, and includes almost all of the critical, unique features. There are many more Cakelisp projects here.

Email macoy [at] macoy [dot] me if you have questions or want to chat about Cakelisp.

Cakelisp: a programming language for games