Hot Loading Code in Nim

Please note that this was written using v0.16.1 of the Nim compiler and on Ubuntu 16.04 Linux.  I'm currently away from my OS X and Windows machines, so I couldn't test this there.  Sorry about that.

One thing that kind of sucks about compiled languages (like Nim) is that you can't modify code and then see it run (almost) instantaneously.  It's why I usually like using Python to punch out algorithms.  Another benefit of scripting environments is you can also "live code," or "hot load," source into a program while it is already running.  This can also be done in programming languages such as C & C++, though it is not as simple.  With something like Nim, we can actually do this fairly easily; let me show you how.

Credit where credit is due: This work is based off of that of Kashyap CK's.  When I was first learning Nim a few months ago, I found this video on YouTube that shows Nim hot loading in action on an OS X environment.  Here's a link to his repo that contains the code in the video.  I didn't bother taking a look at again until this article about game engine development made it's rounds on Reddit, which mentioned hot loading.  Some other bits of my code were based from this thread on the Nim Forums.

For those of you who have never heard about hot loading before, or don't know the technicalities of how it works, I can explain a few sentences.  It all revolves around the magic of dynamically linked libraries.  That's the .dlls on Windows, the lib*.dylib in OS X, and the lib*.so files for us Linux users.  The basic idea is that you write all of your program's actual logic in something that is compiled to a DLL, and you have a separate program that will check for changes to that DLL file on disk and then reload it's code.

To begin, let's talk about how to make a DLL with Nim.  Don't worry, it's very simple.  The first thing to do is that you'll need to attach a few pragmas to whatever proc you want to put into the DLL.  They are exportc, dynlib, and cdecl.  If you look in the Nim manual we can use the pragma pragma to define our shorthand.  Let's use that instead:

{.pragma: rtl, exportc, dynlib, cdecl.}

Now we can just attach {.rtl.} to any proc in a .nim file and then it will be put in the DLL.  The other thing we need to do is compile our .nim file with the --app:lib flag.  Take this file for example:

Note: I'm using strfmt here to make my output a little more pretty.  You can grab it on nimble using nimble install strfmt.  Or you can do something different if you want; I can't really stop you.

When we run nim c --app:lib game, out will pop either a game.dll, libgame.dylib, or libgame.so file, all dependent upon your OS.  That's all there is too it!

Now loading the DLL is a little more involved.  If you don't have any experience with the Nim FFI that's okay.  In fact, most of that FFI stuff probably won't help you since all of the declarations and pragmas are for compile time anyways.  Instead, we have to write the code that will do all of the DLL loading at runtime and execute it.  The built in dynlib module is what we're going to be using.  It provides a nice platform independent interface to using DLLs.

Here's what we have to do:

1. Define a type that matches the prototype of the update() proc in game.nim
2. Open up the DLL using loadLib()
3. Get the address where update() is stored in the DLL
4. Assign that address to a variable, which has that type that we defined in step 1

And if all of that went well, you should then be able to call update() (from the DLL) in any other executable.  Here is the minimal amount of code to do this:

Yeah, this is a bit more complicated than using the FFI, but the benefit of this is that we can reload the DLL if a change was made to it.  

We aren't live coding yet.  One of the core things we need to do is devise a way of having our runner check for changes and reload them if so.  For this we can use the getFileInfo() proc from the os module.  It returns a Time object which has a field called lastWriteTime.  Using that, it's trivial to figure out a change.

While you could check for changes on the DLL file, I prefer to check for changes on the .nim file itself.  I did it like this:

The benefit here is that you'll never have to manually press that compile button.  You can stay in your code editor forever.  As soon as you save the .nim file, it will attempt to compile it, and if it was successful, the reloading process will happen.  This does come with its downside, such as (right now) there isn't any information on what went wrong with the compile, or that compiling can take a bit of time and make the program hang since there is that waitForExit() call.  I'm sure that some of my readers could figure out some others.

The last thing that we need to do is to slap it all together.   Here is a program that will run for 60 seconds.  It will look for that game.nim file, (re)compile it, (re)load it, and run that update() proc.  I've built it using my stopwatch library.  You can grab it via nimble using nimble install stopwatch.  Since this program also uses threading, you'll need to compile with the --threads:on flag.  The full command is: nim c --threads:on runner.

Go ahead and run the runner program.  You'll see it compile game.nim into the DLL for your platform and then call update().  While it's running, try changing the update() proc in game.nim to do something different.  As long as what you wrote was valid Nim code it will recompile the file and reload it.  Now isn't this pretty nifty.

This is most useful for those of us who want to write video games or any other live/real time applications, though with Nim.  I do want to note that this is not the most robust system to do hot loading.  Given enough time we could make something that is a lot better (i.e. actually check for changes in the code, instead of file write times), but I just wanted to show you how it works and how to do it.

Have fun kids.