Don't get me wrong, I like my engine. But at times I get lost in the details, and I yearn for simpler times.

An idea that I've had last year, was to port my engine to the Nintendo Wii. This appears to be a brainfart, but the more I think about it, the more serious I am considering it.

My current biggest headache is the fact that I have no system requirements. You see, if my engine were ever to produce a game, I would like to have it run on as many machines as possible. No matter how garbage your hardware, my game should be able to run on that.

I won't be able to go as low as the original Doom. It is a masterpiece of programming, and it smartly took a lot of shortcuts to be able to run on pretty much everything that has a processor. But exactly because of these shortcuts, the graphics are too low fidelity for what I want to do. I assume everyone who would like to play my game has a machine that is capable to render much more complicated scenes.

Anyway, I still want to depend on as little system requirements as possible. If that sounds vague, well, that is because it is. I repeat myself: I have no system requirements. I have no idea on what hardware I need, to run my future game. And this causes me mental pain.

I could lean on Wirth's law, but where's the fun in that?

I am reading up on cool rendering techniques all the time. Expensive ones, that create high fidelity with little or no required consideration for asset creation. But many new cool features require compute shaders. If your GPU can't run these, then your out of luck. Virtually all modern GPUs support compute shaders, but embedded systems often don't. And I simply don't have the foresight to predict how platforms are going to look like in 10 or more years. Especially with AI currently ruining everything.

At this moment, I am betting everything on Vulkan. Its main criticism is its verbosity. And even though I like the control, the complexity is starting to drain on me. It is so much work to get something to render. And this is motivation enough for people to leave. If Vulkan gets too unpopular, support will end and future hardware won't run anything that uses it. OpenGL is still kicking, but it is approaching its end more and more, and I fear Vulkan will face the same fate.

It's a similar story with the OS. You can completely forget Apple and Android, as they require binaries to be built for specific hardware versions. Longevity is not achievable there. Linux isn't better, as its OS protocols are in constant flux, meaning any binary you distribute won't work on different/newer distros. Windows is stable, and Microsoft has put in great effort into backwards compatibility. However, considering how much slop that company has produced in the last year, I have no trust that Windows will keep being stable.

With such unstable platforms, and me having no idea about system requirements, I am seriously doubting whether PC should be my primary target.

Enter console gaming. Consoles have fixed hardware. Their chipset is known and well defined. As long as hardware of a console exists, code for it will keep runnning. People aren't ditching consoles, like they are changing parts in a PC. People even go to great efforts to preserve them. Also, emulators exist, making console games playable everywhere where an emulator runs. And people are building new hardware, specifically to physically emulate the original stuff.

So why exactly the Wii? Simply put: Nostalgia. I have very fond memories on playing on the Wii. And I think it's capable enough for what I have in mind.

I've been following Kaze Emanuar and his rewrite of Super Mario 64. He is a prime example how non-dead development for the Nintendo 64 is. I don't see why the Wiis story would be different. This is especially true, since the Wii sold more units than the Nintendo 64.

But, porting to the Wii comes with great drawbacks.

First, I need to rewrite my entire engine, or most of it. This would be a lot of work, but a part of me thinks this would be manageable. Much of the work has been gone in learning game engine architecture, not writing it. By that I mean, if I were to start anew, reaching the point I am standing now will take significantly less time, than all the time I have already spend to get here.

The second and probably bigger drawback is, that I would need to switch to C. A Rust port for Wii libraries exists, but over a weekend I only managed to run a Hello World triangle on the Dolphin Emulator, not on original hardware. The error surely is on my side somewhere, but the homebrew compiler just worked out of the box.

This whole situation is kind of ironic. I previously glazed Rust and talked about how I don't understand why someone would rewrite their engine from scratch. And now I find myself exactly in the situation where I am considering to drop Rust and rewrite everything. Fucking hypocrite.

Now, I haven't written C since university, and I am definitely looking at it through rose tinted glasses. Ultimately I think Rust is the better language, because of it's very strict typing and compiler guarantees. But at the very least I imagine C to be much better than C++. I hate C++. I dread it everytime I have to use it.

In any case, I am standing at a crossroad. Continuing with Rust would be the easy choice. I already have a somewhat working engine, even if it isn't feature complete. Working on that is the way of least resistance, and it wouldn't require me to throw my entire progress into the garbage bin. However, having a game run on the Wii would be hella cool.

These thoughts sprung into my head recently, probably because I haven't written a single line of code this year. This gives me time to think. Actually, I have been learning Blender, and I started to make actual assets for my game. Here's a character I am currently modelling:

The closer I get to being feature complete, the less excuses I have to finally start working on assets. For example, I previously postponed the game object system almost an entire year. I did so, because of the fear that with it completed, I had to actually make something presentable. Working on a game engine is a very safe space to be in mentally. I can have a superiority complex, without actually producing a sellable product.

🤓 "Since I am making an engine and you don't, I am automatically better than you" \s

But I did actually finish the game object system, and assets are still waiting to be made. Turns out there is much more to a working game than just game objects.

My current roadmap involves finishing to model the character in Blender, including rigging, UV mapping and texturing. Then I'd like to implement materials, followed by bones, skinned meshes and animations. Somewhere I'd also like to work on lighting and shadows. And the terrain renderer should finally be implemented.

This roadmap is absolutley subject to change. (Which it already has as I am working through the C book…) And if I am going to switch to Wii, then it will definitely delay everything else. But once the character is done, we'll see if I actually try to switch, or if I'm too big of a pussy to try.

Wish me luck!

Overall, the progress feels fractal-like.

Making a custom game is easy actually! You just need two steps:

First, make the engine!
Second, make the game!

So you get to work and consider the structure of your main loop. You work out how startup and shutdown happens. You are feeling fancy and make a restart mechanism. Nice. The mainloop runs 3 things: input, logic, output. And you gotta find out how these 3 talk to each other.

Okay. Let’s do one by one.

In the input you figure out how to handle HIDs and how to remap them to usable actions. You even implement some custom rebind system, that allows the user to rebind controls if they choose to do so. Then you might also think about handling multiple controllers. You might also think to implement an input buffer, for combos and such. Maybe even consider rollback code. But that’s a lot of work, and since you are not planning to make anything multiplayer, let alone competitive multiplayer, any further consideration is out of scope and wont be touched whatsoever.

Then you do logic. You want gameobjects. People talk about ECS a lot. It’s where the money is at. But to this day you don’t understand it and you think it’s a buzzword. Same with anything procedural. Man, the internet loves the word “procedural”. Here’s a fun game: Spend some time in the indiedev-wannabe circles and see how far you come before encountering the word.

Anyway, you set your requirements. You want your gameobjects to be flexible, extensible and performant. You go with flat arrays internally, for cache locality. Ints as ids, for the fastest lookup possible. But this is were it bites you, because of course you are one of the weird kids and chose to build the engine in Rust. And Rust really doesn’t like what concept you have cooked up. But after much pain you have a working prototype, which (hopefully) is as expansible as you think it is. But you won’t test it now. You’ll maybe get to it in 2 years.

Oh, what about output? Ah, that’s were you draw them pixels. You already spent the better part of a year working on this thing and have yet to display something. Because you have a phobia of external libraries, you are going to implement all the math yourself. Vectors, matrices and the beloved quaternions. Especially because you did not choose C++, and all the Rust libraries today are ass anyway, you convince yourself that writing it yourself is better. You make a flying camera, a single cube, that’ll do it.

You want debugging, so you implement Dear ImGUI. A bit hacked together. Can’t use any existing bindings, because your renderer is custom. So the ImGUI implementation has to be custom as well. Anyway, let’s draw some gizmos. Like primitive lines, that’ll make debugging a later game very easy. Also write a gizmo for 3d text. You say to yourself this is absolutely necessary. The renderer you have written a year ago is ass, so it needs to be rewritten. No biggy, gives you the opportunity to use a better Vulkan wrapper. You hate the previous Vulkan wrapper you used, surely Vulkan wrapper 2.0 is better. Once the new wrapper is implemented, you get back to the gizmos, and implement them. They are ugly as hell, but they do their job. You imagine yourself using that in the upcoming game you will write. But we will see about that in 2 years.

Also write a benchmarking tool. It produces HTML. It looks beautiful. This will definitely be helpful. In 2 years.

But then you realize that all this debugging takes a lot of performance. A shipped game is not going to require all of this, and stripping it out will massively improve performance. So you look into conditional compiling and guard every debug feature behind some preprocessor. It works, and you are proud of how many frames you get in an empty scene.

Anyway, time to display some meshes. Fuck, we need to load these in first. So you roll up your sleeves and write an asset system. Turns out all common file formats suck and none are optimized for fast loading. So, you make your own serialization and tinker on an entire asset pipeline, which is capable to convert editor assets to fast loading runtime assets.

Also write an asset compiler. Because you see, IO is expensive, and having a single handle to a file is faster. At least that’s what is claimed, you have yet to do benchmarks. Damn, if only you had a benchmarking tool to test that. Oh well. The compiled thing is a single huge file that holds everything. It has a lookup! Which makes indexing very very fast. Flat arrays for the win.

After some time later you have a barely working asset system. So lets get back to meshes and load them. What kind of format are you going to implement anyway? glTF. That seems to be the standard. So you read the glTF spec from start to finish and write an importer that converts glTF to your mesh format. It’s like a year since you first wanted to render meshes, but now you finally have all the systems to do so. Hurray!

Eventually you say to yourself that you should start to make the game, no? But what’s missing? Oh, not much:

• any gameplay logic
• level loading
• character movement controllers
• skinned meshes
• skeletons
• animation systems
• lighting (bruh)
• sound

Wait, sound? You produced songs in the past! You literally describe yourself as a music enthusiast and you have yet to do anything sound related? Well… it isn’t important for the prototype, so it’s shelved. It’s been shelved for so long actually, that you are starting to wonder if your musical ability took a hit. Anyway, that’s not important for now, so let’s start making the game.

But oh no, you want the character to stand on something. We gotta make the floor first. Prototype is shelved, it’s time for terrain generation baby! This can’t be that difficult! 4 months later you finally have a terrain generator finished, and have yet to render it. God dammit. Time to finally implement materials and lighting. Why now? Because if you were to write the terrain renderer now, you will have to rewrite it again once your entire render pipeline has to be adapted for materials. So the terrain has to wait.

You look back at the goal you set this year. You wanted to have a playable prototype and yet you spent half of the year on a fancy picture generator.

Anyway, how does the engine look like?

Sigh…

Looking at the screenshot fills you with dread. It encapsulates the last 3 years of your life. How can so much work look like nothing? You went the walk, you know all the shit that is required to display that damn image above. Getting to this point is not easy. And yet, it’s demoralizing. To be fair, you have a fulltime job and thus not much freetime. But still.

You check online, to see how others are doing. People are posting their game engines, and most often you just scoff at them. There exist only 3 emotions:

1. The poster did not post their source code. You feel apathy and disrespect towards them.
2. The poster did post their source code, but you quickly realize it’s just an ECS library glued to a renderer library. Sometimes it even uses ImGUI. How nice. You clone the repo, remove all third party libraries, run it through a sloc counter and reveal that the repo has at most 2000 lines of code. Your engine has 39k, many of which have been beaten to death, rewritten multiple times, so you are obviously better. Their screenshots look nice, better than yours. But actually you are superior, because technically they haven’t written their “engine”.
3. The poster actually has something to show, and this is their thousandth update. You give it an upvote, nod in approval, but spend no longer than 10 seconds before going back to your own shit.

The progress really feels like a fractal.

Every tangent, no matter how small it seems, is a major rabbit hole, with tangents on its own. I have written the paragraphs above in the you-narrator, but it’s basically me. I’ve described my journey in a not so chronological order. At every step in the journey, you think you just need to take one step further, only to reveal that between your current position and the step you wanted to take are 100 more steps that you were oblivious to previously.

After all that work I have many loosely connected systems, which by some sort of miracle have not yet collapsed under their own weight. This is either a testament to my software architecture skills, or luck. Anyway, everything still needs to be stresstested, which I still haven’t done, because there is always a new system to write.

There is one somewhat common kind of post you’ll find in game engine dev circles: People rewriting their entire engine. I just can’t relate to that. There is too much stuff going on. Besides, rewriting your code is one of the worst things you can do as a software developer (if we are considering that you are not doing some crypto/AI bullshit, scamming and actually hurting people). Like, sure, a rewrite can make things easier when implementing new features. For example when I am finally getting to implement skinned meshes, materials, lighting, MDI, then my scene renderer is going to face a lot of refactoring. But you know what doesn’t have to change? My gizmo and ImGUI renderers. My startup-, shutdown- and restart logic does not need to be touched. My gameobjects will function exactly the same.

Often I’ve heard the saying that this kind of work is a marathon, not a sprint. However, I don’t agree with that idea. More fitting is like that meme of the guy who climbs a mountain, only to realize it goes up further:

Each dip is a finished system. And reaching one is what I think enlightenment feels like. Months of constant thoughts about a specific problem. All the branching ideas I’ve tried, all the decisions I’ve made. And when the system is done, the thoughts disappear. When before my head was full of voices, now it’s dead silent. It’s these moments where there’s not a single thought in my mind. I am literally thinking of nothing. It is bliss.

But the journey is not over, and after a few days the climb continues. You take the courage, touch the boulder again and start pushing. A new system is started or work on a current one proceeds, and my head begins to fill with thoughts again.

One last thought: “I have to write everything on my own” is a delusional idea. Let’s get back down to reality. This year at SIGGRAPH 2025 I met someone who is writing a Vulkan driver. I thought I was low level, and yet this person does me one better. But after much discussion, even they had to admit that you truly can’t write everything on your own.

You are working with a programming language and its tooling. Someone had to make that. You are working with a text editor and operating system. And you are working with a computer, someone had to build that. And I don’t mean stick the parts together. No, I mean someone had to make the chips on these circuit boards. But even if you are doing a Ben Eater and literally build your own computer, you still have to admit one crucial detail: You are building on top of the knowledge of the ones before you. You are not going to reinvent the graphics pipeline. You will learn how it’s done and then give your own spin on it.

We are really standing on the shoulders of giants.

I am not saying that as an argument against building an engine. My journey gave me understandings and proficiencies most other developers don’t have. But it’s an anchor in reality that one should not lose.

I wanted to write this post for quite some time now, but I never wrote my thoughts down. Then my coworker specifically asked me about serialization. So you can thank him for the existence of this post. Thank you Mark :)

The problem

What are we even trying to do? What even is a binary format? Simply put, a binary format is an array of bytes. Nothing more, nothing less. Classes, structs and objects cannot be saved as files. Neither can they be sent over a network. Why that is, is complicated and out of scope of this blog post. (They keyword here is ABI.) Regardless, what is important is that a byte array CAN be saved or sent. So our challenge will be to find a solution, that translates our object into a byte array and back.

Shouldn't be too difficult.

What's a stream?

Before we start with any kind of serialization, we should be talking about streams. We aren't going to modify the byte array directly. That would be cumbersome and difficult. Instead, we are going to use a container known as a "stream". It will make our life so much easier.

A stream consists of two things: Data and a cursor. The data can be anything: A file, a websocket, a port, whatever. But most importantly for us, the data can be a byte array. The cursor is simply an integer, and it represents the current position in the data. Think of it like the head of a Turing machine. Or the stylus of a record player. I'll use "cursor" and "position" interchangeable from here on. They are pretty much the same concept.

I'll be using the visualization above for all further examples. The boxes represent the data, the byte array, where each cell is a single byte. The red arrow is the cursor, the current position of the stream.

A stream comes with 3 functions: Seek, Read and Write. What these do and how they work in detail, we'll see in a moment. Streams come in many different flavors. Some are read-only, meaning they don't implement Write. Some are write-only, meaning they don't implement Read. For our purposes, we will be using a "MemoryStream". It stores bytes, and it implements all three methods.

Because we will be using a stream in pretty much every example, I want you to understand how a stream works. So we are going to implement one from scratch. It will be barebones, naive and straight forward, but for our use case it will suffice.

All code examples will be in C#, because C# is a language, which Mark can read. If you dear reader have never worked with C# before, then that is fine. We won't be using complicated syntax, so I hope you can follow along.

Let's start by defining our stream:

I prefixed the stream with "Ris", because I am weird and most of the types in my public code are prefixed with the first syllable of my name. It also helps to distinguish my types from others. Anyway, as discussed, the stream consists of two parts: The data and the cursor or position. I also implemented two utility methods, that allow us to create a stream from existing bytes, or to retrieve the bytes once we are done modifying the stream.

We have three more methods to implement. Let's do this!

Seek

Seek does two things: It modifies the position, and then it returns the position after it has been modified. Modifying the position works by adding an offset to one of three locations in the stream:

With this, we can implement the Seek method:

There are a few things to note here. First, notice how the switch treats SeekFrom differently. When applying an offset to SeekFrom.Begin, the offset is pretty much where you want the position to be. When using SeekFrom.Current, we are simply adding the offset to the current position. When using SeekFrom.End, we are adding the offset from the end of our currently held data.

Using positive and negative offsets, we can walk back and forth in the stream. Due to the clamp however, negative offsets on SeekFrom.Begin and positive offsets on SeekFrom.End have no effect.

Why are we clamping anyway? Well, the allowed range for the position is 0 to _data.Length. The bounds are inclusive. Everything outside that range is invalid. We want the stream to be valid in all cases, and thus we need to prevent the position from falling out of the valid range.

Hold on a sec, doesn't _position == _data.Count mean that our cursor is now pointing outside of our array?

Yes. In this case it does indeed point outside the array. But this is fully intentional. It allows us to get the current length of the stream by calling Seek(0, SeekFrom.End). It also makes reading and writing straight forward. And it also means that when _data is empty, _position being 0 is a valid state.

Clamping is one way to keep the stream valid. Another way would be to throw exceptions, that's what System.IO.MemoryStream does. But I don't like exceptions, so I clamp.

Most commonly, seeking is used like this:

• Seek(0, SeekFrom.End) to get to the end to the stream and/or to get its length,
• Seek(position, SeekFrom.Begin) to set the position directly, and
• Seek(0, SeekFrom.Current) to get the current position of the cursor

Read

Now let's see how to read from the stream. As a parameter, Read takes an int that specifies how many bytes should be read. It then reads that many bytes, advances the position, and returns the bytes.

Here's the implementation:

First, we are clamping count. Then we copy the requested number of bytes into a new array. At last, but not least, we update the position and return the read bytes.

Copy operations can be a bit intimidating for new programmers. I hope the example below clarifies things a bit better:

Let's talk about the clamp for a moment. It is there specifically to prevent client code to read more bytes than are actually left in the stream. Reading more bytes than are present is an invalid operation, and we must decide what to do in such a case. Clamping like this provides valid default behavior, but requires client code to validate, if as many bytes were read as expected. Another viable approach would be to throw an exception. The exception comes with the benefit, that no operation will take place. My implementation above will always allocate an array and copy into it, regardless if the operation is sensical or not. But clamping has the benefit that you can read all remaining bytes by calling Read(int.Max) on our stream.

You may have also noticed that I am returning an array. You may find that common stream implementations pass an array as a parameter instead, expecting the read method to fill it. This is done to prevent unnecessary allocations. My implementation will allocate an array for every single read operation that you will make. But the tradeoff is that client code will be significantly simpler. If you want, as homework, you can try to rewrite the read method such that it will take an array as a parameter. Its signature should look like this:

The returned int is to indicate how many bytes were actually read. Whether you opt for a clamped- or exception-based error behavior (to prevent too many bytes being read), even a successful read may not fill your entire array. So make sure to return an int to notify the client code.

Before we continue, I again want to stress that _position == _data.Count is a valid state, in which our stream can find itself in. Assume for a moment that it wasn't, how would you determine if there are bytes left to read?

If we allow _position == _data.Count to be valid, then things become very easy:

Write

Two of three methods implemented. Now let's look at Write. Write puts bytes into the stream at the position of the cursor, and then advances the cursor.

Notice that writing can both grow the stream, as well as overwrite existing bytes.

I want to stress one last time that _position == _data.Count is completely valid. If the cursor is at the very last position of the stream, then being outside the array allows us to append to the stream without overwriting anything:

This also allows appending onto an empty stream, without compensating for any edge cases when implementing it. _position == _data.Count is indeed a very useful hack and very useful behavior a stream can have.

Enough yapping, let's see the implementation:

First, we check whether our stream has enough bytes left to hold the data. If not, we must grow it. This is done by allocating a new array, which is big enough to hold all data, and then copying the old array into the new one. Then, we can copy the actual value into our data array.

Again, copy operations may intimidate the new programmer, so let's visualize them too. Here's an example for resizing the array:

And here is an example for copying the value into the array:

And that's it. This is everything that we are going need. A full implementation can be found here: LINK

You may have noticed that I did not implement the base class. This is on purpose. I am targeting the lowest denominator of readers, and System.IO.Stream is a tad too complicated for my taste. Also, we don't need the entire thing for the following chapters. The methods we have are enough for our use case.

If you are feeling confident in your ability, I leave the implementation of System.IO.Stream as homework for you. Implementing the base class comes with the obvious benefit, that you can use all utility methods for streams on your stream as well.

I am going to use RisMemoryStream for the following chapters. You are allowed to use your stream from your homework. You are also allowed to use System.IO.MemoryStream from the standard library, which has a much more sophisticated implementation, with all the bells and whistles.

Serialization: Baby Steps

Let me introduce you to a new static class:

You may be asking: Is it really necessary to wrap these methods? No, it isn't necessary. But I like to have a uniform API. By wrapping them, regardless if we are reading/writing raw bytes or values like ints and floats, the calls will always look very similar: RisIO.ReadX(stream) or RisIO.WriteX(stream, value). (This will be especially true when we are modifying them a bit later.)

Notice however, that I am throwing an exception in the read method. That is because Read represents a checked operation, meaning it must produce an error if things don't go as expected. Unfortunately, error handling is difficult. And C# is not Rust. So despite me not being the biggest fan of exceptions, I will be falling back to them when the situation calls for it. ReadUnchecked does no checks, and thus client code is at the mercy of whatever error handling the underlying stream uses. Read is a safe alternative. Also notice that Read does not catch the exception of the underlying stream. That is intentional, because exception is exception, regardless of who threw it. In any case, the client is notified on an error, which is good enough for us.

For you C++ nerds out there, yes, this isn't strongly exception safe. If you know what I mean by this, then you are clearly above my target audience. I leave it as a challenge for the reader to argument why it doesn't fulfill the strong exception safety guarantee.

I think strong exception safety isn't of importance here, as I consider any exception a failed deserialization. I like my programs to burn and crash on a (truly) unexpected error. We will generate streams on the fly, and none should outlive a serialization operation. If this bothers you, you can write strong exception safe implementations for this and all the coming methods. I will stick with simpler implementations, even if they are less rigorous.

Integers

Now let's serialize some data! Took us quite a while.

Since classes and structs are made from smaller types, let's see how these smaller types are serialized. Starting with the humble integer. On most platforms, including C#, an int is made up of 32 bits or 4 bytes. So all we need to do is to retrieve these 4 bytes and then put them into our read/write methods. That should be easy. Right?

Jump scare: Endianness

Immediately, even before we serialize our first thing, we are met with an obstacle. Unfortunately, this won't be our last 😓

So what is endianness? Let me answer that question with an example and another question: The int 730797131 is comprised of the bytes 43, 143, 20 and 75. Now, in which order should we store it?

Or?

As it turns out, both approaches are valid and reasonable. So now we are left to choose. Of course we could also scramble them, but that assumes that we know in which order the unscrambled bytes should be. You see, this is a CPU problem, and endianness determines in what byte order different CPUs represent a number. We must solve this. Chances are, that our binary format is stored as a file or sent as a package over the internet, read by a different computer that has a different endianness.

Since there are two orders, there are two types of endianness: Big-endian and little-endian. To figure out which is which, we need to look at the first and the last byte of our number. Sticking with the example 730797131, then 43 is the "most significant byte" (MSB) and 75 is the "least significant byte" (LSB). 43 is the MSB, because it has huge effect on the number. On the other hand, 75 is the LSB, because it has little effect. Look at what happens to 730797131 when you add 1 to either the MSB or LSB:

Adding 1 to the LSB only adds 1 to the whole number. Adding 1 to the MSB however adds 2^24, which equals 16777216!

Here's why that's important: If you store the MSB first, you are using big-endian. If you are storing the LSB first, then you are using little-endian. A good way to remember this: If you are storing the big byte first, you are using big-endian.

: ^)

What you choose is up to you. But whatever you do, stay consistent. Most modern mainstream CPUs use little-endian. But there exist exceptions, notably embedded systems. Older hardware might also use big-endian.

Let's see how this actually looks like in code:

Whenever you are dealing with raw bytes, you must keep endianness in mind. If you have ever read a spec before, which deals with raw bytes, then you know that they always mention what endianness their format uses. Well, at least every spec I've ever read mentioned it. Also, you can directly toggle endianness in most hex editors.

Hex editors

Speaking of hex editors, when writing your own binary format, you should probably start using one. To debug your binary format, you can simply write your serialized bytes into a file and then open that file with the hex editor of your choice. On windows I am using HxD, and on Linux I am using Okteta.

Some hex editors have more features, some less, but most are split into 3 views. On the left you can view the actual bytes in hexadecimal. Also there are the offsets displayed in hexadecimal as well, to ease navigating large files and locating specific bytes at certain positions.

Right next to it, there is a decoded text interpretation. Most of the time, this is some form of ASCII. This decoding is mostly useful when your format uses text. If you are not storing text, the decoded text will be scrambled nonsense. But often the text can help you to spot patterns.

I assume that's why little-endian is practically the standard for modern CPUs. Small integers have their MSB set to 0. If you use little-endian, then it's easy to spot at a glance, where that integer begins. But that is pure conjecture on my part.

On the right side of the hex editor, you will usually find an inspector, which provides gazillions of interpretations of the currently selected byte. It may read multiple bytes to come to each interpretation. After all, the hex editor doesn't know how your binary format works, so it displays all the possibilities. You can toggle endianness and see how the values in the inspector change.

Floats and other numbers

Endianness was quite the big jump scare. But once we are aware of it, floats are very straight forward:

As you can see, the implementation is very similar to that of the integer. I won't list the implementations for all the other numeric types here. If you need to read/write single bytes, chars, doubles, shorts or longs, whether they are signed or not, I am sure you can figure out the correct implementation on your own.

Or you can ask an LLM to do that for you. I heard they are the hot craze right now. Not that I would endorse such steaming shit...

Enums

As far as I am aware, most programming languages use integers for enums internally, including C#. A notable exception is Rust, where an enum is more akin to a union type, which we will come back to later.

An enum value is just a tag to the compiler, which improves readability for you the programmer. But behind the scenes it's just a number. As such, writing an enum is as easy as converting it to an int and writing that:

Since we are using WriteInt to write the number, all endianness problems are already taken care of <3

Reading is done like so:

The code above uses the most advanced syntax of all the examples in this post, because it makes use of generics. If you are not familiar with generics, think of them like this: This method accepts any enum. Now instead of using the specific enum type, T represents every enum possible. No matter what enum you read from this method, the method will resolve it correctly.

Don't worry, we'll walk through this example: First, we read an int. Then we check if the enum is actually defined for that int. If it isn't, we throw an exception. If it is defined, we cast it to the enum and return it.

Instead of throwing an exception, you could return a default value. But I prefer the exception here, because a default value would obfuscate invalid data.

It should be noted, that this approach only works for enums that have unique values. Consider the following enum:

In this case, when writing Fruit.MyFavorite into the stream, you might read Fruite.Orange out of it. This may be desirable, or it may not. In case it is not desirable, you need to read and write strings instead of ints. How to serialize a string we'll see later.

Booleans

Now let's take a look at booleans. A bool can only have 2 values! true and false! How hard can that be? This will be easy!

Narrator: It will not be easy.

New programmers are often surprised to hear that booleans are stored as 1 or more bytes in memory, instead of 1 bit. If you didn't know this, now you know :) The reason why is, because the CPU cannot access single bits. A CPU operates in "words" which are 1 or multiple bytes long.

With our serialization, we face a similar reality: With our current API, we can only serialize bytes, not bits. So whenever we want to save a bool, we waste at least 7 bits of space.

This may seem wasteful, but from my experience, this is a problem which you usually don't want to solve. As it stands, our serialization is "byte aligned", which means all chunks of information start and end with a byte. Once you introduce variable bitlength into your serialization, you face alignment issues. And I cannot stress enough that this is a major pain in the ass, and it really isn't worth solving.

To more effectively use all the space, there are 2 valid approaches: The first one is compression. We will talk about that at the very end of this post. The second is bitfields.

A byte consists of 8 bits. Since a bool only requires 1 bit, it directly follows, that a single byte can hold 8 bools. An int has 32 bits, so it can hold 32 bools! A bitfield is a technique to store bools in an integer. Using bit shift operations and masks you can set, clear and read specific bits and treat them as bools.

However, bitfields are tricky. And the syntax seems arcane for the uninitiated.

I don't recommend a new programmer to deal with bitfields, as they are very easy to misuse. They are so tricky in fact, that Rust, the programming language with possibly the strictest type system of all available languages currently available, doesn't have a type for them. Instead, Rust users recommend external libraries that do bitfields for you.

You can read up on bitfields if you want to, but I will consider any further discussion to be out of scope for this post. If you want to read up on yourself, here's an Wikipedia article and this amazing video by Creel.

My recommendation is, even if it's wasteful, to store an entire byte for a single bool. So we will continue with that.

Ok, but oh no. We are not done yet with problems: What even is a bool? Like, how does the CPU actually represent that in memory? The unfortunate answer is, that bools do not exist. Yes, you read that correctly. As far as the CPU is concerned, only words exist. At best, some special registers that store flags could be considered boolean, but they are used for a completely different use case, compared to the bools in your code. And unless you are writing assembly, you can't actually manipulate such flag registers.

Different programming languages compile bools differently. If you've ever tried to write interoperability between two programming languages, you might have run into this. But it gets worse: Not even the SAME programming language can agree on how a bool should be represented. Different C compilers may produce different assembly. Newer C standards only define what behavior a bool should have, not how it should be represented in memory.

Wild, isn't it?

This absolute disaster has direct consequences for our serialization: Even if our programming language provides a method to convert a bool to bytes and vice versa, we cannot rely on it. There is no guarantee that a different computer or a different program can understand our bool. As such we must take the serialization of bools into our own hands. Luckily, this isn't difficult, as a bool can hold only 2 different values.

There are many different strategies we could use here. We could use a bitfield, but as discussed, not recommended. Instead, I suggest to write a 1 when the bool is true, and 0 when the bool is false. When reading, 1 means true, 0 means false, and every other value leads to an exception.

If you want, you can use a different strategy. But whatever you do, make sure that it is consistent over all computers and programs that use your binary format.

Here's my implementation:

Classes and structs

We have come quite far. So much so, that we can start to serialize our classes and structs! Classes are ultimately just a list of smaller types (we are not going into what methods are or how they work, for that you can look up vtables). So, reading and writing this "field list" in a predefined order is enough to serialize our class. This is very easy actually. Here's an example:

Very straight forward. Very simple. This is what I meant by "creating your own binary format is fast and easy". Sure, there was a somewhat steep learning curve. But once you understand these ideas, and have a collection of utilities like RisIO, you can whip out a new binary format for everything in no time.

What do you say? I did not cover arrays? Strings? Pffff whaat? Who needs those? I certainly don't need arrays. That's for certain. No! Don't look at the scrollbar! We are practically done...

Sized vs unsized types

Welp. We are not, in fact, done. As it turns out there is quite a bit more to cover. But it is true though: Serializing classes and structs is as easy as calling the according IO methods in a predefined order. Everything from here on out covers special types that require specific strategies. It all boils down to sized vs unsized types.

Up until this point, we have only seen sized types. By that I mean, we know how big they are, how much memory they take up. A serialized int has always a size of 4 bytes. A serialized bool is always 1 byte. Last jump scare, I promise: There exist types with unknown sizes.

Let's take the array for example. How many bytes does an array take up? Think about it for a second. The answer is: It depends. If there are no elements in the array, then hurray, we don't need any memory actually. But if there are, idk, 10 items in it, then we require at least 10 times the size of whatever type we are storing.

This is what I mean by an unsized type. An unsized type is a type, that we don't know the size up front. To be specific: A type is unsized, if it's size cannot be determined at compile time.

Once our program runs and the array is actually loaded into memory, it does take up space. This means it does actually have a size, but it must be determined at runtime. To properly serialize an array, we need to store the number of elements, and then serialize each element individually.

Here's how that might look like:

You may notice that these are methods directly on the class itself, instead of RisIO. Well, at the time of writing this post, I have not found a satisfying method signature that covers all the ergonomics that I want in a standalone IO method. As such, I typically serialize arrays like the example above, even if it may be verbose at times.

It should be noted, that often times, classes and structs are sized types as well. If you know the sizes of the fields, then you know the size of the entire class. For example, a class storing an int, a float and a bool has always the size of 4 + 4 + 1 = 9 bytes. As such, it too can easily be stored in an array.

However, once a class stores even a single unsized type, the class will be unsized itself. More often than not, this is fine. Elements in an array are stored back-to-back. Reading one element after the other gives correct results, even if the array element is unsized. But it should be noted that you must deserialize ALL array elements in such a case. You cannot, for example, skip every second element via Seek(n, SeekFrom.Current), because you don't know the size of an element. If you absolutely need this behavior, then you can store the size first, and serialize the class second. The reader should read the size first, and then decide if it wants to skip or not.

Strings

Now we can finally serialize strings. Took us long enough.

I often see new programmers, who work with modern programming languages, taking strings for granted. Some even consider them to be a primitive type!

"Come on man. How hard can Strings possibly be?"

As it turns out, strings are very difficult. If you've ever worked with strings in C before, you know how non-primitive they actually are. Rust has hundreds of different string types! Well, I am exaggerating, but Rust indeed has a lot of different types of strings.

But even if we stay in the realm of C#, strings might still be tricky. Plain text isn't as plain text as you might think. "Plain Text" by Dylan Beattie is a good talk I recommend: LINK. Another good resource is "The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)" by Joel Spolsky: LINK. The rabbit hole goes quite deep, and strings are anything but simple.

But there are very good news for us: Practically, strings are a solved problem.

Thanks to Unicode, serialization of strings is actually very straight forward (as long as we stick with Unicode). To keep things brief, and to summarize Joels post, a string is a sequence of characters. Each character is defined by a codepoint. Codepoints are arbitrarily defined, and must be "encoded" before they can be stored in memory. As long as we know what encoding a given string uses, we can read and understand it.

There are many different encodings, but as long as we use the same encoding in the serialize and deserialize methods, we will be fine. I choose UTF-8, because it is ubiquitous. You don't need to know how UTF-8 works, but you can read up on it if you want. Here's a Wikipedia article, here's the official RFC and here's Tom Scott on Computerphile.

Since strings are a solved problem, serialization is stupidly easy for us:

Instead of serializing characters, we first encode our string into UTF-8 bytes and then store that as a dynamic array. We are storing the length first and then the bytes. To deserialize a string we read the length first and read that many bytes, which are then interpreted as a UTF-8 string and converted into a C# string. Easy :)

In case it isn't obvious, strings are unsized types, because they can have arbitrary lengths.

Dynamic types

Another kind of unsized types are dynamic types, i.e. types that are only known during runtime. Notable examples include polymorphism, by which I mean multiple classes that implement the same base class. This is very common in OOP languages. In other languages union structs fall into the same problem, like Rust enums for example. Here's some code to work with:

Here, Fruit is an unsized type, because any derivation may serialize to a different length. An Apple might serialize differently than an Orange for example. But because of polymorphism, both can be stored in the same List of FruitBasket.Fruits. So we don't know at compile time how much memory a FruitBasket would take up.

To solve this, we store the type as an enum and serialize based on that. This works:

Whenever a Fruit is serialized, we first get its FruitKind and store that first. Then we serialize the Fruit afterwards. When reading, we read the FruitKind first. Using a switch, depending on the FruitKind we are either deserializing an Apple, a Banana or an Orange.

Notice how Deserialize of each Fruit takes a RisMemoryStream instead of a byte[]. This is different from previous examples. The reason for that is because FruitBasket does not know how big a serialized Fruit is. By passing the stream into Fruits deserialization, the Fruit can read as many bytes as it needs.

An odd problem with a banger solution

We are slowly inching closer to the end of this post. So let me present you with an odd, maybe somewhat complicated problem. Up until now, we haven't really used Seek, did we? Well, in this chapter we are going to use it. And for a very cool reason actually. So cool in fact, that I hope this chapter blows your mind.

Let's say you have three classes: FooBase, FooA and FooB. FooBase stores a FooA and a FooB. Assume FooA and FooB have according serialize- and deserialize methods. FooBase looks like this:

Nothing too complicated. Looks kinda like the example in the "Classes and structs" chapter.

But here's my request: I have a stream that contains a serialized FooBase, but I only want to read FooB out of it.

This may seem like an odd request, but let's roll with it. Maybe FooA is very big and very costly to deserialize, so I only want to deserialize FooB and deal with FooA some time later. I don't want to deserialize the entire FooBase, when I just need FooB. How would you implement this?

Well, we definitely need to change both Serialize and Deserialize. As it stands now, our binary format stores no information on where FooB starts in the stream. Currently we have to deserialize FooA, which advances the stream in such a way, that FooB can be deserialized.

One thing we can do, is to store the length of FooAs bytes right before it:

This way, we have information on how big FooA actually is. So we can use it to skip FooA entirely. Here's how that would look like:

Easy. Read the length, and skip that many bytes forward.

I am not satisfied with this solution however. What if we also have FooC, FooD, and so on? If we have thousands of Foos, and we only want to deserialize the last one, then we need to read and skip thousand lengths!

No, I have a better idea: Instead of storing the length of FooA, let's store the position of FooB:

With this, we can directly jump to FooB:

This works. And it is actually quite useful. I use this technique in ris_engine, in which I store all assets in a single file. At the beginning of the asset file, there is a lookup where each asset is stored. Loading an asset is as simple as seeking to the position of the according asset and reading from there. It's also quite helpful if you have quite complicated structs with many unsized types. Storing positions like this is quite the game changer.

And this is where I pull the rug from under you.

If you understood this concept of positions, then congratulations. You now understand pointers.

I am not kidding. This is no joke and I am fully serious. New programmers often struggle with pointers as a concept. From my experience, they even show pride in their unwillingness to understand what a pointer is. The argument I often hear is, that they don't intend to write C/C++, and therefore, pointers are not required to be understood.

But I want to stress again, this position concept we just implemented, is exactly what a pointer is. Somewhere in our stream FooB is stored. At the start of the stream, we have written a pointer, which tells us exactly where FooB is. To read FooB, we use the value of the pointer to seek to the position where FooB actually sits.

The same concept applies to actual pointers in languages like C/C++ and Rust. If you store a simple variable, you have direct access to it. You can read, modify and use it. No pointers required. If you have a pointer however, that variable sits somewhere in memory. Think of memory as a very long byte array, just like our stream. The pointer simply stores an index into that long byte array. To look up the value of our pointer, we look in the array at that index and voilà, we know what our pointer stores.

When people talk pointers, they often say "address" instead of "index". And people say "dereference" instead of "lookup".

I hope you could follow me. I think this was fun. And you can pat yourself on the back, for finally being able to understand pointers.

From now on, it's smooth sailing. The heavy lifting has been done. All that's left is to improve a bit on what we've learned and then discuss some useful strategies.

Making it fat

While we are at pointers now, I want to use the opportunity to widen your mind a little bit.

Right now, when storing a pointer, we are storing just a position. While this is sufficient, if misused, it can produce all kinds of headaches. Like, nothing stops the serializer to read and write outside our intended region. This may cause unintended bugs, but it can also be abused by mischievious actors. As a matter of fact, some of my previous deserialize code does allow such a bug. The astute reader may have caught it already.

Let me bring up the code again, so you have another chance to find it:

Did you catch it? It may be hard to spot, since we are missing a crucial implementation, but the problem is right there. The fruits example from earlier suffers from the same bug actually...

As it stands now, the code is assuming that FooB.Deserialize never seeks, or at least seeks in a way that is invisible to FooBase. Since FooB.Deserialize takes the entire stream as an input, it can seek, read and write wherever it wants, even outside its intended range.

Now we are reading in some place outside of our intended area, which is not good.

This error is called an out-of-bounds error, and it is one of the many reasons why C and C++ are considered unsafe. Pointers in C and C++ are just addresses, and nothing stops you from accessing any memory you want. You can dereference before and after your pointer willy nillingly.

To prevent such out-of-bound accesses, modern programming languages simply don't allow it. C# for example throws an exception if you try to access an array out of bounds. Not all hope is lost for C and C++ though, as modern operating systems utilize some clever tricks, like memory paging. But that's more of a band aid than an actual solution.

In our case, we want to prevent the deserializer to access an entire stream. We don't want it to modify the stream however it likes. The easiest way to accomplish this is to simply not give our serializer a stream, and only rely on byte arrays. But this implies that the one creating the array knows how many bytes a given deserializer needs. In one way or another, we have to store the length of our bytes somehow.

Let me introduce you to the FatPtr. The FatPtr is a fairly simple struct that stores two ints: An address and a length. The address is, well, the address. And the length describes how long the byte array at that address is. Since a normal pointer only stores a position, a FatPtr is "fat", because it stores a length as well.

This implementation also introduces some utility methods, which will be helpful later. For example, we can construct a FatPtr via a beginning and an end. We also define that Address == 0 && Length == 0 means that we are dealing with a null pointer, which might be helpful in some instances. For example, if a field in our data class can be something or nothing, we can use a null pointer to indicate that the field stores nothing.

To support FatPtrs, let's introduce two new methods to RisIO:

The FatPtr only stores two fields, so to serialize a FatPtr we only need to serialize these two fields. And while we are at it, let's modify the regular RisIO.Write method like so:

We seek before and after the write, and construct a FatPtr to it. This FatPtr then points exactly at where we have just written some bytes. This significantly improves the ergonomics of client code, as we will see in the example. But before we take a look at that example, I want to add one last little helper to our RisIO toolbox:

This method takes a FatPtr, which it uses to determine where to seek to and how many bytes should be read. This also will come in handy.

With these tools under our belt, we can now implement a safe FooBase like so:

Now, the deserialize methods of FooA and FooB take a byte[] again. Because of that, FooA and FooB cannot do any out-of-bounds shenanigans.

Some more strategies

While we know enough now to serialize to our hearts desire, there are a few quality-of-life features that are very nice to have.

For example, we have a byte array, yes. But we would like to know what we are actually looking at, without deserializing the entire thing. Our binary format may be huge after all. If a user puts, idk, a PNG file into something that expects our custom format, it would be very nice to stop early before we are deserializing who knows what.

The easiest and most common solution to this problem is what's called a "magic value". A magic value consists of arbitrary bytes, which are written at the very beginning of our format. Thus, it's the first thing we can read.

For example, every PNG file starts with the bytes 137, 80, 78 and 71, which in escaped Unicode spells "\u0089PNG". As another example, every one of my assets in ris_engine starts with the ASCII "ris_".

To serialize, we simply write the magic bytes first and then serialize after. To deserialize, we read the magic bytes and compare if they are as expected. If not, return an error. If they are as expected, continue deserialization. They may be called magic values, but there is no black magic to be found here.

Another reason why a byte stream might not be as expected, is because of a version change. Maybe you are maintaining a program, which is in use for quite a while now and receives updates regularly. In that case the magic value of your format may stay the same, but you need to bump the version to prevent incompatible versions from breaking your program.

You can use different things for your version. A string is probably easiest, but the most wasteful. glTF for example uses a string to identify its version. You can also use an integer, or multiple if you are going the semver route. If you are clever, you can even put a MAJOR.MINOR.PATCH version in a single int, like how Vulkan is doing.

But we may be overthinking things. The version is comparably small in relation to the rest of any format 🤔

Another feature you may want is a header. A header consists of a number of bytes that are always there at the start of your format. The magic value and version is usually part of the header. But a header can hold additional information, for example the number of channels in an audio file or the dimensions of an image file.

Putting all these features into one code example, we might end up with something that looks like this:

A header may be of any type and can contain as many fields as you want. To keep the example short and simple, I chose to use a single string as meta data.

At last, but not least, you might want to be backwards and forwards compatible between different versions. This is a beast of a problem, especially when multiple people over multiple generations are maintaining such a format. Infamously, the .docx file format from Microsoft Word solved this by simply using a zipped XML container. If you have a .docx file lying around, you can literally unzip it. You will get a directory structure, which you can easily view with any file explorer.

I have yet to implement such a format in binary, but I did something like it in XML though. So I have no experience on how you would develop such a format in binary. But I can tell you what I would try!

I would heavily rely on chunks. By which I mean: FatPtrs everywhere. At the start of each chunk, an enum or its "kind" is stored, which determines what exactly this chunk is doing. If a deserializer reads a kind of chunk that it does not recognize, the chunk is ignored. If you are not using FatPtrs, store them right after each other. Each chunk would then also store its size, such that a deserializer, who doesn't recognize a chunk, can skip it.

Compression and encryption

As I've hinted in the chapter about bools, our implementation is quite naive and simplistic, which wastes quite a bit of space. We can try to decrease the size of our data by compressing it. But I have to mention that not all data compresses equally. Some data is resistant to compression. Also, you may be interested in encryption, because you might store sensitive data.

Well, compression and encryption are two entirely different disciplines. Explaining each one in detail is waaaayy beyond the scope of this post. Either one is a rabbit hole on its own. For our use case, only this counts: Both compression and encryption are methods, which take a byte array and spit out a new byte array. Compression will reduce the amount of bytes, and encryption will scramble the bytes.

Since we deserialize into a byte array, compressing and encrypting is as easy as calling the according method on our byte array.

And that's all there is to it. There are many different compression and encryption algorithms out there, many of which have dedicated libraries. You can pick and choose whatever you want. The sky is the limit.

Usually, you don't want to compress or encrypt the entire format. Usually, you want to keep the magic value, version and sometimes even the entire header readable. The reason is simple: You want the receiver to be able to identify what they are looking at, and that will be difficult when the data is mangled.

To take ris_engine again as an example, all my internal formats have a 16-byte magic value at the start of each asset. Each one starts with "ris_" to identify them as one of my assets. The remaining 12 characters indicate what kind of asset one is looking at. Below is a screenshot of my "ris_scene" format. By the mangled nature of the bytes, one can guess that it was compressed. But the "ris_scene" at the start remains readable.

If you want to learn more about compression and encryption, I leave you with these Wikipedia articles:

• Compression
• Encryption

Conclusion

Wow, what a journey. It took some work, but now you should have everything that you need to build your own binary format. I am sure I have written something that you disagree with, and that is fine. Now, you have the knowledge to make adjustments to my code and implement what you actually need or want. And with these tools under your belt, it shouldn't be too difficult to cook up your own binary format.

I claimed in the intro that a custom binary format is easily smaller and faster than JSON. So I will leave you with a benchmark. It compares my serializer with an equivalent implementation using Json.NET by Newtonsoft, a popular JSON library for C#. The benchmark results are below.

(lower is better)

I hope it is evident that a custom serializer easily outperforms a general one.

A complete implementation of the code in this post, including this benchmark, can be found here: LINK

A game object is essentially a frame of reference. It has a position, a rotation and a scale. This is the easy part. The difficult part is this: A game object may have children. This makes game objects hierarchical. Now, a game object references its children, but it also references its parent, enabling the hierarchy tree to be traversed up and down. While convenient for client code, this immediately creates a problem for the implementation. If you've heard anything about Rust, then you know that circular references like this are Rusts kryptonite. Unfortunately, this is the first of many problems.

A game object on its own doesn't do much. To give it behavior, you can attach components to it. These can be a variety of different types. And since most of them directly affect the game object, they also need to reference it. Another circular reference!

It gets worse though: One of the components is the ScriptComponent, which allows custom user code to hook into the engine. Any given script can reference whatever, including itself, its game object, its parent or children, and any other component, script, asset or currently loaded object. This is not only a dependency nightmare, but a polymorphism nightmare as well.

But oh, we are not done yet. My chosen solution has its own set of problems: I chose to use a single scene object, which owns and thus stores all loaded objects. Any references between the objects are solved via handles. These easily copyable handles are simple token objects, which store information on how to get the according object from the scene. But of course, this requires a custom reflection-like solution, such that you can resolve the type of the handle at runtime. And because Rust doesn't have reflection, I was forced to implement it! While a single handle struct might've been enough, I want it to be type safe. I want that when you are trying to get a MeshComponent with a ScriptHandle, the compiler refuses to compile the code. Thus, generics and PhantomDatas everywhere. As a direct consequence, I also use one too many transmutes, sailing very close to the unsafe brink of undefined behavior.

And then, the scene must also be thread safe. Wrap the entire thing in a Mutex and call it a day :^) No, I chose interior mutability, to avoid congestion. For this, every object is stored in a custom thread safe RefCell, with assertions removed for release builds, for maximum performance. (Premature optimization is the root of all evil you say? Never heard of it!) And the engine must hook into the scene in many different ways. All script callbacks need to be called from various locations in the game loop. And somehow the renderer must access the mesh components to render everything. To top it all off, the system must be flexible enough for extensions, such that more components can be added in the future.

Sheesh. That's a lot of stuff. But I've done it. And the API is squeaky clean. Here, take a look:

And the best part about it is, it just works. It runs, the unit tests succeed, and miri doesn't find undefined behavior. You can even double all of that, because all that runs without error in both debug and release builds! It simply works. It just does. And I really want you to appreciate that.

Every now and then, every couple of months, some indie dev team or company shares a hit piece on Rust, how they tried the language and had to abandon it because this, that and whatever reason. And the one reason that is always included is, that the borrow checker is stupidly difficult to work with. Having written the game objects I described above, I can sympathize with this argument. Anything sufficiently complicated is just such a pain in the ass. Rust evangelists will often tell you that "you just don't understand how the borrow checker works", which is honestly such a farse. The borrow checker is easy. I believe any competent programmer will understand it in the first 10 minutes they spend with it. But the matter of fact is, its simple rules are very very hard to satisfy. And yet, if you manage to get your code to compile, it simply works. I can't be more clear than this. Pleasing the borrow checker results in less erroneous code.

This wasn't my first rodeo with hierarchical node systems. I've written one two years ago for my company. This version was written in C# and it is significantly simpler and more complex at the same time; it took me only a few days to get the first prototype working. But over time, problems started to creep up. I was tasked to include more features, which I were promised were absolutely necessary (spoiler alert, they weren't), resulting in messy implementations and spaghetti behind the scenes. Also some bullshit synchronization code was necessary, which needs to sync our nodes to 2 different node systems, and it must be easy to switch on and off, because of course it must. It also produces all kinds of events, whenever anything changes in the node, but this is unreliable and buggy as hell. No one fixes this, because nothing, and I mean NOTHING uses them. Yes, THE central node system, that sits in the center of our entire codebase has this huge complicated non-working deadweight attached to it.

These idiosyncrasies (and many more) make this C# node system such a pain to work with. But we can't fix it, because that requires time and money, which the company isn't willing to spent. So even though the initial implementation was a breeze to get working, over time the issues became apparent and years later we are still applying hacks to get around all the shortcomings. Compare that to the thing I've written in Rust: It was a major pain to get it to compile. But once it did, it just fucking works.

The Rust version frontloaded all the work, while the C# version revealed all the issues during the years.

I don't want to dick ride Rust. As Bjarne Stoustroup famously said, I firmly believe that there are only two types of programming languages: The ones everyone complains about and the one no one uses. So here's a rundown of things I hate about Rust:

Rust has no convenient base error type; you need to download a crate or write your own. Speaking of errors, why can mutexes be poisoned? As far as I know, it's impossible to recover from this, so you might as well segfault. Yet, Rust forces me to handle poison errors anyway...

Rust is also infamous for its absolute dogshit syntax. It's already bad when you take a surface level glance at it, but have you seen the syntax for macros? At times it's worse than bash!

Anyway, who decided to use pipes for closure parameters?

Also, why call it "closure"? Why not simply call it what every single other programming language calls it? Like "lambda", "anonymous function", "function pointer" or whatever?

Everything is so horribly nested! You have a match in a loop in a function in an impl in a mod and you are looking at 5-6 levels of indentation!

for loops are nice at first glance, and they are, until you need a traditional one. For example when you conditionally want to increase i. It gets worse in const code, because for isn't even supported there. The example below is legit code I had to write at one point. It's basically a desugared for loop:

Amazing.

Or why do if statements avoid brackets ()? Sure, it's less to type and one can get used to it. But the problem is, that no other programming language does this! Because of it, I can never build any muscle memory in both Rust and non-Rust languages. Sure, it isn't mandatory and I can use brackets in Rust if I want to, but clippy flags them as bad style. Do I look like someone who configures a fucking linting tool?

And then there is the turbofish. First, really stupid name. Second, you can't convince me that one absolutely needs the two double colons :: to denote a generic type. To use my game objects as an example:

And I can go on and on. I hate Rust. But I continue to use it, because well, the sunken cost fallacy, but moreover because the borrow checker just leads to correct code.

While my professional job is mostly C#, I do get to touch C++ from time to time. And every time I do so, I feel like a gun is pointed to my head. If I make a single incorrect assumption, BOOM, undefined behavior. In the best case it blows up in your face and crashes the program. In the worst case it quietly keeps working, until a customer submits a bug. And then you can't replicate it! I've been there! It sucks! It sucks so much! It's one of the worst bugs you can encounter! You can say skill issue, sure, but not even the greatest C++ wizard can avoid undefined behavior.

"Modern C++ is safe" 🤓 says the C++ enthusiast, unwilling to acknowledge that they are in an abusive relationship. Sure, the newest C++ standard includes safety features, if you find a compiler that actually supports it. But when C++98 is the standard in your enterprise codebase, you're out of luck. Here's the kicker: Even if you use very modern C++, safety is still opt-in. You purposefully need to use the new safety features. Safety is not the default. No one is stops you from doing pointer arithmetic.

In Rust, safety is opt-out. Safety must purposefully be turned off by using unsafe. "If you are using unsafe you are writing Rust incorrectly" 😡 says the degenerate Rust evangelist. I do it all the time. It's no big deal. It's easy and I don't care. My point is this though: Safety is enabled by default. You can tell that a piece of code may cause undefined behavior, because it is clearly labeled by its unsafe block. So, when you do a code review, you can point to it and yell at the junior why the fuck they thought it was necessary to include unsafe code.

In C++ I write undefined behavior by accident. In Rust I write undefined behavior on purpose.

"Rust isn't 100% safe" 🤓 says the educated Rust hater, pointing at bugs in the compiler. I mean, do I even have to argue about that? Theoretically Rust is safe (according to their own definition, which may be different than what you are thinking), but our world tends to be messy and not ideal. However, did you know that seatbelts aren't 100% safe either? It turns out, it's already a huge improvement when a new solution is safer than the current alternative, even if it isn't perfect.

"But Rust is not OOP" 😩 says the clean code obsessed, code smell avoiding programmer. I swear, the paradigm of the programming language doesn't matter, like at all. If C people can do polymorphism, so can you. Yet, OOP bros are acting like everything that isn't OOP is the devil incarnate. I constantly run into videos and articles about "bad code" and "how to avoid it". And they always spew nonsense like "avoid switch statements" and "you shouldn't use bools". What's next, "don't use variables"?!

You think I exaggerate? I am sorry to disappoint you, but these are talking points from messiah Uncle Bob himself:

• https://youtu.be/2IotTzClOAQ
• https://youtu.be/2Q9GRPxqCAk

I swear to God, if I hear one more time that a class should have no more than 100 lines of code I WILL FUCKING EXPLODE! I consistently write code that exceeds 200 lines per file, I'd say around 500 is my average. But I do so because I avoid oneliners, use many, MANY variables, and I like to keep related code in one place. I do not want to spread my code over multiple files, so that future me doesn't need a huge mental map and 100 active tabs to debug it. But OOP bros just cannot shut up about how many lines of code a single method should have. I am seriously starting to question: Do you guys even write software? Or do you just glue frameworks together?

Look, I take great issue with OOP bros trying to sell me "clean code". I am sorry, but I have seen the absolute disaster that these "programmers" produce. I have witnessed widget-based GUI applications written in Python, that despite having less than 7 buttons, somehow struggle to run at 5 FPS. And you are telling me, I am supposed to listen to their disgusting "code smells" and how I am supposed to structure my code?! Come on person, get a grip!

Anyway, this post has absolutely gone off the rails, hasn't it? What was the title of this post? Ah yes, Rust is the future. While writing, ranting seemed natural for this kind of topic. Kudos if you've actually read my incoherent ramblings. Let's come to an end and wrap things up.

Rust has one killer feature: The borrow checker. All arguments that you throw against Rust are immediately invalidated, because your preferred programming language simply does not have a borrow checker. I am going to make the prediction that C++ is going to lose relevance over the next decades. It won't die, simply because of its historical significance and wide adoption. But C++ will fade, for sure. C will never die, because it has transcended into the true and only low-level language (above assembly). C has become the de facto standard for interopability between any two programming languages. Higher level languages like Python, JavaScript and Java will stay around, because they are easy to program in. Garbage collected languages are inherently safe without any hassle. But C++ in particular will lose significance. If it isn't Rust, then another programming language with a borrow checker will take the place of C++. But as it stands today, Rust is the only programming language which is mature enough to compete with C++.

For context, back then I was really into Minecraft, which is the primary reason in my interest in programming nowadays. Anyway, YouTubers back then were using Monstercat music. These YouTubers shall not be named, for reasons that if you know you know. And as the music fanatic (snob) that I am, it was only a matter of time before I would check out this music.

So what is Monstercat? Simply put, a record label. Monstercat was founded in 2011. To this day, they release EDM tracks of different genres from various artists. Initially, they were simply a friends group. But the label expanded massively to include many other artists. Every couple of months, they would release an album, including the previously released songs. Later they would include a continuous mix of all the songs in each album. And it's these albums that I will cover in this post.

The albums are numbered from 001 to 030. I found them between the releases of 015 and 016, and I've been continually following till around 020. So my ratings are biased around that time. Obviously, I was able to check out their previous stuff, but around 020 my interest tapered off. I've been following a few select artists, but most stuff after 020 I went into somewhat blind.

Okay, that's it with the introduction. Without further ado, I will give you my thoughts on each of these 30 albums, in chronological order. At the end of the post, I've included a tier list. Let's go!

001 Launch Week

Playlist: LINK

Ah, humble beginnings. Way before my time and thus 0 nostalgia. 001 is Monstercats first and shortest album, clocking in at only 7 songs, which in total is about half an hour. Later albums eventually become 2 hours long, but since 001 is so short, it's very easy to relisten again and again. But honestly, I rarely do so. It is easily the most amateurish album. The music is not good, and it's only interesting to see where the bar was back then. People gotta start somewhere, and I don't hold that against them. It reminds me a lot of my own music, which is charming in a sense.

So yeah, the album is not good, thus easily bottom tier.

002 Early Stage

Playlist: LINK

Highlights

• Matduke - Rock the House
• Noisestorm - Airwaves
• Feint We Move
• TwoThirds - Lost

More amateurishness! I am considering this album more like a continuation of the previous one, because there are more new people with about the same skill level in music production, with the exception of Matduke and Project 46. These two were definitely ahead of everyone else. Their presence and quality of music gave me a huge whiplash. Noisestorm is also a notable artists. Long before I was aware of Monstercat, I knew about him and his music. Hearing his song Airwaves really hit my nostalgia. But other than that, it's till an amateurish album, without the historical significance of 001. I was constantly thinking, that this is pretty much "Dubstep before it was cool". It doesn't even come close to Brostep, and I am very thankful for that. All my homies hate Skrillex after all. One thing I am surprised about is how similar Stephen Walking sounds like to later in his career. His experience and quality obviously got better, but the style kept pretty much the same.

003 Momentum

Playlist: LINK

Highlights

• Matduke - Shivers
• Stephen Walking - Top of the World
• Feint - Formless
• Obsidia - Android

The quality starts to improve. Again, Matduke is much more ahead than the others. Shivers is the best song on this album and easily one of the best songs on Monstercat ever. I think this is the album where Monstercat is gaining momentum, pun not intended. Like, the music is still amateurish, but it's starting to gain character. I like the songs in the beginning and I really vibe with it. It's amazing. And then it rips the band-aid off with the drop of Obsidia - Android, which whiplashed me into fucking oblivion. It's so disjointed from the first 3 songs, which I find absolutely hilarious. The album drops off a bit in the end and it becomes somewhat boring. Though, Stephen Walking definitely does not drop off. Top Of The World is definitely one of his best songs. I understand why out of his entire discography, this is the one he decided to remaster 2 years later. Absolute legend.

004 Identity

Playlist: LINK

Highlights

• DotEXE - Run Away From Me
• Neilio - Outside This World

Very forgettable. I don't think 004 has a single good track. As such, I don't have much to say about it. The only thing I want to note is DotEXE - Run Away From Me. Its melody reminds me of a German pop song, which I wont name, because the song is garbage and I hate it. However, I really like the melody, and I've considered multiple times to remix the song. But for that, I need to get back to making music, which I can't, because I'm a masochist. Other than that, Neilio - Outside This World is decent and worth mentioning.

005 Evolution

Album Mix: LINK

Highlights

• Arion - Internet Rebellion
• 23 - Fresh
• Noisestorm - Renegade
• Kings Of The City - Make Me Worse (Muzzy Remix)

First album that got a continuous mix. And to be honest, this album is an absolute banger. It's full to the brim with amazing tracks. While still at an amateurish level, the quality is good enough to be recommended wholeheartedly. And the transitions between the songs are very decent, which is commendable. Ahem, 010, ahem. I can't stress enough how much I love this album. Arion - Internet Rebellion probably radicalized me. Today I am very much in favor of abolishing copyright. With his remix, Muzzy introduced me to Kings Of The City, a band which disbanded, but which I hold to a very high regard. Also, 23 - Fresh goes so hard. Tristam is amazing. And so much more songs, that are all "Dubpstep before it was cool".

All around a very solid album. I recommend it. A-Tier.

006 Embrace

Album Mix: LINK

Highlights

• Televisor - Rock The Flock
• Hot Date! & Chrisson - Overcome (This Time)
• Tristam - Talent Goes By
• Project 46 M.O.A.B.

One of the first albums I bought was Monstercat Best of 2012. This was mainly laziness, as it included many past songs. Listening to the Best-of, I didn't need to actually listen to all the previous albums. I am mentioning this, because 006 is the first album that has tracks on this Best-of album. With 001 - 004, I mostly felt indifference and slight annoyance, but with 006 and the albums thereafter, I could do one of these. But other than that, the album is somewhat underwhelming. It has some absolute bangers, like Televisor, who never disappoints. Hot Date! & Chrisson - Overcome (This Time) is amazing. But other than that, I am not really feeling this album. Now typing this out and only seeing my picked highlights, I can't remember anything else about this album.

007 Solace

Album Mix: LINK

Highlights

• Tristam - I Remember
• Overwerk - House
• Feint & Boyinaband feat. Veela - Time Bomb
• Tut Tut Child - Maelstrom

This is THE best album Monstercat has ever released. Period. This is quite a jump, considering that I've called 006 mid. 007 wasn't new when I got into Monstercat, so it has absolutely no novelty. But the tracks are just so amazing that it was only a matter of time before I stumbled over this album. And then, I just couldn't stop listening to it. Every now and then I came back to this album, even before I went on this nostalgia trip. It is honestly that good. This album also introduced me to Overwerk, who genuinely makes good house music. Funnily enough, he wants to be fully independent. He requested his song, House, to be removed from Monstercat. As such, you can only find reuploads on YouTube but not the original. I wont link them, but if you search you'll find it.

Choosing highlights for this one was rather difficult. I don't want to include too many songs per album. So while I absolute love everything on this album, I had to decide, and the songs above are what I landed on.

008 Anniversary

Album Mix: LINK

Highlights

• DotEXE - Hipster Cutthroat
• Muzzy & Day One - Black Magic
• Pegboard Nerds - Rocktronik
• Rogue - Daybreak

I think this album marks a point, where Monstercat officially loses its amateurish status. 007 already was rather good, obviously. But 008 was better sound quality wise, even if it isn't nearly as amazing. The mixing is good, but none of the songs wow me. As such 008 fell somewhat flat for me. But I have to highlight this one song: DotEXE - Hipster Cutthroat. The song isn't that good and somewhat boring, but the first drop is absolutely hilarious. I can never not laugh when I hear it. It's amazing and I won't spoil it for you. Go listen to it.

009 Reunion

Album Mix: LINK

Highlights

• Ephixa - Awesome To The Max
• Rogue - Nightfall
• Pegboard Nerds - Fire in the Hole
• Krewella - One Minute (DotEXE 'Dopest Dope' Remix)

I have very mixed feelings on this album. The start is very weak and forgettable. In the middle it starts getting good and peaks with Ephixa - Awesome To The Max. But then it does very questionable transitions between songs. So yeah, half of the album is good, the other not so much. And the mix isn't a joy to listen to. I don't have much else to say about it, other than 009 really annoys me. Maybe I should listen to the individual songs and not the mix.

010 Conquest

Album Mix: LINK

Highlights

• Feint - Snake Eyes (feat. CoMa)
• Tristam - Truth
• Skifonix - Flatline
• Rezonate - The Phoenix

From bad to worse. 009 is not good, but 010 is somehow worse. I have very bad feelings about this one. In the complete opposite direction of 009, the beginning is strong, but then it just falls of a cliff. The songs are nice, but then the transitions are just such ass. I am just wondering what happened here? 005 was the first album with a mix, and the mixes were decent thus far, but this album just sucks. It sucks so much. The most egregious thing in the entire album however, is the disservice to Feint - Snake Eyes (feat. CoMa). Don't get me wrong, the song is absolutely amazing and one of the best songs on Monstercat ever, but 010 decided put only 45 seconds of it into the mix. Like wtf is this? I am annoyed. I don't like this. I don't like this at all.

011 Revolution

Album Mix: LINK

Highlights

• AZEDIA - Precipitate
• Stereotronique - Stock & Pull
• Eminence - Falling Stars
• Ramases B - Come & Go (feat. Charlotte Haining)

Amazing album. A-Tier. Not much more to say. It's just a very solid album with nothing to complain. The songs are stupidly amazing, and so are the transitions. I recommend 011. Go listen to it.

012 Aftermath

Chaos Album Mix: LINK

Order Album Mix: LINK

Highlights

• Project 46 - No One (feat. Matthew Steeper)
• Pegboard Nerds ft. Splitbreed - We Are One
• Vicetone - Heartbeat (feat. Collin McLoughlin)
• Hellberg - Get Up
• TwoThirds - Origami

The first 2-hour album! Split between two mixes, each being one hour long. This will be the standard for all remaining albums. More music!

Anyway, 012 is decent. B-Tier. Everything is good, but nothing on it really wows me. I keep batting on transitions, but that's because a good one can go a long way. What I mean by that, take the transition between Day One - World's End and Tut Tut Child - Hummingbird (feat. Augustus Ghost) as an example. Here's a link with a timestamp: https://youtu.be/MD3ROlR6k8I?t=312. Honestly, this is THE best transitions on all Monstercat albums. Period. The scales between the songs don't quite match, resulting in some very harsh dissonance. Combine that with the low energy of World's End and it achieves some really deep sadness. But then it cuts, which suddenly resolves the dissonance. Immediately afterwards the vocalist on Hummingbird goes strong and loud. This clash of dissonance being resolved and the mix shifting into such a high gear is simply amazing.

013 Awakening

Dawn Album Mix: LINK

Dusk Album Mix: LINK

Highlights

• Stephen Walking - Short Shorts
• Feint - Lonesong
• Pegboard Nerds & Tristam - Razor Sharp
• Protostar & MakO - No Fire (feat. Rachel Hirons)
• Hellberg & Deutgen vs Splitbreed - Collide

Very good album. I was surprised by how much I liked it actually. I was at first very torn giving it A or B-Tier. But now relistening to the highlights, I am placing it much higher in solid of A-Tier. 013 graced us with this absolute Masterpiece: Stephen Walking - Short Shorts. The music video is simply brilliant. Stephen Walking proved to us that he is indeed worthy of his name. One thing that you might know about me, is that I don't have a driver's license. I am literally walking everywhere. Maybe, just maybe, this video radicalized me to a point that I never considered getting a driver's license. Nonetheless, Short Shorts is funny, and I love it.

Very cool album. Out of all of the ones in this nostalgia trip, 013 is the one I already went back to multiple times. I think I've found a new favorite.

014 Discovery

Odyssey Album Mix: LINK

Journey Album Mix: LINK

Highlights

• 7 Minutes Dead - Delayed Friend Request
• Lets Be Friends - Manslaughter (VIP Mix)
• Mr FijiWiji & Direct - Hysteria
• Falcon Funk - Leophant
• Televisor - Old Skool

Decent Album. Nothing to complain about. But somewhat forgettable. It has not many cool tracks that I celebrate. It's main saving grace is twofold: First, it was one of the albums I owned as a kid, and second, I freaking love DnB. Even though I am fairly familiar with it, I just can't love this album. The transitions are super smooth. Especially between Razihel - Bad Boy and Bustre - Riftwalk, or Mr FijiWiji & Direct - Hysteria and Televisor - Dream Soda. Here are the transitions if you wanna listen:

• https://youtu.be/qNDYQs1vvY4?t=1185
• https://youtu.be/qNDYQs1vvY4?t=3560

Retrospectively, I am surprised by how much DnB is in the Odyssey mix. If it weren't for that, I probably would've given this album C-Tier. One thing that I am very proud about myself is my very broad taste in music. I am not stupid enough to claim that I listen to all genres, but I listen to heck of a lot. But if I had to pick a favorite, it's probably DnB. From Jungle to Intelligent DnB, early Breakcore or modern Breakcore, Neurofunk or whatever other DnB subgenres there are, I just fucking love breakbeats. Funnily enough, I put no DnB songs into 014s highlights lol. Probably because single DnB songs are rarely bangers, and entire mixes are usually much more enjoyable. As a kid, I remember not really enjoying Bustres kind of DnB. But now that I am older, his DnB matches more to what I've linked above, and Bustre is now one of my all-time favorites. How time changes.

Odyssey is definitely the better half of this album. Journey is mid.

015 Outlook

Reflection Album Mix: LINK

Visionary Album Mix: LINK

Highlights

• Project 46 - Motionless (feat. Seri)
• Braken - To The Stars
• Droptek & Tut Tut Child - Drop That Child
• Mitchell Claxton - Wuxia
• Haywyre - Back and Forth

A-Tier. Easily. As the last album before I discovered Monstercat, it was foreseeable that I rank this one highly, no? Good tracks. Decent transitions and oh my god, the nostalgia! I cannot stress enough how much this album hit the spot. But even with my rose-tinted glasses, I couldn't put it into S-Tier. It has some very odd songs, which are really hit or miss. But my nostalgia makes up for it. I enjoy this album very much.

016 Expedition

Arrival Album Mix: LINK

Album Mix: LINK

Highlights

• Rezonate - Canvas
• Tristam - Once Again
• Bustre - Shadow
• 7 Minutes Dead - Peacock
• Lets Be Friends - FTW

Oh boy. The first album after I discovered Monstercat. For me, this hits, and it hits hard. I know all the songs by heart. Even the ones that I didn't like at first have finally grown on me. The songs are amazing. The transitions are great. All around a very awesome package. Out of all albums, choosing the highlights for this was stupendously difficult. Because no matter how I choose, I had to sacrifice all the other songs I loved. If you allow my indulgence, I want to write a small paragraph for each song and why I chose it. Thanks.

Rezonate - Canvas is possibly the most chill song I've ever heard. And it's catchy! While I enjoy calm music from time to time, it isn't my first pick when I want to listen or share "good music". I want to be stimulated, and calm music does the opposite: It relaxes. Canvas is not like that. It penetrates my unfair standard that I've set for my blog post and I always look forward to this song when listening to 016. The best song Monstercat has ever released is Ramases B - Transformations, which is on 016 but not on my list, because it didn't fit the vibe of what I am going for here. But if it weren't for Transformations, Canvas is the best song on 016.

Tristam was always a fan favorite. His music is melodic and grandiose. When I first heard Once Again 10 years ago, it was weird. Like the harmony was something I've never heard before. It didn't resolve immediately and it really isn't that dissonant. It's a flavor that stuck with me. Today I'm into much more weird stuff than in the past, but Once Again was a first for me. The ending of the song is absolutely brilliant. When the last drop is a 10 in terms of energy, then outro is an 11. It always gives me chills.

Bustre - Shadow had to grow on me. As I've said in 014, Bustre was definitely not my favorite. His DnB simply didn't have the punch like all the others, and it is comically simple. The song has drums, and it has bass. But with time, I started to appreciate simplicity. In music and in life aswell. Now, being so deep in the DnB rabbit hole that I am, Shadow is probably one of my most favorite DnB tracks. The drums are brutal. The bass is disgusting. Everything around it gives it an atmospheric feeling. The descending synths in the background are very eerie. But the best part is the second drop, where the bass doesn't even try to play a melody anymore and slowly descends, while the percussion just hammers it into you. It's amazing. It really is.

I do like me some complicated music, Math Rock for example. But for the love of god, I couldn't make or play it. My musical talent peaked in school. I produced a bit of synth stuff, but now I am just programming and my musical ability begins to fade. No matter though, other people can play funky chords for me. 7 Minutes Dead is difficult to describe. His style is so unlike everything else. Different to Monstercat, and different to everything outside of it. I can't compare it to anything I know, I just can't. Peacock doesn't seem to have structure? It has a beat, but it constantly switches, stops playing and starts again. No parts repeat, and melodies weave in and out. Half way in and I think: "Wait, this sounds completely different from how this song started!" But it works. It works so well.

Lets Be Friends - FTW doesn't get a novel like above. It is simply bonkers. Very hype. Very loud. I directly blame this song for the shit I like today and my love for autistic furry music.

017 Ascension

Abyss Album Mix: LINK

Zenith Album Mix: LINK

Highlights

• Tristam & Braken - Frame of Mind
• Haywyre - The Schism
• Droptek - The Covenant
• Aero Chord - Surface
• Pegboard Nerds - Hero (feat. Elizaveta)

Ascension is quite a funny name, considering the cat in the album art is literally underground. But I digress. I have some unreasonable hate for this album. I remember vividly when 017 came out, people really liked it. But it just didn't grok with me. It has some insanely popular tracks, with millions of views each, but I somehow just can't enjoy this album. As such, 017 marks a turning point in my tier list. Every album that I rate below below 017 is an album that I don't quite enjoy. And if I want to listen to Monstercat, I'd much rather pick anything above 017.

Nevertheless, one song I want to single out is exactly Pegboard Nerds - Hero (feat. Elizaveta). This one will always be stuck in my head, because of Corridor Digitals video Superman With a GoPro, which in my opinion is one of the best videos on YouTube of all time. It's a rather simple video, especially nowadays, as drones have become quite ubiquitous. But they weren't that common back in the day, especially for amateur YouTubers. Either way, the effect works and the video is such a joy to watch.

Other than that, Tristam & Braken - Frame of Mind is an absolutely amazing song. It's up there, with the best of the best songs on Monstercat.

018 Frontier

Horizon Album Mix: LINK

Pinnacle Album Mix: LINK

Highlights

• Grabbitz - Here With You Now
• Mr FijiWiji, Laura Brehm & AgNO3 - Pure Sunlight
• Stonebank - Moving On (feat. EMEL)
• Varien - Valkyrie (feat. Laura Brehm)
• Stephen Walking - Shark City

It surprised me. And I really did not expect this. But, 018 is breathtakingly amazing. I remember that I never had any problems with this album. I didn't expect much, and I kinda forgot about it. But I was genuinely surprised by how well everything fits. Relistening to it blew me away so much, I am honestly ranking this one higher than 016.

Here's the thing: Whenever I tell people about what music I listen to, I always stress how important albums are. Not playlists that you create yourself, but albums that are created by the artist. Single songs are highlights at best, but the whole picture is much more enjoyable when it tells a story. A good album has ups and downs, with highs and lows. The lower the lows, the higher the highs. For example, one of my most favorite albums is Cardiacs - Sing to God. This album easily sits in my top 3. It goes so hard, loud and crazy so many times. But it can only go this hard, because it also goes equally silent. You can only be loud in relation to something else, because the listener controls the volume knob.
On the radio, every song competes with every other. As such, the loudness wars are under way with every song trying to be better than the other. The result is that everything is equally loud. The listener turns the volume knob down, and now all the songs are silent. Bummer. But in a well-constructed album, you can deliberately hit pauses, and with them you can reach amazing highs, that you could never do with the stupid radio.

Where was I going with this? Ah yes, 018 tells a story. The transitions are amazing, the songs flow so well into each other, from house, to future bass and back to hard hitting hardstyle. 010 may have good songs, but it absolutely fell flat with the story. 018 may not have that many good tracks, but it makes up for it by having absolutely stellar composition. The highs hit hard, because the Future Bass tracks in between provide a calm and necessary rest.

One of the highlights I want to single out is Grabbitz - Here With You Now. Upon on release, I hated the drop of this song. But repetition legitimizes, and I started to like the song the more I listen to it. Grabbitz in general makes very cool tracks, that have amazing verses and buildups, and then just go fucking apeshit with the drop. It's almost comical how annoying his drops are, but I started to appreciate it.

019 Endeavour

Aspire Album Mix: LINK

Venture Album Mix: LINK

Highlights

• Au5 - Snowblind (feat. Tasha Baster)
• Draper - Coloured Glass
• Astronaut & Far Too Loud - War
• Case & Point - Error Code
• Rootkit - Against the Sun (feat. Anna Yvette)

Mixed feelings on this album. It has some very questionable songs, that are not fun to listen to. They are just so distracting to everything else. But what made me give 019 A-Tier anyway, is because of Didrick. Jesus fucking Christ my dude, his Monstercat Live Performance is pure nostalgia, concentrated up to a lethal level. I wasn't really enjoying 019 as a whole, but then this compilation plays as the last track, and it hits me like a fucking truck. I was so high in my nostalgia trip, and this song just amplified it further. I had to stop everything I was doing and I just listened to the damn song. I had this stupid shit eating grin across my face. I recognized all the catchphrases, the samples, all the melodies, and after it was over I was exhausted. I have never taken drugs before, but that's how I imagine it to be. God damn.

So yeah, the album is mid. Not so good. But this single song elevated it to a higher tier.

020 Altitude

Elevation Album Mix: LINK

Stratosphere Album Mix: LINK

Highlights

• Vicetone - What I've Waited For (feat. D. Brown)
• Stephen Walking - Top Of The World 2
• LVTHER - One Look (feat. Mammals)
• Puppet - Scribble (Extended Mix) (feat. the Eden Project)
• Rich Edwards - Sweetest Addiction (feat. We Ghosts)

Ah, the beginning of an end. After this album my interest in Monstercat dropped. I continued to listen to songs after that, but my consumption gradually declined. I followed a few select artists, like Puppet or Karma Fields, but my interest faded eventually. I discovered Caravan Palace and Modest Mouse, my taste went on a wild journey and I haven't listened to EDM for quite a while after that. Anyway, 020 is full of many amazing house tracks, similar to 011. I wouldn't call 020 peak, but with as solid as it is, I must put it very close to 011. Afterall, 011 was also full of house tracks, so the styles kinda match.

I have to highlight Vicetone - What I've Waited For (feat. D. Brown), which also is the best of the best. Absolutely amazing song. While I haven't returned to 020 that often, I routinely come back to this one exact song.

Also, did I seriously put Top Of The World into the Highlights again? Hell yeah I did!

021 Perspective

Aspect Album Mix: LINK

Vantage Album Mix: LINK

Highlights

• Draper - Pressure (feat. Laura Brehm)
• Haywyre - Insight
• Excision & Pegboard Nerds - Bring The Madness (feat. Mayor Apeshit)
• Stonebank - Stronger (feat. EMEL)
• Ramases B - Mountains (feat. Veela)

The last album I know most songs from, and the last mix I listened to during my Monstercat phase. Back then, it was the first one that did not manage to convince me to buy it. Tracks are mostly forgettable. No nostalgia. Nothing. I don't have much more to say about it. 021 is not ass, but it isn't good either. Vantage is definitely the better half of this album, the other sucks in comparison. But I want to single out two songs: Draper - Pressure (feat. Laura Brehm) is one of Monstercats all time greats. After all this time, I still knew all the lyrics by heart, which is amazing. But speeding up the end of it in the mix really annoyed me. And Haywyre - Insight is the goat. It's Haywyres best song, and since it's actually performed, it's such a joy to watch him go apeshit on his keyboard.

022 Contact

Connection Album Mix: LINK

Essence Album Mix: LINK

Highlights

• Bustre - Combine
• Grabbitz - Way Too Deep
• Puppet & The Eden Project - The Fire
• Rich Edwards - We Are (feat. Danya Nadeau)
• Trivecta - Evaporate (feat. Aloma Steele)

The first album I only know half the songs on. I never listened to the 022 mix in my Monstercat phase. Somewhat forgettable. The tracks don't stick in my head. With as far as Monstercat has come with its quality in sound production, I can't put 022 anything below B-Tier. Puppet is an artist that I've kept following for a bit. With Scribble on 020 and The Fire on 022 he collaborated with and thus introduced me to EDEN. EDEN makes sad music, filled with angst. That shit was emo enough already, and fortunately I didn't go deeper than that in this style of music. Other than that, I have literally nothing else to say on 022.

023 Voyage

Passage Album Mix: LINK

Excursion Album Mix: LINK

Highlights

• Muzzy - Feeling Stronger (feat. Charlotte Colley) (High Maintenance Remix)
• Aero Chord - 4U
• Nitro Fun - Final Boss
• Laszlo - Law Of The Jungle ⚠ VOLUME WARNING ⚠
• Pegboard Nerds - Swamp Thing

For some reason, the YouTube video for Laszlo - Law Of The Jungle is much louder than everything else. It's a cool song, but be careful.

Label me surprised. To my amazement, I recognize quite a few songs on 023. I was certain after 021 most would be new to me. But to my utter shock, quite a significant amount of songs I was already familiar with. And to top it off, it's quite a solid album. I wanted to write everything after 021 off as trash, not worth listening to. But 023 proved to me otherwise. The Passage mix starts very strong, but then at the half time mark it just bombs. I simply hate Rootkit - Concrete Jungle. I mean, wtf is this song?

024 Vanguard

Forefront Album Mix: LINK

Presage Album Mix: LINK

Highlights

• Feint - We Won't Be Alone (feat. Laura Brehm)
• Priority One & Two Thirds - Hunted (feat. Jonny Rose)
• Stephen Walking - OneMan Moon Band

Dogshit album. There I said it. With 023 I had very high hopes to enjoy later parts of Monstercat. Unfortunately, I was greatly disappointed. I boasted about my broad music taste. But there is a specific kind of music, that I vocally hate: Pop music. I hate what it is, how it is made, the people and companies behind it, it's role in society, everything.

As someone who knows how the sausage is made, nothing infuriates me more than people pretending to sound good or talented, when in fact they are not. It's all a large game of pretend. I know one or two people, who sometimes share music with me, and it's always some stupid music video of some person playing the guitar in the video and "their amazing voice", but then the song is the most overproduced bullshit imaginable. Do you know how many effects are put on vocals? More than you can guess. And for what? To sound "natural"? Dude, the moment you put an EQ on a vocal it isn't natural anymore!

I don't hate the effects. Take Draper - Coloured Glass. The pitch correction on that gives it that tactile sound which is quite interesting. Or dial it up to 11 and do what 100 gecs is doing. Layer it, put a chorus on top and you'll end up with something like Fox Stevenson. Everything super artificial, but everything very amazing. No I don't hate the effects. I hate that pop pretends that this is what a natural "talented" singer sounds like. My hate on pop is so significant, that if it remotely sounds "too good", it creates this visceral disrespect deep in my soul. I'd much prefer creative usages of effects, or little to no effects at all.

But this is what 024 is. It sounds too much like pop. And as such I hate it. And the stuff that does not sound like pop is just bland. 024 has one or two good songs, especially Feint - We Won't Be Alone, which is one of Monstercats all time greats. But everything that is not in my highlights is trash. If 001 - 004 wouldn't be so amateurish and low in sound quality, 024 would be bottom tier.

025 Threshold

Collision Album Mix: LINK

Encounter Album Mix: LINK

Highlights

• Grabbitz - Better With Time
• Pierce Fulton & Puppet - Boy and the Beast
• Savoy - Love Is Killing Me (feat. Chali 2na)
• Varien & Mr FijiWiji - We Are The Lights
• F.O.O.L - Need You

I like this album. Very much actually. 024 really gave me the scare, and so did the album cover of 025. The album covers were always somewhat meaningless, but with 025 - 030, Monstercat introduced this evil cat and something of a little story. This scared me, because before listening to 025 - 030, I thought the evil cat represented Brostep. Fortunately, this is not the case, ahem 028 ahem, and I really liked what 025 ended up being. Grabbitz again is absolutely phenomenal with Better With Time. If I wasn't already, now I really started to love Grabbitz and his music. Also I want to mention Karma Fields, which I was very familiar with already, and hold to high regard. His and MORTENs song Stickup (feat. Juliette Lewis) doesn't hit me as odd. It's that experimental unique style that he puts on the table.

All in all, 025 is a very great album, and I am really giving it S-tier. I fully expected to write off all albums after 021 as trash, especially after the 024 scare, but I am delightfully surprised.

026 Resistance

Defiance Album Mix: LINK

Submission Album Mix: LINK

Highlights

• Karma Fields - Edge of the World
• Karma Fields - Greatness (feat. Talib Kweli)
• Puppet - Vagabond
• San Holo - They Just Haven't Seen It (feat. The Nicholas)
• PYLOT - A Race Against Time

Too much like pop, simmelar to 024. 026 is the better album of the two, but not by much. It's annoying, bland and simply forgettable. I don't recommend it. Since I chewed 024 out already, there is not much more to say about 026.

Karma Fields is amazing and definitely the best thing about this album. The others just don't hit as hard. I also want to touch on San Holo for a moment. His music is also very different from anything that came before, and his song Victory is simply amazing. Victory is on 022, but didn't make the cut in the highlights, because the other artists were simply better. Quite a shame. In 026 San Holo easily made the list, because 026 provided absolutely no competition.

027 Cataclysm

Trials Album Mix: LINK

Tribulations Album Mix: LINK

Highlights

• Melano - Traveller
• Marshmello - Alone
• Ramases B - Neon Rainbow
• Trivecta - Into the Limelight
• Notaker - Infinite

Oha. This is the album with Marshmello on it. Honestly, I don't know much about the guy, Alone is literally the only song I know from him. Probably because it was on Monstercat lol. And despite my vitriol against pop, I don't dismiss this song. Honestly, I am even giving it a pass, and somehow, it fits very well inside the entire album. Even though it's somewhat cringe, It's decent as a standalone song.

So while Marshmello stole the show, 027 in it's entirety is nothing that I hate. As such, it gets solid B-Tier. Not above 017 though, as even though there is nothing bad on it, there is not much good on it either. And my 0 nostalgia towards 027 is really dragging it down.

028 Uproar

Tyranny Album Mix: LINK

Supremacy Album Mix: LINK

Highlight

• WRLD - By Design

The bottom of the barrel. We have reached it. I absolutely despise this album. Even more than the atrocious 010 continuous mix. Where to even begin?

I vividly remember EDM going into a very crazy direction, aka Brostep. But as I said already, all my homies hate Skrillex. So, first things first, 028 doesn't start well with the absolute worst drop I have ever heard in my entire life. It isn't fun, rhythmically interesting, or enjoyable in any way. It's just some batshit crazy drop and that's it. The album has more songs like this, and the other half is more pop shit. One questionable song after the other, and it just doesn't stop. Out of all the albums on this blog post, this one I really don't want to listen to.

And the ironic part is, that 028 doesn't want to live me alone. It's in my recommendations to this day. It pops up every day. It probably does this, because I am still flirting with the idea to just listen to it 100 times, so that the repetition-legitimization kicks in. And it does have probably the coolest cover out of all the Monstercat albums, even to this day, with their newer stuff. It's glorious and evil. The red tint really makes it look menacingly.

I noted in 025 how I was scared about the evil cat subplot, because I thought it represented this awful kind of Brostep. 028 is probably the main reason for that fear. The first song just doesn't go down well. I did check on Monstercat a few times, even after 021. But 028 was the last thing before my nostalgia trip. I really thought this is the end of Monstercat, and Brostep took over.

029 Havoc

Fury Album Mix: LINK

Retribution Album Mix: LINK

Highlights

• DROELOE - Bon Voyage
• Koven - Silence
• Aero Chord x Fractal - Until The End (feat. Q'AILA)
• Varien - TEVA833
• Stonebank - The Only One (feat. Ben Clark)

Okay, things took a rather sad turn with 028. But there is light at the end of the tunnel: With my interest into Monstercat completely dead after 028, I was able to hear 029 and 030 completely blind. Literally the only song I knew was Karma Fields - Sweat, because as should be clear by now, I really like the music from this dude. But if you've never heard Sweat before, it really does not help the image of Monstercat going into a weird and bad direction.

However, to my utter surprise, 029 is actually very cool. Instead continuing with Brostep, it pivoted hard into Growlstep. And Growlstep is fucking amazing. Instead of just doing stupid ridiculous drops, it's relishing in the filthy and disgusting basslines and reshapes them into nice harmonics and very catchy rhythms. So while 029 is probably the weirdest Monstercat album between 001 and 030 (maybe even all time, honestly I have no fucking clue about anything after 030), it's definitely one of the better albums. Similar to 013, I now routinely come back to it and I think I've found a new favorite.

But as much as I like 029, it has proven itself to be strange enough. When shared with friends, they demand me to stop and request something else. Funny how that turned out.

Solid A-Tier. Very good album.

030 Finale

Overture Album Mix: LINK

Epilogue Album Mix: LINK

Highlights

• Gareth Emery & Standerwick - Saving Light (feat. HALIENE)
• Darren Styles & Gammer - Feel Like This
• Ramases B - Virtuality
• Unlike Pluto - Someone New (feat. Desi Valentine)
• PYLOT - Shadowtask

And thus we have reached the end. And it isn't a bad ending at all. 030 is quite a stellar album! In retrospect, the evil cat subplot does make sense. Monstercat has gotten somewhat weird and poppy between 025 and 029. But 030 completely escapes that. This album really feels like old Monstercat, but with modern high-quality mixing. So even though I knew literally no songs on it, it did feel somewhat nostalgic.

I want to mention the squeaking chair in Hush - Pilvorm, which I find absolutely hilarious. I don't know, something about originally irritating, but looping sounds, which blend into the music, are just very fun. Repetition legitimizes after all. Reminds me a lot of Cowboy Bebop OST 1 - Piano Black.

The absolute highlight here is the last song, Gareth Emery & Standerwick - Saving Light (feat. HALIENE). It's so triumphant and confident, tying the story together and showcasing the last victory over the evil cat. But it also marks a change, as this song definitely sounds more mainstream than anything else on the label. Also it's absolutely bathed in reverb. What is this? SLIFT?

Bonus <3

Uncaged Vol. 1

Album Mix: LINK

Highlights

• Unlike Pluto - Everything Black (feat. Mike Taylor)
• Darren Styles - Us Against the World
• Stonebank - Ripped To Pieces (feat. EMEL)
• Duumu - Illuminate (feat. Styleaf)

After 030, Monstercat rebranded. This was probably for the better. As 025 - 030 demonstrated, Monstercat wanted to bring in many new people and styles, and multiple channels allowed them to specialize. Monstercat also got rid of the 2 mixes and combined them into one. I was always listening to both mixes anyway. Highlights were pretty difficult to find for this album, because Monstercat has perfected their mixing to such a level, that it's very difficult to tell songs apart. Where does one song end and where does another start? As someone who doesn't know any of the songs, the whole mix really does feel like some continuum, and I condone that.

However, even though I did give some albums after Uncaged Vol. 1 a fair shot, I decided that Monstercat simply isn't for me anymore. I have my nostalgia, and will go back to 007, 018, 016 and 025 many times, but everything after Uncaged Vol. 1 doesn't interest me in the slightest, regardless how good the music may be.

Before I leave, I want to mention Unlike Pluto - Everything Black (feat. Mike Taylor). When Mike sings "black diamonds" in the buildup before the drop, I mishear "black garbage". I can't be the only one, no? Doesn't change that the song is an absolute banger though. But I find it it still hilarious.

Conclusion

And with this, my thoughts are concluded. Below is a tier list, combining every of my thoughts into an easy to reference image. Tracks to the left in the same tier are considered to be higher. I.e. 007 is better than 018.

The real treasure was the friends we made along the way.

Since I can remember, I was fascinated by video games and how they work. I want to make my own. A few years back, it was only natural for me to get sucked into the game dev rabbit hole. I did what the internet told me. I participated in game jams, engaged in subreddits, watched other people talk about game design on YouTube. But over time, my views drastically changed. So much so, that I started to write my own game engine, something that no one recommends. But with how my views changed, this is the only path that I want to take.

Many beginners are not aware that making a game is a huge endeavor. How difficult can it be to draw a triangle on the screen? The community is aware of this issue. To counteract, their advice is tailored to keep the expectations low and the workload manageable. This directly leads to many corners being cut.

For example, People recommend to build your game with preexisting, all-purpose game engines. Don't focus on project or code architecture and just get things running. Don't reinvent the wheel, use the first library or asset you can find that does what you want. Fail fast, i.e. don't learn upfront, don't try to understand how things work and only investigate when things fail. Burn yourself out in game jams, to gain experience. And the best of all: Don't work on what you want. Work on smaller things that neither you, nor anyone else cares about.

I presented these a bit mean spirited, but I can't help but feel that this advice is harmful. First, it generates terrible code, that is neither maintainable nor performant. You'll struggle a lot to get things running and you will bite your teeth out when you hit awful bugs. Second, you will learn slower, as per this advice you are not actually trying to understand what you are doing. And third, it does not make you happy, because you will be spending more time on focusing on finishing a project, instead of enjoying the process itself.

Let's play devil's advocate. Assume the worst-case position for my argument, that I've constructed a strawman. Assume I misrepresented what everyone is saying. And assume I did not address the actual advice, which still floats around in the community. In this case I want to ask the question: Why is there so much failure in this community? Every day in different parts of the web, people share their post mortems, about how they are burnt out, how much work all of this is, how it is not fun or that their end product was not as successful as they anticipated. At one point, one should stop and ask themselves: Surely, the failure rate would be much smaller, if this advice actually works, right? However, if there is a reason why people fail, why then is the advice not tailored to this reason?

The answer is, as I've lined out, that the advice is essentially harmful in itself. I don't assume bad faith. People are not evil, and I think what they are trying to say is well meant. But because of how communication on the internet works, well meant advice got distorted. It morphed into unhelpful buzzwords and short, snappy phrases, which misguide the beginner.

For example, take the advice to not work on what you want, but on many small things. Most big successes are from people who already have years of experience. Usually, they worked a long time in the field already, or boast a portfolio of smaller projects. The wrong way to connect the dots is to assume more games equal more success. The reality is however, that people did not gather all this experience to have one single success, but rather because it is their passion. The huge work they put into it is because they can and want to. When they do eventually have a breakaway game, usually it's because of that exact experience, which allowed them to put all their strengths into one unfathomable good product.

Or take for example the advice to not reinvent the wheel. People recommend this, because programming is a wide and deep field. You can pick a single subject and spend your entire life only studying this one thing. So, when you are doing a project, and you need a solution for something you don't fully understand, you can build on top of existing knowledge. You can save a lot of time and energy, by using other people's work. But this advice gets distorted to never try to build things on your own. It leads to people not wanting to do stuff, and them assuming everything is solved nowadays. In the worst case, people really don't understand what they are doing, and they burn out on the fact that there might not be a solution to their problem.

Knowledge is like a drawer. Whenever you learn something, you put something into it. When you forget, you don't really lose anything. Forgetting is more like losing the note on one of its doors, such that it's difficult to tell what's behind it. However, a circumstance or event can trigger you to look into it, allowing you to access the knowledge. It goes without saying, the more knowledge you have, the larger your drawer becomes. Over time, you will burn stuff inside it. Mostly details, for example mathematical formulas or syntax of a programming language. But that's not necessarily a bad thing. You may forget details, but you retain the concepts. Making room for different knowledge allows you to use the space more efficiently. You can always look up details later.

Students often ask, why should we study, when everything can be looked up? The simple and unsatisfying answer is that you don't know what you don't know. It can feel like a real waste of time, when you don't see how a given knowledge can be useful. But putting more stuff into your drawer is always useful, because you look inside it every single day. It affects how you think. Let me ask you this: How exactly are you looking up a mathematical formula, when you are not even aware of its existence? How exactly are you solving a problem, when you can't even pinpoint what the exact problem is? Sure, you can ask people online for help, or one of the hot new text generators. But when your problem is too niche, even they will struggle to help you.

And this is why I take great dismay with "don't reinvent the wheel". I am not reinventing anything. I am learning how things work. I look up what other people have done and then give my own spin on it. I am putting things into my drawer, growing it in the process. When you lose yourself in the mindset to never solve problems and always use preexisting solutions, you will struggle. Period. A related issue is what the community calls "tutorial hell". People who are in tutorial hell follow too many tutorials too closely, never deviating from them. People in this situation really struggle to get the ball rolling on their own, because they miss the concepts to do so.

So, what about me then? I am talking out of my ass and I have yet to say a single thing about my game engine. Well, after 2 years of work my engine is far from being in a presentable state. It looks primitive. And what it can do is easily achieved by a few clicks with every other engine on the market. At this point, I am imagining that this whole project will take an even greater portion of my life. But God damn, it's the single coolest thing I've ever worked on, by a long shot. I take great pride in saying "yeah, I did that", and I can't do that for the things I've done in Unity. I understand and have full control over my code. I actually do own my code, which isn't given in today's copyright hellscape. Literally the only downside to writing your own thing is the huge amount of time and effort required to do so. But I am planning big (maybe too big), so I am going to spent a lot of time and work on this anyway. By writing my own game engine, I am just front loading this work, experiencing it now instead of later.

Of course, I have this one great amazing innovative never seen before best of all time game in my head. I hope it will succeed, like everyone else does for theirs. But it's not my primary incentive anymore. I am having fun. And I amassed so much underlying knowledge and experience, that I feel I can work on anything now. I don't think I could've said that 2 years ago. Ultimately, you'll only learn by going out of your comfort zone. And by doing something as difficult as this, I am practically always outside my comfort zone.

I am not really trying to convince you of writing your own game engine. What I am trying to say is exactly the cheesy quote at the very top of this post: The real fun is the things you learn along the way, not the end product. You should enjoy what you are doing, using the end product as a justification to do so. And in my case, this is writing a game engine. For you, the end product may not be some giant undertaking, but at least it should be something you care about. And you have to learn. Always learn. At least try to understand why your copied solution works the way it does.

If you understand what you are doing, and have fun doing it, then you will have a much better time. You will regret nothing.

Unfortunately, I did run into a very severe bug, which is able to deadlock the entire system. Considering I made my JobSystem a big deal, I feel obligated to share you this rather interesting bug 😊

The Bug

Without going into the nitty gritty, here's a basic rundown of what the JobSystem does and how it works:

The JobSystem allows "Jobs" (function pointers, closures, lambdas, anonymous methods, or whatever your language of choice calls them) to be submitted to a buffer. Worker threads dequeue Jobs and execute them. With one worker per CPU core, this system is able to utilize all cores of your CPU, without the user ever needing to spawn additional threads for concurrency.

The submit function is global, and thus easily accessible. The pseudocode looks something like this:

First, a future is created. A future is a utility, that allows client code to wait and retrieve a Jobs return value, once it's done. The job is wrapped into another job, such that the future can be set. Then the wrapped job is enqueued. If the enqueue failed, the Job is simply invoked. Finally, the future is returned.

Unfortunately, this code is based on a flawed assumption: If a Job failed to be enqueued, then calling it immediately may lead to a deadlock.

Let me elaborate: Enqueuing can fail for 2 reasons: Either because the buffer is full, or because another thread holds a lock inside the buffer. These failures are by design. They enable a lock free behaviour and allow predictable performance. However, this comes with the consequence, that we may have a Job which was not enqueued. And now we need to decide what to do with that Job. When I wrote this code, I naively assumed I can just execute the Job. Afterall, as long as the system is making progress somehow, everything is fine. The client code should not care whether the Job is executed now or later.

But the following code breaks:

If an enqueue fails, this leads to the Job being executed directly. It's as if the call to submit isn't even there. The thread responsible for setting the future, the thread itself, will never set the future, because it's waiting on the future being set. Deadlock.

The Fix

Unfortunately, this has no satisfying solution. My engine is written in Rust, and to say it mildly, I am really struggling to get safe global state working. My latest prototype produces client code like this. It could be argued that the client code is unsound, and it shouldn't be legal in the first place, but because my latest prototype is somewhat promising, compromises have to be taken.

My current fix looks like this:

It's very similar to the original code. Basically, submit the wrapped job until it succeeds. When the enqueue fails, call run_pending_job(). This function is a utility, allowing client code to run an enqueued Job. This is helpful, because if the current thread cannot make progress, the entire system can still make progress by working on something different. If no enqueued job exists, this function yields, freeing up the core for another thread.

This solution is not ideal, because this can still deadlock. It is still invoking Jobs, which may block the calling thread. There is always the possibility to simply not invoke a pending Job. But that would lead to a busy spinlock, using CPU resources without making progress. Admittedly, it would be safe, but there is the potential for a performance hit. Considering that my flawed clientcode only exists in debug builds of my engine, I am preferring the risk of a potential deadlock over a potential performance hit.

An more sophisticated solution would be to have a dedicated dispatcher. It would assign each Job an ID, telling them on what threads a Job is allowed to run on. But this requires more time and effort. And at the moment this bug is not justification enough for a dispatcher to exist. So I am sticking with the less likely, potential deadlock.

Conclusion

I am kind of amazed that it took this long until I found a major bug in my job system. In my previous post about the JobSystem, I emphasized how important it is to stress test something as critical and central as this. It only goes to show, how much good and extensive tests prevent bugs. Since then, I've also integrated miri in my testing, which gives me even more confidence in the stability of my system.

It also highlights once again how reliable Rust is. When I am working with C#, implementing something similar, I can't tell you how many times my code breaks just a few weeks or days after implementation. But nevertheless, my Rust code did in fact produce a bug. And unfortunately, that means I must take down my little PHP script on my original post 🥲

Nonetheless, I did get some value out of it. A single commenter pointed out that actually, Rust does have a utility to capture the stack trace. Well, it isn't built into Result, and you still need to call the function manually, but it exists. And poof. My biggest argument against Result just evaporated.

Capturing the stack trace allows for some beautiful ergonomics: Calling file!() and line!() isn't as important anymore, as all information is on the stack trace anyway. This means, no macro for conversion between errors is necessary, and now it actually makes sense to implement a generic From, that converts every std::error::Error into a RisError. Sure, it adds one more frame to the stack, but who cares? The entire thing is captured anyway ¯\_(ツ)_/¯

I find it quite silly that so many people recommended to just use some crate. Especially since the entire implementation, plus some utility, fits into a little over 100 lines of code. Again, it was just a single person who realized the actual problem. I am really getting is-odd vibes from this situation.

Enough condescending talk about redditors. Here's the new, better RisError:

GitHub permalink

There are only two kinds of languages: the ones people complain about and the ones nobody uses.

I use Rust and I plan to continue, so I've earned the right to complain about Rust. But this won't be your average Rust vs Go blogpost. (Why are there so many of those? What the hell?) I won't be complaining about the borrow checker. Neither about traits, lifetimes, macros or compile times. These are fine. No, I will be complaining about Result. That is right. I will be complaining about one of Rusts most beloved features.

Rust prides itself on its robust and strong type system. If a function returns an object, you can be 99.99998...% sure that this object actually exists. (I am not saying 100%, because unsafe Rust exists. And who knows if some dependency you are using is doing some funky C++ shit under the hood.) If this object may not exist, it will be wrapped in an Option. Before you can access it however, you must first check, whether this Option actually holds your object. And if it doesn't, your object really doesn't exist and you cannot access it.

Result is quite similar, but a notch above Option. Like in Option, your object may or may not exist, but if it doesn't, it holds an error. This error (hopefully™) describes why the function couldn't return your object. And just like Option, before you can access your object, you must first check whether it actually exists or not.

Everything is safe and sound.

But then you start to use Result, and it turns out it's pretty clunky to use and just plain annoying.

The problems emerge

First of all, a lot of things can fail. Let me repeat: A LOT OF THINGS CAN FAIL. And you need to handle each and every Result you encounter. This means, everyone gets a match-statement, and bloated boilerplate ensues.

"Aha!" says the Rust evangelist, "Simply use the ? operator!" Ah yes. That removes the match. But it introduces more complexity somewhere else: Now the client function must also return a Result, and its error type must be the same as the calling function. This introduces problems the moment you call functions that return different errors. Now you'll have to convert errors to other errors, otherwise the compiler won't be happy.

"Well, just use From!" says the evangelist. Sure, let's hide the boiler plate code somewhere else, because this obviously fixes the issue. Also, do you really want to implement From between all the hundreds of error states, that your program can find itself in? I don't think so.

"I hear you, I hear you. What about map_err() then?" I mean yes, that works. It's obvious, straightforward and understandable. But it becomes quite annoying when you have to type it hundreds of times.

"Ehh, unwrap?" Cool. Why even use Rust in the first place?

Ultimately there is no silver bullet, and you will most likely use a combination of these solutions.

It keeps on giving...

Okay, usability aside, what else is there to hate about Result? Here's something: They give little information about what actually went wrong. Ideally, you want the error to implement std::error::Error, such that you can log it or something. You probably also want it to be an enum, such that it gives you a human readable and computer friendly way to tell what went wrong. Boilerplate over boilerplate.

Look, for the API of your library, it makes sense to cover all possible failure states. But from my perspective, building a game engine, an executable, I only care whether the given operation succeeded or not. And when it fails, it should tell me where and why. Result simply doesn't provide that information. The uncool thing about Result is, that anything can be an error. The error may not implement std::error::Error, and it may not be an enum. The SDL2 crate for example made the amazingly annoying decision, to only return Strings as errors.

Compare that to exceptions in C#, which I am very familiar with. When an exception is thrown, you get an entire stack trace. You know which function called which, and it's usually fairly easy to pinpoint where and why an exception was thrown. Not so with Rust errors. When a "stream read failed" error bubbles up to my entry function, my first reaction is annoyance, because I have no idea where it occurred. It's not like I don't read streams in like 100 different places, no?

In Rusts defense however, when building C# as Release, most debug symbols are stripped away, and you get error messages that are undecipherable and less useful than C++ linker errors. Because you know, the linker at least tries to tell you what went wrong. Exceptions in C# Release builds just spit digital vomit at you.

The antidote?

Nonetheless, all problems I've described thus far, drove me to the point where I attempted to solve them by my own custom error type: RisError. It stores a message, it's file and line where it was created, and the std::error::Error that caused it. The file and line are not as useful as an entire stack trace, but it's better than nothing. I also created a few macros, that make creating and converting errors into it easier. Now, everything that fails returns an RisResult<T>, which essentially is just a Result<T, RisError>. Thus I can use ? everywhere, and use macros whenever something is not an RisError.

At this point, you may notice that this looks awfully like exceptions. Abort execution at anytime... Possibly (hopefully™) catch it somewhere up the call stack... Pretty much a goto, but god knows where it will end up... And you may ask, why even go through the trouble and not just use exceptions in the first place? To that I say: RAII still holds. And RAII is very useful.

With this system, if an error occurs, I can still execute code. This allows me to gather system information, properly log failure states and ensure that the current state of the game is saved, such that no progress is lost. A try-catch (or std::panic::catch_unwind as the Rust people call it) doesn't work: Building as Release, exceptions are disabled and they are turned into aborts.

Even if I enable exceptions in Release, or are somehow able to catch aborts, working with exceptions in memory managed languages is infamously difficult. It's so difficult in fact, that people coined the term exception safety. If I keep avoiding exceptions, everything that implements Drop will be cleaned up properly.

And it is now, where I have to mention, that I am absolutely annoyed by the fact that the Rust people decided to rename exceptions to "panics". When talking to non Rust people, I always have to preface: "Ackchyually, Rust has exceptions, they just call it something different".

I understand why the Rust people decided to rename them. Because exceptions are rarely exceptional. If you are anticipating an error with try-catch, is it really an "exception"? But the irony is, in my case, exceptions now really represent exceptional state! It's state that I did not account for. State that allows the engine to crash without properly shutting down. It's so backwards, and I hate it. I hate it so much.

Conclusion

If there will be a successor of Rust, I'd wish for an error system in between of Result and exceptions, which goes alongside the other two. Something that generates a stack trace and unrolls like an exception, but allows RAII to properly free all held resources.

The compiler is smart. Couldn't the compiler generate a stack trace at compile time? I mean like generics, it could just figure out all the functions that call an error, and just place that function path as a compile constant string into the error, no? But I guess recursion must then be forbidden, because that damn halting problem exist. God fucking dammit. I curse you, halting problem!

Nevertheless. As much as I hate my current solution, it is usable. And that is all I need for now....

Because people will be asking, below is the implementation of RisError. I called the conversion macro unroll, to differentiate it from unwrap, and because RisErrors can be chained, like exceptions unrolling a stack.

GitHub permalink

I was trying to change this, by actually doing a beginner friendly intro to Quaternions, but it turned out to be much more work than anticipated. I made this visualization tool, which is something. It's inspired by the book "Visualizing Quaternions" by Andrew J. Hanson and it's in my opinion better than any other visualization that I've found online.

Unfortunately, the tool is rather worthless if you don't have a mentor that explains it to you. So the idea was that I also make a video to specifically explain the tool and give a good, simple explanation of quaternions. An explanation that isn't riddled with formulas, doesn't require an understanding of the 4th dimension, and doesn't ask of a deep understanding of mathematics. Such an explanation is indeed possible: I've held a presentation at my job to do exactly that. But the slides are rather low quality, and use many questionable pictures which violate copyright. (On that note, fuck copyright.)

If I wanted to make something that would go on YouTube, instead I had to make custom pictures and diagrams, write a proper script, record and edit a video. Problem is, the visualization tool already took a month of work, and this video will take probably another. Also also, I advocate to use Quaternion.AngleAxis and Quaternion.LookRotation and I discourage everyone to use Euler Angles ever again. But I want to keep things as general as possible. I don't want to say "just use the provided functions by Unity". Instead I want to provide proper formulas, because 1: Not everyone is using Unity. And 2: People may want to write their own quaternion library, and this just doesn't fly.

This is all a lot of work.

It is at this moment, where I truly realized the difference between people who make, and people who talk. On YouTube you find a lot of people who talk about how to program, how to make a game the "proper way". But often the advice these people preach is shallow and meaningless. The worst kind of these people are video essayists and critics, who talk about what makes a game good, but never made or even considered making one. On this note, huge respect for Mark Brown from Game Maker's Toolkit, who actually tried to put his knowledge to the test and tried making one.

This whole makers vs talkers thing reminds me about the book "Clean Code" by Robert Martin, which is full of questionable advice, which the author himself doesn't actually follow. It also reminds me about No Man's Sky, how it was ruined by the fact that they talked about it too much, but then fixed it by shutting up and actually fixing it: https://youtu.be/O5BJVO3PDeQ

I decided that I want to be a maker, not a talker. As such I scrapped the idea of a video, and instead focus more on my engine. So here's what I've been up to: I implemented a basic Vulkan based 3d renderer into my engine :)

Because of this, my brain is now full of transformation matrices instead of quaternions. Next steps would be build on top of it more. At the time of writing, I am missing index buffers and depth testing. Then I would also want to start with my resource system, such that I can actually import 3d files. Then I would also want to put my shaders into files, such that I can recompile and hotswap them at runtime. Nice stuff.

Until then, radio silence 🙊

First off, I can't make backups. I have yet to experience an SQL Injection Attack, but eventually someone or something will drop all my tables. This is no big deal for my blog, since I store according files separately. In the worst case, I have to fill in the database by hand. But with emails, I can't do that. Assume for a second my tables are dropped, and in that exact time you want to unsubscribe from my newsletter. If I had a backup and restore it, and send the newsletter for a new blogpost, suddenly you receive an email from me, even though you unsubscribed. This is an opportunity for you to sue me, and I don't want to be sued.

Also, people who want to steal the data are free to attack me. I am no IT security professional. I do have basic knowledge, and I did put some safety measures in place, but I am not arrogant enough to say that my system is failsafe. The systems I had were good enough to an extent, such that to my current knowledge, I haven't leaked a single email. But if I do manage to leak an email anyhow, I am the one who is responsible. Again, you could sue me if that happens, and I really really don't want to be sued.

😓

All this is pain. And it has led me to delete my newsletter. At the time of writing this post, all emails stored on my database are already eradicated. Maybe, if the development of my game engine is far enough, such that I can release a working product, I may introduce a newsletter again. But then I am probably in a position where I would pay a professional service for that. There is no way I am ever touching something sensitive like storing user data ever again.

While cleaning up my legal mess, I used the opportunity to get rid of Disqus as well. I am proud to say that I don't use trackers or cookies. I think that's honorable, considering that tracking people has become the norm. And this non-tracking also allowed me to not show you an annoying popup whenever you visit my website. For Disqus however, I displayed a warning that they will track your data. I feel also very proud about this solution, as this thing doesn't even load Disqus, when you don't press the button.

Unfortunately, no one uses the comments. Probably because I have no reach, but probably also because that warning deters that lone wanderer that found me. If this is an indicator for anything, then that people simply don't want to be tracked. Therefore, Disqus serves zero purpose, and I deleted it as well.

I also updated my privacy policy, to reflect these changes. And to make it more user friendly, I added a section at the top for non-lawyers. A less dry and quick summary of the whole document.

After these adjustments, I am very happy about the state of my website. A major headache gone. Now literally the only thing that collects data is my contact form. But since this sends an email, it should be pretty obvious that this is data that is stored somehow somewhere, and I think there is nothing controversial about that.

Now my website is exactly how I always described it: A blog and project-collection.

I had a lot of doubt in the beginning. During my time studying, I was making music on and off, it was never a big deal. But then I had a job, and suddenly the free time was gone. The game engine is definitely a life goal, so I am definitely going to pursue that. Though, whenever I was working on the engine, there was always this nagging thought at the back of my mind: "Bro? What's up with music? You haven't opened FL Studio in 2 years!" It drove me crazy. Purchasing gear and feeding the GAS didn't help, and it made the doubt so much worse.

In October I finished my JobSystem. While crude and fundamental, it works. But I still had to write the blogpost, which was just. So. Much. This JobSystem and its accompanying blogpost are possibly a master thesis on their own, but fuck it. A degree is nothing but a piece of paper. Once the post was finally done, the literal next step of my engine would be the 3d renderer. I really hit a milestone, and it was the perfect moment to take a break.

And now, after the album is composed, mixed, mastered and uploaded, I want to document, how I made the album.

The Polybrute

Let's start with this baby, the work horse. It's responsible for the vast majority of sounds in the album. But god damn it, it is such a diva. If you are new to synths, there's this preconception that analog synths are "warmer", "better". I mean, sure the thing is much more fun than a VST on your computer, and the Polybrute definitely sounds different from anything out there, but this god damn thing just didn't want to stay in tune.

I have the Polybrute for almost 2 years now, and one year ago it started to be funny. It went out of tune so badly, that every sixth voice wasn't played in the right ear. I haven't turned it on for months, but then I fired it up in October. And would you know it? No tuning issues whatsoever! The thing acted like it never had any problem. I felt like I was being gaslighted by a friggin synthesizer. But then winter hit, and suddenly the tuning issues returned. This was such a big problem, that I had to plan my entire day around it.

First, I never turned the Polybrute off. Second, I heat my studio to 24,5°C (32°F). I know I know, this is ridiculously warm. And if you still believe in the carbon footprint, this temperature is ludicrous, but it was necessary to keep the Polybrute in tune. Also, I always kept the door of my room shut, to get a more consistent temperature. Once these preconditions are met, my average day looked like this:

1. Wake up.
2. Ventilate my flat (now everything is cold and the Polybrute is unusable for the next 1-2 hours.)
3. Go to work after closing the windows.
4. Back home, tune the Polybrute.
5. Make music.
6. Ventilate the room (everything is cold again).
7. Go to sleep.

Most weekends I would only ventilate on Sunday evening, because waiting for the synth to warm up and then tuning it took just way too long and was always such a hustle. The Polybrute has an automatic tune function, which would've been awesome, if it would work... I had better luck disabling the tuning and just stick with the calibration. Remarkably, the VCOs never went out of tune. You know, the thing that people praise about analog synths, that their pitch is slightly off and thus produces these warm and lush tones. Yeah, the VCOs were always perfectly in tune, but the Steiner Filter and the VCAs refused to stay in line.

So, afte all these inconviniences, does it sound good? Well, pretty much every sound on this album, with notable exceptions, is made the Polybrute. It's the fat farting bass synth, and the screaming, grainy pad in "02 Insecurities". It's literally every sound in "03 Chasm Pt1", even the organ! It's the tremolo bass in "04 Chasm Pt2". And "05 Nightmare" pretty much reuses most of these patches, only with the addition of an arpeggio and a solo lead sound. The Polybrute sounds amazing and perfectly fits my taste, it makes all the tuning issues absolutely worth it :)

Note to self: Always make music during summer.

The other instruments

While the Polybrute was the bread and butter, I made use of a handful of other instruments. Most notably:

• 909 samples for the drums,
• Instance of 3xOsc, only used for risers,
• Pianoteq, for the Piano (duh) and
• A sample-based choir (it's the default one in FL Studio 20).

I pretend to be someone who listens to all genres. Even though I try to listen too as many different styles of music as possible, if I had to pick a favorite, it's probably DnB by a longshot. And my favorite subgenre therein is Intelligent DnB. While I love the sampled, shuffled break beats in the music I listen to, I somehow don't want to use that in the stuff that I make. I really prefer programming my drums, no matter how fake or MIDI they sound.

Up until this album, I've always used samples of acoustic drums. But this time I wanted to try using electric drums. I chose the 909, which I think sounds the best (heresy to the 808, I know). But only after I was done with the first track, "02 Insecurities", did I notice that I really don't like the sound after all, and that I still prefer my drums to be acoustic. But now I had a dilemma to face. I could go back, replace the 909 with acoustic samples and mix the drums again from the ground up. Or I could simply use the finished mixed drums for everything thereafter. I chose to not re-mix my drums. Though I am a little bit bothered by it, I don't really hate it. So it's fine.

While keeping the same project file for multiple songs reduced much mixing work, organization pretty much flew out the window. In the end, instruments weren't in the tracks they were supposed to go, and I had patterns named "solo #24" or something, or like 100 separate vocal recordings. It was insane. Nonetheless, with everything being mixed already, every new melody only had to be programmed, which saved a lot of time :)

Let's talk about the other instruments. The instance of 3xOsc isn't remarkable. I used it for risers, because risers are trivially easy to make with sliding notes. But only internal FL Studio synths support sliding notes. Maybe if I would've been less lazy, I could've made some really sick risers with the Polybrute.

Pianoteq isn't remarkable either. It's a good sounding piano, with the benefit that it doesn't clog up my computer with piano samples. I don't care much about 100% authenticity. It sounds like a piano, as such the small size is a big plus. In case you are interested in specifics, I used Pianoteq 7 with the Grand C. Bechtstein DG piano.

The choir isn't remarkable either. It literally is the "Choir Ahh" preset of DirectWave. It doesn't sound good, but it doesn't sound particularly bad either. Again, I don't care about 100% authenticity. It sounds like a choir, and as such it is sufficient.

But you know what also isn't remarkable?

My terrible singing

In the middle of January, I bought some acoustic material! To kill the awful awful reverberation in my studio. I don't own much furniture, that you would find in the typical home. No couch, no cupboards, no bookshelves. Just a desk, a computer and audio equipment. As such, the acoustics in my room were absolutely terrible. The reverb was around 5 seconds long, which is quite amazing, when you consider how small my room actually is 🙈

But I did my homework, and I know that gluing things on my walls is more damaging in the long run. So instead, I bought (stupidly expensive) general absorbers that can be mounted on microphone stands. If you were to permanently install acoustics, you can get much more and better stuff for a couple of bucks. If you are a DIY person, you can also build absorbers yourself, which apparently is very easy. Building them yourself is even cheaper than the cheapest stuff you can buy. But I'd cut myself even just thinking about using power tools.

But the garbage I bought has the obvious benefit that I can move them around. A recording booth on the go. Even though the acoustics in my studio are now much better, I still have my PC in my studio, which my microphone picks up. So, instead I simply moved the absorbers into my bedroom and recorded my vocals there :D

During the making of the first song "02 Insecurities", I didn't have these absorbers yet. Instead, I sang into my closet. If you pay attention, you can actually hear it. It sounds like I am singing into a wooden box :P

But thanks to the absorbers, in "04 Chasm Pt2" and "05 Nightmare", the acoustics are much more clean. As clean as the recordings are though, I still hate how my singing voice sounds. I never practiced singing, so that's probably the biggest problem here. Nevertheless, to "fix" it in the last part of "05 Nightmare", I've put heavy EQ and a bit crusher on top, so now it sounds like I am singing through a telephone. In retrospect, going to all the effort to get squeaky clean recordings, just to ruin them with effects after, was a big waste of money. Oh well, at least with these absorbers randomly standing around in my studio makes me feel like a professional now ¯\_(ツ)_/¯

With the acoustics taken care of, now I only had to actual sing. I simply programmed a piano to play the melody I wanted to sing. As a metronome, I used the 909 hi-hats, and that's it. Something like this:

You can hear how crude it is. I even have the metronome lasting for much longer than was actually needed 🙈, but it does its job. I had one of these tracks for every part I wanted to sing. I downloaded them to my mobile phone, and listened to them with headphones.

For the last solo, where I actually wanted to sing lyrics, I had to plan out the words and how they fit with the melody. I did a lot of experimenting, writing down the lyrics in a word document and programming a piano parallel to it. I needed like 2 weeks to get the lyrics done. Once I was happy with them, I printed them out for quick reference. That's the sheet of paper lying on the drying rack, in the photo above.

Finally, I had to mix them, which wasn't that hard honestly. EQ, compression, reverb, the telephone effects. I don't want to pretend that I am a good singer, so be rest assured that I used autotune. (In the industry, we actually call it "pitch correction", and FL Studio calls it Newtone.)

Sampled quotes

In "02 Insecurities" and "04 Chasm Pt2" I definitely sampled what other people were saying. Did I breach copyright? Possibly. Do I care? No. I have little respect for copyright, much less of big companies abusing it. I won't write an essay about abolishing copyright law, but if you are interested from where I am coming from, here is a detailed take.

Okay, opinions about copyright aside, I am actually going to credit where I got the samples from. The first quote is from John Carmack, on an Interview with Lex Friedman. Carmack is a legendary programmer. He is the founder of id Software and he is probably most famous for being the programmer behind Doom, the original from 1993.

The second quote is from Brian Karis. He is the lead developer behind Nanite, the feature in Unreal Engine 5, which generates LODs at runtime. I actually only learned of his existence because of Nanite. But considering how stupidly impressive Nanite is, this dude got brains.

The third quote is by a man which I honestly don't know. I've come across this clip of his on Reddit some time ago. I kinda vibed with it, with the insanity and determinedness behind the voice. Though I had to alter the audio quite a lot, removing the censored bleeps and rearranging it into something more coherent.

Also also, I really want to credit all the ideas I stole. I am mentioning it, because there is a high chance you missed it, but the melody in "04 Chasm Pt2", which is played at the same time as the enraged man rants, is not mine. It's from Undertale by Toby Fox. More specifically the song "But the Earth Refused to Die".

All other references and similarities are pure coincidence, or simply because I included them unconsciously. Pure originality doesn't exist anyway.

• John Carmack: https://youtu.be/I845O57ZSy4?t=18426
• Brian Karis: https://youtu.be/NRnj_lnpORU?t=2933
• Enraged man: https://redd.it/sj210o
• Toby Fox – But the Earth Refused to Die: https://youtu.be/ML6OV8fG74w

Thumbnail

To say it bluntly, the cover art was generated by AI. Specifically: I used DALL·E for the job. Unlike previous albums, I had absolutely no ideas for a cover. So I threw some keywords at DALL·E and watched it go.

First I tried "drum and bass album cover, black noise", but I really hated that DALL·E tried to put text on the cover. Even specifically adding "without text" to the prompt, DALL·E just couldn't help itself to add text.

Then I tried "black digital noise" and later specified it to "corrupt noise, gloomy athmosphere, without color, abstract, cartoonish". While generating super cool textures, it didn't scratch my itch and thus I threw the idea out of the window.

My third attempt was "speaker of a subwoofer, distorted, high contrast perlin noise, cell shaded 3d graphics", but I didn't like these, because they were too basic.

But then I hit gold, once I used the prompt "the world through the eyes of a schizophrenic person, in black and white", and beside the two creepy looking people, the other two pictures really hit the nail on its head. It was a really close call between the dude with the missing head and that distorted forest, but ultimately ended choosing the forest, because the missing head was a little to depressing for me.

Below is a selection of the thumbnails I generated. I left some out, due to them being very similar to the ones you already see here.

I was quite lucky actually, getting very good results on my 4th attempt or so. For example, the thumbnail for my engine required lots and lots of trial and error. I spent like 3/4 of my credits for colorful engine pictures, but only like 5 for my album :P

Up until this point, the album also didn't have a name yet. After seeing the headless dude and the forest, I stumbled onto "The World Between My Mind And Reality". I think it fits :)

Miscellaneous

Finally, let's talk about some miscellaneous stuff. Things that I want to mention, but for one reason or another could not or simply didn't want to put in any category above. So let's go:

The big swell in the beginning is a simple Piano key stroke, but with many many effects on top of it. So many in fact, that it generates these interesting noisy artifacts.

The organ sound in Chasm Pt1 and 2. That is 100% the Polybrute. I used modulated noise to generate the wind sound effect. With the longest possible release on the VCA envelope, such that the noise can ring out. But to stop the VCO when a key isn't pressed anymore, I used the MOD envelope to modulate the VCO mixer.

That sound after John Carmack and Brian Karis, is the Polybrutes delay effect. If you don't play any sound, and put the feedback of the delay up to 100%, the noise floor eventually gets so loud that you can hear it. Once it's at maximum loudness, turn down the delay time to get that pitch shift. Once I recorded it, I only had to correct its pitch, such that it is in tune with the rest of the song.

The first song "02 Insecurities" is never silent. I recorded the ambient noise of my PC and placed it into the song. In the later songs I didn't do that, because first, the organ sound is noisy enough already. And second, the last song "05 Nightmare" has so much sound stacked on top each other, that it was practically meaningless. I wanted the silent parts in it to really be silent.

Conclusion

And there we have it. Now obviously, I didn't go into much detail, like how I mixed and the exact plugins I used, because quite frankly, I find that boring. And honestly, at the end of the day, mixing is more decision making. If you want to learn mixing, you probably should watch Dan Worrall instead.

As for what the album means and what I want to say with it, eh. It is what it is. Death of the Author or what have you. I let the art speak for itself.

Anyway, there is one idea that really caught me on fire, ever since I first came across it. Pretty much most game engines that you will come across, as well as any tutorials that may claim to teach you how to program one, they all hinge on the same basic, single threaded main loop.

This is fine, until you realize that today's machines have multiple processors. If the entire engine runs only on one thread, then no matter how many cores your processor has, only the speed of one actually determines how fast your game is running.

Naïve me, and multiple people that I've seen online, all seem to come to the same idea eventually: Simply divide the engine into different threads! For example, have one thread that runs the input and logic, and have another thread that runs the output. But this is not an ideal solution. One job will be more expensive than the other. Either the logic, or the output (most definitely the rendering) will take a larger amount of time to compute. So to have things synced up properly, one must wait for the other, wasting CPU cycles. To solve this, maybe you think you simply have to divide the engine into even more threads. But this becomes very complicated very quickly, with diminishing returns.

No. What I want is what Jason Gregory proposes in his book "Game Engine Architecture": A Job System.

The idea doesn't sound so difficult at the first glance: You spawn a thread for every available core. You have a queue where you can push jobs to. The worker threads then constantly try to dequeue jobs from it, and run them. A job is simply a piece of code, usually a pointer to a function. The potential of this idea is huge: In the best case, it gives you access to 100% of the CPU 100% of the time. Sure, we need additional job system logic, which overall is more work. But since we can do multiple things at once, this overhead is definitely worth the effort, and reduces the computation time in total.

Thread Pool Theory

One of the best books about programming I've ever read, is "C++ Concurrency in Action" by Anthony Williams. Even though I made the switch to Rust a few months ago, the book was still immensely helpful for understanding and designing multithreaded stuff. And the job system that I've implemented is inspired by the thread pool that Anthony proposed in his book. So before we dig into my spaghetti code, it's probably a good idea to first understand how a thread pool works. By that I mean, we'll go and look at Anthony's thread pool.

Naïve Thread Pool

In the most naïve way possible, a thread pool is fairly easy to implement: Have one concurrent queue and share it among every worker thread.

This is primitive, but it should work. Nonetheless, this design has a huge problem: Congestion. This naïve solution has multiple threads that are all constantly trying to dequeue jobs from the job queue. This means that the queue must be absolutely thread safe. This begs the question, how does something become thread safe? Simply put, it prevents simultaneous accesses from different threads. There are various techniques to accomplish this. There exist blocking and non-blocking methods, but ultimately, a thread safe object turns multithreaded accesses into "single threaded" ones. (My explanation isn't quite correct, concurrent accesses on a thread safe object still happen on different threads. But since all accesses happen in order and never at the same time, they may as well be single threaded. You get my point.) This is a huge huge problem, because we wanted a system that could run things in parallel, and now this queue alone is preventing that entirely.

To fix this, we somehow need to avoid threads attempting to access the same data. Luckily, I don't have to be smart to figure this one out, and Anthony proposes a better solution.

Local Job Buffers

The key thing to realize, is that jobs can submit other jobs. Jobception. If we enqueue these to the shared job queue, we have an even bigger congestion problem. No. Instead, each worker thread has its own job queue, where it can submit and dequeue jobs.

Now, a worker thread only attempts to dequeue from the global job queue, if its own local queue is empty. Also notice that only the worker thread has access to its queue, so the queue doesn't even need to be thread safe!

This is a much better solution, but it is still prone to problems. Imagine again that we have a logic job and an output job, that are both submitted to this job system. One worker may receive the logic job, and another receives the output job. Since jobs can push other jobs, and these are enqueued to the local buffer, logic jobs will always stay on thread A, while render jobs will always stay on thread B. But just like before, one job is much more expensive than the other one. And even though we have enough jobs for everyone, because they never leave their worker thread, we get this situation again:

Anthony comes to the rescue one more time!

Stealing Jobs

The final design proposes that every worker thread actually has a reference to the local queues of the other threads, such that workers may steal jobs from each other.

This implies that the local job queues must be thread safe again. Also, we stop talking about queues, and more about buffers. More precisely, a deque. If the container that holds the jobs where a simple queue, we would run into the congestion problem again. But to allow for maximum parallel work, jobs are pushed and popped from the same end of the buffer, while stealing is done from the opposite end. This ensures that even though multiple threads access the same buffer, they happen at different locations on the buffer, so stealing and popping can happen simultaneously. The only way we could run into congestion, is if the buffer were empty. But in this edge case we simply don't care, because what is a worker thread going to do if it is empty? Wait? Since there isn't anything to do anyway, we can absolutely afford to congest.

While a bit more complicated, this solution is much more sophisticated. Full credit to Anthony. And it is this thread pool, which I decided to build upon.

My Job System

Now that we have a basic understanding of how a sophisticated job system looks like, let me present you my altered version. In total, my system is built from 5 parts, which we will tackle in order:

1. Job.  Simple wrapper around a function pointer. Until I figure out how to store function pointers in a collection, a wrapper is required.
2. JobBuffer.  Buffer that holds Jobs. It isn't really a queue, neither is it really a deque, it's more akin to a weird stack, but at the same time it isn't. It's specially designed for this job system, and isn't even thread safe if used incorrectly, but we'll talk about it later.
3. JobSystem.  Collection of globally available functions, to setup worker threads, submit jobs and run pending jobs.
4. JobFuture.  Utility to wait and return values from jobs.
5. JobCell.  Utility, to pass references into a job. Rusts ownership rules made this absolutely necessary and a real headache to design.

Without further ado, let's jump into some Rust code 😊

Job

As stated above, I honestly don't know how to store function pointers in a container elegantly. In Rust, every function is unique. Even if two functions were to have the same signature, they are still considered different from each other. Thus, such two similar functions cannot be stored in the same collection, unless you put them into some sort of wrapper. To make my life easier, I simply created a wrapper that stores a single function pointer. Since every Job is syntactically the same, they can be effortlessly stored in a container.

To be able to invoke the function pointer, it is stored in an Option, such that when calling it, we can take ownership of it. It should go without saying, that this Job can only be invoked once, because once we moved the function pointer out of Option, there is no function pointer left.

JobBuffer

There is one thing that you must know, when designing interfaces for thread safe constructs: The more flexible the public interface, the more race conditions you introduce. While impractical, assume the following example: A worker thread only wants to pop jobs of the buffer, if they are for example render jobs. To program something like this, you would have a peak method, that checks the next job without popping it off the buffer. The caller then can check the job. If it's the right kind, pop it. Client code would look something like this:

This code would be fine in a single threaded context, but this introduces a race condition. Between job_system::peak() and job_system::pop(), another thread may pop off a job. Thus, even though the if-statement may be true, a different job is popped off than what was peaked at. This means, render_job may actually not be a render job.

The simple solution is to keep your interface to an absolute minimum. Besides its constructor, my job buffer only exposes 3 methods: Push, pop and steal. Peaking and indexing are not implemented, because they cause race conditions and aren't used by my system.

An iterator may be helpful, but iterators are notoriously difficult to make thread safe. Whatever mechanism you use to prevent other threads to access your collection, it somehow must live at least as long as the iterator. This introduces many difficult questions, like who owns the locking mechanism, or how to sync it with the iterator? It's probably best to just not implement it.

is_empty() may be useful, and I even have the use case for it, but ultimately it's redundant. Because steal() and pop() can simply return None when no job exists, is_empty() provides truly redundant information.

count or length are also useless: Either there are jobs in the buffer that can be popped off, or there are none. There will never be a case, where I would want to know how many jobs are currently stored in the buffer.

It should become a bit clearer now, why I am calling it a buffer, and not queue, deque or stack. It is lacking seriously in features. Also, it isn't technically a queue, nor stack, because jobs can be popped from both sides. It isn't a deque either, because it can push only from one side.

Okay, that were a lot of preambles, but how to actually build such a thing? Hang in there, we gotta talk collections first. We will see code soon enough.

I was quite surprised how many different designs there are. I went through at least 10 different prototypes before I landed on the buffer that I am currently using. Initially, I wanted to write a doubly linked list. Those come with a hefty number of drawbacks and they are stupidly difficult to write in Rust. But I wanted one, because it can hold infinite items in theory. When a buffer is full, and a job is pushed, what should happen? For a long time, I didn't want to answer this question, hence I was only considering a buffer with theoretical infinite size.

But eventually, while going for a walk, I realized something: When a buffer is full, I can simply invoke the job. The caller shouldn't care whether the submit function blocks or not. All that matters is that the job system is making progress somehow. If the caller needs to be interrupted to invoke another job, then this is totally fine. Even though the caller may not make progress, another job will be, therefore no resources are wasted.

My choice to implement the JobBuffer fell onto the ring buffer. A ring buffer is just an array or vector, with one or two cursors, keeping track where the current head and tail are. It's called a ring buffer, because the cursors wrap around the array, thus it effectively has no start and end, like a ring. Once allocated, no further allocations are necessary to use it. Also, no nasty pointer issues arise. And to top it all off, a ring buffer can be entirely coded in safe Rust, which is always a big bonus.

To familiarize yourself with it, here's a simple playground I've built in JavaScript, where you can familiarize yourself with the concept:

We're finally done with the preamble, let's jump into code:

This is the entire struct. it stores the current head and tail, as well as a vector of jobs.

This definition may confuse you. Why is the head an UnsafeCell<usize>? Why not a Mutex<usize>, like the tail? Why does the job vector also store mutexes? Isn't locking the head and tail safe enough?

Pushing and popping will happen on the head. Stealing will happen on the tail. I hope it should be fairly obvious why the tail needs to be a mutex. Multiple worker threads may steal from the same buffer. As such, the tail needs to be protected. Okay, but doesn't this also apply to the head? Well, no. Notice that only the worker thread that owns the buffer is pushing and popping jobs. Because a single thread is calling push and pop, there will never ever be the case, that push() or pop() will be called simultaneously. This means, as an optimization, we can simply leave the head unprotected. This is what I meant earlier, that this buffer isn't thread-safe, if you use it incorrectly. But since we aren't using it incorrectly, we are perfectly safe and sound.

Okay, but why do the items in the vector need to be protected? Doesn't that mean that we have to lock 2 mutexes to access a single entry? Yes, and that's one downside of this design. But till now, this didn't cause too much performance issues, so I am not worrying about it yet. While I can't get rid of this extra mutex, you should still understand why it is needed: When the buffer is full or empty, the tail and the head point to the same node, and thus two different threads may access the same node simultaneously. This needs to be prevented. We can't lock both head and tail, because that would mean if one thread were to call steal(), a second thread cannot call pop(). We want that 2 threads can work on the same buffer at the same time. Thus, the node itself must be stored within a mutex.

The constructor should now be very straightforward. We create a vector with a fixed capacity, and insert empty nodes. Notice that we return an Arc<Self>. This makes the client code a bit tidier, because this buffer is never used on its own, it will always be duplicated somehow.

Push is a bit more complicated. Notice the unsafe block in the first line of this method. As I've explained earlier, this doesn't cause problems, because this will only ever be accessed by a single thread. We then lock the node at the current head. Notice how this is a try_lock, not a lock. This avoids blocking, meaning the job system can progress, even if the push failed. If we managed to lock the node, we can then see if there is a job inside or not. If there is a job already, the buffer is full. If there is no job, then we can overwrite it and update the head.

Note that the BlockedOrFull error takes ownership of the job. This allows the not-pushed job to be returned to the caller, so that they may invoke it.

Pop is very similar, but the new head is computed first and then the node is locked. Note that it uses lock, not try_lock. A worker thread attempting to pop a job literally has nothing else to do. So to make progress, the best option is to block until we can access the node. Then, whether or not there is a job, we return the job or an IsEmpty error.

Again, steal is very similar. But unlike the head in push and pop, the tail needs to be locked, because it is protected by a mutex. Notice how this one is using a try_lock and not a lock. This is due to there being number of CPUS - 1 amount of buffers to steal from. If we can't steal from this buffer, we simply steal from the next one. There is no need to block. After that, we then lock the node. If we were successful, we compute the next tail and return the job, otherwise we return an error.

At last, but not least, we need some utility stuff, so that this buffer compiles and can actually be shared between threads:

And there we have it. A highly specialized buffer to store jobs.

This JobBuffer is about 100 lines of code, but to test it I wrote 1180 lines! (including whitespaces)

This buffer is the core of my job system, which is the core of my game engine. So it must be water tight. I really can't afford this buffer to blow up, under any circumstance. As such, Anthony Williams recommended himself, to write as much tests as possible, to test every thinkable edge case, no matter how rare. What if one thread is pushing while another is stealing? What if 100 threads are stealing on an empty buffer? What if 1 thread is popping from an empty buffer? What if one thread is pushing and popping on a full buffer, while 100 threads are stealing?

But the painful, repetitive work was absolutely worth it. To this day, I haven't encountered a single issue with the JobBuffer, and it's been 530 days, since its implementation. And I am quite confident that it does its job properly. Maybe I will encounter a bug in the future, but I seriously don't think I will encounter a serious problem, ever.

Narrator: The moment the buffer blows up, he will regret to have said this.

EDIT 17th March, 2024. The system did, in fact, blow up:
https://www.rismosch.com/article?id=i-found-a-bug-in-my-job-system

JobSystem

You may or may not have figured out, that the JobBuffer described in the previous section is the local JobBuffer, which is owned by a worker thread. What about the globally shared JobBuffer? For better or for worse, I got rid of it.

Anthony Williams' thread pool seems fine, but I was always asking myself: Who is pushing onto the global buffer? I feel like Anthony's thread pool is designed for a general purpose. It most likely finds use in some enterprise project. Maybe a user is requesting a search or find feature, and then the program would initiate a thread pool to do just that. Maybe there will be additional threads that also push onto the thread-pool. Nonetheless, I really want to do what Naughty Dog did with their engine: Jobify the entire thing. EVERYTHING will run on this job system, with a few significant exceptions: Startup, shutdown, logging and IO. Everything that runs on the job system will push local jobs.

I also thought about who would own the main game loop. This too went through many different design iterations, but I realized soon enough that the main thread can also be a worker thread. The main thread can own a local JobBuffer too, and it would execute a "GodJob", that lives for the entirety of the program. When the god job ends, it waits for all currently enqueued jobs. After all workers have been ended, the engine is allowed to shut down.

Let's start with some structs:

First, we have a thread local variable. It stores an Option, because a thread that wasn't initialized as a worker thread cannot serve as a worker thread.

The WorkerThread currently stores 3 things: The JobBuffer that it owns, the JobBuffers from the other workers, and an index, that uniquely identifies this thread. The index is mainly used for debug purposes, and it's a utility that directly comes with my job system. As such, the index isn't linked to the platform, the OS, or anything of that nature.

JobSystemGuard is the struct that will be returned when the job system is initialized. If this guard is dropped, the job system will be shut down. Due to its singleton-like behavior, it didn't feel right to have a struct that represents the job system, similar of how you would design an OOP singleton. Instead, it gives the caller the opportunity to drop it at a specified time. The JobSystemGuard stores all handles of every spawned worker thread, and an AtomicBool to signal when worker threads should be shut down.

The init method may look daunting at first, but it's actually very straight forward. It takes 2 values: The size of the JobBuffers and how many threads should be spawned. First, we clamp the threads to the number of processors that your machine has. This is to prevent spawning more threads than cores.

Then we calculate the affinities. Affinity allows you to lock a thread only to a specific core. For example, if a thread has affinity 0, 1 and 2, then it will run only on core 0, 1 and 2. Every worker thread should run on a different core, otherwise worker threads can interrupt each other, which may result in a performance loss.

Assume your PC has 12 cores. If you were to spawn 5 worker threads, my code would estimate the affinities like so:

After we've calculated the affinities, we simply create one JobBuffer for every worker thread. We also create the done flag.

Since the main thread is also a worker thread, we will be spawning threads - 1 number of workers. For each we clone the affinities, the buffers and the done flag. Once spawned, each thread will first set itself up and then run itself.

Just right after that, the main thread is doing the same, but it also collects the handles and puts them into the JobSystemGuard. The job system is now setup and already running!

Now let's see how it's dropped:

First, we set the done flag to true, so every worker thread knows it's time to stop. The main thread then empties its local buffer, meaning it is popping all jobs left in the buffer and running them. Then we take the handles, and join each worker thread. With some logging thrown in, that's all that drop does.

Let's take a look at all the utility functions, that the previous two code snippets were using:

This method duplicates the array of JobBuffers, thus every worker thread can have a copy of all available buffers.

Setting up a worker thread is a bit more involved. First, we set the affinity. This is simply a wrapper around the Windows API, nothing special.

Then, we choose our local buffer. This is simply the buffer at the worker threads index. Meaning, the local buffer of thread 0 will be at index 0. The local buffer of thread 42 will be at index 42.

Using 2 for loops, we then get the buffers from which will be stolen from. We start at index + 1 and then wrap around. This diagram illustrates how the buffers will be arranged, for worker thread 2 on a machine with 5 cores:

After the busy work, we then simply set the thread-local variable with the necessary information.

Running the worker thread is very straight forward. While the done flag is false, we are stuck in an endless loop, running jobs that are still waiting to be executed. If the done flag is true, we empty the buffer, like the main thread does when JobSystemGuard is dropped.

To run a pending job, we first pop from our local buffer. If it is successful, we invoke it. If our local buffer is empty, we steal from the other buffers instead. If this is successful, we just stole a job and can invoke it. If no job was popped or stolen, we yield, to give room for other threads.

run_pending_job() is public, because as you will see in the sections JobFuture and JobCell, it is quite useful to let others run jobs as well.

Emptying the local buffer is also very straight forward. We are stuck in a loop, popping jobs from our local buffer and running them, until it's empty.

Now we've seen several functions that pop and steal jobs. So now it's time to show what these functions are doing in detail:

Popping literally does nothing except calling wait_and_pop() of its local buffer. If for whatever reason this method is called from a non-worker thread, then this method returns nothing and prints an error instead.

Stealing is pretty much the same, except that because we have a vector of buffers, we iterate through them. The first steal that succeeds breaks the iteration. Just like pop_job(), if this is called from a non-worker thread somehow, this method will print an error.

With all the setting up, tearing down and dequeuing taken care of, let's look at the thread index:

This simply returns the thread-local index. If the calling thread is not a worker thread, -1 is returned and an error is printed.

I reserved the submit function for last, because it uses JobFutures, which I'll explain in the next section. But here it finally is:

This function takes a closure, that can return a value. This is the job to be run. And it returns a JobFuture, which can return the jobs returned value. You may remember, that Job was a very very simple wrapper around a function pointer, which doesn't even return anything. That is intentional, to make my life easier. Instead of handing the closure directly into the job, and thus forcing me to somehow accommodate for its return value, it's much easier to simply wrap the closure into another closure, that doesn't return anything.

submit() first creates the future. This includes a SettableJobFuture and a JobFuture. Soon you will see, that they share a mutex, but the split between "settable" and "non-settable" prevents client code to set the future themselves.

Then it creates said closure, which sets the future with its return value. After that, we simply call push() from our thread-local local buffer. Recall that pushing returns the job, when the buffer is full. In that case, we store it in not_pushed and invoke it later. At the very end, we return the non-settable JobFuture, so that the client may wait on the job that was just submitted.

If you read this far, and understood everything, I have to commemorate you. To come to this point, I've read several books already. Even though this blogpost is already incredibly long, it still condenses a lot of information, and I seriously applaud you if you could follow along with everything. But we aren't done yet, there are still 2 things that we need to talk about.

JobFuture

Submitting a job is an asynchronous operation. Submitting takes O(1) time, but the job itself will be executed on any core in some arbitrary point in the future. But what if you need to wait for the job to continue? For example, my Input system runs mouse, keyboard and gamepad in parallel, but my remapping system requires that these 3 systems are already computed. So before any remapping can take place, the input job needs to wait for the 3 children.

This is what the JobFuture is for. It gives client code a handle, which can be awaited. As you've already seen, it is split into a SettableJobFuture and a non-settable JobFuture. Let's just jump straight into it!

Inner is simply the data to be stored. It stores a bool is_ready, which indicates whether the job is finished or not. data is the value, which was returned by the job. It is stored inside an Option, because sooner or later we need to move the data out. The Future can be in these 4 states:

SettableJobFuture and JobFuture literally store the same: An Arc<Mutex<Inner<T>>>. Thus, they are virtually identical. Because they are different structs however, they can implement different methods. For example, only SettableJobFuture has a constructor, and it returns both a SettableJobFuture and a JobFuture. Also, only SettableJobFuture can set its value. JobFuture can wait on a value, but SettableJobFuture cannot. Let's see:

SettableJobFuture is very straight forward. It initializes the Arc<Mutex<Inner<T>>> which will be shared between the two structs. It then creates each struct and returns them. set() simply locks the mutex, sets is_ready to true, and overwrites data. Notice how set() consumes self. This directly means that once SettableJobFuture is changed, it might as well be dead.

Let's look at the await-able JobFuture next:

Just like set() in SettableJobFuture, wait() also consumes self. This results in the fact, that taking data will always succeed. We can't wait() a second time, when it was awaited already. This is why the None branch of wait() includes an unreachable!() and why wait() can afford to return the value directly, instead of returning an Option.

wait() calls wait_and_take(), which spinlocks until data is set. Considering the reactions I got from a previous blogpost, people somehow seem to hate spinlocks and immediately consider them unsafe. But let me assure you, that without a spinlock, this future wouldn't work.

Assume for a second, that wait() uses a locking mechanism, like a condition variable or a mutex. In that case, waiting for the future would block the entire thread. This is problematic, because a blocked worker thread cannot run jobs. Ideally, the worker thread would do other things while the calling job is waiting. As such, a spinlock-like construct is the best choice for implementing such behavior.

This is what wait_and_take() does: It first calls try_lock() on the mutex, which isn't a blocking operation. If the lock was successful and the future was set, we can take the data inside. If the lock was not successful, because it was poisoned, then this is unrecoverable and we throw an error. If the future was not set, or the lock resulted in a TryLockError::WouldBlock error, then we run a pending job and attempt the whole procedure again.

In a nutshell, wait() will be stuck in a loop, running pending jobs, as long as the job is not completed, thus effectively blocking the calling job and progressing the entire job system. With this taken care of, what else might be missing from this job system? Well...

JobCell

At this point, I thought I was done with the job system. I seriously didn't expect that Rusts borrow checker would be a major killjoy. I've spent already a lot of time and effort to get to this point. But then I tried using it in my main loop, and the compiler simply said no.

The entire purpose of jobs is that they can take stuff, like objects or references, and then do work on them. A job will probably mutate data, or simply just immutably reference it to generate new state. But here's the last obstacle: Because the entire job system stores everything statically, thanks to its singleton-like nature, everything that is moved into a job must be statically borrowed. Something that is locally owned or referenced cannot be borrowed for a static lifetime. As much as the compiler is concerned, if a reference is moved into a job, it enters the nirvana, never to return. And the compiler simply doesn't allow that.

A workaround must be found.

To overcome this problem, I really had to scrape the bottom of the barrel. I have tested numerous solutions, all of which weren't satisfactory. Either I'd took a major performance hit, or the solution would panic randomly, or I'd introduce undefined behavior. Nothing worked, and prototype after prototype failed.

Before you say it, no, Arc<Mutex<T>> does NOT work. Remember what I've written in the previous section JobFuture: If a mutex were to block, then the entire worker thread blocks, meaning no progress will be made. I could write a spinlock around try_lock(), but this really isn't practical. Even if I were to create an elegant interface to spinlock a mutex, a spinlock still adds major overhead, which simply isn't worth it for just passing a single value into a job. This is especially true when I plan to spawn numerous jobs, all of which may reference their outside in one way or another.

What I want is an easy way to pass things into a job, with no overhead at all. Eventually, this led me to UnsafeCell, and a much much deeper understanding of how Rusts ownership rules work.

First, let's tackle mutable references, because as you will see shortly, these are actually quite easy to work around. I was expecting that these are the difficult ones, but with a little hack they were pretty trivial to solve. Instead of passing a mutable reference into a job, you can simply pass ownership into the job.

Test your knowledge! Does the following code compile?

Hover to reveal the answer:

No, this doesn't compile. That should come as no surprise. We are moving a value out of wrapper! If you are accustomed to Rust, this should stick out like a sore thumb, and you would immediately realize that passing a &mut into consume() is probably what you wanted in the first place.

But what about this one? Does this compile?

Hover to reveal the answer:

Yes, this does actually compile. This surprised me when I first saw this. We are still moving a value out of wrapper, just like the first code snippet, so why does it compile? The reason why the first one didn't compile is not really because we moved something out. Rather, we tried to use a value which was already moved. If you remove the println!() in the first example, we are not using wrapper after its move and the code compiles just fine. This implies that it's A-okay to move child values out from a parent. We just gotta replace it before we use it again. Instead of replacing it with an arbitrary value, we can replace it with the value we moved out! Remember that the JobFuture allows us to return something from a job. Any ownership that we may pass into a job we can return afterwards.

To illustrate this technique, here's an excerpt of my input logic, showcasing the part of the keyboard job:

Ignore Ref for a moment 🅑. It most definitely is not the Ref you are currently thinking about. We will talk about it in a second.

Notice how the run() function takes ownership of an InputData object 🅐. It also returns an InputData object 🅒. This method alone is already demonstrating the technique. It takes ownership of InputData and later returns it. The first thing we do is to pass ownership of current.keyboard to a local variable 🅓. Then we move the local variable and ownership into the job 🅔. Now the job owns the keyboard and can mutate it however it wants. With an additional gameloop_state, the job also returns the keyboard 🅕, such that when the job is awaited, the caller once again has the ownership of keyboard in a local variable 🅖. To satisfy the compiler, we move this local variable new_keyboard back into current.keyboard 🅗.