Optimizing pipelines through specialization
When designing how my video editor Kama Studio should work, I had one very important requirement in mind: previews should be instant. Considering that I wanted to be able to edit even 4K 60FPS video, it was clear to me that I needed to utilize the GPU somehow to handle complex effect pipelines.
While thinking about it whilst I was doing some random bs throughout the house, I came up with an idea: “What if I could stitch some shaders together and have all effects run that way?”. Welp, so I did. I decided to make pretty much all video effect shaders, through the power of naga, able to achieve that goal. Each plugin and effect is namespaced, so there are no conflicts, and they all get stitched together into a single combined shader at the end. This is what allows my video editor to reach the performance it does, and as such allows the shader compiler to not only better optimize the code, but also reduce memory bandwidth and needed passes.
So I was thinking, what if you could apply this to other programs? Turns out, you can!
By using various backend compiler technologies, such as LLVM or even cranelift, you can generate some code on the fly and make things run significantly faster. Browsers do something like this through JIT, where JavaScript is compiled when needed, allowing you to play browser games with decent performance. If you have a deterministic data structure, like Kama Studio’s effect graph for instance, you already know how everything should be hooked up together and the possible values you can have, so you can emit code representing that entire pipeline. Many programs can benefit from this, for example DAWs, where you can chain different effects on top of one another and then even make the backend compile to the user’s CPU’s specific features while also avoiding expensive operations that come from a generalized architecture.
Let’s say that we have a basic effect chain for a microphone:
Input -> Noise Reduction -> EQ -> Limiter -> Output
Instead of passing the audio chunk individually through each effect, what we can do instead is combine this exact chain into one function and let our compiler backend of choice optimize it as if it were regular ol’ code.
If we compare that to a generalized pipeline, things get a bit uglier performance-wise. You start to run for example into walking nodes dynamically, and due to the amount of indirection in that, it can turn out rather slow. You may also need to check types and configuration at runtime as well, adding even more overhead, and you may get branches that could literally never be taken depending on what parameters are set.
Specialization gets rid of a lot of that – it takes what is essentially runtime knowledge and turns it into compile-time knowledge, where branches can disappear, calls can be inlined, constants propagated, locality improved, registers allocated more effectively, etc.
You can take this idea even further, and have parameters that would normally be some simple variables, be turned into constants, and the ones that are changing dynamically to remain as variables. Though depending on the program, if you are going to have external influence over parameters such as scripts, for example, this may not be such a good idea.
This isn’t all sunshine and rainbows though, as there are some real downsides. Those are in terms of invalidation and recompilation.
You can have an entire effect chain but then you want to add a new effect; this means that suddenly you have to recompile the whole thing again. You may also have multiple variants consuming even more memory. So you need to be careful depending on your requirements. For example, if you want to make a DSP for a microcontroller, it may not even be worth the effort implementing a code generation system, and might as well put each effect behind a flag, and have the specialization in the actual code.
Ultimately, if you know about the pipeline ahead of time, consider turning that runtime knowledge into compile-time knowledge. This way you give the compiler opportunities that a generalized architecture shrimply cannot provide.