Compiling assets performance

I have spent too much time trying to improve the compiling performance. That is enough now, I am quitting it. The priority of it is quite low anyway. I need to work on scenes and actors, I need to develop the runtime..

Well, you all know how this happens sometimes. You are so much into the problem that you stop seeing the whole picture for some time. You just can not stop doing it because it is not done. On a rational level you understand that there are more important things to do, but you think: duh, just one more day.
Etc.

This is what happened to me. The good news is that the Asset Manager is pretty much done. I will allocate "just one more day" tomorrow to make an installer, and create something downloadable. This is not a release yet, this is just a preview that gives you a feeling of what Puppeteer Asset Manager is.

The compiling performance is still an issue, but it is not the end of the world if I fix it after the release.

Compiling in Puppeteer is a process of creating runtime files - the files that are used in the game. Assets in the Asset Manager contain some data that is not needed in the game. For example, asset names, channel names for animations, bone names for skeletons. There is no need to waste the game memory for it. Another thing to consider is compressing animations. The Asset Manager uses the raw animation data. It is a good idea to compress it before using in a game.

Thus, Puppeteer uses different asset data files in the Asset Manager and in the runtime. As I already mentioned, converting the Asset Manager assets to the runtime files is called compiling in the Puppeteer.

Compiling in Puppeteer is quite similar to C/C++ compiling. Puppeteer compiler creates a kind of obj file for every asset - the extension is ppo. Then it "links" them to a partition files - extention ppp. It also creates a runtime file - ppr. Ppr file has a general information about asset types, partition dependencies, etc. Ppp and ppr files are the files that are used in the game.

Initially I created a very simple single thread compiler, that created ppo files one by one, and then generates all the ppp files. Compile then link approach.

The performance was just fine on a stress test project - 21-22 minutes. I thought making it multithreaded could reduce the time to 7-8 minutes. That didn't happen. All I could achieve in 4 threads was 15-16 minutes.

Linking is a bottleneck. I am using the next algorithm now:

- Collect all the dirty assets to an array.
- Based on the dirty assets array, collect all the dirty partitions to an array.
- Start 4 threads. Every thread function:

1. Check the partition array for a partition to compile. If it is not empty take one partition, remove it from the array and goto 3.
2. Check if the assets array is not empty. If it is not take an asset, remove it from the array, compile it and goto 1. If the array is empty - exit.
3. Go through all assets of the partition. For every asset. If it is in the dirty list - compile it and add to ppp file. If it is not in the dirty list - just add the ppo file to the ppp file. If another thread is compiling it right now - compile another asset and check again after it. If there is no another asset to compile, just sleep a few milliseconds. After the partition is compiled, goto 1.

For example suppose we have 1 partition with 1000 assets. In this architecture the first thread takes the partition, the other 3 threads are compiling the assets. So linking and compiling happens at the same time. If a needed file is not ready, the linker stops and helps with the compiling.

The problem is that if we change just one asset, the linker still has to add all 1000. Compiling compile their one asset and exit, and the rest happens in one thread. Considering that every asset is 30k in my stress test - this is generating 30Mb file from 1000 other files. That definitely takes some time.

Incremental linker is the solution, but not now. I have spent enough time, and after all I do not expect 64000 assets projects in the nearest future :-)