Puppeteer demo

I finally found some time to upload the Puppeteer setup files.

This is the Puppeteer release number zero. I call it demo-09.10. The puppeteer versions will be called after the year and the month they will be released. For example, if I will make a release on February 2010, it will be called puppeteer-10.02.

This demo includes the asset manager itself, a simple skeleton editor and a simple animation editor.

All the scenes, controllers and others are not ready yet, so they are excluded from the demo.

Download the Asset Manager demo.

EA

EA is still laying people off. They don't do huge events like resigning 200 employees or closing huge studios. They are just laying off small groups of people. One of my friends Yulay has been resigned recently.

What is wrong with the company? It looks like Titanic now. Huge, leaking and out of control. The stocks continue to go down. I think they are doing worse then the other companies in the industry.

Well, may be I'm wrong. I don't exactly know how other companies are doing :)

My own impression is - this is all about management. EA has too many managers. And the situation is getting worse because they resign artists and software engineers. It doesn't look like they are resigning managers. Again, may be I'm wrong. I don't see the whole picture. This is just how it looks from my point of view.

Once when I was working there, I was looking to the intranet to find who are the managers of my manager. I was really surprised by the number of managers above my manager. There were the chain of 10. John Riccitiello was a manager of a manager of a manager of a manager.. oh I'm getting tired saying it 10 times .. of my manager.

Reflecting to the Soviet army.. Why Soviet army? Because I don't know much about other armies. Considering I'm a soldier. My nearest boss is a sergeant. Then a master-sergeant, then a lieutenant or a captain, then a major or a colonel, then a major-general or a lieutenant-general, then a colonel general, then a marshal, then the commander-in-chief. Eight levels.

Do you know how big was the Soviet Army? About 10 million. And it had less management levels than EA. What are all these people doing?

Well, I promised not to discuss EA here. Sorry.

Idling

I'm taking a two-week break.

I have signed a short contract - developing a pretty small and simple application for iPhone.

Just want to earn some money and take a short break from the Puppeteer.

I'll be back as soon as the contract will be done.

Compiling assets performance

I have spent too much time trying to improve the compiling performance. That is enough now, I am quitting it. The priority of it is quite low anyway. I need to work on scenes and actors, I need to develop the runtime..

Well, you all know how this happens sometimes. You are so much into the problem that you stop seeing the whole picture for some time. You just can not stop doing it because it is not done. On a rational level you understand that there are more important things to do, but you think: duh, just one more day.
Etc.

This is what happened to me. The good news is that the Asset Manager is pretty much done. I will allocate "just one more day" tomorrow to make an installer, and create something downloadable. This is not a release yet, this is just a preview that gives you a feeling of what Puppeteer Asset Manager is.

The compiling performance is still an issue, but it is not the end of the world if I fix it after the release.

Compiling in Puppeteer is a process of creating runtime files - the files that are used in the game. Assets in the Asset Manager contain some data that is not needed in the game. For example, asset names, channel names for animations, bone names for skeletons. There is no need to waste the game memory for it. Another thing to consider is compressing animations. The Asset Manager uses the raw animation data. It is a good idea to compress it before using in a game.

Thus, Puppeteer uses different asset data files in the Asset Manager and in the runtime. As I already mentioned, converting the Asset Manager assets to the runtime files is called compiling in the Puppeteer.

Compiling in Puppeteer is quite similar to C/C++ compiling. Puppeteer compiler creates a kind of obj file for every asset - the extension is ppo. Then it "links" them to a partition files - extention ppp. It also creates a runtime file - ppr. Ppr file has a general information about asset types, partition dependencies, etc. Ppp and ppr files are the files that are used in the game.

Initially I created a very simple single thread compiler, that created ppo files one by one, and then generates all the ppp files. Compile then link approach.

The performance was just fine on a stress test project - 21-22 minutes. I thought making it multithreaded could reduce the time to 7-8 minutes. That didn't happen. All I could achieve in 4 threads was 15-16 minutes.

Linking is a bottleneck. I am using the next algorithm now:

- Collect all the dirty assets to an array.
- Based on the dirty assets array, collect all the dirty partitions to an array.
- Start 4 threads. Every thread function:

1. Check the partition array for a partition to compile. If it is not empty take one partition, remove it from the array and goto 3.
2. Check if the assets array is not empty. If it is not take an asset, remove it from the array, compile it and goto 1. If the array is empty - exit.
3. Go through all assets of the partition. For every asset. If it is in the dirty list - compile it and add to ppp file. If it is not in the dirty list - just add the ppo file to the ppp file. If another thread is compiling it right now - compile another asset and check again after it. If there is no another asset to compile, just sleep a few milliseconds. After the partition is compiled, goto 1.

For example suppose we have 1 partition with 1000 assets. In this architecture the first thread takes the partition, the other 3 threads are compiling the assets. So linking and compiling happens at the same time. If a needed file is not ready, the linker stops and helps with the compiling.

The problem is that if we change just one asset, the linker still has to add all 1000. Compiling compile their one asset and exit, and the rest happens in one thread. Considering that every asset is 30k in my stress test - this is generating 30Mb file from 1000 other files. That definitely takes some time.

Incremental linker is the solution, but not now. I have spent enough time, and after all I do not expect 64000 assets projects in the nearest future :-)

Another container code to share

This time it is a C# red-black binary tree.

I wrote it when I considered reimplementing TreeView and ListView standard controls. Their performance was a bottleneck. I thought that making similar controls that use less memory allocations and better sorting algorithms could solve it. However after I reduced the number of posted messages, the performance became pretty decent.

Thus the tree is not used in the Puppeteer at this moment in time, but still it is a pretty good red-black tree. The performance was improtant, so I tried to avoid memory allocations as much as possible.

The Inverse Law of Bug Complexity

"The harder a bug is to track down, the simpler the fix tends to be." (c) James Tauber

I was implementing the c# DirectX rendering loop. As everyone who was doing it I read Tom Miller - Rick Hoskinson discussion, and copied this piece of code to Puppeteer:

public static class Native
{
    [StructLayout(LayoutKind.Sequential)]
    public struct Message
    {
        public IntPtr hWnd;
        public int msg;
        public IntPtr wParam;
        public IntPtr lParam;
        public uint time;
        public System.Drawing.Point p;
    }

    [System.Security.SuppressUnmanagedCodeSecurity]
    [DllImport("user32", CharSet = CharSet.Auto)]
    [return: MarshalAs(UnmanagedType.Bool)]
    public static extern bool PeekMessage(out Message msg, IntPtr hWnd, uint messageFilterMin, uint messageFilterMax, uint flags);
}

Guess what? I forgot to put "struct Message" declaration. The compiler didn't complain. It took System.Windows.Forms.Message for a function argument instead. Tom's loop used PeekMessage to know if there were messages in the queue. The message itself was not needed in the loop, so compiler couldn't notice the difference. So did I.

Basically as far as it was not used, there was no difference. But there was a small problem: sizeof(Native.Message) was 28; sizeof(System.Windows.Forms.Message) - 20. Oops.

It didn't crash under debugger because.. To be honest, I do not know why. May be because of some special heap alignment under debugger, may be because the things it corrupted were not that critical.

Guess how much time it took me to track this down? I can not answer myself. The first time I saw this crash was in April. I could run Puppeteer under the debugger, so I did not push it hard. I thought I could fix it later, but still - I fixed it only last week.