Floating point numbers

Sometimes we all need to deal with the floating point weights. A normalized values between 0.0 and 1.0.

Every time I deal with it I think about normalizing it to percents 0%-100% instead of 0.0-1.0. Why?

Because human beings, including myself, always think linear. I think: 0-100 has a better precision than 0-1. Our floating point numbers are discrete. There are more possible values between 0 and 100 than between 0 and 1. So our calculations are 100 times more precise with the percents. Right?

Wrong. Absolutely wrong. The IEEE-754 floating point format does not put numbers uniformly. This is not an even distribution. The distance between two floating point values depends on the module. For example, if we use a 32-bit float, the closest number to 100.0 is (100.0 + 2 ^ (-17)) . The closest number to 1.0 is (1.0 + 2 ^ (-23)).

The density around 1.0 is 64 times higher than the density around 100. The distance between closest floating numbers grows exponentially.

Even though there is more numbers between 0 and 100, the precision for a reasonable numbers like 50,60 etc is better when they are normalized to 1.

The moral is - don't be afraid to normalize. Normalization never cause loosing precision. More than that, if you are doing extensive calculations with large floating point numbers, it makes sense to normalize them, make calculations, and de-normalize back at the end. There is a big chance that the accumulated error is lower in this case.

Still idling

No, I didn't quit Puppeteer.

I'll get back to it as soon as I get rid of the current contracts.

I'm developing for iPhone now. Honestly speaking I don't like Objective C. IMHO, this is a poorly structured and unsafe language. I don't like Mac either. And particulary I don't like XCode.

The only thing I like about this project: I earn some money and I earn some XP to port Puppeteer to iPhone later.

This is all I have to say: The Puppeteer is still alive, but I need to spend some time on other tasks...

Puppeteer demo

I finally found some time to upload the Puppeteer setup files.

This is the Puppeteer release number zero. I call it demo-09.10. The puppeteer versions will be called after the year and the month they will be released. For example, if I will make a release on February 2010, it will be called puppeteer-10.02.

This demo includes the asset manager itself, a simple skeleton editor and a simple animation editor.

All the scenes, controllers and others are not ready yet, so they are excluded from the demo.

Download the Asset Manager demo.

EA

EA is still laying people off. They don't do huge events like resigning 200 employees or closing huge studios. They are just laying off small groups of people. One of my friends Yulay has been resigned recently.

What is wrong with the company? It looks like Titanic now. Huge, leaking and out of control. The stocks continue to go down. I think they are doing worse then the other companies in the industry.

Well, may be I'm wrong. I don't exactly know how other companies are doing :)

My own impression is - this is all about management. EA has too many managers. And the situation is getting worse because they resign artists and software engineers. It doesn't look like they are resigning managers. Again, may be I'm wrong. I don't see the whole picture. This is just how it looks from my point of view.

Once when I was working there, I was looking to the intranet to find who are the managers of my manager. I was really surprised by the number of managers above my manager. There were the chain of 10. John Riccitiello was a manager of a manager of a manager of a manager.. oh I'm getting tired saying it 10 times .. of my manager.

Reflecting to the Soviet army.. Why Soviet army? Because I don't know much about other armies. Considering I'm a soldier. My nearest boss is a sergeant. Then a master-sergeant, then a lieutenant or a captain, then a major or a colonel, then a major-general or a lieutenant-general, then a colonel general, then a marshal, then the commander-in-chief. Eight levels.

Do you know how big was the Soviet Army? About 10 million. And it had less management levels than EA. What are all these people doing?

Well, I promised not to discuss EA here. Sorry.

Idling

I'm taking a two-week break.

I have signed a short contract - developing a pretty small and simple application for iPhone.

Just want to earn some money and take a short break from the Puppeteer.

I'll be back as soon as the contract will be done.

Compiling assets performance

I have spent too much time trying to improve the compiling performance. That is enough now, I am quitting it. The priority of it is quite low anyway. I need to work on scenes and actors, I need to develop the runtime..

Well, you all know how this happens sometimes. You are so much into the problem that you stop seeing the whole picture for some time. You just can not stop doing it because it is not done. On a rational level you understand that there are more important things to do, but you think: duh, just one more day.
Etc.

This is what happened to me. The good news is that the Asset Manager is pretty much done. I will allocate "just one more day" tomorrow to make an installer, and create something downloadable. This is not a release yet, this is just a preview that gives you a feeling of what Puppeteer Asset Manager is.

The compiling performance is still an issue, but it is not the end of the world if I fix it after the release.

Compiling in Puppeteer is a process of creating runtime files - the files that are used in the game. Assets in the Asset Manager contain some data that is not needed in the game. For example, asset names, channel names for animations, bone names for skeletons. There is no need to waste the game memory for it. Another thing to consider is compressing animations. The Asset Manager uses the raw animation data. It is a good idea to compress it before using in a game.

Thus, Puppeteer uses different asset data files in the Asset Manager and in the runtime. As I already mentioned, converting the Asset Manager assets to the runtime files is called compiling in the Puppeteer.

Compiling in Puppeteer is quite similar to C/C++ compiling. Puppeteer compiler creates a kind of obj file for every asset - the extension is ppo. Then it "links" them to a partition files - extention ppp. It also creates a runtime file - ppr. Ppr file has a general information about asset types, partition dependencies, etc. Ppp and ppr files are the files that are used in the game.

Initially I created a very simple single thread compiler, that created ppo files one by one, and then generates all the ppp files. Compile then link approach.

The performance was just fine on a stress test project - 21-22 minutes. I thought making it multithreaded could reduce the time to 7-8 minutes. That didn't happen. All I could achieve in 4 threads was 15-16 minutes.

Linking is a bottleneck. I am using the next algorithm now:

- Collect all the dirty assets to an array.
- Based on the dirty assets array, collect all the dirty partitions to an array.
- Start 4 threads. Every thread function:

1. Check the partition array for a partition to compile. If it is not empty take one partition, remove it from the array and goto 3.
2. Check if the assets array is not empty. If it is not take an asset, remove it from the array, compile it and goto 1. If the array is empty - exit.
3. Go through all assets of the partition. For every asset. If it is in the dirty list - compile it and add to ppp file. If it is not in the dirty list - just add the ppo file to the ppp file. If another thread is compiling it right now - compile another asset and check again after it. If there is no another asset to compile, just sleep a few milliseconds. After the partition is compiled, goto 1.

For example suppose we have 1 partition with 1000 assets. In this architecture the first thread takes the partition, the other 3 threads are compiling the assets. So linking and compiling happens at the same time. If a needed file is not ready, the linker stops and helps with the compiling.

The problem is that if we change just one asset, the linker still has to add all 1000. Compiling compile their one asset and exit, and the rest happens in one thread. Considering that every asset is 30k in my stress test - this is generating 30Mb file from 1000 other files. That definitely takes some time.

Incremental linker is the solution, but not now. I have spent enough time, and after all I do not expect 64000 assets projects in the nearest future :-)

Another container code to share

This time it is a C# red-black binary tree.

I wrote it when I considered reimplementing TreeView and ListView standard controls. Their performance was a bottleneck. I thought that making similar controls that use less memory allocations and better sorting algorithms could solve it. However after I reduced the number of posted messages, the performance became pretty decent.

Thus the tree is not used in the Puppeteer at this moment in time, but still it is a pretty good red-black tree. The performance was improtant, so I tried to avoid memory allocations as much as possible.

The Inverse Law of Bug Complexity

"The harder a bug is to track down, the simpler the fix tends to be." (c) James Tauber

I was implementing the c# DirectX rendering loop. As everyone who was doing it I read Tom Miller - Rick Hoskinson discussion, and copied this piece of code to Puppeteer:

public static class Native
{
    [StructLayout(LayoutKind.Sequential)]
    public struct Message
    {
        public IntPtr hWnd;
        public int msg;
        public IntPtr wParam;
        public IntPtr lParam;
        public uint time;
        public System.Drawing.Point p;
    }

    [System.Security.SuppressUnmanagedCodeSecurity]
    [DllImport("user32", CharSet = CharSet.Auto)]
    [return: MarshalAs(UnmanagedType.Bool)]
    public static extern bool PeekMessage(out Message msg, IntPtr hWnd, uint messageFilterMin, uint messageFilterMax, uint flags);
}

Guess what? I forgot to put "struct Message" declaration. The compiler didn't complain. It took System.Windows.Forms.Message for a function argument instead. Tom's loop used PeekMessage to know if there were messages in the queue. The message itself was not needed in the loop, so compiler couldn't notice the difference. So did I.

Basically as far as it was not used, there was no difference. But there was a small problem: sizeof(Native.Message) was 28; sizeof(System.Windows.Forms.Message) - 20. Oops.

It didn't crash under debugger because.. To be honest, I do not know why. May be because of some special heap alignment under debugger, may be because the things it corrupted were not that critical.

Guess how much time it took me to track this down? I can not answer myself. The first time I saw this crash was in April. I could run Puppeteer under the debugger, so I did not push it hard. I thought I could fix it later, but still - I fixed it only last week.

Hashtable

Download

This is a simple C++ hashtable implementation that I am using in the Puppeteer Runtime. It is doing coalesced hashing, which is a combination of open addressing and separate chaining.

Hashtable is using "const char*" strings as for the keys. I didn't implement a general keys to keep the code simple. It is not a big deal to change it to using a template parameter for the keys, but the code becomes harder to read. Anyway, the Puppeteer always uses the strings, so keeping it simple makes more sense to me.

Hashtable does not copy the keys, so if you are going to use it, make sure their lifetime covers the lifetime of the hashtable. The reason is obvious - performance. In fact, the performance is the highest priority for this hashtable. This is the main reason why I decided to make my own implementation. Most of the existing hashtables are making a copies of the keys, and this produces a lot of unnecessary memory allocations.

Hashtable is also supplied with 11 different hash functions. Here is the code that finds the best hash function for a given set of strings:

using namespace puppeteer;

template <HashFunction Hash>
int TestHashtable(const char** aStrings, unsigned aStringCount)
{
    Hashtable<int, Hash> hashtable(aStringCount);
    for (unsigned i = 0; i < aStringCount; ++i)
    {
        hashtable.Add(aStrings[i], i);
    }
    return hashtable.GetCollisionCount();
}

int main()
{
    const char* strings[] =
    {
        "aaa",
        "bbb",
        "ccc",
        "1",
        "2",
        "3",
        "sgfdsgfsd",
        "vcxvx",
        "dc dcd ",
        "   ",
        "4456"
    };

    printf("AdditiveHash: %i collisions\n", TestHashtable<AdditiveHash>(strings, sizeof(strings) / sizeof(*strings)));
    printf("XorHash: %i collisions\n", TestHashtable<XorHash>(strings, sizeof(strings) / sizeof(*strings)));
    printf("RotatingHash: %i collisions\n", TestHashtable<RotatingHash>(strings, sizeof(strings) / sizeof(*strings)));
    printf("ShiftAddXorHash: %i collisions\n", TestHashtable<ShiftAddXorHash>(strings, sizeof(strings) / sizeof(*strings)));
    printf("OneAtATimeHash: %i collisions\n", TestHashtable<OneAtATimeHash>(strings, sizeof(strings) / sizeof(*strings)));
    printf("BernsteinHash: %i collisions\n", TestHashtable<BernsteinHash>(strings, sizeof(strings) / sizeof(*strings)));
    printf("XorBernsteinHash: %i collisions\n", TestHashtable<XorBernsteinHash>(strings, sizeof(strings) / sizeof(*strings)));
    printf("SdbmHash: %i collisions\n", TestHashtable<SdbmHash>(strings, sizeof(strings) / sizeof(*strings)));
    printf("FnvHash: %i collisions\n", TestHashtable<FnvHash>(strings, sizeof(strings) / sizeof(*strings)));
    printf("ElfHash: %i collisions\n", TestHashtable<ElfHash>(strings, sizeof(strings) / sizeof(*strings)));
    printf("JenkinsHash: %i collisions\n", TestHashtable<JenkinsHash>(strings, sizeof(strings) / sizeof(*strings)));

    return 0;
}

Partitions

Partitions is a puppeteer term for the runtime files. These files that are generated by the Asset Manager by compiling assets. They are containing the minimum asset data. Minimum means that all unnecessary information like asset names, tool tips, etc have been stripped. The game loads and unloads them using Puppeteer Runtime library.

Partitions can be created in the Asset Manager. Users can assign their assets to a different partitions to put them to the different runtime files. The rules for the partitions are:

1. Partitions can depend on each other. The dependencies are acyclic. Basically Puppeteer shows partitions in a form of a tree. For example:

Here the Scenes 1, 2 and 3 depend on the Base. Which means, the partition Scene 1 can't be loaded into the game before the Base is loaded. The game can work like this:

• User starts the game. It loads all the assets from the Base.
• User enters the first scene. The game loads the Scene 1.
• User leaves the first scene and enters the second. The game unloads the Scene 1 and loads the Scene 2.
• User leaves the second scene. The game unloads the Scene 2.
• User leaves the game. The game unloads the Base.
  

The depth of the tree is not limited. There might be sub-scenes and sub-sub-scenes. As many as needed to use memory most efficiently.

The assets from Scene 1 can reference the assets from the Base, but they can not reference assets from the Scene 2. Because there is no guarantee Scene 2 is loaded when the Scene 1 assets are trying to use the reference. There is a guarantee that the Base is loaded though.

For the same reasons the Base assets can not use any assets from any Scenes. Puppeteer Asset Manager always enforces these rules. It just denies creating the references that violate them.

2. Every asset can be assigned to only one or zero partitions. When an asset has no partition, it inherits the partition from the container it belongs to. If the container doesn't have it, it inherits it from the upper level container. Etc, etc. The root container is always assigned to a partition.

Why does the Asset Manager not allow assigning an asset to many partitions? Because of the complexity. The rules become too hard to maintain. What if user is trying to assign an asset to the Base and to the Scene 1 at the same time? Assets from Base are allowed to use this asset because it is a member of Base, but they can not use it because it is a member of the Scene 1. At the same time the asset can use other assets from the Scene 1 because it is a member of the Scene 1, but it can not use them because it is a member of the Base.

I decided to make it simple. An asset can be assigned to only one partition. However, there are exceptions.

3. There is a special partition - Free Partition. If an asset is assigned to a Free Partition, the Asset Manager puts it to every partition that references this asset. Thus such assets can get to more than one partitions.

Let me illustrate the usage of the Free Partition with an example:

Suppose you have a fantasy game. The main character is a hero who fights orcs, trolls and goblins. The Scene 1 is populated by the orcs; Scene 2 - by the goblins; Scene 3 - by the trolls.

You have the next skeletons:

• Hero Skeleton.
• Orc Skeleton.
• Goblin Skeleton.
• Troll Skeleton

You also have the next animations:

• Walk
• Run
• Jump
  
• Hit
• Shoot
  
• Block a hit
• Block a shot
  
• Dodge a hit
• Dodge a shot
  

Suppose your hero can walk, run, hit, block a hit and block a shot. The hero is needed in all three scenese, so the hero assets should be placed to the Base partition. The Base contains:

• Hero Skeleton.
• Walk
• Run
• Hit
• Block a hit
• Block a shot

The orks can walk, run, hit, dodge a hit and dodge a shot. Walk, Run and Hit are already in the Base. The Base is always loaded when the Scene 1 is loaded, so there is no need to put them to the Scene 1:

• Orc Skeleton
• Dodge a hit.
• Dodge a shot.
• Scene1 Asset. Don't mix up with the partition Scene 1. This is an asset that references all the assets used in the scene. It organizes them to work together.

The goblins can walk, jump, shoot and dodge a hit. That makes the Scene 2:

• Goblin Skeleton
• Shoot
• Dodge a hit
• Dodge a shot
• Jump
  
• Scene 2 Asset
  

Wait a second. "Dodge a hit" has been already assigned to the Scene 2. An asset can be assigned to only one partition. A possible solution is to assign it to the Base. It solves the problem . Now both Scene 2 and Scene 1 can use it. But what if trolls can not dodge a hit?

This means that the animation "Dodge a hit" is not used by any other asset in the Scene 3, but it is still loaded into the memory. The right solution is to put it to the Free Partition. In this case the Asset Manager finds that Scene1 Asset and Scene2 Asset are both referencing "Dodge a hit", and it puts it to the Scene 1 and the Scene 2 partitions. The Base and the Scene 3 do not have any assets that reference "Dodge a hit", so it is not included.

The trolls can walk, run, jump, dodge a shot, block a hit and hit:

• Troll Skeleton
• Dodge a shot
• Jump
  
• Scene3 Asset
  

Note that all three scenes are using "Dodge a shot" animation. If we put it to the Free Partition, the Asset Manager puts it to all three Scene partitions. This is not good because this animation is unloaded and loaded every time the hero changes scenes. Unfortunately the current version of Puppeteer is not smart enough to move it to the Base. It is in my plans but there are many tasks with a higher priority. Thus we need to move Dodge a shot to the Base manually.

The final asset assignments are:

• Hero Skeleton - Base.
  
• Orc Skeleton - Scene 1.
  
• Goblin Skeleton - Scene 2.
  
• Troll Skeleton - Scene 3.
  
• Walk - Base.
  
• Run - Base.
  
• Jump - Free.
  
• Hit - Base.
  
• Shoot - Scene 2.
  
• Block a hit - Base.
  
• Block a shot - Base.
  
• Dodge a hit - Free.
  
• Dodge a shot - Base.
  
• Scene1 - Scene 1.
  
• Scene2 - Scene 2.
  
• Scene3 - Scene 3.

Basically you can simplify it by putting all assets except the Scene Assets to the Free Partition, and let the Asset Manager create all the right partitions using asset dependencies. But as I already mentioned, the current version does not distribute them in the most optimal way.

Last Week

It was not a quite productive week. I continued stress testing. Found a number of critical bugs, fixed them. Found a number of performance issues, fixed them too. Boring.

Reworked the callback system again. I did it a few times already. Hopefully this version is the final one. At least I do not see any significant weaknesses in it.

To be more specific, I reworked the way the messages were suspended/resumed. When Puppeteer executes a bulk operation, it normally suspends callbacks before starting and resumes them at the end. This is a standard performance approach.

This is how callbacks were implemented before the last week. The code here is simplified to be more clear:

public event EventHandler<ProjectEventArgs> OnProjectChange;

internal void Event(EventsMode aMode, object aSender, ProjectEventArgs aArgs)
{
    if (mSuspendEventsCounter > 0)
    {
        if (mSuspendedEventsHead == null)
        {
            return;
        }

        SynchronizationContext context = Environment.SynchronizationContext;
        if (context == null)
        {
            if (OnProjectChange != null)
            {
                OnProjectChange(aSender, aArgs);
            }
            return;
        }

        SendOrPostCallback handler = delegate(object aObject)
        {
            PuppeteerEventArgs args = aObject as ProjectEventArgs;
            if (OnProjectChange != null)
            {
                OnProjectChange(aSender, aArgs);
            }
        };

        if (aMode == EventsMode.Asynchronous)
        {
            context.Post(handler, aArgs);
        }
        else
        {
            context.Send(handler, aArgs);
        }
    }
}

internal void Event(ProjectEventArgs aArgs)
{
    Event(Environment.EventsMode, aArgs);
}

public int SuspendEvents()
{
    if (mSuspendEventsCounter < 0)
    {
        mSuspendEventsCounter = 0;
    }
    return ++mSuspendEventsCounter;
}

public int ResumeEvents()
{
    if (mSuspendEventsCounter <= 0)
    {
        mSuspendEventsCounter = 0;
    }
    else
    {
        --mSuspendEventsCounter;
        if (mSuspendEventsCounter <= 0)
        {
            Event(this, new ProjectEventArgs(ProjectAction.Reset));
        }
    }
    return mSuspendEventsCounter;
}

As you can see the callbacks can be asynchronous, and they are thread safe. They are always executed in a synchronization context. Environment is an initialization class for the Puppeteer Asset Manager. This is how it is initialized in puppeteer's Main:

...
Environment.SynchronizationContext = SynchronizationContext.Current; Environment.EventsMode = EventsMode.Asynchronous;
...

Which means all events are executed in the main thread, and they are asynchronous. When the ResumeEvents is called it is sending Reset event. The controls reset their content on this event. The biggest drawback of this approach is that controls are loosing their state. For example, suppose you have a tree view of assets and containers:


Now you copy some assets to another container, and at the end of the operation discover that all brunches are closed an the selection is lost:

Not quite good. I tried to remove suspend/resume. All the messages are asynchronous anyway, so the performance hit during the bulk operation should not be too hard. Indeed, it is not. It hits me after the bulk operation. The SynchronizationContext.Post function is using Windows message queue to deliver the callback, so the event queue becomes overflowed. It still survives, but it takes a significant time for a program to return to a normal state.

The solution was to re-implement suspend/resume:

internal void Event(EventsMode aMode, ProjectEventArgs aArgs)
{
   if (mSuspendEventsCounter > 0)
   {
      if (mSuspendedEventsHead == null)
      {
         mSuspendedEventsHead = aArgs;
         mSuspendedEventsTail = aArgs;
      }
      else
      {
         mSuspendedEventsTail.Next = aArgs;
         mSuspendedEventsTail = aArgs;
      }
      return;
   }

   SynchronizationContext context = Environment.SynchronizationContext;
   if (context == null)
   {
      while (aArgs != null)
      {
         aArgs.Call();
         aArgs = aArgs.Next;
      }
      return;
   }

   SendOrPostCallback handler = delegate(object aObject)
   {
      PuppeteerEventArgs args = aObject as ProjectEventArgs;
      while (args != null)
      {
         args.Call();
         args = args.Next;
      }
   };

   if (aMode == EventsMode.Asynchronous)
   {
      context.Post(handler, aArgs);
   }
   else
   {
      context.Send(handler, aArgs);
   }
}

// SuspendEbents stays the same

public int ResumeEvents()
{
   if (mSuspendEventsCounter <= 0)
   {
      mSuspendEventsCounter = 0;
   }
   else
   {
      --mSuspendEventsCounter;
      if (mSuspendEventsCounter <= 0 &amp;&amp; mSuspendedEventsHead != null)
      {
         PuppeteerEventArgs args = mSuspendedEventsHead;
         ProjectEventArgs suspendOutput = new ProjectEventArgs(this, ProjectAction.SuspendOutput, this);
         ProjectEventArgs resumeOutput = new ProjectEventArgs(this, ProjectAction.ResumeOutput, this);
         suspendOutput.Next = args;
         mSuspendedEventsTail.Next = resumeOutput;

         mSuspendedEventsHead = null;
         mSuspendedEventsTail = null;

         Event(suspendOutput);
      }
   }
   return mSuspendEventsCounter;
}

So all events since Suspend is called are collected to a linked list. Resume sends the whole list in one message. It also wraps them to a pair of SuspendOutput/ResumeOutput events. Controls can react on them by sending WM_SETREDRAW message.

It is not a huge change, and definitely it is not a rocket science. But it took a significant time for figuring it all out. This is just a good example of what I was doing last week.

Last week

I was doing stress testing of the Asset Manager.

I created a project with 65,536 animations 100K each. Comparing with the real projects: FIFA in EA has about 30,000 animations. Standard animation size is about 20K-30K.

I was trying to compare the performance with ANT. ANT is a similar tool that EA is using for sports games. Unfortunately I don't have an access to ANT right now. So I couldn't compare them correctly. I just tried to compare my memories with what I'm having.

The results were quite good.
1. Saving/loading single asset time is not noticeable. I don't even set the mouse cursor to hour glass.
2. Loading the project time is not noticeable either.
3. Saving all takes 17 minutes.
4. Compiling all assets takes 21 minute. Compiling is creating runtime files. I didn't implement multithreaded version of compiling yet. But the assets were implemented keeping multithreading in mind. I'm hoping to reduce this time to 7-8 minutes by using 4 threads. It's planned for this week. So I'll see it very soon.

From what I remember this is much better than ANT numbers. Single asset loading took some noticeable time. Sometimes it even popped up a dialog with a progress bar. Full compiling of FIFA project took about half an hour. Again as I said - I don't remember the exact numbers.

The main secret is - I'm not using XML serialization. There were some plans to change the serialization in ANT. May be they've done it. I don't know.

Basically I'm a huge opponent of XML. I think using XML is the best method to make your application slow and memory consuming. Every time I read about advantages of XML, I find nothing reasonable. The standard argument is "it's human readable". Nice. Look at this simple document (a random file taken from the windows system directory):

<xml version="1.0">
<scpd xmlns="urn:schemas-upnp-org:service-1-0">
<major>1<major>
<minor>0<minor>
<actionlist>
<action>
<name>magiconname</name>
</action>
</actionlist>

Is "scpd xmlns" human readable? Hmm...

Doesn't matter. A better question is: how our application benefits from the human readable format? Please don't substitute the question. I'm not asking you how the programmers benefit from human readable format. After all what is our goal: a good application or a happy development team? Well some people choose a happy development team. Considering they are a part of this team, I can't blame them for it.

But here is what application gets:
This text size is 219 bytes long. Considering unicode it's even bigger. Now look at the data. A binary writer can put all the significant data of this file into 4 bytes. XML is 55 times bigger. Which means it takes 55 times more memory, and loading is 55 times slower. If this is not a waste of memory, file space, or network traffic - you should tell me what is a waste.

What is Puppeteer (continued)

The main idea of Puppeteer is to create a simple tool for making games.

Create a few clips using MB or Blender; import them to Puppeteer; assign them to different characters; set up the transitions from one animation to another; attach keyboard/mouse/controller events to these transitions; and you are set.

Now you can load your assets to the testbed and play. The AI and animation are fully described by Puppeteer assets.

The process should be as simple as possible. At the same time Puppeteer should be able to handle a large number of assets and large memory amounts. It should not cause any problems when the project is growing.

That's the general requirements. At the same time I want to mention what Puppeteer is not:
1. Puppeteer is not an animation or skeleton editor. Blender, Maya, or MB do it better.
2. Puppeteer doesn't do skinning and rendering. That might change in the future releases, but I don't see it coming soon.

What is Puppeteer

As I mentioned before, the Puppeteer is an animation engine for video games. But not only the engine. It also has a visual pluginable tool for combining and blending animations, customizing transitions, editing scenes, and many more.

Puppeteer consists of 3 main parts: Puppeteer Asset Manager (PAM), Puppeteer Runtime (PR) and Puppeteer Modules (PM). The animation engine is implemented as a Puppeteer Module.

Puppeteer Asset Manager

What is asset management? Let's start by defining what the asset is. An asset is a program data resource that:
1. Is unavoidably required for a normal application run.
2. Never changes during the application lifetime.

The examples of assets are:
1. Config files.
2. Icons and dialog layouts.
3. Meshes and animations in video games.

the best known and the simplest Asset Management software is the Visual Studio resource editor. However, when it comes to video games the assets can be quite big to be linked to the game executable or dll. They are normally shipped as separate files, and the game loads and unloads them as required.

Ideally, an asset manager software should:
1. Allow grouping assets for an easy visual navigation.
2. Provide a simple way to edit assets of different types.
3. Provide a 3-d preview for assets that can be previewed in a 3-d space. For example, meshes, skeletons and 3-d animations.
4. Maintain assets relationships including referencial integrity and constrains.
5. Be extendable. It should provide a clear plugin API for creating new asset types on top of the existing ones. For example creating an asset that references two or more animations and blends them together using different weights. Or an asset that references a number of animations and plays them in a sequence.
6. Provide assets grouping and packaging into the runtime files, which are the asset packages loaded by an actual game.

There is a number of asset management engines for video games on the market. However I am not aware of any good open source asset managers. Collada Asset Manager has announced plans to go open source. But it's not there yet. There are some additional reasons why I think Puppeteer is better than than Collada's product, but I would rather not go into the details just yet.

Puppeteer Runtime

The PR is a static C++ library. As I have mentioned already PAM packs assets into the runtime files, and the PR provides API for loading/unloading these files. It also provides an interface for Puppeteer Plugins for instantiating assets.

Once a game loads a puppeteer runtime file, it can create assets using asset IDs provided by PAM. Puppeteer runtime doesn't make any assumptions of how these assets are used. It's up to the modules to provide the code that uses the data. All PR is doing is loading/unloading and making sure the assets are correctly delivered from the PAM.

Puppeteer Modules

Every Puppeteer Module consists of two parts:
1. A dot net DLL to be used with the PAM.
2. A static C++ lib file to be linked into the game.

Both libraries implement the same assets. The standard Puppeteer package comes with two modules:
1. Animation Core Module.
2. Animation Standard Module.

I also have plans for adding Rendering Core Module. But not in the first release.

Animation Core Module implements the following assets:
1. ChannelsAsset. Defines a set of animatable channels. Every channel has a type, which is either: float, weight, translation, scale, rotation, rotation-translation, or scale-rotation-translation.
2. SkeletonAsset. Defines a skeleton hierarchy and attaches every joint to one of the channels of the ChannelsAsset. It references ChannelsAsset for it.
3. AnimationAsset. A set of key-frames that animate a subset of channels from the ChannelsAsset. Note that Animation doesn't reference Skeleton. They both reference Channels instead. So Animation can animate different Skeletons as long as they belong to the same Channels.
4. ControllerAsset. A base class for all animation controllers. AnimationAsset is the only controller in this package. Standard Puppeteer Plugins will be focusing on implementing controllers.
5. CharacterAsset. Represents a game character. It bounds Skeleton and a set of Meshes with a static scaling array. (The Meshes will not be implemented in the first release, so currently it only bounds skeleton with scaling).
6. TransformationAsset. I don't know mush about this asset yet ;) I didn't implement is yet. May be I will change the name later on, the "transformation" is too long. Anyway, this is a base class for transformation assets.
The runtime calls them after controllers do their job. Transformation assets correct the pose before sending it to the rendering part of a game. Possible implementations are: HumanIkTransformation, FootPlantingTransformation, PhysicsTransformation. And more specific transformations like head tracking, setting emotions/attitude, etc. None of these will be implemented in the first release. But they all are on the to-do list.
7. SceneAsset. A set of character assets with initial positions. It also attaches an initial animation controller to every character. And sets Transformations.

Animation Standard Module currently implements only one asset - the BlendController. It blends a number of animations together. Animators can set a synchronization points for every animation, and the BlendController makes sure all animations reach this point at the same time.

For example animator may want to blend fast running character with a slower running character. The problem is that the steps in two animations are usually not synchronized. So animators may set a sync point when the foots touch the ground. And the controller will slow down or speed up certain animations to make sure they reach these frames at the same time.

Additional controllers I'm planning to add to the Standard Plugins in the nearest future are: BlendArray, BlendSpace and Locomotion. They all will be wrapping BlendController.

Current Status

1. Asset Manager - completed. I am working on optimizations and stress testing.
2. Runtime - almost completed. Everything is implemented but not tested yet.
3. Modules - under development. Channels, Skeleton and Animation are done. I'm working on a Controller, Character and Scene.

Overture

It all started in February 2009. The event that I was expecting the whole winter finally happened.

I was laid off from EA Canada. For those who don't know - EA is Electronic Arts (с) - the biggest video games producer in the world.

Now I'm going to disappoint you. I'm not willing to tell you why I was kicked out. And I'm not willing to condemn EA. Or amaze the world with an "unpleasant truth" about them.

Basically there is nothing bad I can tell about the company. It is a great company to work for. I liked it there. Obviously I was disappointed by being resigned. Was it my fault? Am I not good enough? Was it just a bad luck? I don't know. I am not thinking about it too much. I believe, these thoughts are counterproductive. I was the weakest link. Bye-bye.

As you might expect, I got a pretty good severance package. They paid me for five months. In addition my wife Inna started earning much more than she did before. She owns a small but growing accounting company. Doesn't matter - this blog is not about it.

The essential thing was - I found myself free, well-off, and available for any new endeavors. And I asked myself - what do I want?

The first thing I wanted was a vacation. So I spent about a month idling. That was really good.

Then I started thinking - what is next? New job? My greediness told me - go and get a new one. But my other part rebelled. I thought it was a great chance to get out of this hamster wheel called "permanent job", so I decided to start my own project.

I had 3 ideas.
1. To make a game for iPhone called Amoeba. Players control amoeba by leaning the iPhone and capturing different objects. This is pretty similar to Katamari, but I thought amoeba is more fun than a simple ball - it's sticky, green and ugly. It can eat things with a champing sound.

I abandoned the idea because it's too small and simple. I feel that iPhone market is already taken. There are too many people developing and selling. No more wild money for small programs.

2. Another idea was a bigger game. A very simplified version of World of Warcraft. Teams of real people work together to achieve some missions. The fun part is that all group members are crazy. Seriously mentally sick. How? Very simple. The game shows them fake group members, fake monsters, displays fake chat messages. Sometimes puts them to wrong chats. I mean most of the group members, monsters and messages are real. But some of them are fake.

That makes everybody to behave slightly inadequately. The real challenge for players is to find out what is real and what is not. The higher is your level the more fake things you see.

I was never serious about this idea. I knew I couldn't make it. I'm not an artist. My skills as an animator are pretty weak. And even the programmer's work is too big for one person. But I liked the idea. This is the game I would want to play.

3. And the final idea was to keep doing what I was doing in EA. I liked the project very much. I didn't like my role in the team. I didn't like the way I was treated in the team. But the project itself was fun.

We did an animation engine called ANT. (It stands for Animation Tool). It was... Actually it is an internal EA tool for animators. EA is not selling or distributing it. I'll skip explaining details here. I'll describe Puppeteer in my future posts. It's pretty similar. So you'll get a good feeling of what it is in the nearest future.

Again the concern is: this project is too big for one person. Well.. This is free and open source pluginable software. I'm hoping to get help from the community.

But before that I have to create the core architecture. This is what I am doing right now.

In my next posts I will describe what the animation engine is, and the core architecture of Puppeteer.

Introduction

Well, lets skip "hi all".

My name is Sergei Zotin.

English is my second language. So if you expect a great literature, beautiful phraseology, or something like that - the Internet is large. Don't quit searching - you'll find what you want. But not here.

If you want to correct my language - you are more than welcome. But don't expect me to become a great orator. I'm not. My English is broken, and there is not too much of a chance to fix it.

Most of the time I'm going to speak about the program I'm writing. It's an animation engine for video games. It's called Puppeteer.

This is an open source software available for commercial and non-commercial use. You'll be able to use the code, change the code, and even delete the code.

However, the Puppeteer is not ready yet. I expect to complete and publish it in November 2009. I'll report the progress here on a weekly basis. I allocated Mondays for blogging in my schedule.

Starting from today.