Horde3D

Yes, would be better.


The samples are public domain / "do whatever you want". Having a good sample which shows the benefits of threads would be great.

This is a very interesting thread...

Codepoet, my first try would also have been to generate a list of nodes that need to be updated. But just going over the direct children of the root node sounds like a clever idea and should be reasonable for most cases (except when someone puts an additional group-node to the root which contains all of his scene). I also like DarkAngel's idea very much to outsource the threading to an external layer and hence keep the engine core slim and free from additional dependencies.

Another interesting thing could be OpenMP. Did you already consider using it? This would have been my first test. It offers in general a very convenient way to parallelize for-loops and could fit quite nicely to the idea of just treating the root's children (which are handled in a for-loop anyway) as parallel.

I'm actually doing this in my engine

I have a RootNode class of my own (to allow multiple root nodes / scenes) -- in it's constructor it creates a Horde group node with the Horde root node as the parent.

I am using a quick hack, which gives each camera a RootNode parameter. It saves all the visibility setup between rendering scenes, but it would also let you handle this case efficiently.

_________________
Tristam MacDonald - [swiftcoding]

I have never worked with OpenMP so far. But replacing my thread pool with OpenMP pragmas should be straight forward.
The advantage of it would be that we don't need a thread API. But then applications that already use threads can't as easily control it like with DarkAngel's idea.


The general solution is to use a list of all nodes, but that comes with a huge overhead. But with sufficiently large scenes and enough available cores that would probably be the way to go.
When the root scene node has exactly one child node, we can treat that as a special case.


I really like that idea, way better than making it a special case.

With OpenMP it is very easy to control the maximum number of threads used for a parallel section. That variable could just be exposed as an engine option.

I am trying to see if i can understand what you guys are trying to design. Basically you want to implement a producer/consumer pattern that uses a thread pool pattern (dynamic number of threads) to process each branch of the graph tree, using statistical run-time data to optimize the size of the thread pool. Is this the general idea of the design?

BTW, this online event: Jon Olick on Parallelism and Engine Architecture, November 9th - http://forums.aigamedev.com/showthread.php?t=1535 may be of interest.

I wonder if it makes sense to optimize Horde3D for more than one core if one would intend to create a GameEngine. If you are using Horde3D only for visualization issues, I guess it might perform better. But for a gameengine you have physics, ai, sound, etc. that could use the remaining CPU cores as well. I tried to do that in the GameEngine published to the Community SVN. In this approach all entities contain several components. Each type of component has a component manager and all component managers will be updated in parallel. After all calculations are finished the state of the entities' components are synchronized using an event system. In small application examples the result was not that impressive, because most CPU power was taken by the physics, while all other component managers did their calculations much faster and had to wait for the bullet physics engine.

Sorry if that comment was a bit off topic Generally I really appreciate the work done by codepoet and just wanted to focus a little bit on the usage of Horde3d in an application.

It's an interesting research topic nonetheless, personally i am following your idea Volker, and using the cores for game engine processing, this appears to be the trend on 1st order game parallel systems. Targeting quad cores, i have divided the cores into: Rendering (Video and Audio), Data (De)compression, Logic/Animation/AI and Physics.

BTW, did you ever play around with the idea of dividing each of the components into smaller tasks/chunks, and use dynamic scheduling so that the system did not have to wait for a specific thread (in this case the physics thread). This to me looks like the hardest thing to do. Create a division of tasks and data chunks that works well, for the time being i am using a tolerant rendering algorithm, so it uses the correct available data not the most current (though hopefully in most cases the data will be the same (best+current))...

Maybe one could use bullet's integrated multithreading support to improve performance.

Here is a quite interesting presentation:

http://www.lrr.in.tum.de/~klug/events/G ... DGames.pdf

For a game engine that has a similar design as the Smoke framework (using a job pool for modules and smaller subtasks) it makes really sense to use callback functions as DarkAngel proposes.

Volker: I don't know if it's a good idea to add the dependency on bullet if the program which uses Horde3D does not use it other than for threading.
Regarding physics, AI, etc. you are right: These tasks can use the CPU more efficiently, especially if they don't need to do complicated scatter / gather operations.

So I wont do further work on this patch, but it was a nice experiment. If anybody is interested in trying it out and adapting the code, it should be easy enough to follow what I did.

No no, I didn't want to use bullets multi threading system for Horde3D. I just wanted to mention that one could use Bullet's multi-threading system within bullet. If I remember correctly, it is not active by default and so all physic calculations will be done in one thread.

ok

It definitely was, thanks a lot! I think such experiments are very valuable even if they don't result in a new feature immediately.