Horde3D

We are currently thinking about taking out the scene tree (transformation hierarchy) from the Horde core. Why?

The classical example for a scene tree (or graph) is a car where the four wheels are attached to the chassis. This was how a virtual car might have been built in the early 90s but who would do it like that today? Either the car is completely modelled in a DCC tool and the wheels are part of the model as static meshes or the wheels are physicalized. In the latter case, a physics engine would completely take care of computing the transformation of the wheels and the hierarchical transformation on 3d engine side would just be disturbing.

Sure, the scene tree can be convenient if you want to quickly attach a camera, light or particle system to an object (like in the knight sample). However, if you consider a game engine where all interactive objects are in some way represented as entities, you need to somehow store the relation between objects (e.g. camera is attached to character) on a higher level than that of the graphics subsystem anyway. If the 3d engine is doing the transformation in that case, it can even complicate things because then the game or physics has to read back the transformation after horde has updated the scene tree. It would be easier in that case if the game would compute the transformation of the attached camera and just submit the matrix to horde.

So I begin to think more and more that is not a good idea if the 3d engine (and even less the renderer) is modifying transformations. Without a scene tree, all horde scene objects would just be added directly into a flat list. Besides making the game engine integration strategy clearer, this would also help to further simplify the horde code and would definitely be beneficial for performance, as the scene tree does not have to be updated every frame (an operation which causes a lot of cache misses).

The change would have the introduction of model resources as dependency. Along with it, we would probably give more control over the model update as well. It needs to be possible to explicitly and immediately apply the skeletal animation to a model, so that the joint matrices can be read back to manually attach an object to a joint. A simple function updateModel could take care of that and control as well when morphers and software skinning should be applied. BTW, if anyone really needs it, it would be possible to have a scene tree in the utility library.

At the moment this is just an idea but if it gets realized it would mean a relatively drastic change to the way horde is working now, so I want to ask first if there are any major concerns.

I've actually thought to myself for a while that the Horde library is kind of 2 libraries glued together -- one half is a rendering API and the other is a simple scene graph

I don't think I've any objections to the scene graph part being removed and just focusing on the clean graphics API part

The main complication is that with a flat list, the application/client will really have to perform scene management itself. This isn't a problem, but it means there will need to be a mechanism for the app to tell horde how to traverse/cull a scene.
For example, say I've got 100,000 model nodes and in the app I've got a neat data structure to quickly cull them down to a visible set of 100 (for any camera viewpoint).
My Horde pipeline contains different viewpoints (e.g. main camera and shadow-casting-light viewpoints). Whenever Horde switches viewpoints, it should query my application for the correct visible set.

The first thing I think of when presented with this is some kind of call-back function -- this is fine for me, but -- what kind of problems does this present for binding with other languages?I'd recommend doing this so people can still easily get started with Horde

Me too, and I would also suggest to release a Beta 5 before doing such a change, because the editor probably has to be refactored dramatically when the scene graph would be removed, and that could take some time. So we could at least release a Beta 5 with a working editor as an official release.

I'm also in favor of having a scene graph in the utility library when it's removed from the core. That's ideal for small test applications or the knights / chicago examples.

Python bindings can also support callbacks, but I don't know how fast / slow that would be.
D and Java would also work.

Actually, Horde provides four areas of responsibility at the moment: a basic 3d engine taking care of culling, a rendering front- plus backend, a low-level animation system and a basic asset conditioning pipeline. I think we shouldn't get completely rid of the 3d engine part. Even if we remove the transformation hierarchy, Horde can still have a spatial graph internally which does the culling based on some culling primitives provided by the application (portals, anti-portals, etc.). I think that making Horde a purely submission based renderer would put too much burden on most smaller projects that would have to do the culling then by their own.


We could make the culling part optional (still part of the core though) and provide a mechanism that the user can specify render queues and which objects should go into what queue. In contrast to the transformation hierarchy (scene tree), the culling which in the end is just generating the render queues, is more orthogonal and wouldn't affect the performance or API style so much when not used.


I'm not a fan of callbacks, especially if they are C-style. But the main reason why I would be against them is that they will probably not work too well with multithreading. If we stick to a push approach (the user updates the transformations explicitly), Horde could double buffer data internally and completely transparent to the application.


Yeah I agree. That would also give everyone a chance to build up a clean repository that can automatically be converted with the new batch processing of ColladaConv once the changes are in effect


This is true, for very small projects it is just convenient to have.

What about calling it 1.0 (final)? And then resume with a "normal" version scheme?

Really, just calling it a beta like current versions doesn't represent the situation. If you want it to be seen as unstable, use either a 0. version scheme or like the linux kernel odd numbers for development, even for releases (1.2 would be the next stable release then). I don't quite like the horde3d versioning scheme atm -.-


Language bindings often don't like C style callbacks - but they even less like C++ style interfaces.

Btw. Multithreading in Horde3D would be quite interesting as I currently have a mess of code mapping horde3d into thread safe classes. That would be quite hard though and might be a serious performance hit (at least in some functions).

In any case, one probably could make Horde3D quite a bit more modular.

For us the scene graph isn't really useful. We are using physics based nodes.

I think horde3d should have a (maybe multiple) visibility system. It would
be great to have multiple options: render everything, Simple frustum culling, BSPtree, Octree, Portal system, PVS, ...
I think instead of callbacks these should be implemented as extensions or as part of the core.

A Final 1.0 version would be great. I think the "beta" has a wrong message to the users,
especially because Horde is in beta stage since 2008-04-12. I've used Horde3d, and it was quite stable. More stable then some commercial engine...
So you should really use a different versioning scheme, something like phoenix64 suggested.

There could be a new horde type, SceneGraph (like Node/Resource), which the user can add renderables to (Models/Emitter/2D), and Horde can query a render queue from. The most basic SceneGraph could just return the full list of renderables (unculled). Extensions can add new SceneGraph types with culling based on trees, portals, etc. Each SceneGraph type can define it's own culling primitives, like sectors/portals/trees/nodes/leaves...
This would move scene management out of the Horde core into extensions, leaving horde just to process flat lists of renderables.
A "standard extension", could implement a basic scene hierarchy similar to the current node-based techniques.

Seeing Nodes are part of the current scene-management techniques, it would be possible to redefine what goes into a render queue at this point... A Renderable object could have a type enum, a pointer to a Model/Emitter/Terrain/2D object (are these resources now?) and a StateBlock object. StateBlocks could be filled with transformation, animation and shader-uniform data. The leaf-nodes of SceneGraph objects would usually contain lists of Renderables. After merging render-queues gathered from the SceneGraphs, the renderables could be sorted by their type enum and separate ranges passed to drawModels/drawParticles/etc. The material (and animation) data could possibly have been written to a StateBlock in advance, so that there's no branching out to calculate it in the middle of rendering.
For (single-pass) forward-lighting, a list of lights could have also been written into the StateBlock...
Actually, gathering the lights that affect each model before rendering them would solve the lit-alpha ordering issue

Although there might be some details that may have to be discussed further, I love the idea of different scene graph implementations using extensions.

I agree that the current naming scheme is not optimal but for now we are stuck with it. Horde 1.0 final will be released when the engine is good enough, not just for changing the naming scheme


Do I understand correctly that you are always referring to spatial graphs when talking about "scene graphs" and not any transformation hierarchies? Having spatial graphs as extensions is a good idea and the basic infrastructure for them is already existing with the dedicated SpatialGraph class where individual scene nodes can be registered, independent of the scene graph.


This is something I was thinking of as well some time ago. A clearer distinction between renderer frontend (high-level, platform-independent pipeline handling) and backend (low-level, abstracted view of draw calls) would be clean and help to improve performance. If the backend which pushes the drawcalls to the GPU command buffer was simple enough, it could even act directly as the API abstraction layer (we would have different backends for GL, GL ES, D3D, etc.).

Yeah -- a spatial structure that given a view-point can return a list of visible models.

just instead of moving scene graph to some kind of a utility lib, just make a h3dLockSceneGraph() function which will immediately "if (locked) return;" in every scenegraph funcion.

So you're working soon on scene graph exclusion soon? I think it's a *great* idea, and I'm really excited about it because it would bring my vision of beloved engine closer.
For me, as you mentioned, it would be ideal to have division between:

renderer
culling
animation
particles
gui
<and so on>

I hate this huge blobs which are most of engines. It's a great thing you guys are developing Horde3D Engine, but one of my worries was that horde3d might go in this direction. It would block possibility of alternative solutions, and potential developers work.

So, one more thing (btw. reason why i am writing this reply). Even if i wont mention concrete implementation ideas, it would be great of not being bogged down to C++ in implementation of this components.

Thanks for listening,
Bye

Any progress on this? I'm really looking forward this change.

Any progress on this?
and what are your plans for the future of the team?