Horde3D

Ok, we managed to modify the structure so that the terrain can be compiled without having to be merged with the SDK directory. This makes especially our internal development more productive.

Thanks for the good wishes... I think I'm recovering now

It's good that you no longer need to copy files around. But you still have to alter the engine code and build files in order to specify which extensions you want to use.

It would be great to have a plugin architecture where applications can load and register plugins without any changes to the Horde3D core. Since templates are not object code, I don't see why they need to be exported in a DLL, unless you want to export particular specializations of a templated function or something. As for classes, I'm sure I've seen this done in Windows DLL code. What problems did you encounter?

PS. I also don't really understand why there are now two Extensions collections - one inside Horde3D, the other outside. Can't we keep Terrain inside Horde3D/Extensions? The extensions will be closely tied to particular versions of Horde3D, so it makes sense to keep them inside the same code tree in case of branching/tagging/etc.

Having a real plugin architecture was the original idea. We didn't try out anything but Volker did some investigations and found the following article:

http://www.unknownroad.com/rtfm/VisualStudio/warningC4251.html

I hadn't read everything in detail (was busy with my facial rendering project) but from what we understood it seems to be impossible to export any STL containers except vectors. Might be there are other ways but due to our limited time we decided to make the extension mechanism instead of the plugin system. Personally I still think it is acceptable to add the three lines of code to integrate the plugin/extension. Also Horde has this pure C-interface so I don't like the idea to invest a plenty of effort to get all the internal classes exported just for the plugins. But as already mentioned, if someone knows a good way to realize that we could try to improve the system.



You must have missed an update that we did yesterday evening. The internal Extensions directory has been removed. We have the Extensions outside because that way we can just check out the Horde SDK from the svn without any extensions. I think this makes sense since the terrain is also distributed as a separate archive (to keep the SDK small and be able to make individual releases of the terrain extension which we expect to be updated more frequently).

Due to current problems with the terrain extension under Linux I want to write a few things from my perspective:
For any project of a reasonable size which wants to use Horde3D it certainly won't be a problem to compile a special version of Horde3D with all needed extensions enabled as a single shared library. That just goes hand-in-hand with having a project local copy of a library to avoid constantly updating to the current upstream version which might break things. As a default solution for small projects one could just enable all extensions and ignore them.

It would be a great advantage if the build system could be used to generate, maybe using macros, the egExtensions.cpp from the requested extensions and link everything as needed. A single build system used for all platforms would make this certainly easier to implement.

I have also been thinking along these lines. CMake would be the preferred build-system for this, as it is the only one with absolutely full MSVC and Mac support out of the box. It is also very friendly to anyone with a makefile background.

_________________
Tristam MacDonald - [swiftcoding]

I think we need to reopen the issue of dynamically registering plugins - here are the reasons, off the top of my head:

• Static plugins force the recompilation of the whole engine to add/remove a plugin
• Static plugins are automatically covered by lgpl, so there is no way to integrate closed-source software as a plugin
• Dynamic plugins are trivial on Linux, Mac, and even possible on Windows using mingw

OK, so the problem with dynamic plugins is MSVC, in particular exporting classes which contain a std::container from a DLL. Microsoft seems to say this is not possible, so we had perhaps better avoid the problem altogether (I wonder about this though - OGRE's public interface is littered with STL classes, and it works fine with MSVC).

As far as I can tell, the only STL class that is widely used publicly in Horde's C++ interface is std::string, which can be exported from DLLs. Most of the rest of the containers are private or protected in their surrounding class, and so with a little work we can hide these entirely using the PIMPL idiom (or a similar technique).

There are a couple of classes (such as PipelineCommand), which do publicly use std::vectors, and for these we would either have to rewrite them substantially, or place them out of reach of the plugin API.

I am pretty sure this is achievable, to provide a Plugin-API that encompasses most of Horde's internal functionality, and pretty sure it is worth the effort - what do you all think?

_________________
Tristam MacDonald - [swiftcoding]

I think you are right. I'm currently working on a component based game engine that makes intensive use of plugins, even with MSVC I couldn't find any big problems so far. I'm using a slightly adjusted version of the Ogre DynLib class to load the plugins.

So for future versions I don't see any reason why not to change the plugin system to dynamic libraries. But I think we will first finish the 1.0.0 stable.
So far there are still some small things we want to fix. But I think we can finish the 1.0.0 release within the next two weeks.

The next steps I'm planning are a configurable maximum number of log messages (already integrated in my own version) and the dynamic loading of plugins. Next to that, it would be nice if we can enhance the occlusion culling to the mesh layer. Currently only whole models are considered, which is not very efficient when using e.g. a city model that consists only of several meshes instead of one model for each house.
Apart from that, the collada converter thing is still something to be improved.

I'm really looking forward to this, since that's exactly how my scene is made.

This was a misunderstanding between Volker and me. Horde supports occlusion culling for meshes. The only limitation is that one mesh of a model can't occlude another mesh of the same model. But if you use different model nodes it's all fine.

Now I'm confused My city environment is one model with a lot of meshes. Does occlusion culling work then?

No, and my comment above is not in contrast to what Nicolas said. If you have a city model and there is another model e.g. a character behind one of the city meshes the character will be recognized by the occlusion culling. If you have a house mesh behind another one, the house mesh won't be recognized by the occlusion culling because it is a mesh within the same model.

Do you have a link about this MSVC / STL exporting problem? I primarily use MSVC for work, so I can test some implementations if need be.

As far as I know, MSVC will happily export anything from a DLL, including STL classes. The problem with doing so is a Windows/DLL limitation. On Windows, each DLL is assigned it's own heap.

Here's an example: "DLL A" initialises a std::string to "a", and then "DLL B" assigns that same string to "foo". This means that DLL A allocates some memory in DLL A's heap. Then DLL B tries to free that allocation from DLL B's heap, which causes heap corruption in DLL B and a memory leak in DLL A.

All of this occurs simply because DLL A and DLL B both link to the STL statically - if the STL was in it's own dynamic library, then all STL allocations would occur on the heap of that DLL (but I don't think that's possible to do...).

So in short, you *can* export anything on Windows, you've just got to be careful not to take ownership of any dynamically allocated memory that is exported.

For example, it would be absolutely fine to export a read-only std::vector, because then memory-ownership won't be transferred:

__declspec(dllexport) const std::vector<int>& GetVector();

I don't have a link, and your explanation agrees with my understanding of the problem. I think we should be able to compile an extension into a DLL (probably the Terrain extension), and see if it works.

_________________
Tristam MacDonald - [swiftcoding]

So I thought, that's why occlusion culling within a model would be a great feature. When exporting a scene
with a lot of buildings from 3DSMax it's very hard to make every object a separate model.

Occlusion culling is not completely free and adds some overhead. So if you have a scene with no occlusion and enable occlusion culling it can even run slower than without. If it is only enabled for models I think we have a good compromise for most cases.

Ideally the preferred way to build your city is to create each building in Max and assemble the city in a scene tool like the Horde editor. I know that this is not always practicable. But even if you have one single model for your city you can still have occlusion for your entities like cars or people.