Horde3D

If it is possible, I would suggest to use DDS. A custom texture format is quite uncommon among engines and makes development for users more complicated since you always need custom tools (with DDS you can directly use the common Photoshop plugins) and it is a pain to debug texture assets. I think DDS supports most of the required features: compressed/uncompressed images, mipmaps/no mips, 2D/Cubemaps/Volumetric textures and as far as I know also floating point textures (but probably a less efficient compression for high dynamic range than the exponent based HDR format). It also does not seem to be overly complex and rather straightforward to load.

To the XML/binary discussion: As attila points out we have some reasonable line between binary and textual/XML data. For reasons of productivity, I think it makes sense to use XML/scripts for small files like materials that are also often modified by hand or opened for debugging. There is still much room for performance improvements: The XML parser we use at the moment is probably one of the fastest DOM parsers (clearly faster than TinyXML) but there is even a much faster way of parsing XML documents called non-extractive parsing (the idea is to just store references to the raw XML file instead of copying all names and attributes). I could also offer to replace XML by a custom, more optimal and readable syntax; something like this:

The XML doesn't bother me as such, what does bother me is that many of those attributes cannot be created or set programmatically (aside from generating XML and reloading resources).

As for DDS, support would be great, but I don't think I will be able to implement it in the near future, as I have no software that produces DDS, and no need for it either. I do have a patch to allow loading of PVRTC, but it isn't very well integrated.

_________________
Tristam MacDonald - [swiftcoding]

Normally it should be possible to set all attributes of the XML Node programmatically. Which one are you missing?

It is more the creation end - say I need to synthesise a pipeline at runtime (based on the hardware caps), there really isn't an efficient way to do this.

_________________
Tristam MacDonald - [swiftcoding]

That's true, concerning the pipelines we don't have a complete C Api for handling them, yet.

As you hinted earlier, my Horde-pipeline wrapper class saves it's state into a temporary XML buffer which is passed to horde. This is terribly inefficient though!

xml stuff should be moved to horde3d utils
core shouldn't be parsing text data, imho...

for the image formats; it should only support the formats supported by gl(and gl ext which contains compressed ones?). Also move the image upload functions(png jpg etc) to horde3d utils.

I wish

Ive not read all four pages of the thread, though before reading I was about to post on the forums to see if Horde supports DDS, or an equivalent. I have to agree that I think DDS is more suitable for generalised use due to the fact its already heavily supported, and has range of plugins for most mainstream art programs (Photoshop, Gimp, PSP).

But if does get introduced I think its important to make sure its done as fully as possible, supporting as many features such as custom mipmap levels and the numerous channels (easier said than done .

As for a new custom format perhaps Valve would be a good example, as for their own format (VTF/VMT) they use is a standalone application to import standard saved images (tga, jpg etc) to create their own VTF image files. Not that ive used it great deal but basically you can build as many channels as you want, apply filters to be used at runtime and many other things.

Either way I don't think TGA, PNG cut the mustard anymore.

_________________

Let's bring 'em out! Any old iron! Any old iron!
A door opens and a homewife brings out a rather sophisticated-looking ground-to-air missile system, and dumps it on the cart.
Thank you.

I was thinking a bit about a dds loader. Implementing it should be quite simple but there are some conceptual challenges, specific to dds integration and textures in general.

1. How to store meta information for textures, e.g. if it should have mips or not?
a) Store it in the material files. That's the current solution but different material files could have conflicting options, so the first config found is just used atm.
b) Store the information directly in the texture. DDS is great here since it supports all the required meta data (mips, cubemap, ...) but what about other formats? Tiff is the only one that has robust meta data support, no chance for tga or hdr.
c) Have some global textureConfig.txt file which assigns the configuration to a texture name. Hmm, I don't like that so much.
d) Encode options in the file/resource name with some conventions. For example mytex--nmips-ncomp.tga could mean no mips and no compression.

So the cleanest solution would be to just have dds in the engine but that is too restrictive for a general purposer graphics engine.

2. What to do with automatic power-of-two conversion? It is currently there to avoid problems on older ATI cards which have only limited support for NPOT textures. But it won't work well with dds since the dds would need to be decompressed, converted and compressed again. We could just drop the automatic conversion and make it the responsability of the application to use valid textures for the desired target hardware.

3. What about the current vertical cross cubemap layout? Should we drop it since dds has native cube map support? If not, what about a vertical cross stored as dds? This would require decompression again.

That's just the problems that came into my head after some quick reflection, I fear there will be some more when we get deeper into the implementation.

I've just started implement a dds loader, just to see the problems. I'm ill so I have some free time.

I removed the stbi loader. I think this should be in an extension or in utils. It could convert the image to dds data.

1. I think for dds it should be uploaded as it is in the file. For example if there are only 3 mipmaps in the file the opengl texture should have only 3 mipmaps.
For non-dds images I would be happy with the d) option except that in the given example the parameters would be the default paramaters (nomip, nocomp, 2d). So the name should be postfixed only if mipmap, compression or cube conversion is needed.

2. For non-compressed dds textures the conversion could be kept. But for compressed textures the speed and quality is a big problem so I wouldn't support it.
I don't know OpenGL but D3D has a conditional pow2 texture support. If this caps flag is set (it is true for all ps20 hardware) it supports non-pow2 texture without mipmaps. I i remember correctly OGL has texture_rectangle but with different UV mapping. I don't know if this has changed and has something similar in OGL too.

3. I would use the native dds cube texture support. The non-dds images the name postfix "-cube" could be used.

Some problems that I found:
- The vertical flip of textures. This is a costly operation. I think compressed textures can be flipped without decompression (just have to flip the indices inside the block). Moreover as far as i know D3D and OGL uses different mapping. To solve this UV's in vertices could be mirrored too, but I don't know what could be the best solution.

- In D3D the caps tells that the cards supports ARGB8 format instead of OGL's RGBA8. I don't know if this is just a different naming convention or different byte ordering. For float format ABGR16F is exposed.
After a little investigation I found this on an OpenGL forum: So it seems that OGL and D3D uses different naming convention (D3D's ARGB8 -> OGL's BGRA and D3D's ABGR16F -> GL's ARGB) The forum said that Ati preferres RGBA, but d3d caps tell that only old Ati cards supports the ABGR8 format. I can confirm that Nvidia has some problem with uint16 formats, i have some bad experience with this.

marciano wrote:


I wonder how could this be accomplished. I implemented a DDS only version of Horde3D(also an external stbi image loader). If someone is interested, I've attached it. Could contain some bugs, and some feature is missing, but it works well for our models. Currently I do the vertical flip in the vertex program, but I wonder if there is a better way. I wouldn't like to flip the texture as it was in horde3d...

Is this one of the reasons you are trying to multi-thread horde3d, so you can dynamically load diff pipelines?

Horde already supports dynamic loading of many different pipelines (you don't need multi-threading to do this).

Thanks for that patch attila. I managed to get my version of Horde3D working with dds files but had to modify it in order to get it to work.



This doesn't seem to work right with dds files exported from GIMP using the Gimp DDS plugin (version 2.0.6, although the same issue was in 1.22). I commented out the line for now and everything still works, but I know it's a very hacky fix. What was the reasoning for this check?

All right, so I have a new desktop on the way, which will equip me with both a decent video card (Radeon 4870 HD), and Windows Vista - once I have that up and running I would like to revisit this question. Free time may be a limiting factor, but I am going to need to get on-the-fly ST3C compression running in order to continue with the planet renderer.

A question first, does anyone have any further developments with integrating DDS support since this thread? Any progress would be great to hear for inspiration.

My gut feeling is that we should strip the power-of-two conversion out of the engine, as it is a required part of the GL 2.0 spec, and the engine already specifies GL 2.0 as a requirement. Failing that, power-of-two conversion should be disabled for DDS, as it would require decompression. Additionally, DDS files should use their native cubemap support, while other textures continue to use the vertical cross.

The simplest way I see to implement this is to add an extra texture upload hook to the renderer, to support compressed textures. This way the DDS loader can be implemented without making huge changes to the existing hooks and loaders.

_________________
Tristam MacDonald - [swiftcoding]