Horde3D

I have been working with Horde on a scene with medium complexity (30,000 polys, 30 materials, 50 textures and 12 shaders). However my loading times have seemed quite long (over 2 minutes in debug), so I decided to investigate. The vast majority of the time is taken loading textures, so I experimented with adding texture compression. I have found this to be a big win, and allows me to iterate changes much quicker. I thought it might be useful to share what I've found and it would be great to discuss how this could be implemented in the main source.

My current scene contains 50 textures, these are roughly evenly divided between tga and png. This takes just over 90 seconds to load on a debug build and just under 45 seconds on a release build. The majority of the textures are 512x512, the biggest are 2048x1024. The size of the textures on disc is a surprisingly large 100MB. The majority of this data comes from tgas, although more compressed textures like .png in general take longer to load, as it is bound by the decompression process. My system is a 1.66Ghz dual core intel with a 8600 GT graphics card running Vista, however I have tested the same and similar scenes under a variety of machines with similar results, predictably its proportionally faster in line with the processor speed.

I replaced the stbi loading code with the SOIL library which is based on the stbi texture loading code Horde uses, but adds support for .dds texture loading. I also added a script that will compress all the textures referenced in a .dae file and copy them into the correct directory. Direct Draw surface textures are designed for use with DirectX, but are compatible with OpenGL. However the openGL coordinate scheme stores textures bottom to top, rather than top to bottom in the y axis. Compressing the textures to .dds took the data size from 100MB down to 17.5 MB, and this also now contained all mip-maps. Initially I loaded the .dds files and then flipped them at run-time before uploading them. This halved the texture loading time from 45 seconds to 21 seconds on a release build (results under debug were similar). I then took the Nvidia texture compressor and added the ability to flip the textures as they were being compressed. Now loading the textures directly, without flipping them, they loaded in 9 seconds on debug build down from 90 seconds, which makes a huge difference!

At the resource loading stage I hacked in a check for texture resources, which will check for the filename with a .dds extension before trying to load the actual file.
To implement this system in a more maintainable way, that could be used in the main branch of Horde, there are a few options:

Create a custom file format at run-time:
Horde could load a custom compressed file format (similar to .dds, or just a flipped .dds file) if it exists on disc, otherwise it could create this file whenever it loads any other image type. Whilst this doesn't require an offline tool and is platform independent, it means that loading times will be unpredictable - something that loads in a flash and doesn't even need a loading screen on my machine, will hang for 2 minutes on somebody else's machine as 50 .tgas are loaded, compressed and written to disc.

Just support loading of .dds:
Moving over to use SOIL seems like a good idea, doing so will give support for .dds textures. Combining this with a simple run-time flip, can half the loading time of a typical scene. This is the simplest solution, but it gives less than half the possible speed boost available. I also don't know how good support for .dds file creation is on other platforms. Another downside is that in general Artists don't use .dds files - really compression is best supported by the engine not the content.

Add a cross platform texture compression tool:
Support an external tool that pre-compresses textures, or add our own. This is quite easy to make using something like SOIL or freeimage. Horde could then either support .dds and a tool that compresses and pre-flips textures, or create a new texture format that was similar to .dds but was already flipped.

If there's a general consensus on how best to implement this change, I would be happy to do so and ideally integrate it into the main branch.

This may be true on your machine, but laptops in particular may well be bandwidth-bound by their slow hard drives, and the CPU will idle waiting for larger files to transfer.

This would be my favoured option. DDS is already well supported by most tools and engines, and is a very decent format. I would suggest providing a cross-platform compression tool (with the option to flip on creation), for those who don't have an art package capable of generating them.

_________________
Tristam MacDonald - [swiftcoding]

This is an interesting topic, I was already thinking about that a few months ago. For a fully professional and integrated game engine I would go for a mixture of all three propositions. The engine core just supports loading of dds but there is an external command line tool that can convert other image formats to dds. This tool is called by the engine when the dds file of a texture is missing (or not up to date). The tool is only invoked by the engine in the production/development stage. For your final game you would ship all textures in dds format. The nice thing is that such a command line tool can also be used to create automatic builds from the source data that can be used for shipping.

But of course this defines a very strict art production pipeline which is not an optimal thing for an open source graphics engine that will be integrated in many different game engines and also used for small and quick projects. For example you would have to know your package file format so that the command line tool can integrate the files directly in your pak files. So I would also opt for solution three in our case. The engine supports loading of all image formats including dds and you have a special external tool which can convert images to dds. Now you can use that tool before you release your application to convert all assets to dds or you can integrate it in the content pipeline of your game engine as you wish. Since you are supposed to do the file loading yourself in Horde you can use this tool to realize what I have described in the first lines. But you can also just ignore all that and continue loading slow jpges for your mini project

afaik etqw used uncompressed raw data at development stage
but in the release they compressed textures as dds, compiled scripts etc
It clearly reduces load times(and other stuff like reading data directly into objects)

btw isnt dds just a container?

Yes, but it supports DXT and ST3C compression directly, along with pre-generated mipmaps (including for cubemaps).

_________________
Tristam MacDonald - [swiftcoding]

DXTC/S3TC are useful formats, and it would be great to have support for them. A lot of machines are more disk-bound than cpu-bound though, so it might also be worth adding support for gzip compression or similar as well. (eg .dds.gz)

Try it out on a few dds files, and take a look at the numbers. Your gzip is unlikely to compress the file by more than ~5%. DXT, S3TC and JPEG are all specially tuned for image data, and you can't generally gain much by compressing them further with a generic compression algorithm.

_________________
Tristam MacDonald - [swiftcoding]

Yes, gzip doesn't help much for detailed textures, but when the process is disk limited, it is effectively free. Also, since DXTC uses a fixed compression ratio, gzip will be much more effective for low-detail textures, such as those which have large areas of solid color. Fonts, specular maps, mostly-flat normal maps, etc.

iirc point of dxt(and s3tc) compression is, it is stored in compressed form in memory and it is decompressed only when it is used. so dxt compressed images takes less video memory.

it is different from jpg, zip, png or other compression. you can use those to gain disk space.

I think zipped data which contains dxt compressed images will be optimal
means less disk space(may be a little decrease in load times) and less video memory

In practice, yes, but I chose those examples because jpeg and dxt (which is just another name for s3tc) are very similar algorithms, both using lossy block-compression.

You are right about the disk space - I managed to achieve 30% additional compression on my terrain textures by zipping the dds files.

_________________
Tristam MacDonald - [swiftcoding]

DXTC and JPEG both compress blocks of pixels, but that's about where the similarity ends... for one thing, JPEG uses a ZIP-like compression pass on its final data. Neither DXTC nor its DDS wrapper do this, so a zip layer is much more helpful.

The main reason to use DXTC over JPEG is that the data remains compressed in video memory. Otherwise, JPEG gives better quality for typical photographic RGB images for the same file size.

With any form of optimisation, it is important to gather real-world metrics.

I don't believe zipping a DDS will decrease its loading time, it will increase it in any case I have been able to test. It would be useful for others to post metrics if they find different results. My original findings for texture compression were all performed on a laptop, I would be interested to know if other people find worse HDD read results on a machine that can run Horde.

Note from my findings that the loading times go from 45 seconds in Release down to 9 seconds when loading compressed textures directly. This 9 seconds is pretty close to the actual time taken just to read the textures. You might expect that if the textures took 30% less size, the raw data would load 30% faster. However this is of course not the case. In general there isn’t much speed difference between loading a 1MB file and a 2MB file. Loading time increases rapidly if you increase the number of files, but is not linearly affected file size (this is generally true regardless of your drive speed).

In fact when I was loading 100MB of texture data, the actual time taken to load the raw data was not that much greater than before – about 14 seconds compared to 9 (the rest of the time is spent decompressing the texture format – I did not count the time required to then compress to DXT if texture compression was enabled for the upload). Also note that loading the texture data and performing a simple flip, increased the loading time from 9 seconds to 20 seconds, even in release. Zip decompression should prove much more expensive than this. In fact, a zipped .dds should load in roughly the same time as a normal .png or .jpeg if my metrics are correct. Of course a .jpeg will take longer if it is then DXT compressed when it is uploaded.

I realise that these figures will vary per machine and situation, but I think will remain basically the same. Performing load-time decompression will not be free unless the loading and decompression processes are totally asynchronous. Whilst possible, this is difficult to add. Also In debug it is likely that you will always be CPU bound, and when I’m debugging is when I want to have the quickest turn around on load times.

One big saving would be to add support for something like PhysicsFS, this would mean that all assets for the application needed can be loaded in a single .zip file, and referenced as resources directly from memory. This would be a big win for Horde apps, I believe, as it loads many small files (materials in particular).


Right now I want to add texture compression support to Horde. DXT is a well established format for this and the only one that can be directly loaded into memory. Extra, more experimental methods, could be added on top of this. However the relative merits of different compression methods is a large and complex subject that probably deserves a thread of its own.

Sounds good - and integrating SOIL would be a very good first step in whichever case, especially since it has basically the same interface as the current STBI loader.

_________________
Tristam MacDonald - [swiftcoding]

I am getting ready to integrate SOIL into the main source tree, to add DDS support.

From a code point of view it is just a matter of replacing stbi_load_from_memory() with SOIL_load_image_from_memory(), in egTextures.cpp, however, SOIL is comprised of a couple of extra files, in addition to the main STBI file. We are talking 6 headers and 4 .c files here, so which would be best:

• merge the whole lot into utImage.h/.cpp - hard to maintain
• add the files directly to the source directory - messy
• create a new subdir in Dependencies - different kind of dependency to GLFW
• add the files to the shared sources directory - seems fairly sane

_________________
Tristam MacDonald - [swiftcoding]

I think support for DDS files will definitely make sense.

@swift: Thanks a lot for your quick initiative!
I took a look at soil and found that there is one c file (and a header) used for dds loading. I would suggest that we continue using stbi directly and just extract the dds loading code from soil. So we don't need all the additional files. The routine for uploading compressed images should be implemented in the renderer, probably by extending the RendererBase::uploadTexture2D function. Would you see any advantages for integrating the complete soil code?

In general, I was thinking about changing the texture coordinate system, so that images are defined as in DX (vertically flipped compared to GL). This would make loading dds files much simpler (we don't need to flip them which is not trivial without uncompressing them but possible) and we could remove the vertical flip from the loading code of other images. This would improve loading times. If there are plans for supporting DX somehow in the future we will need a standardization anyway. So why not use the one with the best performance. But the ups and downs of that should be discussed in more detail.