Horde3D

I'm thinking about a regression test sample. With it we could test the current state of the engine (hopefully finally it could be get out of beta state). And also test new features that after proper testing could get into the main trunk. For example it would help me a lot with gles2 and d3d11 renderer implementation.

After some googling I found: http://aras-p.info/blog/2011/06/17/test ... ars-later/

What do you think about this? Is there anyone who would like to work on this? I think it should be written in c/c++ or in an easily embeddable script language like lua to be able to test on many platforms.Of course this is a huge project, but after creating the regression framework adding new tests could be simple and could be done by many people.

I agree that this would help keeping Horde3d stable and would make life a lot easier if we would have multiple rendering backends. Unfortunately I won't have time working on this.

Are you going to test static scenes without anything moving around? If so, we can setup a framework to load multiple scenes to test different stuff and compare their output with still images of correct outputs stored on disk. If output difference from reference images is less than a given error tolerance, they are passed.

I think most of the tests could be static. For me (to test the renderers) the tests should be as simple as possible. Not some eye-candy high polygon stuff.
Just simple rectangles/cubes for test texture wrapping modes/filters, different compressed texture formats, shadow mapping, render to texture, particle rendering,forward/deferred rendering ...
I was thinking about doing the tests manually. With some descriptive text and maybe a shot the user should be able to decide if the test passes or fails.

Automatic tests would be better but harder to implement as comparing images is not trivial. from the unity3d forum

remember that if you want to be exact in content, you have md5 and things like that.

Hi there, this left me thinking and after thinking, I decided to take a shoot... so I have make https://github.com/tyoc213/secret-wokie2 wich uses the free travis for opensource projects.

There

[*] I have setup XFVB for run OpenGL (it is not horde, it is only a simple GLFW example).
[*] added 2 jpegs wich I compare using a javascript installed via npm or via python


You can see the CI in one of:

[*] https://travis-ci.org/tyoc213/secret-wokie2 or
[*] http://travis-lite.com/tyoc213/secret-wokie2


The linkgs to the img cmp are:

[*] https://github.com/HumbleSoftware/js-imagediff and
[*] https://github.com/kaikuehne/mirror.git
[*] and only included as a comment somewhere perhaps https://github.com/bitlyfied/js-image-similarity but I think it needs something that run the js so for the moment no command line


NOTE: that those image comparison libs should work also locally... so if you want to run some test you can do it also stand alone... the python one reports: 1.23975555556 for example... I don't know exactly what it means...


Hope this help you out somehow GL && HF.

Thanks for sharing, but I am not quite sure if comparing this way could work among different hardwares. Sometimes a small difference means a bug, for example see http://www.crytek.com/download/Playing% ... hadows.pdf on page 44-45, other times a small filtering/antialias difference in hardware is not a problem.

A yeah, you should have a "base case" for each possible configuration of HW/drivers and resolution I guess.

This is only a "simple" setup for let run it on a CI server with a headless X server setup. And a raw test for some image comparison tools.

http://humblesoftware.github.io/js-imagediff/ this one let you output the image differences as a difference over black or use it on command line with a tolerance value (mmm, haven't tested it on cmd line only for equals) which returns: true, false.

While https://github.com/kaikuehne/mirror outputs a number.

https://github.com/bitlyfied/js-image-similarity it explains a little about "average hashes".


If you see the examples, they are not as hard as the ones on the linked pdf about shadows .

http://aras-p.info/blog/wp-content/uplo ... -tests.png

I think with this little apps is enough to get at less a basic local regression test (you compare your outputs with your own hardware and move the correct images to a "correct" dir and output images to a "testagainst" dir so you could run for each image the compare, and write a little script that does the work).

-------------

If you really want to build the database for test across different HW, not only your own.


I was thinking that is necessary to build a little server that can get "hits" from different users at time they run the test... so you will be able to collect different images on different hardware without need to own each possible, and it would use those specific case for compare to others alike, or insert a new entry on it.

If possible I was thinking something on the lines of: https://github.com/tobami/codespeed/ which is used in http://speed.pypy.org/ for example, they use a interface, where you only post results and the server handles all the internals... like I know If Im correct is that this server doesn't run the test, it only gets "hits" after some tests with https://github.com/tobami/codespeed/#saving-data and something that post the result on the lines of https://github.com/tobami/codespeed/blo ... _result.py so you can run the test anywhere.

Sure, you need to trust the users posting results (or give them a key to be able to upload results)....


For example, if not images, they can post "the averages" or the whole compressed images on a zip for comparision on the server against a similar case... (also at start if there is a original test case) the local test can connect and download the images that are of the similar spec the user is running on.


Don't know... yeah, it is a whole "little project" .


----

Searching a little more I just found: http://www.ogre3d.org/tikiwiki/Visual+U ... +Framework and a link to another image comparision: http://pdiff.sourceforge.net/ another option in compare images.

I also suggest to integrate a classic static code regression test suite.

I really suggest CATCH, an open source, header-only test framework:

https://github.com/philsquared/Catch