Horde3D
http://horde3d.org/forums/

JSON instead of XML
http://horde3d.org/forums/viewtopic.php?f=1&t=708		 Page 1 of 2
Author:  jimbo [ 21.04.2009, 21:31 ]
Post subject:  JSON instead of XML
Would it be an idea to move to JSON for the data files? It's more lightweight, human readable, widely supported and easy to convert from XML. The shaders have already moved away from XML. An example:

Code:
<Material>
 <Shader source="model.shader" />
 <ShaderFlag name="_F03_ParallaxMapping" />
 <Sampler name="albedoMap" map="ball.tga" />
 <Sampler name="normalMap" map="ball_normal.tga" />
</Material>

Code:
    material:{
        shader: {source:"model.shader"},
        shaderflag: {name:"_F03_ParallaxMapping"},
        sampler:[
            {name:"albedoMap", map:"ball.tga"},
            {name:"normalMap", map:"ball_normal.tga"}
        ]
    }

or even more tight:
Code:
    material:{
        shader: "model.shader",
        shaderflag: ["_F03_ParallaxMapping"],
        sampler:[
            {name:"albedoMap", map:"ball.tga"},
            {name:"normalMap", map:"ball_normal.tga"}
        ]
    }
Author:  marciano [ 22.04.2009, 23:14 ]
Post subject:  Re: JSON instead of XML
JSON looks interesting but I see two problems: it does not seem to support comments and it would be even more difficult to reuse it as simple scripting language (required for pipelines). Although Horde has a lot of XML, I am not a huge fan of XML per se. It is just used because it is very practical and everyone knows it. But I would be open to replace it by another (also custom) syntax if that would bring a real advantage.
Author:  DarkAngel [ 23.04.2009, 03:03 ]
Post subject:  Re: JSON instead of XML
AFAIK json comments are just /* comment */

XML isn't very appropriate for a scripting language either, because strictly speaking, the order that elements are stored in a file is supposed to be irrelevant (XML stores sets of data, not sequences of data).

What about in the long run if all of Horde's XML functionality was moved into a "file to class translation" module (responsible for converting data from disc into internal classes/objects). Then perhaps the file formats (XML/JSON/binary) could be changed with Horde extensions?
Author:  Siavash [ 23.04.2009, 03:46 ]
Post subject:  Re: JSON instead of XML
This is a little offtopic : Is there any way to compile XML files to hide the Shaders, Scenes and ... from intruders?
Author:  DarkAngel [ 23.04.2009, 07:25 ]
Post subject:  Re: JSON instead of XML
Siavash wrote:
This is a little offtopic : Is there any way to compile XML files to hide the Shaders, Scenes and ... from intruders?
Yes, Horde's resource loading code doesn't actually implement file-loading (it only loads resources from memory, not from files).

Horde3D utils is the part that actually loads files - so you need to replace this utility file loading code to use a "virtual file system" (which could be encrypted to stop people from looking at the files).
If you're interested in this, I would suggest using PhysFS as a starting point (it implements a .ZIP virtual file system).
Author:  jimbo [ 23.04.2009, 09:38 ]
Post subject:  Re: JSON instead of XML
Optimally I would like s-expressions, but that might scare people...
Author:  swiftcoder [ 23.04.2009, 21:00 ]
Post subject:  Re: JSON instead of XML
If you are going to move away from XML, I would suggest skipping the intermediate stage of JSON, and moving all the way to YAML.
Author:  marciano [ 24.04.2009, 11:25 ]
Post subject:  Re: JSON instead of XML
DarkAngel wrote:
AFAIK json comments are just /* comment */

I also though that at first but it does not seem to be in the spec, at least that's what I found by some googling.

DarkAngel wrote:
XML isn't very appropriate for a scripting language either, because strictly speaking, the order that elements are stored in a file is supposed to be irrelevant (XML stores sets of data, not sequences of data).

Yes I know, but it sort of works :)

DarkAngel wrote:
What about in the long run if all of Horde's XML functionality was moved into a "file to class translation" module (responsible for converting data from disc into internal classes/objects). Then perhaps the file formats (XML/JSON/binary) could be changed with Horde extensions?

I think we should stick to some well-engineered but fixed file formats in Horde. Although I like flexibility, too much of it can hurt. It would become nearly impossible to write tools if everyone uses his own formats (e.g. different material syntaxes using XML or JSON). If someone really requires a different format, it is easy to change the loading code but we shouldn't encourage that. Much of an open source engine like Horde is the community projects and tools around it, so I think we should not do anything that complicates this development.

swiftcoder wrote:
If you are going to move away from XML, I would suggest skipping the intermediate stage of JSON, and moving all the way to YAML.

I would say it is ok to stick with XML for now. Our XML files are not that big and I think they are readable enough. The first example that jimbo brough is slightly unfair since the XML did not contain any indentation. The big advantage is that everyone knows XML and has a parser for it (when creating small tools).
Concerning performance: I started writing my own extremely lightweight XML parser some time ago. It is non-extractive (just stores references instead of copying data) and uses node pools to minimize the number of memory allocations. It should be really fast so I don't expect any performance problems with XML files.
Author:  swiftcoder [ 25.04.2009, 03:13 ]
Post subject:  Re: JSON instead of XML
marciano wrote:
Concerning performance: I started writing my own extremely lightweight XML parser some time ago. It is non-extractive (just stores references instead of copying data) and uses node pools to minimize the number of memory allocations. It should be really fast so I don't expect any performance problems with XML files.
There are quite a number of these already out there - you might want to have a look at rapidxml and fastxml, before committing to maintaining a custom solution ;)
Author:  attila [ 25.04.2009, 18:10 ]
Post subject:  Re: JSON instead of XML
We are using http://code.google.com/p/pugixml/ in our title. In our test this was the fastest and required the less memory, we haven't checked fastxml though.

I think XML is the best solution, the worst I can imagine is a custom format. There is no need to reinvent the wheel... (Simplicity rulez. :) )
Author:  marciano [ 25.04.2009, 18:59 ]
Post subject:  Re: JSON instead of XML
swiftcoder wrote:
There are quite a number of these already out there - you might want to have a look at rapidxml and fastxml, before committing to maintaining a custom solution ;)

Thanks for the links! Especially rapidxml looks very interesting. BTW, my own parser is already working (was just a longer evening of work) but it only takes a quite limited subset of XML so I would not be against using something more proven (like rapidxml). But I guess the XML parsing is not a bottleneck for anyone at the moment so I would consider it as one of the later optimization steps.

attila wrote:
We are using http://code.google.com/p/pugixml/ in our title. In our test this was the fastest and required the less memory, we haven't checked fastxml though.

Pugixml looks also interesting, although according to this (biased) benchmark, rapidxml seems to be a bit faster. But the difference is really marginal.

attila wrote:
I think XML is the best solution, the worst I can imagine is a custom format. There is no need to reinvent the wheel... (Simplicity rulez. :) )

The only real motivation for a custom format would be that we could create one that can embed binary data. So model and animation files could also become more readable. But anyway, that's not essential and XML has the other advantages of popularity and broad support so we will probably keep it.
Author:  swiftcoder [ 26.04.2009, 19:00 ]
Post subject:  Re: JSON instead of XML
attila wrote:
I think XML is the best solution, the worst I can imagine is a custom format. There is no need to reinvent the wheel... (Simplicity rulez. :) )
I am not disagreeing with your overall point, but XML is not really designed for simplicity - if that were the only goal, JSON or YAML would be better, being specifically designed for simplicity, and also widely used (FaceBook, Google, etc.).

As is, we don't really derive any benefit from using XML, since a) nobody wants to hand-craft these files, b) our format isn't widely supported by 3rd party tools, and c) we aren't using Schemas, DTDs, etc. (the one area where XML truly excels).
Author:  DarkAngel [ 27.04.2009, 03:57 ]
Post subject:  Re: JSON instead of XML
marciano wrote:
I think we should stick to some well-engineered but fixed file formats in Horde. Although I like flexibility, too much of it can hurt. It would become nearly impossible to write tools if everyone uses his own formats (e.g. different material syntaxes using XML or JSON). If someone really requires a different format, it is easy to change the loading code but we shouldn't encourage that. Much of an open source engine like Horde is the community projects and tools around it, so I think we should not do anything that complicates this development.
That's a good point..

It's pretty obvious though that everyone has a different preference for file format though!

Would you be interested in non-human readable formats (e.g. binary material files, etc)?

If there were two standard file formats (human/XML and binary), which were interchangeable and could be converted from one to the other, then people could use YAML etc in their own work-flow, as long as they write their own YAML<-> binary converter.

Then if someone needs to view a YAML file in XML form then a binary interchange format makes it possible: YAML -> binary -> XML.
Author:  swiftcoder [ 28.04.2009, 15:37 ]
Post subject:  Re: JSON instead of XML
DarkAngel wrote:
Would you be interested in non-human readable formats (e.g. binary material files, etc)?

If there were two standard file formats (human/XML and binary), which were interchangeable and could be converted from one to the other, then people could use YAML etc in their own work-flow, as long as they write their own YAML<-> binary converter.

Then if someone needs to view a YAML file in XML form then a binary interchange format makes it possible: YAML -> binary -> XML.
Binary files would be nice on the face of it, but since by production time you tend to have all these files in a compressed, packed-archive format, it doesn't really matter. Also a binary format means we need to deal with endian issues, etc.
Author:  fullmetalcoder [ 28.04.2009, 19:44 ]
Post subject:  Re: JSON instead of XML
swiftcoder wrote:
Also a binary format means we need to deal with endian issues, etc.

I don't think endianness is the biggest issue with binary files. It requires some care during the design phase (either indicating the endianness used upon storage in the file or using a fixed endian and converting upon read/write on machines that need it) but is relatively easy to deal with.

In my experience storage of recursive data structures and backward/forward compat are much bigger concerns than endianness (none likes to maintain several loaders/writers just to keep support for old file formats).
Page 1 of 2 All times are UTC + 1 hour
Powered by phpBB® Forum Software © phpBB Group
https://www.phpbb.com/