Horde3D

Next-Generation Graphics Engine
It is currently 19.04.2024, 03:04

All times are UTC + 1 hour




Post new topic Reply to topic  [ 8 posts ] 
Author Message
PostPosted: 22.07.2008, 00:22 
Offline

Joined: 02.01.2008, 18:34
Posts: 14
I am finding that in most of my shaders, the LIGHTING context is unique, but the AMBIENT and SHADOWMAP contexts are the same (they normally only change for skinning). In a scene with 15 shader.xml files each with 3 contexts, Horde actually uploads 45 sets of opengl vertex and fragment shaders (one per context).

I changed contexts so they could be shared between shaders. This brings the number of shaders uploaded in my case down to 19. This took my loading time for shaders from 20 seconds down to 8, and also took my framerate from under 40 fps, up to between 43 and 45.

I have implemented these shared contexts in a very quick and hacky way, but I'd be interested in doing it properly if it could be rolled into the main branch.

For now, I have added a check when compiling a shader - to see if another shader already exists that is complied with the same set of <InsCode> tags (and has no <DefCode> tag). If so, then these shaders share the shaderObject id between them (so one shader is uploaded between all shared contexts).

I think a more maintainable way to implement this feature is to support contexts as resources, however this potentially creates many more files, and a less obvious workflow for shaders.

What do people think? Is this a worthwhile feature and how best should it be implemented?

Another interesting problem that this could potentially solve is where you wish an object to not cast shadows (but still receive). Currently you would duplicate a shader and remove the SHADOWMAP context, but this increases the number of duplicate contexts that get compiled - here again you wish for materials to share at the context level.


Top
 Profile  
Reply with quote  
PostPosted: 22.07.2008, 22:45 
Offline
Engine Developer

Joined: 10.09.2006, 15:52
Posts: 1217
Yeah that's definately a problem that needs to be adressed in the future. In general, the goal for the post 1.0 era will be to optimize the engine for larger projects. Faster loading times (e.g. support for dds), better shader permuation handling and low level optimizations (maybe some basic SSE support) should be part of that. This will not happen in the next weeks but some concept studies (maybe in form of patches) are very welcome of course.


Top
 Profile  
Reply with quote  
PostPosted: 24.07.2008, 15:22 
Offline

Joined: 03.07.2008, 01:23
Posts: 50
Maybe the "Context" tag in the shaders could have a boolean "shared" attribute. When the Context tag with the shared attribute set is first perceived (in loading order), it would be created in a shared singleton class. If the shared attribute is not set, a usual context will be created just like currently.

If the shared context is already there, nothing will be done. Afterwards references to contexts with that name, whether specified as shared or not, would be resolved to either a local or the shared instance:

First the "local" contexts are searched like currently and when the context isn't found the shared container is used as a fallback.

I'm not sure about technical details but the same works with sharing render targets. Loading order shouldn't matter in that it is sufficient to just define the Context everywhere you need it, but with the shared attribute it will be created only once.

Then it is up to the user not to use conflicting shared contexts, e.g. the first one will always be used and not created again. Error reporting could naturally be done after comparing the code in them... maybe by calculating a hash of the code's contents and comparing that during loading.

Or possibly if you omit the code completely, the shared instance would still be used if it exists. That way you wouldn't have to duplicate shared code whenever you have a context around... This actually sounds a bit like having the contexts as stand-alone resources that shader "compilations" only refer to. However in this case you could just load a bunch of shader resources that actually have code for a single context, and the rest that you actually use would refer to those via shared contexts.

So, IMO separating shader resources from context resources is currently a waste of space and concepts. They both fit to the shader concept, if you simply allow contexts to be shared. However if you want a clearer solution, introducing an explicit "ContextLink" tag or something instead of recycling the old "Context" when referring from "empty" contexts might make it clearer.

@blue: I would be interested to see your implementation


Top
 Profile  
Reply with quote  
PostPosted: 05.09.2008, 15:33 
Offline

Joined: 19.11.2007, 19:35
Posts: 218
@cantele, a pool of shared contexts would work well enough for several cases but what about clashes where two different context's named AMBIENT need to be used by several different groups of objects? Shared pools instead of a single shared pool would probably be more ideal, based on the filename of an included resource specified in the shader xml (basically the same as #include).


Top
 Profile  
Reply with quote  
PostPosted: 12.09.2008, 21:01 
Offline

Joined: 19.11.2007, 19:35
Posts: 218
What about something along the lines of:
Code:
<Shader>
     <Context id="LinkedContext" link="linked.shader.xml" />
     <Context id="NormalContext">
               <VertexShader>
                         Code
               </VertexShader>
               <FragmentShader>
                         Code
               </FragmentShader>
     </Context>
</Shader>


Modifying the shader loader and storing the linked contexts as a lookup in the ShaderResource class. Lastly modifying
Code:
ShaderContext *ShaderResource::findContext(const string &name)
{
   for( uint32 i = 0; i < _contexts.size(); ++i )
      if( _contexts[i].id == name ) return &_contexts[i];

   //include the link lookup here

   return 0x0;
}


That's a simple solution and deals with name clashing by leaving it to the responsibility of the technical artist. In the event that a linked context can't be found (such as if the linked shader hasn't been loaded) then findContext() will still return 0x0 and nothing will be drawn, no crashing. I haven't put any thought into shader compiling issues or things like differences in material uniform names / textures / etc (that should be the responsibility of the technical artist || content creator IMO).


Top
 Profile  
Reply with quote  
PostPosted: 12.09.2008, 21:20 
Offline

Joined: 18.05.2008, 17:47
Posts: 96
hash it!


Top
 Profile  
Reply with quote  
PostPosted: 12.09.2008, 21:36 
Offline

Joined: 19.11.2007, 19:35
Posts: 218
There's already so many string lookups done by iterating through the resource vector that another one isn't going to hurt. Hashing would make sense in the event of a resource management overhaul, but there's the issue of "resourceType.xml" showing up an awful lot in strings, making djb2 or sdbm the only suitable hash algorithms.


Top
 Profile  
Reply with quote  
PostPosted: 13.09.2008, 00:16 
Offline

Joined: 19.11.2007, 19:35
Posts: 218
Here's a patch that should take care of that. No one commit this to the community branch until its been tested thoroughly. You can also see some of the stuff for AlphaTest and Alpha to Coverage in there, ignore it (you'd also be in for a surprise if you did, should you scrutinize the if tests). It's untested and doesn't do anything unless you add the necessary code to the egRenderer.cpp in setMaterialRec.

It works using the XML syntax I described previously, instead of the full blown container <Context></Context>. It's only lightly tested, I haven't tried using multiple links or other odd combinations, such as linking to a link (EDIT: Just tested that). It does add the linked file as a resource, therefore it chokes and spits it at the log if it can't find the resource, of course it does not explicitly load the data.

Code:
<Context id="CONTEXTID" link="xxxxx.shader.xml" />


Code:
Index: egShader.cpp
===================================================================
--- egShader.cpp   (revision 4)
+++ egShader.cpp   (working copy)
@@ -149,7 +149,22 @@
 
    _contexts.clear();
 }
+ShaderContext *ShaderResource::getLink(const std::string &name)
+{
+   for(uint32 i = 0; i < _links.size(); ++i)
+   {
+      if(_links[i].contextName == name)
+      {
+         Resource *t = Modules::resMan().findResource(ResourceTypes::Shader, _links[i].shaderResName);
+         if(t)
+         {
+            return (((ShaderResource*)t)->findContext(_links[i].contextName));
+         }
 
+      }
+   }
+   return 0x0;
+}
 
 bool ShaderResource::raiseError( const string &msg, int line )
 {
@@ -232,44 +247,77 @@
    {
       if( node1.getAttribute( "id" ) == 0x0 ) return raiseError( "Missing Context attribute 'id'" );
       
-      ShaderContext sc;
-
-      sc.id = node1.getAttribute( "id" );
-      
-      // Config
-      XMLNode node2 = node1.getChildNode( "RenderConfig" );
-      if( !node2.isEmpty() )
+      if( _stricmp(node1.getAttribute( "link", "" ),"") == 0)
       {
-         if( _stricmp( node2.getAttribute( "writeDepth", "true" ), "false" ) == 0 ||
-            _stricmp( node2.getAttribute( "writeDepth", "1" ), "0" ) == 0 )
-            sc.writeDepth = false;
-         else
-            sc.writeDepth = true;
+         ShaderContext sc;
 
-         if( _stricmp( node2.getAttribute( "blendMode", "REPLACE" ), "BLEND" ) == 0 )
-            sc.blendMode = BlendModes::Blend;
-         else if( _stricmp( node2.getAttribute( "blendMode", "REPLACE" ), "ADD" ) == 0 )
-            sc.blendMode = BlendModes::Add;
-         else if( _stricmp( node2.getAttribute( "blendMode", "REPLACE" ), "ADD_BLENDED" ) == 0 )
-            sc.blendMode = BlendModes::AddBlended;
-         else if( _stricmp( node2.getAttribute( "blendMode", "REPLACE" ), "MULT" ) == 0 )
-            sc.blendMode = BlendModes::Mult;
-         else
-            sc.blendMode = BlendModes::Replace;
-      }
-      
-      // Code
-      node2 = node1.getChildNode( "VertexShader" );
-      if( node2.isEmpty() ) return raiseError( "Missing VertexShader node in Context '" + sc.id + "'" );
-      if( !parseCode( node2, sc.vertShaderFracts ) ) return raiseError( "Error in VertexShader node of Context '" + sc.id + "'" );
+         sc.id = node1.getAttribute( "id" );
+         
+         // Config
+         XMLNode node2 = node1.getChildNode( "RenderConfig" );
+         if( !node2.isEmpty() )
+         {
+            if( _stricmp( node2.getAttribute( "writeDepth", "true" ), "false" ) == 0 ||
+               _stricmp( node2.getAttribute( "writeDepth", "1" ), "0" ) == 0 )
+               sc.writeDepth = false;
+            else
+               sc.writeDepth = true;
 
-      node2 = node1.getChildNode( "FragmentShader" );
-      if( node2.isEmpty() ) return raiseError( "Missing VertexShader node in Context '" + sc.id + "'" );
-      if( !parseCode( node2, sc.fragShaderFracts ) ) return raiseError( "Error in FragmentShader node of Context '" + sc.id + "'" );
+            if( _stricmp( node2.getAttribute( "blendMode", "REPLACE" ), "BLEND" ) == 0 )
+               sc.blendMode = BlendModes::Blend;
+            else if( _stricmp( node2.getAttribute( "blendMode", "REPLACE" ), "ADD" ) == 0 )
+               sc.blendMode = BlendModes::Add;
+            else if( _stricmp( node2.getAttribute( "blendMode", "REPLACE" ), "ADD_BLENDED" ) == 0 )
+               sc.blendMode = BlendModes::AddBlended;
+            else if( _stricmp( node2.getAttribute( "blendMode", "REPLACE" ), "MULT" ) == 0 )
+               sc.blendMode = BlendModes::Mult;
+            else
+               sc.blendMode = BlendModes::Replace;
 
-      _contexts.push_back( sc );
+            //Alpha Depth Test
+            if( _stricmp( node2.getAttribute("alphaTest","false"),"true") == 0 ||
+               _stricmp( node2.getAttribute("alphaTest","0"),"1") == 0)
+            {
+               sc.testAlpha = false;
+               if( _stricmp( node2.getAttribute("alphaToCoverage","false"),"true") == 0 ||
+                  _stricmp( node2.getAttribute("alphaToCoverage","0"),"1") == 0)
+                  sc.alphaToCoverage = true;
+               else
+                  sc.alphaToCoverage = false;
+            }
+            else
+            {
+               sc.testAlpha = true;
+               if( _stricmp( node2.getAttribute("alphaTestMode","greater"),"less") == 0)
+                  sc.alphaTestDirection = false;
+               else sc.alphaTestDirection = true;
+               sc.alphaTestVal = (float)atof(node2.getAttribute("alphaTestValue","0"));
+            }
+         }
+         
+         // Code
+         node2 = node1.getChildNode( "VertexShader" );
+         if( node2.isEmpty() ) return raiseError( "Missing VertexShader node in Context '" + sc.id + "'" );
+         if( !parseCode( node2, sc.vertShaderFracts ) ) return raiseError( "Error in VertexShader node of Context '" + sc.id + "'" );
+
+         node2 = node1.getChildNode( "FragmentShader" );
+         if( node2.isEmpty() ) return raiseError( "Missing VertexShader node in Context '" + sc.id + "'" );
+         if( !parseCode( node2, sc.fragShaderFracts ) ) return raiseError( "Error in FragmentShader node of Context '" + sc.id + "'" );
+
+         _contexts.push_back( sc );
+         
+         node1 = rootNode.getChildNode( "Context", ++nodeItr1 );
+      }
+      else
+      {
+         ShaderLink t;
+         t.contextName = node1.getAttribute("id");
+         t.shaderResName = node1.getAttribute("link");
+         _links.push_back(t);
+         Modules::resMan().addResource(ResourceTypes::Shader,t.shaderResName,0,false);
+         node1 = rootNode.getChildNode("Context",++nodeItr1);
+      }
       
-      node1 = rootNode.getChildNode( "Context", ++nodeItr1 );
    }
 
    compileShaders();
Index: egShader.h
===================================================================
--- egShader.h   (revision 4)
+++ egShader.h   (working copy)
@@ -60,7 +60,11 @@
 
 typedef SmartResPtr< CodeResource > PCodeResource;
 
-
+struct ShaderLink
+{
+   string   contextName;
+   string   shaderResName;
+};
 struct ShaderCodeFract
 {
    PCodeResource   refCodeRes;
@@ -91,6 +95,12 @@
 
    // RenderConfig
    bool                  writeDepth;
+   bool                  testAlpha;
+   //true is > false is <
+   bool                  alphaTestDirection;
+   float                  alphaTestVal;
+   bool                  alphaToCoverage;
+
    BlendModes::List         blendMode;
    
    // Engine uniform and attribute locations
@@ -130,6 +140,7 @@
    static string            _vertPreamble, _fragPreamble;
    
    vector< ShaderContext >      _contexts;
+   vector< ShaderLink >      _links;
 
    bool raiseError( const string &msg, int line = -1 );
    bool parseCode( XMLNode &node, vector< ShaderCodeFract > &codeFracts );
@@ -150,12 +161,14 @@
    bool load( const char *data, int size );
    void compileShaders();
 
+   ShaderContext *getLink( const string &name );
    ShaderContext *findContext( const string &name )
    {
       for( uint32 i = 0; i < _contexts.size(); ++i )
+      {
          if( _contexts[i].id == name ) return &_contexts[i];
-      
-      return 0x0;
+      }
+      return getLink(name);
    }
 
    vector< ShaderContext > &getContexts() { return _contexts; }


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 8 posts ] 

All times are UTC + 1 hour


Who is online

Users browsing this forum: No registered users and 42 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group