Horde3D

Next-Generation Graphics Engine
It is currently 12.05.2024, 22:07

All times are UTC + 1 hour




Post new topic Reply to topic  [ 14 posts ] 
Author Message
 Post subject: Shader system updates
PostPosted: 11.12.2008, 23:24 
Offline
Engine Developer

Joined: 10.09.2006, 15:52
Posts: 1217
Back in the days when I had to write some more advanced shaders for my final university project, I realized that the current system has some limitations which make it not very convenient to use for more complex shaders. So I started experimenting with some new ideas. We finally decided now that it would be a good idea to take them over to Horde 1.0, even though it is a major change in the way shaders are written.

The updated shader system has three new main features:

  • Shader code is no longer written in the CData section of an XML file. This makes the shader files cleaner and more readable. Contexts are defined in a special FX section (still in XML). The change also makes it possible to share code within a single shader file.
  • The shader system has a recursive #include mechanism; the days of the XML InsCode and DefCode elements are over. Code resources can also include other code resources now.
  • Automatic shader combination generation (ubershaders) based on flags: this is the most important change. You can now use #defines for flags with a special naming convention in your shaders and set them in material files. Horde will automatically compile the required shader combinations for you. This can save you from a lot of code duplication and makes it much easier for users to enable and disable shader features.

Here is an excerpt from the new model.shader which is some sort of standard shader used for geometry. It looks much nicer with proper syntax highlighting (I have a style definition for Notepad++ if anyone is interested).

Code:
[[FX]]
<!--
// =================================================================================================
   Model Shader
   
   Supported Flags:
      _F01_Skinning
      _F02_NormalMapping
      _F03_ParallaxMapping
// =================================================================================================
-->

<Context id="ATTRIBPASS">
   <Shaders vertex="VS_GENERAL" fragment="FS_ATTRIBPASS" />
</Context>

<Context id="SHADOWMAP">
   <Shaders vertex="VS_SHADOWMAP" fragment="FS_SHADOWMAP" />
</Context>

<Context id="LIGHTING">
   <RenderConfig writeDepth="false" blendMode="ADD" />
   <Shaders vertex="VS_GENERAL" fragment="FS_LIGHTING" />
</Context>

<Context id="AMBIENT">
   <Shaders vertex="VS_GENERAL" fragment="FS_AMBIENT" />
</Context>
      

[[VS_GENERAL]]
// =================================================================================================

#ifdef _F03_ParallaxMapping
   #define _F02_NormalMapping
#endif

#include "utilityLib/vertCommon.glsl"

#ifdef _F01_Skinning
   #include "utilityLib/vertSkinning.glsl"
#endif

uniform vec3 viewer;
attribute vec2 texCoords0;
attribute vec3 normal;

#ifdef _F02_NormalMapping
   attribute vec3 tangent, bitangent;
#endif

varying vec4 pos, vsPos;
varying vec2 texCoords;

#ifdef _F02_NormalMapping
   varying mat3 tsbMat;
#else
   varying vec3 tsbNormal;
#endif
#ifdef _F03_ParallaxMapping
   varying vec3 eyeTS;
#endif


void main( void )
{
#ifdef _F01_Skinning
   mat4 skinningMat = calcSkinningMat();
   mat3 skinningMatVec = getSkinningMatVec( skinningMat );
#endif
   
   // Calculate normal
#ifdef _F01_Skinning
   vec3 _normal = calcWorldVec( skinVec( normal, skinningMatVec ) );
#else
   vec3 _normal = calcWorldVec( normal );
#endif

   // Calculate tangent and bitangent
#ifdef _F02_NormalMapping
   #ifdef _F01_Skinning
      vec3 _tangent = calcWorldVec( skinVec( tangent, skinningMatVec ) );
      vec3 _bitangent = calcWorldVec( skinVec( bitangent, skinningMatVec ) );
   #else
      vec3 _tangent = calcWorldVec( tangent );
      vec3 _bitangent = calcWorldVec( bitangent );
   #endif
   
   tsbMat = calcTanToWorldMat( _tangent, _bitangent, _normal );
#else
   tsbNormal = _normal;
#endif

   // Calculate world space position
#ifdef _F01_Skinning   
   pos = calcWorldPos( skinPos( gl_Vertex, skinningMat ) );
#else
   pos = calcWorldPos( gl_Vertex );
#endif

   vsPos = calcViewPos( pos );

   // Calculate tangent space eye vector
#ifdef _F03_ParallaxMapping
   eyeTS = calcTanVec( viewer - pos.xyz, _tangent, _bitangent, _normal );
#endif
   
   // Calculate texture coordinates and clip space position
   texCoords = texCoords0;
   gl_Position = gl_ModelViewProjectionMatrix * pos;
}


[[FS_ATTRIBPASS]]
// =================================================================================================

#ifdef _F03_ParallaxMapping
   #define _F02_NormalMapping
#endif

#include "utilityLib/fragDeferredWrite.glsl" />

uniform sampler2D tex0;

#ifdef _F02_NormalMapping
   uniform sampler2D tex1;
#endif

varying vec4 pos;
varying vec2 texCoords;

#ifdef _F02_NormalMapping
   varying mat3 tsbMat;
#else
   varying vec3 tsbNormal;
#endif
#ifdef _F03_ParallaxMapping
   varying vec3 eyeTS;
#endif

void main( void )
{
   vec3 newCoords = vec3( texCoords, 0 );
   
#ifdef _F03_ParallaxMapping   
   const float plxScale = 0.03;
   const float plxBias = -0.015;
   
   // Iterative parallax mapping
   vec3 eye = normalize( eyeTS );
   for( int i = 0; i < 4; ++i )
   {
      vec4 nmap = texture2D( tex1, newCoords.st );
      float height = nmap.a * plxScale + plxBias;
      newCoords += (height - newCoords.p) * nmap.z * eye;
   }
#endif

   vec3 albedo = texture2D( tex0, newCoords.st ).rgb;
   
#ifdef _F02_NormalMapping
   vec3 normalMap = texture2D( tex1, newCoords.st ).rgb * 2.0 - 1.0;
   vec3 normal = tsbMat * normalMap;
#else
   vec3 normal = tsbNormal;
#endif

   vec3 newPos = pos.xyz;

#ifdef _F03_ParallaxMapping
   newPos += vec3( 0.0, newCoords.p, 0.0 );
#endif
   
   setMatID( 1.0 );
   setPos( newPos );
   setNormal( normalize( normal ) );
   setAlbedo( albedo );
   setSpecMask( 0.1 );
}


At the moment the new format is completely orthogonal to the old one, so old shaders compile without any problems. The changes can be found in the svn repository. Please tell us your opinion on the updated system... :)


Top
 Profile  
Reply with quote  
PostPosted: 12.12.2008, 02:55 
Offline

Joined: 19.11.2007, 19:35
Posts: 218
Looks interesting.

In your example, would "FS_SHADOWMAP" and "VS_SHADOWMAP" typically also be present in the shader file?


Top
 Profile  
Reply with quote  
PostPosted: 12.12.2008, 05:38 
Offline

Joined: 19.03.2008, 01:22
Posts: 79
small bug in CodeResource::assembleCode()
Code:
-      std::string &depCode = _includes[i].first->assembleCode();
+      std::string depCode = _includes[i].first->assembleCode();

I find it interesting that MSVC would accept the original as valid..


Other than that, it looks good.


Top
 Profile  
Reply with quote  
PostPosted: 12.12.2008, 12:19 
Offline
Tool Developer

Joined: 13.11.2007, 11:07
Posts: 1150
Location: Germany
Vurlix wrote:
Code:
-      std::string &depCode = _includes[i].first->assembleCode();
+      std::string depCode = _includes[i].first->assembleCode();

I find it interesting that MSVC would accept the original as valid..

It is indead a little bit confusing, since you get a copy from the assembleCode method returned but store that copy as a reference. Not sure what will be the result when compiling this with MSVC but your solution seems to be the more correct one. I fixed it in the SVN. Thanks for the hint.


Top
 Profile  
Reply with quote  
PostPosted: 12.12.2008, 16:51 
Offline

Joined: 22.11.2007, 17:05
Posts: 707
Location: Boston, MA
The new system looks very nice, and fixes pretty much all of my gripes about the previous system. Good work!

_________________
Tristam MacDonald - [swiftcoding]


Top
 Profile  
Reply with quote  
PostPosted: 12.12.2008, 22:55 
Offline

Joined: 26.08.2008, 18:48
Posts: 120
It looks great, (I haven't updated the code because I modified horde3d's image loader so it will be a little work to merge the changes. And hoping for dds support in SVN :) ) A shader cache would be also great but AFAIK you can't compile glsl to binary.

Has anyone checked d3d11 ctp? It seems like nextgen hardwares will support interfaces to handle the multiple combinations of shaders. Also d3d11 supports cards with shader 2.0 support and above. Maybe I will give a try and create a d3d11 port...

Keep up the good work!


Top
 Profile  
Reply with quote  
PostPosted: 13.12.2008, 01:51 
Offline
Engine Developer

Joined: 10.09.2006, 15:52
Posts: 1217
Great to see that you seem to like the update :)

@AcidFaucet: The other contexts are also in the file, the example is just an excerpt.

@Vurlix: Right, according to that link it is not ISO 99 standard. But a const reference should work and be more optimal since it avoids an additional heap allocation for the string.

@attila: Unfortunately a shader cache is not possible for the reason you tell: GLSL has no precompiled binary shaders


Top
 Profile  
Reply with quote  
PostPosted: 13.12.2008, 03:22 
Offline

Joined: 19.03.2008, 01:22
Posts: 79
As far as I know, you can't return references or pointers (const or not) to stack objects. Or is there an exception to this rule? One workaround would be using a string reference passed as parameter for the output..


Top
 Profile  
Reply with quote  
PostPosted: 13.12.2008, 04:01 
Offline

Joined: 22.11.2007, 17:05
Posts: 707
Location: Boston, MA
marciano wrote:
@Vurlix: Right, according to that link it is not ISO 99 standard. But a const reference should work and be more optimal since it avoids an additional heap allocation for the string.
Named Return Value Optimisation ought to take care of this without any effort on your part, so a straight copy should be just as efficient as the const reference.

_________________
Tristam MacDonald - [swiftcoding]


Top
 Profile  
Reply with quote  
PostPosted: 06.01.2009, 12:05 
Offline

Joined: 20.12.2007, 00:18
Posts: 23
This looks good, I just have one question.

If you have 3 materials, A, B and C, with flags defined as:

A: _F01_Skinning, _F02_NormalMapping
B: _F02_NormalMapping
C: _F03_EnvMapping

After all shader combinations are compiled, how many shadow shaders will you end up with in this instance? Presumably it will compile shadow shaders for each combination, but these will actually be the same as the shadow shaders do not use any of the flags. Does it do anything internally to remove these duplicate shaders or will the application end up using 3 different shadow shaders for these materials?


Top
 Profile  
Reply with quote  
PostPosted: 07.01.2009, 23:17 
Offline
Engine Developer

Joined: 10.09.2006, 15:52
Posts: 1217
Manny wrote:
After all shader combinations are compiled, how many shadow shaders will you end up with in this instance? Presumably it will compile shadow shaders for each combination

No it should not. Each shader context knows the flags it is using. This flag bit mask is anded with the requested combination mask. So if no flags are used in a context, the context mask is 0 and the result is combination 0 regradless of what flags are requested.

More generally speaking, maybe it should be explained that the flags have a special naming scheme: _F<digit><digit>_<ReadableName> (e.g. _F01_Skinning)
The ReadableName is just for convenience and competely ignored by the shader system, solely the number identifies the flag.
So _F01_Skinning is the same as _F01_Whatever.


Top
 Profile  
Reply with quote  
PostPosted: 15.02.2009, 16:11 
Offline
Engine Developer

Joined: 10.09.2006, 15:52
Posts: 1217
We finally integrated the last part of the new shader system. The changes are:

- Uniforms need to be declared explicitly in the shader FX section now. This makes it easier for programmers and tools to track which uniforms are used. But the real advantage is that uniforms can have default values now.

- The current texture unit configuration got replaced by sampler declarations. You are no longer required to keep track of texture unit indices and can just assign a texture to a arbitrarily named GLSL sampler uniform (also declared in the FX section). This is much more convenient especially when using material links or complex pipelines where the texture unit setup is scattered over several files.

- Some texture related parameters (coordinate address mode and filtering) are defined in the FX section now and no more in the materials. This is similar to the way it is done in d3d fx and cgFX and makes the whole system cleaner.

Unfortunately the changes make it impossible to continue supporting the previous shader format. So you will need to port/convert your own shaders when Beta3 is out. This is actually not really hard but nevertheless, here some notes to make that task easier:

Code:
Shader Porting Notes

- Create a new file with the extension .shader
- Add a new section [[FX]] at the beginning

- For each context $CCC in your old shader
    - Create new code sections after the FX section for the vertex and fragment shaders, e.g.
     [[VS_$CCC]]
     [[FS_$CCC]]
    
     Please note that with the new system it is possible to share code sections between contexts, so if the vertex shader is the same for all contexts, it is enough to have a single code section for it. It is also not required that the shaders have VS or FS as prefix. This is just a suggestion.
    
   - Copy the old shader code into the new sections
      - The content of the <DefCode> blocks can just be copy&pasted
      - The <InsCode code="$CODE"> blocks simply become #include "$CODE"
   
   - Create a new XML Context element in the FX section:
      <Context id="CCC">
        <Shaders vertex="VS_CCC" fragment="FS_CCC" />
      </Context>
    
   - Copy the XML RenderConfig node of the old Context to the new Context

- Declare all of your custom uniforms in the FX section. You can give them default values if desired.

- Add sampler declarations for all used custom textures. You are no longer required to name the GLSL samplers tex0 to tex11 but can have an arbitrary name. Please take a look at the sample shaders, they show quite clearly how things are done now.

- Be aware of the fact that you can use #ifdefs now to write an übershader. This can help to merge many different shader files into a single one.


Top
 Profile  
Reply with quote  
PostPosted: 21.07.2009, 22:23 
Offline
Engine Developer

Joined: 10.09.2006, 15:52
Posts: 1217
Another update to the shader system:

The XML based FX syntax got replaced by a syntax that is similar to CgFX. I think this is a good upgrade. The FX sections look a lot more readable and familiar now. Many tokens are slightly different from CgFX but the basic syntax structure is the same (including annotations). Since CgFX / D3D FX is the de facto standard for shaders, this change should make it easier and more convenient for people to get started with Horde shaders.

Example:
Code:
// Samplers
sampler albedoMap = sampler_state
{
   Address = Clamp;
   Filter = Bilinear;
};

// Uniforms
float4 myUniform <
   string UIDesc = "This is my uniform";
> = {0.5, 0.5, 0, 0};

// Contexts
context AMBIENT
{
   VertexShader = compile GLSL VS_GENERAL;
   PixelShader = compile GLSL FS_AMBIENT;
}


Top
 Profile  
Reply with quote  
PostPosted: 07.12.2009, 23:37 
Offline
Engine Developer

Joined: 10.09.2006, 15:52
Posts: 1217
It is possible now to specify a default texture for a sampler.

This makes the material system more robust since it is no longer required that all samplers are configured in a material. If for example no normal map is specified, a default flat one is just used without any visual artifacts. It is also easy now to make the model shader support a single color value since the color uniform would just get multiplied with the default white texture if no albedo map is specified.


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 14 posts ] 

All times are UTC + 1 hour


Who is online

Users browsing this forum: No registered users and 4 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group