Horde3D

I apologize ahead of time for any of the typos and mindless babble that low sleep and lots of coffee induces that may pop their way in here:

I've spent the past few hours hacking together some basic capabilities to use multiple pipelines. Thus far I haven't tested anything too heavily. There are also probably better ways to do it as this was the result of a 3 hour "can't sleep" research and hack fest. Despite that, I only notice a 5 fps hit against 160fps average. I haven't check mem consumption but vector should be pretty good on keeping it down.

It's done through turning vector< PipelineCommand > _pipelineCommands; into a vector<vector<PipelineCommand> >.
Assuming Y is vertical and X is horizontal this means that each Y is its own unique pipeline. Nothing overly spiffy has been done yet in regards to dealing with renderbuffers and such. As renderbuffers are left untouched it should therefore be possible (in theory) to use a renderbuffer from a different pipeline without ever defining it for your current pipeline. If you screw up and don't load that buffer then you're pretty much SOL at the moment.

A few extra little gootchy-woos are added, all of the default functions exposed to a program using Horde still work as expected , but some new ones have been added configPipeExtension(), renderPipe(), and renderMultiPipe(). configPipeExtension loads a pipeline explicitly as an extension. renderPipe(int) allows the scene to be rendered only through the chosen pipeline. renderMultiPipe(int*,int size) is designed to take an array and render through any setup of pipes that are loaded (e.g. 0,1,8,4,1 etc) in the order specified in the array.

So you could render using pipe 1 through a designated camera into a buffer which you could use as a reflection buffer for for pipe 0 through another designated camera. Provided you do the alternation of the active camera and use the appropriate render function in your application.

I have not tested it yet (still need a better pipe setup to try it out) but I've also added a function which will set the data of a uniform of from a pipeline in memory. Kind of redundant since you could ignore this capability of pipelines and get straight to the material's uniform.

Right off the bat:
Problems:
-There's no way to remove render buffers associated with a specific pipeline (unless you toast everything)
-Once you load an extension pipeline, you're stuck with it (unless you toast everything again)

Other things to solve: (+ is gauranteed, - is not)
+Thorough debugging and optimizing (maybe some use-case testing)
-Hook up support for multiple pipelines being defined in a single xml file
-Selective loading of them perhaps?
+Dealing with the other issues of render buffers
-names for the pipes would be nice
+General management of the pipes
-Maybe some "smart camera stuff" like swiftcoder suggested in a prev. post of his/hers/its/I-don't-know
-It's not directly related (well it kind of is) but implement that desired ability to access buffer data in materials
-A "random pipe" function could be amusing

What can this approach do (maybe with some other work):
-Toolset case example: Highlighting things by class
-Game case example: damage blur post process effect
-General case example: imposters, reflections, etc
-Really general case example: general case stuff it'd be useful for

Now for mandatory screenshots:

First: Using only the regular old pipeline


Second: using pipelines 0 and 1 (turned ON/OFF via spacebar)


The contents of the pipeline loaded as an extension:


I'm going to work on solving the thorough testing problem and a wee bit of restructuring today.

If anyone would like to help speed things along I'd certainly appreciate any pipelines, materials, shaders you can whip up for either of the existing demos.

Also any advice, commentary, thoughts, misc. errata, and anything shy of fish with worms are welcome.

For Pipeline 0 (base)
Here's the knight blurred but without the blooming going on:


For Pipeline 1 (extension)
Here's the knight with the blooming:


The extension does not setup the buffers that it uses. It gets those from the base pipeline.

I'm glad to see that you are trying out some ideas!

Might be I should talk a bit about the original idea of stages. A stage is not only meant as a tool for grouping commands for better readability but also to accesss that group (that's why they have IDs). Originally I wanted to implement an API function with which you can enable or disable the stage. So it would easily be possible to switch on/off bloom or some other postprocessing effects during run-time.

As I understand you have done something similar but with another approach, one that offers more flexibility but that also introduces more complexity (you already said yourself that you have to watch out for render target dependencies between files). Since you can assign a pipeline to a camera it is already possible to calculate reflections. That's good but I think the real design challenge concerns the render targets. How do you handle instances of a pipeline (two cameras with the same pipeline, e.g. for two reflective objects), sharing of buffers between pipelines, access to buffers in materials and so on.

Having read yours and swiftcoder's suggestions in the thread about the cameras I also came up with some ideas for a simple system. I will post them soon in that thread. I'm hoping that you can contribute more ideas from the work you are doing at the moment

doh!

Stages, that seems so obvious now.

That's definitely the way to do it, less work in code, cleaner interface, fewer hidden hack functions, order of execution can even be arbitrary instead of linear (which solves and/or simplifies many problems). Then there's all the cool things that you could hook in with the data used for arbitrary stage execution.

When I saw Horde first time, and was forced to extend its functionality,
Pipelines were tuned along with other things.

For my purposes of that moment, required Pipeline/RenderTarget-dependent abilities
( multicamera/stereo rendering,
thumbnailing,
low-res icon creation (legible, with simplified postprocessing - no shadows, no bloom and so on...) for platform/application-specific GUI elements,
hi-res printing,
simultaneous handling of different overlapping screen viewports (of the same or different scenes) with arbitrary Z-order and opacity,
application-driven realtime-changing postprocessing
and other)
were achieved with minimum efforts mainly by splitting Pipeline term into two concepts:

1) RESOURCE-DERIVED concept "Pipeline"(only action sequencer), whith no-instance dependent data(in accordance with Horde Resource ideology) - that is, it owns RenderBuffers(RB) only as references(array indices) rather then real objects GLhandles/references/pointers;
and
2) (Multiple instances)High-level tool for rendering, "Painter", (actually performing rendering on a set of private surfaces with aid of low-level rendering tool - Horde's Renderer) which mainly owns:
a)private surfaces for rendering - real RB instances with application-(un)controlled sizes (it may be main-window-size-dependent, for example, for final result output; or as intermidiate surfaces for engine inner purposes, like real-time reflection construction).
RB's references/pointers are kept in an array of the same dimensions and in the same order, as RT's are in the current attached (to the Painter's instance) PipelineResource's array of RT's (see below).
b)Bool elements container(array) of the same size as the PipelineCommand array in the current attached PipelineResource; it was used to activate/deactivate any stage (or even separate PipelineCommand).
c)PPipelineResource (in v0.14 terminology) smart pointer.

Painter's data(bool elements array, draw surfaces,...) can be managed (constructed/destructed) for example, during PipelineResource attachment/detachment to this Painter (by appropriate scanning of PipelineResource).
Instance/pipeline separation and Resource-deriving of Pipelines probably will help to improve Horde inner architecture and reduce code size in projects of moderate/large complexity, where Resource data/logic is separated from multiversion/multiformat serialization code.

Sorry I'm so inconsistent on your forum.
B.Regards.

While we're on the topic of pipelines, I thought I'd point out that "loadPipelineConfig" taking a file-name just doesn't seem to fit in with the rest of the API.
In my local copy of Horde I've hacked this function so it takes an ASCII string containing the contents of the file (instead of the path to the file), but in the long-term I think pipelines should become a proper Horde Resource just like textures, models, shaders, etc...

Full agreement there.

Me too

@Spectator: Nice to see you again here and thanks for sharing your ideas! Separating the pipeline instances from the pipeline definition is certainly a good thing. I already had some ideas but I will think about the whole subject again.

@DarkAngel: Right, that's not consistent. I think there is even a TODO in the code which says that the file access needs to be removed. I will change that for the next version in some way...

Thanks for that stages tip, it's way cleaner now. Enable/disable works, Stage id is used as the name of the stage and a state attribute added to enable or disable it at default.

Arbitrary order of stage execution also there and not requiring special variants of the render call, just setup calls.

I've got buffers exposed to materials now, though I hit some issues there:

I had to make the assumption that pipelines are always loaded before materials. Therefore when the material is loaded a render buffer of the name is looked up to return a pointer to the render buffer that is stored in a vector of a MaterialBufferReference struct also contains the desired buffer index and the assigned tex unit.

That look up required an additional function in the pipeline class getRenderBuffer that returns a void*. I don't like that void*, but if the buffer doesn't exist than 0x0 is returned and a nasty red error message states that there's a "BAD RENDER BUFFER : " and names it. That's tested and works. But I still don't like it.

<RenderBuffer buffer="NAME" unit="#" bufIndex="#" />

I'll post some code for anyone that wants to try it out once I've got some the kinks worked out.

In your current implementation you have access to all buffers of a pipeline form the materials?

One idea I recetnly had is to bind a pipeline to a Texture2D resource. You first create the texture in your code (also specifying the width and height) and bind the pipeline to it after then. The current semantic in the pipeline is to render to the framebuffer if no rendert target is bound. This would be changed to render to the associated output buffer, which can be a texture. The advantage is that this system requires no changes to the material system and doesn't have any special cases. A texture in a material is always handled in the same way, wheter it is loaded from a file or coming from a pipeline. The drawback is that you don't have access to the render targets of a pipeline from outside. But I think in most cases that's not necessary since the pipeline render targets act as temporary buffers to generate the final output.

If you have finished your code I would be interested to have a look

I think I'm having issues with texture coordinates for the material, though its hard to tell when you binding the blur buffer to the Tex unit 0 for the floor. Seems like it may be locking to u = 1 and v = 1.

Dynamics are definitely working, but I'm thinking I botched something (not an OpenGL guru) somewhere or am overlooking something.

Here's a basic overview of the major parts of code:

Whoops.

A bit more testing reveals that it's binding garbage as a texture. Code logging time.

I deserve a dumbass reward. Everything was fine, GL_CLAMP_TO_EDGE just plays tricks.

adding



After the buffer is bound in setMaterial() in egRenderer.cpp solves the problem.

Also if anyone goes tries to implement from my prior code note post, you'll be better off having the void* point at the RenderTarget instead of the RenderBuffer, opens up more options.

I also forgot that in the material load the pipeline().getRenderBuffer() is just a simple function that iterates through the rendertargets and looks for one whose ID matches the name specified in the XML file.

No significant loss of performance.

Major limitation is that reloading a different pipeline could really botch things at runtime, a bit more error checking could minimize and a reference counter could get rid of it.

======================================

That sounds like a great approach, doesn't sound trivial but it wouldn't have some of the issues my hack has (specifically reloading a new pipeline wouldn't break anything, textures would just be as they were at the last update).

Here's a link to the relevant code files. This should be more useful for anyone wanting to mess around / improve / use the code in the forum post without having to fill in the gaps themselves.

I suggest using WinMerge or some other merge tool to highlight the file differences for you.

http://mihd.net/h06zpm

A quick rundown of XML changes and additions:

Pipelines:

<SetCamera node=integer />
Changes the active camera to the node with the stated integer index. I would advise that if you using this code for release purposes that you use a data structure of your divising to store cameras. This is proof of concept, nothing more.

<Stage id="name" state="bool" >
Stage tags have an optional state attirbute which determines whether this stage is ON or OFF by default.

Materials:

<RenderBuffer unit=# buffer="string" bufIndex=# />
Binds the named buffer to a texture unit using the color buffer specified by bufIndex. THE PIPELINE MUST BE LOADED BEFORE TEXTURES ARE LOADED. This problem can be solved via a form of reference counting.

API Changes

All existing programs will work perfectly fine. The addition is a "RenderConfig" namespace which contains the functions:

clearStageExecList();
erases the arbitrarily ordered rendering of stages

addStageExec(int stageID);
Takes the integer value of the stage (>=0) and pushes it to the back of the arbitrary execute list.

addStageExecByName(const char* name);
Same as the above only it accepts the id of the stage.

setStageActive(int stageID, bool OnOff);
Enables or disables the stage specified.

getStageActive(int stageID);
Returns the state of the specified stage.

setExecListActivity(bool OnOff);
Sets whether arbitrary execution is enabled or disabled.

getExecListActivity();
Returns true if arbitrary execution is enabled, false if not.

getIndiceOfStage(const char* stageName);
Returns the indice of the named stage. Use it to manage things on the application side.

toggleStageActive(int stageID);
Super simple, change the state of a stage to the opposite of its current state.

Advice:

Adding a <AlphaTest state="bool" value=# mode="GL_LESS" or "GL_GREATER" /> is a worthwhile addition to the pipeline syntax.
Enabling or disabling depth test is also worthwhile.



Let me know if I overlooked anything in packaging this up.