PDA

View Full Version : new shader FX framework proposal



tweakoz
12-27-2004, 11:07 AM
hi list ,

I have been implementing a shader framework (ala CgFX or HLSLFX ) for my engine orkid
( see http://www.tweakoz.com/orkid/ ).

In this process I was thinking it would be best to standardize this sort of thing.

so would anyone care to collaborate on a specification ?

personally, what I need out of the fx framework is support for lower end hardware (Arb Vertex Program and Arb Texture Combiners) all the way to the high end (GL Shader Language). I would prefer the higher level constructs stay similar to CgFX and HLSL Fx (though I wouldnt mind extending it).

Here is are 2 examples of orkid shader scripts
for DX8(maybe even DX7) class hardware.
http://www.tweakoz.com/orkid/ui.fx
http://www.tweakoz.com/orkid/dflow.fx
these scripts actually work inside orkid, though I have not implemented every feature.

currently the fx implementation is dependant on my framework, but if there is enough interest I would consider breaking it out and 'componentizing' it.

thanks,

mtm

jwatte
12-27-2004, 02:00 PM
What's wrong with ".FX"? You can allegedly already run "FX" files on OpenGL using CGFX (which allegedly is compatible).

No need re-inventing a standard that already exists. Just support the target "glsl" instead of "vs_2_0" or whatever.

tweakoz
12-27-2004, 05:43 PM
try supporting arb texture combiners (dx7 class hardware) with CgFX. try supporting GLSL with CgFX (on ATI), try supporting Arb Vertex Program on an ATI card with CgFX, matter of fact try supporting any ATI card with CgFX. try using CgFX with Suse Linux 9.1 (it wont link).

I like CgFX enough, I chose to use it at work for our maya wysiwig pipeline using only nvidia cards ;< . I'm in contact with the NVidia CgFX team trying to get my issues with it fixed; that said, I cant wait forever for them to fix it, and I'm not going to count on them to making it portable to anything other than new nvidia hardware.

I'm not suggesting 'reinventing the wheel', if you notice the format I came up with, it is for all intensive purposes CgFX, but with lower end targets and a few other convenience features. Also, lets not confuse an implementation of an FX framework with a specification of an FX framework. CgFX is an implementation, not a specification , I'm not even sure I can find a specification other than the DX9 implementation's documentation - the CgFX doc's are certainly lacking...

so again my question rephrased : is anyone interested on standardizing an FX format for a wider range of hardware , say dx7 to dx9 class hardware ?

mtm

Korval
12-27-2004, 05:53 PM
The ARB is apparently (from the most recent notes postings) already working on making their own standard. Not surprising, since they made glslang, even though Cg's syntax was becoming a de-facto standard.

valoh
12-27-2004, 06:25 PM
I'm certainly interested in a high level GPU configuration resp. GPU-Engine (already experimenting with this on my own ;) ). But if you ask me, forget pre pixel shader 2.0 hardware. It isn't worth the effort.
Personally I'm more interested in higher level APIs. Haven't looked in the latest directx updates with the effect file script construct, but imo FX files are only the beginning of the migration from static 3d engines to generic, data driven engines. You'll need higher level scripting (describe complete render algorithm with lots of passes via scripts/data/whatever), component management (which data is compatible with which shader components), generic mesh formats holding the various data of various algorithms and so on...

tweakoz
12-27-2004, 08:52 PM
But if you ask me, forget pre pixel shader 2.0 hardware. It isn't worth the effort.
If I could afford a new laptop or new mac graphics card I would forget the pre 2.0 stuff, but I cant afford it right now, and Im trying to make sure my engine runs everywhere. it was actually easier for me to write the gl side fx framework stuff than it would have been for me to change all the higher level rendering code to a new path. my main development platform is HLSLFX / DX9, so my engine pretty much uses an abstracted form of the common BeginBlock/BeginPass/EndBlock/EndPass/CommitChanges interface as the main rendering methodology.



but imo FX files are only the beginning of the migration from static 3d engines to generic, data driven engines. You'll need higher level scripting (describe complete render algorithm with lots of passes via scripts/data/whatever), component management (which data is compatible with which shader components), generic mesh formats holding the various data of various algorithms and so on
I agree, but I need something I can work with today. Since I have 'rolled my own' I can add any higher level construct I want to it, but I was just trying to share some workload and make (or even join) a standard. Heck, I'd even offer orkid as an open source replacement for FX Composer with which to test the higher level functionality. It wouldnt take much effort at all to turn orkid into a cross platform fx composer replacement, and by its very nature (open source) it would potentially offer much more funtionality than fx composer can (I was planning on this eventually anyway). At any rate im not trying to advertise orkid here (I did that in another thread ;>). I just see the need for a STANDARD format with which to author effects, be it simple vertex shaders, pixel shaders, passes, techniques, and whatever else you think of that should be data driven.



The ARB is apparently (from the most recent notes postings) already working on making their own standard. Not surprising, since they made glslang, even though Cg's syntax was becoming a de-facto standard.
kewl, I'll check it out, hope it doesnt deviate too far off from the Cg stuff(unless of course it's clearly better).

writing cross platform stuff is a PITA.
(even if your using open GL ;>)

eeks,

mtm

tweakoz
12-27-2004, 11:25 PM
The ARB is apparently (from the most recent notes postings) already working on making their own standard. Not surprising, since they made glslang, even though Cg's syntax was becoming a de-facto standard.
From the Arb Meeting Notes


Metafile Working Group

GLSL alone doesn't define bindings to other attributes apps need. Metafiles include additional attributes/information about shaders and their parameters. Primary question: does this even belong in the ARB?

Sony has introduced "Collada" XML schema which has been picked up by some content providers. Also inclues a rudimentary scene graph. This could be extended for GLSL.
lol collada ;> - I started implementing a collada importer into orkid a few weeks ago. I think it makes sense for models, scenes, animations and the such. for shaders/effects Im not so sure, I dont want to be editing shader code in XML style by hand. I started out implementing the gl fx stuff in orkid with XML before I decided I didnt like it and I wanted it to use C style formatting as opposed to XML style formatting.

see http://www.tweakoz.com/orkid/modeler.gx to illustrate the point I am trying to make.

I guess I realize an xml editor tool could be constructed which would hide the nasty details of structuring stuff like this in an xml file, and present a pretty UI for doing so. I guess I'm just stubbornly attached to the C formatting style.

mtm

CrazyButcher
12-28-2004, 02:29 AM
this looks interesting, I am working on something simila (not that highlevel so) for a engine project with a friend.

http://www.planetquake.com/polycount/cottages/crazybutcher/code/evac2shdscript.txt

the material definition stuff I came up with, found its roots in q3 shader script. Since mostly I will have to deal with non shader-hardware. Hence the "combiners" are not programmable but predefined, that way I can optimize the generation of passes/multitexture

The programmable support and techniques is the latest addition and more as a testbed with shaders, since the majority of material will be fixed function.

It is not based on .fx since I found .fx a bit too "codeish" and looked for a more "artist" friendly script, well ideally this wouldnt matter when there are graphical tools to make those material shaders, but it will take us a while to do that...
Idea is haveing a relative simple "non-coder" friendly gameengine, e.g the gamescripting is done with LUA and doesnt require c/c++ compile... anyway never did something alike before so its mostly learning experience hehe, I come more from the artists side, as thats what I mostly did before this.

jwatte
12-28-2004, 05:18 PM
I also think that the CgFX support for Radeons is woefully weak (and thus, we don't use it). However, I think the .FX file syntax works JUST FINE for all the other cases that you suggest. Note that CgFX doesn't implement these, but that doesn't mean that the file syntax is bad.

We already have a standard floating around: .fx files. Let's just use that, rather than inventing Yet Another File Format. Shader metafiles start feeling like raw raster image formats. There really only needs to be ONE, yet there are 20 or more, all in more or less widespread use.

AndrewM
12-28-2004, 06:21 PM
But if 3dlabs or ATI get their sticky little fingers into it, it wont be anyhing like .fx

MickeyMouse
01-17-2005, 01:06 AM
It would be very nice if someone took .fx file format and did something like CgFX, but without all those bugs.

Even better would be if an ARB stated common C API on top of OpenGL for handling FX effects. They could be handled just like OpenGL through drivers common for certain vendors, so they could even be specifically optimized.

Doing the job by NVIDIA or ATI alone will never state any fine standard, not to mention whether it will ever work at decent speed on other than 'the right' cards ;)

CrazyButcher
01-17-2005, 08:59 AM
there is a engine with its own Cg runtime, which also partially supports Ati Radeon 8500

http://xengine.sourceforge.net/

zed
01-17-2005, 10:46 AM
any ideas why sony are behind collada, my conclusion is this destined to be on the ps3, i mean why else

tweakoz
01-18-2005, 03:18 PM
Originally posted by jwatte:
I also think that the CgFX support for Radeons is woefully weak (and thus, we don't use it). However, I think the .FX file syntax works JUST FINE for all the other cases that you suggest. Note that CgFX doesn't implement these, but that doesn't mean that the file syntax is bad.

We already have a standard floating around: .fx files. Let's just use that, rather than inventing Yet Another File Format. Shader metafiles start feeling like raw raster image formats. There really only needs to be ONE, yet there are 20 or more, all in more or less widespread use.im definately trying to stay close to the basic FX file format (you will notice my technique blocks and uniform declarations are syntactically identical to CgFX/HLSL), but what do I do for example when trying to support NV texture combiners, ati fragment shaders, arb fragment programs, or other paths that are not supported by CgFX? either I have to make a new path (and hence deviate from the standard), or try and algorithmically convert supported paths over to the unsupported path ( i wont be writing my own arb_fp to register combiner compiler, thankyou..) .

Again Im not suggesting deviating from the FX standard, I just want interop with any fragment or vertex pipeline extension required.

mtm