id's "Rage" engine, Tech5 is OpenGL based

[Korval]

It really takes no more thinking than "What would you prefer; C or IA32 machine code, if you were to make some code run on any of a number of CPU's?". As it seems the plain C code case is out of the picture (as that'd make it "source code", and with pissing contests and secrecy we can't expect that) there's no way to fix that flaw. What remains is another compiled representation - though still ISA-independent.

If your goal is to make your code run optimally anywhere, then your goal is not the secrecy of the source code. You can have multiple goals, but one has to have primacy. And if the goal is to be able to compile for any arbitrary architecture into an optimal form, then you're not going to achieve secrecy.

And if that's your goal, then you need a language that preserves high-level constructs. It must preserve function calls, high-level looping constructs (not merely a jump function), structs, etc.

A good (read: proper) intermediate represenation would benefit all - even if some implementations had non-optimal optimizers to turn this into optimal code for their hardware.

Perhaps, but such a thing is really just glslang. Maybe an easier to parse form of it (high-level assembly), but that's all the savings you're going to get.

The problem of interest in this thread is that programs with thousands of glslang shaders take minutes, in some cases hours, to start. We want to find a way that can decrease this time.

The intermediate language issue only came up as a possible solution to this problem, and it is only useful as a solution to that problem. An intermediate language that is high enough level to remain optimal in virtually all conditions will require approximately the same compile time as glslang. This no longer makes it a solution to the problem, and thus it becomes useless.

Which is the point. You can decrease compile time (on some hardware) by making a low-level shader language. But in doing so, you make it so that only certain hardware can compile the shader optimally.

In general, I would say that readback of a fully IHV-dependent binary blob is the 100% safest way to achieve faster compile times. While the initial compile will be slow, you can pretend that it is program install time. The concern is that later changes (swapping graphics cards, etc) will force a lengthy compile, and even a driver download can cause the system to want to recompile the shader.

The modification of this method to hold textual glslang code in addition to IHV-dependent data is an alternative that exists primarily to make the binary blobs IHV-neutral. That is, a blob written by driver X can be read in driver Y without problems, though it may recompile the shader. The only problem there is that it is not possible to know if the shader will actually be recompiled or whether it will be loaded precompiled.

The third alternative is to hope that IHVs will cache compiled versions of our shaders and load them when our string matches with one in the cache. The primary disadvantage is that it isn't really a solution, as it does not rely on GL spec-defined behavior (since spec-defined behavior begins and ends with a render context).

There are two things that these three alternatives have in common that the intermediate language one doesn't. First, it retains glslang's advantage with regard to optimization. Second, and most importantly, they have a chance of actually being implemented. The ARB is not going to create a brand new shader language, and they certainly are not going to go the full D3D route with shader model nonsense.

So the intermediate language "alternative" is simply idle fantasy.

[knackered]

I simply want my shaders to build faster, therefore I'll take the unspec'd driver caching idea for now. They could even extend it so each IHV keeps an online-cache, the driver uploads the shader text to the server, the cache checks the hardware/driver config, scans the appropriate database, returns the binary blob. I'm going insane.

[Zengar]

[quote]Originally posted by Korval:

The expression (where M and V are matrices):

M-1 * V * M

... will be optimal in any IL that has matrix operations and types as part of the standart library.

And while it is true, that GLSL is ultimatively a high-level IL, it is not so optimal from compiler's point of view (C clones are parser-unfriendly) . Shaders are very small compariong to real-world program. I really believe that if one could step parsing and basic analysis, including them into IL, it would speed up the whole thing.

P.S. I have great suspects that Nvidias drivers first compile to assembly via their Cg compiler and then compile assembly again.

[Korval]

will be optimal in any IL that has matrix operations and types as part of the standart library.

If you transmute this into a form of assembly language, you get these opcodes, at the highest possible assembly-like level:

I really believe that if one could step parsing and basic analysis, including them into IL, it would speed up the whole thing.

Sure. As I pointed out, you would get some performance improvement from it (though how much is entirely up for debate) However, as I also pointed out, it's not going to happen.

[Zengar]

I was not speaking of assembly langauges. Nor do I believe assembly languages are good IR - because they separete code and data (variable mutability) which makes the dataflow difficult to analyse.


Insead of

Originally posted by Korval:

tempMat t1;
InvertMat t1, M;
MultMat t1, t1, V;
MultMat t1, t1, M;

I propose something like

@1 : Mat = InvertMat(M)
@2 : Mat = MultMat(@1, V)
@3 : Mat = MultMat(@2, M)

Here variables are assigned only once, so compiler can create precise data/controlflow graphs - actually, the IL code would define a graph.
And if the compiler knows an optimisation (rewriting) rule in a form

Multmat(Multmat(InvertMat(X), Y), X) -> SomeOptimisation(X, Y) where X, Y are Mat

is can very effectively apply it by replacing old part of the graph with new one. The semantics won't change (referential transparency).


Even more, this would alow some optimisations (like dead code or common subexpression elimination) to be performed before feeding the end compiler - in the shader to IL conversion step.

Such representation is very powerful, the only real "problem" is branching, but it may be solved in a rather efficient manner.

[sqrt[-1]]

Zengar's suggestion coupled with user controlled binary dumps and I would be 100% happy.

Compile to "Zengar IL" to remove dead code, do all pre-processing and basic code folding. (solves most of my concerns about shoddy driver code parsing)

User controlled binary dumps would be great for loading speed.

[PaladinOfKaos]

An IL shouldn't be necessary - I just want an asynchronous compile (so I can start a shader compile, then start loading other assets associated with that material), and I want the "linking" stage to be as minimal as possible for the hardware - it should just be a validation that VS output matches FS input on DX10-class hardware.

Binary dumps might be nice, but I'd rather handle the caching myself. Compile the first time it's used, and tag the dump with a hash of the renderer string. Heck, the first compile could be done during the install - there's lots of CPU idle time during that period of disk access.

[Korval]

I propose something like

That's really no different from the assembly form I mentioned. Indeed, it's (slightly) more involved to parse, and the goal was to minimize parse time.

I just want an asynchronous compile

Well, in GL 3.0, all object creation, except for templates, is asynchronous.

I want the "linking" stage to be as minimal as possible for the hardware

Well that's never going to happen. Even in regular C/C++ code, linking involves optimizations like inlining, global optimizations, etc. This only becomes more important when a couple of opcodes of wasted space can mean running acceptably and not (or not running at all).

[PaladinOfKaos]

Originally posted by Korval:
Well that's never going to happen. Even in regular C/C++ code, linking involves optimizations like inlining, global optimizations, etc. This only becomes more important when a couple of opcodes of wasted space can mean running acceptably and not (or not running at all).

Do any drivers actually do constant-propagation across shader stages? And is it really worth the hassle of not being able to easily mix'n'match various vertex and fragment shaders?

[Zengar]

Originally posted by Korval:
That's really no different from the assembly form I mentioned. Indeed, it's (slightly) more involved to parse, and the goal was to minimize parse time.

Hm, no, not at all. First, my suggestion was not an assembly but a graph - there are no variables, but only expressions. With other words, code = data it produces. Second - it is not about the text form, as the matter of fact forget the text form. The IL as I visualise it is a graph and should be stored in a compiler-friendly format - as a graph for example Thus, parsing and semantics analysis won't be needed, because of the graph's nature.

But of course, this is just theoretical talk If ARB and/or hardware vendors don't want to make such a step I can't force them We know that IT is a very slow in adopting new technologies - we still use programming languages that became morally obsolete several decades ago.