In an effort to do simple nonphotorealistic lighting I am trying to use the NORMAL_MAP_ARB texture coordinate generation mode and a texture matrix to do the dot product with light direction. the result of the dot product then indexes a 1-dimensional texture map. I have found that if I only enable NORMAL_MAP texgen on the S and T coordinates, everything seems to work all right (at least initially). However, when I turn on texgen for R, suddenly all the texture coordinates I get (regardless of the texture matrix I use) are screwed up (I think they are all .5 or something). I was unable to find any explainable cause for this. Even using an identity matrix (which should map only the incoming S coordinate to the outgoing S coordinate) saw the texture coordinates being screwed up merely by enabling or disabling texgen on R.
Having given up on the full texgen for a bit I started looking at mapping the -1 to 1 range of the S values being generated to the 0 to 1 range of the texture map. Clearly the first row of the texture matrix should be: .5 0 0 .5 to achieve this, but I quickly found that using .5 0 0 k for any value of k always gave the same results. .5 0 k 0 gave odd results that almost implied that texgen on R was already being performed ( it perturbed the generated coordinates in strange ways as k increased). Finally I found that by turning off texgen on T and using .5 .5 0 0 as the first row of the texture matrix yielded texture coorinates ranging from 0 for completely left-pointing normals and 1 for completely right-pointing normals. This, however, makes no sense whatsoever, as it implies that the T texture coordinate defaulted to 1 after I turned off texgen for it. Does the number of dimensions in the texture being used (1D versus 2D versus 3D and cube map) have any effect on the behaviour of the texture matrix, or the default values of particular texture coordinates? I doubt this, as the problems with using texgen on R persisted even when I used a 2D texture instead. I am working on a Radeon DDR 64MB, latest drivers. Any help at all would be appreciated.