Apply circular alpha state to quad

[nokama]

Hi All, my first post, and, well, I'm a complete noob to shaders. I'm learning how to use Ardor3D (java scenegraph), and am making a terrain engine for it. Currently I'm playing around with a skydome, and have placed a quad above the ground for clouds texturing. I found a cool cloud shader on the net, and have successfully applied it to my cloud quad. Now, for a nicer view, I need to alpha blend the quad in a circular fashion, so I can adjust things so where the quad intersects the skydome, I don't just get a harsh cutoff point of cloud texturing v skydome background.

I found a good tutorial online, that used the equation for a circle to apply a blend via the mix function from the texture colour to a given colour. So I thought I'd use this to adjust my quad shader to give me what I want. Anyway, it doesn't work and my lack of understanding to things doesn't allow me to troubleshoot yet, so I'm hoping someone could please have a look at it and see what I'm doing wrong.

Vertex Shader:

uniform vec3 eyePosition;
uniform vec3 sunDirection;
uniform float scale;
 
 
varying vec2  oUv;
varying float oGlow;
 
void main(void)
{
   vec4 position = gl_Vertex;
 
   // get vertex-position
   gl_Position = gl_ModelViewProjectionMatrix * position;
 
   // get vertex-normal
   vec3 normal = gl_Normal;
 
   oGlow = pow (clamp (dot ((position.xyz - eyePosition), sunDirection), 0.0, 1.0), 7.0);
 
   // UV-coordiantes
   oUv = vec2 (gl_MultiTexCoord0.xy) * scale;
 
   gl_TexCoord[0] = gl_MultiTexCoord0;
 
 
}

Fragment Shader:

uniform sampler2D cloud;
uniform sampler2D detail;
uniform vec3  sunDirection;
uniform float time;
uniform vec4  fogColor;
uniform vec4  cloudColor;
uniform float density;
uniform float cloudInvScale;
uniform float detailInvScale;
uniform vec2 cloudSpeed;
uniform vec2 detailSpeed;
uniform float circleRadius;
uniform vec2 textureSize;
uniform float tolerance;
 
varying vec2  oUv;
varying float oGlow;
 
void main(void)
{
 
   vec2 cloudOffset = cloudSpeed * time;
   vec2 detailOffset = detailSpeed * time;
   vec2 circleRegionCoords = fract(gl_TexCoord[0].st);
 
   float a1 = texture2D (cloud, (oUv + cloudOffset) * cloudInvScale).b;
   float a2 = texture2D (detail,(oUv + detailOffset) * detailInvScale).b;
   //float a3 = texture2D (mask, gl_TexCoord[0].st).a;
 
   vec3 p_sunDirection = normalize (sunDirection) * -0.01;
        p_sunDirection.x = p_sunDirection.x;
        p_sunDirection.y = -p_sunDirection.z;
        p_sunDirection.z = 0.0;
 
   vec4 p_fogColor = fogColor; 
 
   vec4 oCol = cloudColor * (p_fogColor *2.0);
   oCol.a *= clamp (a1 + a2 - 1.0, 0.0, 1.0);
   oCol.rgb *= 1 - oCol.a * density;
 
   vec3  absorption = vec3(0.0);
         absorption += clamp (
                         texture2D (cloud,  (oUv + sunDirection.xy * 0.2 + cloudOffset) * cloudInvScale).rgb +
                         texture2D (detail, (oUv + sunDirection.xy * 0.2 + detailOffset) * detailInvScale).rgb - 1.0, 0.0, 1.0);
         absorption += clamp (
                         texture2D (cloud,  (oUv + sunDirection.xy * 0.4 + cloudOffset) * cloudInvScale).rgb + 
                         texture2D (detail, (oUv + sunDirection.xy * 0.4 + detailOffset) * detailInvScale).rgb - 1.0, 0.0, 1.0);
         absorption += clamp (
                         texture2D (cloud,  (oUv + sunDirection.xy * 0.6 + cloudOffset) * cloudInvScale).rgb + 
                         texture2D (detail, (oUv + sunDirection.xy * 0.6 + detailOffset) * detailInvScale).rgb - 1.0, 0.0, 1.0);
         absorption += clamp (
                         texture2D (cloud,  (oUv + sunDirection.xy * 0.8 + cloudOffset) * cloudInvScale).rgb + 
                         texture2D (detail, (oUv + sunDirection.xy * 0.8 + detailOffset) * detailInvScale).rgb - 1.0, 0.0, 1.0);
         absorption += clamp (
                         texture2D (cloud,  (oUv + sunDirection.xy * 1.0 + cloudOffset) * cloudInvScale).rgb + 
                         texture2D (detail, (oUv + sunDirection.xy * 1.0 + detailOffset) * detailInvScale).rgb - 1.0, 0.0, 1.0);
 
   oCol.rgb *= 1.0 - clamp (absorption * 0.2, 0.0, 1.0) * density;
 
   oCol.rgb *= oGlow;
 
 
 
	//blend between fragments in the circle and out of the circle defining our "pixel region"
	//Equation of a circle: (x - h)^2 + (y - k)^2 = r^2
	vec2 powers = pow(abs(circleRegionCoords - (textureSize/2)),vec2(2.0));
	float radiusSqrd = pow(circleRadius,2.0);
	float gradient = smoothstep(radiusSqrd-tolerance, radiusSqrd+tolerance, powers.x+powers.y);
 
 
   gl_FragColor = mix(oCol, vec4(0.0,0.0,0.0,0.0), gradient);
 
}

If someone could point me in the right direction, or maybe even test it I'd be really grateful, I intend to learn more about this stuff, it seems the way forward for fast effects.

[todayman]

What about this doesn't work? The shader doesn't compile, link? It doesn't look right, and if so, how does it look?

Something you might try is to modify the alpha component instead of mixing to a blackish nothing - vec4(oCol.r, oCol.g, oCol.b, gradient or 1-gradient or something).

Also, if the roof of your skybox is blue, you could discard all fragments from the cloud quad that are blue sky, though again I don't know how this would work with the edges.