normal for square

im trying to set up lighting for my scene and apparently its not working…below is my calculate normal function to get normal for a square… is it correct?


void normal (point p1,point p2,point p3,point p4, point &normal)
{
    point a, b,c;
 
    // calculate the vectors A and B
    // note that v[3] is defined with counterclockwise winding in mind
    // a
    a.x = p3.x - p2.x;
    a.y = p3.y - p2.y;
    a.z = p3.z - p2.z;
    // b
    b.x = p2.x - p1.x;
    b.y = p2.y - p1.y;
    b.z = p2.z - p1.z;
    // c
    c.x = p1.x - p4.x;
    c.y = p1.y - p4.y;
    c.z = p1.z - p4.z;
 
    // calculate the cross product and place the resulting vector
    // into the address specified by vertex_t *normal
    normal.x = c.x*((a.y * b.z) - (a.z * b.y));
    normal.y = c.y*((a.z * b.x) - (a.x * b.z));
    normal.z = c.z*((a.x * b.y) - (a.y * b.x));
 
    // normalize
    float len = (float)(sqrt((double)((normal.x * normal.x) + (normal.y * normal.y) + (normal.z * normal.z))));
 
    // avoid division by 0
    if (len == 0.0f)
        len = 1.0f;
 
    // reduce to unit size
    normal.x /= len;
    normal.y /= len;
    normal.z /= len;
}

If you have a rectangle with points p1,p2,p3,p4 and you want to calculate the normal, you only need two vectors. Pick any three adjacent vertices, for example, p1,p2, and p3.

a = p3 - p2
b = p1 - p2
normal = a cross b

Looking at the formula from wikipedia, you’ll see the cross product only uses two vectors. Get rid of c. Also, if you make a as p3-p2, b needs to be p1-p2, not p2-p1. That will point your normal in the wrong direction. Or you could keep it that way and do do “b cross a” instead of “a cross b”
This equation can be factored to form

a × b = (a2b3 − a3b2) i + (a3b1 − a1b3) j + (a1b2 − a2b1) k = (a2b3 − a3b2, a3b1 − a1b3, a1b2 − a2b1).