Where can I get math and algorithms for surface smoothing? It goes like when you get for example 30x30 height data array and you render it using 2 triangles for a Quad, surface looks triangular like because 30x30 is quite small resolution. To smooth it there are techniques to split a triangle to 4 triangles a so on for a given quality. And there is some interpolation for new height points and its not linear as in this case you’ll have more triangles and no smoothing.
Some keywords :
for grids : 2D cubic spline (bspline, bezier, patch, nurbs)
for general mesh : catmull-clark (corner cutting)
It can be interpolation (passes through control points) or approximation (generally smoother).
You can also try so-called ‘fractal subdivision’, to add random detail, quite realistic for terrains.