Because the former will use an unsigned normalized texture internal format (e.g. GL_RGBA8), thus the unsigned byte data that you pass in will be normalized to the [0,1] interval while this is not the case with GL_RGBA32F.
You can only have non-normalized integer texture formats if you use GL_RGBAxI or GL_RGBAxUI enums (e.g. in your case this should be GL_RGBA8UI).