The performance degradation is due to NVIDIA power saving feature (a.k.a. Powermizer). When the driver detects low load it decreases the GPU or memory clock. You can disable it NVIDIA control panel. You can monitor the clocks using GPU-Z to be sure that the tests are running with max speed.