I'm trying to get my camera to zoom towards a particular object in my scene. The camera is not pointing at the object, but I want its apparent position on the screen to remain fixed as I zoom. I've read in several places that I can get this effect by translating the camera along the vector between itself and the object, without changing the camera's orientation. Seems simple enough, but I'm running into an issue where it seems to drift off course. I've got a few gifs to better explain.

In this first gif, i'm zooming in on the target. Hardcoded coordinates, so the position of the cursor doesn't actually matter. At every step of the zoom, I record the translations. At the end, I return to the start position (again, hardcoded) and display the previously recorded translation data as red lines. Those lines match what I would expect, a vector from the camera position to the object.


Here's the relevant code for the zoom
Code :
void Simulation::ScrollCallback(GLFWwindow* window, double xoffset, double yoffset)
{
	ImGuiIO& io = ImGui::GetIO();
	if (io.WantCaptureMouse) {
		ImGui_ImplGlfwGL3_ScrollCallback(window, xoffset, yoffset);
		return;
	}
 
	// zooming out 
	float zoomDirection = 1.0f;
	// zooming in 
	if (yoffset > 0.0)
		zoomDirection = -1.0f;
 
	//target for the zoom. Hardcoded for now to eliminate souurce of error
	glm::vec3 cursorPosition = { 597.84, 438.7, -15.2 };	
 
	float x = cursorPosition[0] - graphics.xTranslate;
	float y = cursorPosition[1] - graphics.yTranslate;
	float z = cursorPosition[2] - graphics.zTranslate;
 
	float deltaZ = .1;
	if (zoomDirection < 0) 
	{
		graphics.debugPoints.push_back(graphics.xTranslate);
		graphics.debugPoints.push_back(graphics.yTranslate);
		graphics.debugPoints.push_back(-graphics.zTranslate);
 
		graphics.xTranslate -= zoomDirection * deltaZ * x;
 
		graphics.debugPoints.push_back(graphics.xTranslate);
		graphics.debugPoints.push_back(graphics.yTranslate);
		graphics.debugPoints.push_back(-graphics.zTranslate);
 
		graphics.yTranslate -= zoomDirection * deltaZ * y;
 
		graphics.debugPoints.push_back(graphics.xTranslate);
		graphics.debugPoints.push_back(graphics.yTranslate);
		graphics.debugPoints.push_back(-graphics.zTranslate);
 
		graphics.zTranslate -= zoomDirection * deltaZ * z;
 
		graphics.debugPoints.push_back(graphics.xTranslate);
		graphics.debugPoints.push_back(graphics.yTranslate);
		graphics.debugPoints.push_back(-graphics.zTranslate);
	}
	//return to start position
	else 
	{
		graphics.xTranslate = 0.0;
		graphics.yTranslate = 0.0;
		graphics.zTranslate = -2000.0;
		debugIndex = 0;
	}
 
	std::cout << graphics.xTranslate << " " << graphics.yTranslate << " " << graphics.zTranslate << " " << std::endl;
}
My projection and view matrix, which is just built up from the xTranslate, yTranslate, and zTranslate. I've removed all rotation and all other translation trying to debug this.
Code :
	graphics.projection = glm::mat4();
	graphics.projection = glm::infinitePerspective(
		(float)glm::radians(90.0f),
		(float)graphics.width / graphics.height, // aspect ratio
		0.1f // near clipping plane
	);
 
	graphics.view = glm::mat4();
	graphics.view = glm::translate(graphics.view, { graphics.xTranslate,graphics.yTranslate,graphics.zTranslate });
And nothing special in my vertex shader.
Code :
#version 450 core
layout(location = 0) in vec3 position;
layout(location = 1) in vec3 inputColor;
 
out vec3 vertexColor;
 
uniform mat4 view;
uniform mat4 projection;
 
void main()
{
	gl_Position = projection * view * vec4(position, 1.0f);
	vertexColor = inputColor;
}

For this next gif, I go back to the initial start position and start stepping through graphics.debugPoints, translating the camera according to that data. It is the same array that I use to generate those red lines, but the camera position and the red lines seem to diverge. I'm also pretty sure that I haven't messed up a negative sign somewhere.



This is the code for the debug zoom

Code :
void Simulation::KeyCallback(GLFWwindow * window, int key, int scancode, int action, int mods)
{
	// == d, for debug
	if (key == 68 && action == GLFW_PRESS) {
		cycleDebug = true;
		graphics.xTranslate = graphics.debugPoints[debugIndex];
		graphics.yTranslate = graphics.debugPoints[debugIndex+1];
		graphics.zTranslate = -graphics.debugPoints[debugIndex+2];
 
		std::cout << graphics.xTranslate << " " << graphics.yTranslate << " " << graphics.zTranslate << " " << std::endl;
 
		debugIndex+=3;
	}
}

Any ideas/advice is much appreciated.