I’m part of a team creating a linux based render farm that will be running an OpenGL based renderer. Not specifically Nvidia’s Gelato, but a custom OpenGL renderer we’ve created; (I reference Gelato so readers familiar with the concept get my gist without a lot of explaining.)
Our goal is 100% hardware acceleration, so we plan on various optimizations beyond the scope of this question (such as scene rendering per node, rather than single frame rendering, plus other things…).
There are a few conditions:
We’ll be using Nvidia Quadra graphics cards
Each machine in the farm is 64-bit
There’ll be at least 20-40 machines, possibly more in the farm
All machines will be headless, yet we want acceleration (is that an issue?)
Anyone recommend a specific Linux distribution for our goal?
Would it make a difference (OpenGL acceleration is the key) to select a desktop version versus a server version of a given distribution?
Thus far, our investigations point toward Ubuntu Server as a good distribution… anyone think otherwise?
thanks for the suggestion, but it’s not the info we’re seeking… it almost seems like we’d get more bang for the buck with 32-bit systems… any suggestions anyone?
Your requirements don’t really narrow it down. FWIW, we run 40+ node diskless boot Linux OpenGL rendering systems on Gentoo and SuSE without issue. SuSE has very nice developer-oriented packages if that’s important to you, with boatloads of pre-built packages available. Nice if you don’t like the Gentoo rebuild-the-world mindset. But you can probably do what you want on pretty much any distro nowadays since 64-bit CPUs have been out for so long.
Why would you say that? Without knowing the exact needs of your renderer, can’t offer any advice there. But think maintenance.
However, I can tell you it’s quite frustrating when your renderer first hits that 3GB virtual memory limit (closely tied to 32-bit addressing). If this doesn’t ring a bell, you need to research it before you go buy and install your system. Google for 3GB/1GB split. Note that 3GB is virtual, not physical memory – that is, your app’s max physical memory consumption is a good bit less than that when you hit the 3GB limit.
OTOH, install a 64-bit distro on a 64-bit CPU, and your VM limit is well above 1 terabyte. Just shove in more RAM chips if needed and use them.
Thank you for your highly useful information. Of course, the 3GB/1GB memory limit makes perfect sense. I’m surprised we let that issue slip by our evaluations. Our renderer is spec’ed for a 16GB RAM foot print too. (duh!)
I made the 32-bit systems comment based upon recent difficulties we’ve had getting a 64-bit evaluation system up and operating with hardware acceleration. We’re a bit handicapped, in that our linux admin had not setup linux accelerated OpenGL before, and our OpenGL programmers are used to having their systems administered for them, so little help came from that direction (I’m on of them. Never maintained a linux box on my own before.)
I second that comment. For some distros like SuSE, you can just update your driver under the system package manager like any other package (YaST). Though the driver you get isn’t the absolute latest.
But if you’re a GL developer and want to flip between driver revs or track the absolute latest driver, it’s trivial – hard to get easier than:
Download driver *.run file
“sh *.run”
Answer a few menu prompts
Only catch is to make sure the kernel-source package is installed first.
Just FYI I’ve been running 64-bit SuSE for years with the 64-bit NVidia drivers on an AMD Athlon 64, and they’re solid as a rock. At work we run on 64-bit Intels with no problems there either. If you hit problems, definitely suspect your admin or system config first.
Never maintained a linux box on my own before.
Well, welcome to Linux! Learning curve might be a little steep at first if you have to drop down to system stuff, but it smooths out and is worth it. I can’t imagine going back. Wife (an accountant) even runs Linux now. The other OS safely confined away from viruses to a VMWare session, which is never run anymore…
If you have any other questions (general Linux, etc.), feel free to PM me, and I’ll help you out.
Thanks for the great information Dark Photon & ZbuffeR!
Although my “user” status is “newbie”, that’s because I lost access to a prior account… I’ve been coding 3D since '85 and was an early user of IrisGL…
I’m sure the linux admin stuff will come quickly. Not like I have to learn DirectX or some odd crap like that
One thing you might want to consider is running Gentoo with your own local portage server. Then you could easily do updates with shell scripts.
If you are going to roll your own localized distribution (ie from scratch, which I did once), currently I believe the most stripped down and easily adjustable distributions is Crux, but it is 32bit only I think.
Assuming you had your own ultra stripped down Linux system, and you have some kind of really fast log time 1:N network data distribution implementation (which probably is going to be a requirement for the render farm anyway), the idea of simply running all the nodes from a mounted ram disk (ie common binary file distributed in the network) would make maintenance easy. Would be really easy to test new system updates on say a few nodes at a time simply by updating the ram disk image, then after verification send the ram disk image to all nodes. Would also be easy to roll back updates if a problem was found simply by loading older ram disk images.
