Just for you hardware geeks

[SloYerRoll]

I have a client that does allot of HD rendering and the beating he's giving his machines is starting to take it's toll on them.

So he's asked me to build him a cluster for rendering..

The hardware is actually the easy part. It's developing the code that will make them all talk and render FCP files etc that will be the tough part.

Here's some snaps of the hardware that's going into this. I won't bother you w/ boring peripherals that aren't as cool as this stuff.
*Sorry the snaps aren't a bit better. I didn't have the patience to set up a real light rig and have all this electronic goodness around me. :D

48GB of DDR3 RAM: (it's actually 56GB there, but my 8GB of RAM made it a perfect pyramid)
http://jbritt.smugmug.com/photos/317813091_d7NT4-M.jpg

Another shot:
http://jbritt.smugmug.com/photos/317815714_bKNeM-M.jpg

Intel quad core processors, Gigabyte motherboards (x6):
http://jbritt.smugmug.com/photos/317815224_8LXm6-M.jpg

All components out of box:
http://jbritt.smugmug.com/photos/317816369_bdgUh-M.jpg


All components assembled:
http://jbritt.smugmug.com/photos/317817010_hpXXU-M.jpg

My partner in crime. He's actually the hardware guru. I just do the software:
http://jbritt.smugmug.com/photos/317811981_FDHZQ-M.jpg

Each one of these is a screaming machine on it's own. I'm going to create a rack mount enclosure and parallel them all together so they will act like one serious kick tail computer that can handle big time rendering.

I wish I could keep this thing :drool

The really cool thing about this is after the kinks are worked out. I can parallel 3 to 300,000 computers. It will be completely scalable depending on the needs of the render farm. Currently the largest cluster parallels 160,000+ processors. It's used by NASA and the Govt.

[Awais Yaqub]

They look like gold bricks :wow

[peestandingup]

*Nerdgasm*

[ziggy53]

You're gonna be able to heat a small country with the dissipation from that monster.

Did you say "Final Cut Pro"?

[peestandingup]

When you get this monster together, I want benchmarks on how long it takes FCP to render full res HD files.

Would the software/OS even be able to take full advantage of this setup??

[SloYerRoll]

Did you say "Final Cut Pro"?Yes I did. I won't have FCP running on this machine. But it will be hooked up to an Apple Xserver and can grab the videos and do whatever rendering is needed for different outputs. It's all controlled from the command line. but I'll probably have someone build a GUI for the client since he's not the most tech savvy person :wink

[SloYerRoll]

When you get this monster together, I want benchmarks on how long it takes FCP to render full res HD files.
Ya, I'm gonna stress test this thing to really see what it can do.

Would the software/OS even be able to take full advantage of this setup??Yes, I'm going to load Linux Red Hat or Solaris on there. I haven't gotten that far yet. I'll squeeze every watt of juice out of this rig. I promise!

[SloYerRoll]

You're gonna be able to heat a small country with the dissipation from that monster.The heat sinks on the quad cores are SWEEEET!! They are really beefy too. Intel did a great job designing these. There are a total of 4 mounting pins for the fan/sinks. The first two go in really easy, but the last two are REALLY hard to mount. I felt like i Was gonna break the MOBO every time. Pretty scary when your dealing w/ $$ like this. I'm glad my friend Tom was around. I would have chickened out and not put the pressure needed to mount these if I didn't see him do it first.

I REALLY hope that none of these are DOA. I'm not going to know until about a week from now when all the hardware is mounted and I install the OS's.

[SloYerRoll]

They look like gold bricks :wowIt's DDR3. It may as well be gold bricks! :rofl

[SloYerRoll]

*Nerdgasm*Not nerd. Geek :D

[cmorganphotography]

:cry it's... so.... beautiful...:bow

[SloYerRoll]

You're gonna be able to heat a small country with the dissipation from that monster.BTW Ziggy:
My plans only look like there will be about 650w being used at peak rendering time. So It will get warm, but I think I'm doing something wrong if it gets hot.

I'll make sure to tell you if my thoughts are wrong though.

[sirsloop]

Time to make a borg cube and go folding! :thumb

[cmorganphotography]

Time to make a borg cube and go folding! :thumb

Lower your sensors and surrender your V-chips. We will add your photographical and technological distinctiveness to our own. Your f-stop will adapt to service us. Resistance is futile.

[zacker]

Pfffftt... not that cool, I mean, My dell came with "3-D Pinball'..




lol,lol,lol...

seriously, thats gonna be a sweet machine.. whats something like that run?$$

[SloYerRoll]

whats something like that run?$$Hardware alone cost 4k & big change. You can probably shop around and shave it down to 3k. Or you could just parallel 3 machines.. You get the idea on how cost is relative to your needs.

Development $$ TBD. :D

[Rhuarc]

Add into the mix a 9800GX2 for each machine, figure out a way to run Crysis split across all of those machines and you would have a dream gaming rig!

[SloYerRoll]

Add into the mix a 9800GX2 for each machine, figure out a way to run Crysis split across all of those machines and you would have a dream gaming rig!Only need one video card. You could SLI a few cards together. But even one high end VC would be able to red line Crysis and not even twitch. All the real work happens in the processors and is dependent on VC clock speed primarily.

This machine won't need anything but an on board video card since the video card isn't really doing any heavy lifting. I'm tossing a 8800GTX in there anyway just cuz I can :P

[mackidbrendan]

you should put some of that in the Pay it Forward Thread :barb :barb :barb
:barb :barb :barb :barb :barb :barb :barb :barb :barb :barb :barb :barb :barb

[SloYerRoll]

So I'm doing a write up of this in another place. But FWIW:

I've decided to install Mandriva (http://www.mandriva.com/)on these boxes. It's a flavor of Linux that has been around for quite a while.

What's really amazing to me is the GUI on these open source OS's is just as good and in some places beats the crap out of the mainstream versions. i.e. Leopard & Vista

[Rhuarc]

Only need one video card. You could SLI a few cards together. But even one high end VC would be able to red line Crysis and not even twitch. All the real work happens in the processors and is dependent on VC clock speed primarily.

This machine won't need anything but an on board video card since the video card isn't really doing any heavy lifting. I'm tossing a 8800GTX in there anyway just cuz I can :P

VC? You mean one high end video card? I don't knowif you are talking on the setup you are creating, but on a regular gaming machine crysis on it's highest settings at max resolution with max AA and AF will bring any modern video card to it's knees. Maybe the RAW horsepower of this machine makes up for that?

[SloYerRoll]

VC? You mean one high end video card? I don't knowif you are talking on the setup you are creating, but on a regular gaming machine crysis on it's highest settings at max resolution with max AA and AF will bring any modern video card to it's knees. Maybe the RAW horsepower of this machine makes up for that?VC= Video Card :D
Crysis (http://www.ea.com/crysis/) is brutal on a video card though. I run an 8800GTX and it still clips when I max everything out. Processor is a Intel Core 2 Extreme X6800 (overclocked to 3.73GHz)

I know you know this, but for the sake of anyone reading this thread..
The video card is a major component for a graphics intensive game such as Crysis (http://www.ea.com/crysis/). But the processor is vitally important as well. I have a few meters in my sidebar that track all my machines vitals and I can see it when I'm playing a game (sidebar is on my second monitor) and the processor screams since it's not a 2 dimensional video like a movie. If you turn one direction or move back and forth, these graphics are being rendered through the processor and being delivered to the video card at an astonishing rate.

Wendell,
This is far from my area of expertise, but I'd imagine if you had a nice quad core that could really render a game quickly. You could get away with a lesser card since the VC isn't really doing as much heavy lifting as opposed to a beefier VC w. a meager processor.
Is this really the case?

[Rhuarc]

VC= Video Card :D
Crysis (http://www.ea.com/crysis/) is brutal on a video card though. I run an 8800GTX and it still clips when I max everything out. Processor is a Intel Core 2 Extreme X6800 (overclocked to 3.73GHz)

I know you know this, but for the sake of anyone reading this thread..
The video card is a major component for a graphics intensive game such as Crysis (http://www.ea.com/crysis/). But the processor is vitally important as well. I have a few meters in my sidebar that track all my machines vitals and I can see it when I'm playing a game (sidebar is on my second monitor) and the processor screams since it's not a 2 dimensional video like a movie. If you turn one direction or move back and forth, these graphics are being rendered through the processor and being delivered to the video card at an astonishing rate.

Wendell,
This is far from my area of expertise, but I'd imagine if you had a nice quad core that could really render a game quickly. You could get away with a lesser card since the VC isn't really doing as much heavy lifting as opposed to a beefier VC w. a meager processor.
Is this really the case?

Yes and no. First the game has to actually be optimized for Quad Core. Secondly, the processor usually handles the processing of things like physics, AI, etc... The higher end graphics are processed primarily by the 3d graphics card. I have actually seen Quad SLI (yes, 4 video cards) running in a quad core processor machine only able to run Crysis at 15 to 20 fps. When you try to run it at 1920x1200 or eve higher (256, etc...) AA at 8 or 16x, AF at 16 or 32x, and everything else on max it absolutley crushes the machine. As far as I know a game like Crysis doesn't gain much from parallell processing (which correct me if I am wrong, is what these machines are going to be doing)

[DJT]

first picture made me hungry. Thought it was a pyramid of chocolate candy bars. Thought they were going to be used to give energy as you put this thing together.

You better keep most of us on this forum away from this machine because it'll be shorted out in no time as we all drool over it.

[SloYerRoll]

Yes and no. First the game has to actually be optimized for Quad Core. Secondly, the processor usually handles the processing of things like physics, AI, etc... The higher end graphics are processed primarily by the 3d graphics card. I have actually seen Quad SLI (yes, 4 video cards) running in a quad core processor machine only able to run Crysis at 15 to 20 fps. When you try to run it at 1920x1200 or eve higher (256, etc...) AA at 8 or 16x, AF at 16 or 32x, and everything else on max it absolutley crushes the machine. As far as I know a game like Crysis doesn't gain much from parallell processing (which correct me if I am wrong, is what these machines are going to be doing)It's a good thing I like COD4 WAY better than Crysis then :D

Thanks for the knowledge!

Cheers,
-Jon

[ziggy53]

So I'm doing a write up of this in another place. But FWIW:

I've decided to install Mandriva (http://www.mandriva.com/)on these boxes. It's a flavor of Linux that has been around for quite a while.

What's really amazing to me is the GUI on these open source OS's is just as good and in some places beats the crap out of the mainstream versions. i.e. Leopard & Vista

LINUX is becoming very mature, but driver issues still abound and you have to do your homework regarding hardware before building the machine.

Even Adobe is recognizing the market now and at one time Corel even had their own (pretty good) installer for a dialect of LINUX.

ILM uses LINUX renderfarms (something over 1000 nodes I hear.)

For render fans, here's a homebrew 24 core, 48GB memory machine running in (basically) a filing cabinet (186 Gflops):

http://helmer.sfe.se/

Sounds familiar?

[SloYerRoll]

LINUX is becoming very mature, but driver issues still abound and you have to do your homework regarding hardware before building the machine. This is why I went w/ Mandriva. I fired up the first 3 machines last night and it recognized all 8GB or RAM, all 4 cores, etc, etc.. The guy I am building this w/ swears by ATI (video cards) but ATI support is junk in the *nix world. So we went w/ my brand, NVIDIA :thumb

http://helmer.sfe.se/

Sounds familiar?This is where my client got his idea from :D
It's a shame the pictures are garbage and the documentation is next to non existent. Even the Drqueue site doesn't provide much clear documentation. But that's the nature of open source software outside the realm of the major distros like Ubuntu and others.You either make the code or write about it. It's next to impossible to do both.