Cells produce their own proteins in organelles known as ribosomes.

[Worklog] David's 2015 Folding Rig "lol no sli bridge"
#16
Let me introduce my new folding rig:

[Image: zkWe56F.jpg]

It's based around four GTX 980 Ti watercooled cards. Parts are the same as my first post in this thread, except I did use Corsair LPX memory as hiig suggested. Weighing in at 55 pounds (25 kg), I've seen this beast draw as much as 960 watts of power.

The case is cooled by three fans in the front and one in the back, and the CPU cooler and each radiator has its own fan. This keeps the cards around 40-45 C. This leaves a bit of overhead for GPU overclocking although I do not have plans to do that at this time. I was surprised to find the quad core CPU fully loaded just serving the four GPUs, but it stays around 50 C.

The build process was fairly straightforward, as I've done a number of builds before. Four GPUs is a lot and it just took a while and left a ton of packaging. Mounting the radiators in the top went fine, as there are holes there for fans and radiators of various sizes. The front radiators are mounted with zip ties as there are only holes for 140 mm fans, not 120 mm fans. The hard drives cages potentially block air to those front radiators, but this hasn't been noticeable in elevated temperatures.
I'm running Fedora 23 (MATE spin). Originally I tried using the nvidia drivers that are packaged in the rpmfusion repos, but they do not include the necessary openCL libraries. In the end I had to install the binary package directly from Nvidia, and everything worked after that.
This computer is a bit noisy at full load, but not unbearable. I was able to sleep with it in the room no problem. But today I moved it down to the crawlspace where it can run and not disturb anyone. I also learned that UPS's don't like having large amounts of power drawn from them, either from the battery protected side or just the surge protected side. It woke me up at 2 am with a loud beeping until I turned the computer off, and I ended up having to put the computer on a plain power strip.

As I currently live with my parents, I made it clear to them how much power this draws and have offered to pay for it, but for now they are paying for the power.

I was planning on using more pictures but one really shows it all, if you have any questions or want to see more pictures let me know.

See my stats for scores! The current estimate from the client is 2.1 million PPD, but it will take a few days to really see.
Reply
Likes:
#17
Oh I'm liking what I see very much! I'm actually quite impressed with the peak power draw not even hitting 1KW as well. Has this been benchmarked yet?

One thing I do find strange though. Why did Corsair go with making their power cables look like ribbons? Seems rather unwieldy. Or maybe I'm just biased and prefer cables bundled rightly with mesh coverings and heat-shrinked tips.
[Image: sigimage.php?FAHUser=hiigaran&FAHTeam=21...hangeling1]
Reply
Likes:
#18
(2016-01-03, 09:36:04 AM)hiigaran Wrote: Oh I'm liking what I see very much! I'm actually quite impressed with the peak power draw not even hitting 1KW as well. Has this been benchmarked yet?

One thing I do find strange though. Why did Corsair go with making their power cables look like ribbons? Seems rather unwieldy. Or maybe I'm just biased and prefer cables bundled rightly with mesh coverings and heat-shrinked tips.

I haven't run any benchmarks, anything you'd like to see? (it would have to be for linux)

The ribbon cables are probably cheaper. Doesn't bother me except it did make getting the back panel on a bit of a pain as they were trying to lie sideways instead of flat.
Reply
Likes:
#19
I'd probably try to fold them like in the image below and secure them with some hot glue or something...

[Image: folded-ribbon-cable-assembly.jpg]

Nothing in particular with the benchmarks. I'm mostly curious about the absolute peak power consumption and whether or not you have hit the highest temps you could get.
[Image: sigimage.php?FAHUser=hiigaran&FAHTeam=21...hangeling1]
Reply
Likes:
#20
Damn that looks down right sexy. I was hoping to hit the numbers you plan on when I built my last rig but have only hit a quarter of your PPD. I forgot to take heat in to account and think that's what's keeping me back. Looks like you're making up for my derp moment though for the team, keep it up!
"The Dark Gods call out for suffering to be appeased, so I go to work." -EZM
Reply
Likes:
#21
(2016-01-03, 11:54:25 AM)hiigaran Wrote: I'd probably try to fold them like in the image below and secure them with some hot glue or something...

Oh good idea. Although right now what I have works fine, so I think I'll leave it. Aesthetics was never one of my primary concerns.

(2016-01-03, 12:17:55 PM)EZM Wrote: Damn that looks down right sexy. I was hoping to hit the numbers you plan on when I built my last rig but have only hit a quarter of your PPD. I forgot to take heat in to account and think that's what's keeping me back. Looks like you're making up for my derp moment though for the team, keep it up!

Yeah heat was a big concern of mine, which is why I went with the watercooled cards. Open and even blower GPUs don't do well when stacked.
What is your rig? (Feel free to post here, in the What's your rig? thread, or a new thread, depending on how much you want to discuss it.)
Reply
Likes:
#22
(2016-01-03, 03:50:50 PM)davidfg4 Wrote:
(2016-01-03, 11:54:25 AM)hiigaran Wrote: I'd probably try to fold them like in the image below and secure them with some hot glue or something...

Oh good idea. Although right now what I have works fine, so I think I'll leave it. Aesthetics was never one of my primary concerns.

Yeah, it's not something I'd bother doing if I was in your position, unless I was really bored. Perhaps if it was a purely air-cooled system and I needed unimpeded airflow...
[Image: sigimage.php?FAHUser=hiigaran&FAHTeam=21...hangeling1]
Reply
Likes:
#23
I pulled what I think is the key system info from F@H below and wrote out if I screwed up the pic. Right now only the CPU is water cooled and the GPUs are air cooled. I'm looking for water cooling for them but no one have any. Twilightangry2

i7-5960x @ 3GHz with 32G of RAM running 8.1 with 4 GPUs all being GeForce GTX Titan Z.

ps. Took me long enough to figure out how to get the screenshot to work. Twilightblush

[Image: ADuzk1l.jpg?1]
"The Dark Gods call out for suffering to be appeased, so I go to work." -EZM
Reply
Likes:
#24
(2016-01-03, 06:59:24 PM)EZM Wrote: I pulled what I think is the key system info from F@H below and wrote out if I screwed up the pic. Right now only the CPU is water cooled and the GPUs are air cooled. I'm looking for water cooling for them but no one have any. Twilightangry2

i7-5960x @ 3GHz with 32G of RAM running 8.1 with 4 GPUs all being GeForce GTX Titan Z.

Hmm, that does seem a bit low, but within the possible margin if they are throttling because of heat. Just to make sure, you are using a passkey to get the quick-return bonus points? And if you are doing CPU folding on that same machine, I would limit it to four threads in order to leave four cores for the four GPUs. (Yes I know the cpu is hyperthreaded for a total of 16 threads, but folding is intensive enough that hyperthreading may not help.)
Reply
Likes:
#25
(2016-01-03, 09:55:50 PM)davidfg4 Wrote:
(2016-01-03, 06:59:24 PM)EZM Wrote: I pulled what I think is the key system info from F@H below and wrote out if I screwed up the pic. Right now only the CPU is water cooled and the GPUs are air cooled. I'm looking for water cooling for them but no one have any. Twilightangry2

i7-5960x @ 3GHz with 32G of RAM running 8.1 with 4 GPUs all being GeForce GTX Titan Z.

Hmm, that does seem a bit low, but within the possible margin if they are throttling because of heat. Just to make sure, you are using a passkey to get the quick-return bonus points? And if you are doing CPU folding on that same machine, I would limit it to four threads in order to leave four cores for the four GPUs. (Yes I know the cpu is hyperthreaded for a total of 16 threads, but folding is intensive enough that hyperthreading may not help.)

How do you limit the cores on a node in the FAH client?
Reply
Likes:
#26
(2016-01-22, 05:07:31 PM)TehLizard Wrote: How do you limit the cores on a node in the FAH client?

[Image: uDmOJ8n.jpg]

1. Hit "Configure"

2. Go to "Slots" tab

3. Highlight "cpu" and click edit

4. Change "number of CPU slots" from -1 (auto) to desired number (4 or 8)

Unfortunately, without a little script work, cpu affinity isn't really an option as each WU is tasked as a new process. Some people report that task manager can hold the affinity, some say that the fah client "core locking" works, but I have found neither to be true with any of my systems. I stopped SMP folding not long after, so I never really looked into 3rd party programs.
[Image: sigimage.php?w=2793449&p=81403693&u=6281...b=octavia1][Image: sigimage.php?e=pinormous&b=vinylscratch1]
Reply
Likes:
#27
(2016-01-03, 06:59:24 PM)EZM Wrote: I'm looking for water cooling for them but no one have any. Twilightangry2

Wow. I'm shocked that a waterblock is so hard to find... I found one! Just one....
[Image: sigimage.php?w=2793449&p=81403693&u=6281...b=octavia1][Image: sigimage.php?e=pinormous&b=vinylscratch1]
Reply
Likes:
#28
(2016-01-22, 10:54:32 PM)pinormous Wrote:
(2016-01-03, 06:59:24 PM)EZM Wrote: I'm looking for water cooling for them but no one have any. Twilightangry2

Wow. I'm shocked that a waterblock is so hard to find... I found one! Just one....

I do not know wither to hug you, slap you or do both at the moment. Pinkiecrazy

Hug, I think... yes hug. Heart
Huge thanks for the link!

Sigh, now to wait.
"The Dark Gods call out for suffering to be appeased, so I go to work." -EZM
Reply
Likes:
#29
Thanks Pinormous, that was exactly what I needed. Going into idle when I'm just using chrome or a single thread VM is pointless with the 4 cores on a i7.
Reply
Likes:
#30
Well, I figured this would happen at some point...

So after the third power outage in the past week I checked on the server and noticed that one of the GPUs was missing from F@H. After standard debugging I determined that GPU #2 had failed. I have started the RMA process and have shipped it off. EVGA's warranty seems excellent (3 years), and they even approved my RMA in 12 minutes on Sunday night. Are they staffed by robots? But I'll see how they respond when they get it and if they send me a new one.
Reply
Likes:




SOON