need help with monster watercooling build - cooling 4 GPUs

M

Mizugori

[H]ard|Gawd
Joined
Mar 25, 2004
Messages
1,240
I am building a rig to essentially serve as a GPU supercomputer. It's sole purpose will be to perform work using raw GPU processing power. As such, I plan to put 4 7950s in it, and they will be at high utilization 24x7, so I want to watercool them.

This will reduce noise (fans running at high speed are very loud on most graphics cards in my experience) and cool them better than an air solution. Also, I want to be able to do this inside a computer case - several GPUs in close proximity = HOT.

I have wanted to foray into watercooling for a while but never took the plunge. I read through the basic stickies on several forums but I could really use some advice.

I have confirmed that EK makes waterblocks to fit the cards I intend to use (thankfully they are ref design cards.)

I spoke with the folks at FrozenCPU and they told me that I should use 4 radiators (that is, 4 fans - like 4 single fan rads or 2 duals or 1 quad.) They also suggested to me that I could run the tubing to the gpu waterblocks in such a way that the water would not be going from card 1->card 2->card 3->card 4 (My concern was that by the time the water reaches card 4 it will be warmer and thus cool card 4 less efficiently than card 1.)

If I want to set it up that way - so the water isn't going directly from card 1 to 2 to 3 to 4 - do I have to use 4 TOTALLY SEPARATE rads, or can I still use a quad or two duals?

Also, how can I determine which rads will potentially fit in my case? I am looking at this case - http://www.nzxt.com/product/detail/120-H630.html The description mentions "The H630 offers exceptional support for a wide range of liquid cooling solutions, including the ability to mount 240mm, 280mm and 360mm radiators in both top and front positions." But that would lead me to believe it can only hold 3 120mm fan radiators (or does it mean I could have up to 360 in the top and up to another 360 in the front?)
 
I just built a loop to cool 4 GPUs. If you check this thread you can see exactly what I ordered. http://hardforum.com/showthread.php?t=1799525&highlight=

You definitely do not want four individual radiators. You want at least 480mm of total radiator capacity. So yes, you could buy a single 480mm radiator. One of the best ones per the reviews is the Alphacool UT60 480mm. However, not many cases fit a 480mm radiator. So what radiator you get depends a lot on your case size. The case you linked looks like it can do two 360mm radiators max. However, if you do that you're going to lose all of your drive cages. You may want to go with a bigger case. I have a Corsair 900D, which is as big as they come, but it's completely full with radiators and my cards. If I were you, I would definitely get more than 480mm totaly radiator capacity. That's the minimum capacity you should have really, and that doesn't even count you putting your gpu in the loop. Personally, I would get at least 720mm radiator capacity total. Think about it this way, if you build a high-quality waterloop with plenty of cooling, it can last you for 5+ years without really having to change things except for the gpu blocks.

EK blocks work fine, they're the cheapest but some people prefer the build quality of different brands. I bought EK blocks. What you want to buy are one of the blocks from the following link.
http://www.frozencpu.com/cat/l3/g57...cts-EK_Blocks_-_VGA_ATI-EK_HD_7950-Page1.html

To connect the four cards, you simply buy this. It makes it very easy. You just screw this into the top of the four waterblocks. http://www.frozencpu.com/products/1...e_QUAD_Semi-Parallel_CSQ.html?tl=g57c645s2062

To run four gpu's, you're going to need a special motherboard. Very few motherboards have four PCI-e x8 slots. They are usually in the $300+ range. What CPU are you using?
 
you can get parallel water distribution blocks for multiple GPU's and use a single 4x120mm rad if you want.

Just googled it, EK calls them terminal blocks, whatever, they make a quad semi parallel but it looks OOS everywhere so I'd recommend going with 2x dual parallel terminals, you would ideally want 2 2x120mm radiators or some Y splitters for your tubing.

I think it'd probably be better to have two independent loops though, since you'll probably need a second pump anyways, this way it is easier to work on the loop if anything should go wrong.

With the case you linked it looks like you can fit a 3x120mm in the front if you take out all the HDD cages and a 2x120mm or a 2x140mm on top, though you may need to remove the CD drive cage, I can't tell.

With that case I would probably put 2x gpu's on 3x120 on front and 2x gpu's on 2x140 on top so I could keep the fans a lower speed.

EDIT: looks like allen found the 4 one in stock, i'd still go two loops but that is me, if you're mining you're probably looking at keeping costs down.
 
Thanks! To clarify, the GPUs are just doing calculations, they only need 1x lane, so I don't need like a gaming motherboard or a powerful cpu and I only intend to watercool the GPUs. I also don't need a hard drive - linux booted off a usb drive is the likely O/S that I will be using. So far I am looking at:

mobo:
http://www.asus.com/Motherboards/M5A99FX_PRO_R20/

cpu:
any piece of crap I can throw in there

ram:
any, probably 4gb - if I try to overclock the video cards, does system ram (ie the ram I stick in the motherboard) matter?

Also, do I need a reservoir?

If I use multiple rads (like 2x 480 or 2x 360) I don't quite understand how it would work. Would the water flow through one radiator then through the next one then to the GPU waterblocks?
 
Mizugori said:
Thanks! To clarify, the GPUs are just doing calculations, they only need 1x lane, so I don't need like a gaming motherboard or a powerful cpu and I only intend to watercool the GPUs. I also don't need a hard drive - linux booted off a usb drive is the likely O/S that I will be using. So far I am looking at:

mobo:
http://www.asus.com/Motherboards/M5A99FX_PRO_R20/

cpu:
any piece of crap I can throw in there

ram:
any, probably 4gb - if I try to overclock the video cards, does system ram (ie the ram I stick in the motherboard) matter?

Also, do I need a reservoir?

If I use multiple rads (like 2x 480 or 2x 360) I don't quite understand how it would work. Would the water flow through one radiator then through the next one then to the GPU waterblocks?
Click to expand...

I would go with dual chanel ram, 2 * 2GB sticks. Look for a good freq and low cas (CL).

With a large radiator you don't NEED a res but they make things easier such as filling the loop. The pump HAS to have a constand supply of water.

You can go, Res > Pump > GPU 0 > GPU 1 > GPU 2 > GPU 3 > Rad 1 > Rad 2 >

Or, Res > Pump > GPU 0 > GPU 1 > Rad 1 > GPU 2 > GPU 3 > Rad 2 >

The order of blocks doesn;t realy make much difference. Just make sure the res is physicaly higher then the pump so the pump has constant water.
 
With water cooling the last card will be hotter, but we are talking maybe 4-5 degrees, and when you're dealing with 40-55 degrees a few more is NOTHING to a GPU. Just find the best loop that works for you and your budget.
 
Mizugori said:
Thanks! To clarify, the GPUs are just doing calculations, they only need 1x lane, so I don't need like a gaming motherboard or a powerful cpu and I only intend to watercool the GPUs. I also don't need a hard drive - linux booted off a usb drive is the likely O/S that I will be using. So far I am looking at:

mobo:
http://www.asus.com/Motherboards/M5A99FX_PRO_R20/

cpu:
any piece of crap I can throw in there

ram:
any, probably 4gb - if I try to overclock the video cards, does system ram (ie the ram I stick in the motherboard) matter?

Also, do I need a reservoir?

If I use multiple rads (like 2x 480 or 2x 360) I don't quite understand how it would work. Would the water flow through one radiator then through the next one then to the GPU waterblocks?
Click to expand...

A single loop is a lot easier than running two loops with a y-splitter and two gpu block connectors. Just go with the quad bridge that I linked. You don't have to worry about serial/parallel for the bridge because they only make one configuration of the quad bridges. It's half serial, half parallel. It also shows 7 in stock on the website, and that's the exact model you need for EK waterblocks. You have to be careful with bridges, because only certain models are compatible with certain waterblocks, but that's the right one.

In regards to the GPU temperatures, with four waterblocks the first two run parallel, and the last two run parallel, with a serial connector inbetween. Basically this means that the first two have identical temperatures as do the last two cards. In my loop, my last two gpu's are 2C higher than the first two cards.

The problem with the motherboard that you linked is that the PCI-express slots are not evenly spaced. In order for waterblocks to work with the bridge, they must be exactly evenly spaced. If you get any motherboard that's not built for that, even if you use the 1x PCI slots, the spacing will not work out. Furthermore, the 1x slots will not supply enough power for the GPU. Secondly, if you use PCI-e extenders to give power to those slots, well then the spacing won't work out again. You have no choice really but to go with a full quad-slot motherboard.

I would definitely get a reservoir. It's extremely useful for bleeding out air. You put the reservoir before the pump so it receives the excess air. I use a pump/reservoir combo which works well and is easy.

This loop setup can be really easy. Lets say you got the case you linked. The setup would look like this. 360mm radiator -> pump/reservoir combo -> 240/360mm radiator -> cpu block -> gpu quad bridge (if you get a bridge, you just connect it going in and out) -> the radiator at the start. A very simple loop and easy to set up.
 
Last edited:
Thanks!!! How can I find a motherboard with the right spacing of the PCIE slots? Any recommendations? Also for the pump/reservoir combo?
 
Youd be much better off selling all your 7950's, not buying any watercooling, and upgrading to 290's. All I'm gonna say. At the cheapest, you are looking at what ~$100 a pop per gpu block, another $30 for the connector, another $120 for the pump/res, and another $100 for the radiator + $50 for odd's and ends. So yea, $700 for just the watercooling side, On a rough day, you could get $150 per 7950, so in total you would have enough money to buy almost 4 - 290's over watercooling the 7950's.
 
Mizugori said:
Thanks!!! How can I find a motherboard with the right spacing of the PCIE slots? Any recommendations? Also for the pump/reservoir combo?
Click to expand...

The motherboard you need depends on what processor you want to go with.

I bought a XSPC kit, which saved a lot on costs. I like it a lot. You get a 360mm kit that has everything that you need. All you would need to buy in addition would be a second 240/360mm radiator, a few feet of tubing, a few extra compression fittings, and the gpu blocks/bridge.

http://www.frozencpu.com/cat/l3/g30...s_-_Brands-XSPC_Water_Cooling_Kits-Page1.html
 
sok0 said:
Youd be much better off selling all your 7950's, not buying any watercooling, and upgrading to 290's. All I'm gonna say. At the cheapest, you are looking at what ~$100 a pop per gpu block, another $30 for the connector, another $120 for the pump/res, and another $100 for the radiator + $50 for odd's and ends. So yea, $700 for just the watercooling side, On a rough day, you could get $150 per 7950, so in total you would have enough money to buy almost 4 - 290's over watercooling the 7950's.
Click to expand...

I am not sure what you mean. I do not have any of the parts yet - I would be buying the 7950s. Which cost about $400 a piece and sell even used for about that much due to the high demand for them that currently exists. The only reason I want to watercool is because I am not willing to do a ridiculous open air design such as setting up the computer on a workbench or in a jury-rigged case made out of plastic milk crates. I need a completely enclosed solution. Based on my reading, putting 4 7950s or other similar cards in one case and running them at full load will create a serious, serious heat problem. So I was looking into watercooling to get around this problem.

I am planning on buying 7950s because according to what I have read, they are the best ratio of electricity used : amount of hashes calculated per second.
 
Mizugori said:
I am not sure what you mean. I do not have any of the parts yet - I would be buying the 7950s. Which cost about $400 a piece and sell even used for about that much due to the high demand for them that currently exists. The only reason I want to watercool is because I am not willing to do a ridiculous open air design such as setting up the computer on a workbench or in a jury-rigged case made out of plastic milk crates. I need a completely enclosed solution. Based on my reading, putting 4 7950s or other similar cards in one case and running them at full load will create a serious, serious heat problem. So I was looking into watercooling to get around this problem.

I am planning on buying 7950s because according to what I have read, they are the best ratio of electricity used : amount of hashes calculated per second.
Click to expand...

Honestly, if you're buying the cards, might as well go with 290's. I thought you already owned the cards. The 290's will keep their value over the years better and are less overpriced comparatively. You want four reference 290's since you're watercooling anyways.

You can get the 290's on Amazon for $430+ tax. You just have to watch for them. I've bought eight 290's from Amazon doing this. Here's how you do it. Have a page on Chrome open set to autorefresh every 2 minutes. You can get a plugin for chrome to autorefresh. Then do an Amazon search for R9 290. Now you need to filter it down so you can easily see the 290's. So on the left, put in $400-$500 as your price range, and then click on "4GB VRAM" to filter all the Nvidia cards out that erroneously show up. Also click on "prime" shipping. This should narrow your search down to R9 290's specifically sold by Amazon.com which will be the cheapest. Keep it on autorefresh all day, if any cards show up in the window, it means you should snag them. Some of them come with BF4, which you can resell online for $25 easily.

No point buying a 7950 for $400. No point at all. If you want to go with a 7970 you can do the Amazon search/refresh technique I mentioned. My friend bought four 7970's for $300 from Amazon two days ago. If you're watercooling, the heat isn't an issue, and just go with the R9 290's. Although if you do, you absolutely must get 720mm radiator capacity minimum. I have 1200mm radiator capacity and they get quite warm with four R9 290's.

If you make the step to watercooling, it's a huge increase in cost over going air. But if you do it right, your loop will last for years. So might as well get a quality loop now that will run 6+ years for you. In my opinion, get the 290's, get a bigger case, get two 480mm radiators.
 
Last edited:
perfect lol - so my question i guess is, wouldn't the 290 use a lot more electricity and thus cut into your profit margin?
 
Radeon HD 7950 gets ~600Khash/sec @ ~300W = 2.00KHash/Watt
Radeon R9 280X (HD 7970) gets ~700Khash/sec @ ~340W = 2.06KHash/Watt
Radeon R9 290 gets ~900KHash/sec @ ~330W = 2.73KHash/Watt
Radeon R9 290X gets ~1000KHash/sec @ ~360W = 2.71KHash/Watt
 
sok0 said:
Radeon HD 7950 gets ~600Khash/sec @ ~300W = 2.00KHash/Watt
Radeon R9 280X (HD 7970) gets ~700Khash/sec @ ~340W = 2.06KHash/Watt
Radeon R9 290 gets ~900KHash/sec @ ~330W = 2.73KHash/Watt
Radeon R9 290X gets ~1000KHash/sec @ ~360W = 2.71KHash/Watt
Click to expand...

thanks where did you get those numbers??? I have been trying to do an honest comparison but I don't know where the numbers come from aside from the litecoin hardware comparison page which doesn't really show how much power each is using. Why does everyone seem to prefer the 7950 if the 290 is clearly more bang per watt??
 
Mizugori said:
thanks where did you get those numbers??? I have been trying to do an honest comparison but I don't know where the numbers come from aside from the litecoin hardware comparison page which doesn't really show how much power each is using. Why does everyone seem to prefer the 7950 if the 290 is clearly more bang per watt??
Click to expand...

The 7950's have been cheaper. Before black friday they sold for $130 on Newegg. Also you can run four of them without a 1600watt power supply. Honestly the energy prices are negligible. Also, according to my Corsair AX1200i, each R9 290 was only pulling 300W, not 330. Under water they will pull even less because they are running cooler and there is less leakage.

With the current price increases and the limited supply because they're no longer made, the 7950's are not even close to worth it. Get the 290's, or watch for some 280's on Amazon for $310. The 290's will be easier to find and will hold their value better over the years.

Most 290's get 880 hash, but a few are just better than the rest and can push 950 hash.
 
Are those stock clock numbers? With WC i can imagine the 290 getting 1Gh/s+ with some good clocks.
 
sok0 said:
Radeon HD 7950 gets ~600Khash/sec @ ~300W = 2.00KHash/Watt
Radeon R9 280X (HD 7970) gets ~700Khash/sec @ ~340W = 2.06KHash/Watt
Radeon R9 290 gets ~900KHash/sec @ ~330W = 2.73KHash/Watt
Radeon R9 290X gets ~1000KHash/sec @ ~360W = 2.71KHash/Watt
Click to expand...

I am curious about these numbers bc this is kind of the real key here. I can't find anything that suggests a 2950 would ever pull 300watts. Here is one link that shows it pulling 175-188watts at full load doing bitmining: http://www.tomshardware.com/reviews/radeon-hd-7950-review-benchmark,3207-9.html
 
Warrior said:
Are those stock clock numbers? With WC i can imagine the 290 getting 1Gh/s+ with some good clocks.
Click to expand...

The cards can go a lot higher in clocks. However, most of them just prefer exactly 1000 core. I've tried raising the clocks even a hair over 1000, and hash starts to drop a LOT. There's an optimal setting for each card. Only one of my four cards likes being over 1000 core, and it gets 950 hash.
 
Mizugori said:
I am curious about these numbers bc this is kind of the real key here. I can't find anything that suggests a 2950 would ever pull 300watts. Here is one link that shows it pulling 175-188watts at full load doing bitmining: http://www.tomshardware.com/reviews/radeon-hd-7950-review-benchmark,3207-9.html
Click to expand...


That's probably accurate. However, at 10 cents a kW the energy costs are insignificant compared to the increased hash rate and how much better the cards will hold their value when you resell them since they are the newest gen.
 
I also recommend parallel gpu cooling loop configuration to give your gpu's the coolest and most even temperatures. There is some great advice from others in this thread :).
 
Warrior said:
Are those stock clock numbers? With WC i can imagine the 290 getting 1Gh/s+ with some good clocks.
Click to expand...

Most people see better hash rates when downclocking if anything. Most people also seemed to report around 900khs on 290x.

Also four 7950s on a single 480. Lol, good luck with that.

dejacky said:
I also recommend parallel gpu cooling loop configuration to give your gpu's the coolest and most even temperatures. There is some great advice from others in this thread :).
Click to expand...

The temperature at any point in a loop stays pretty much the same. It doesn't heat up right away.
 
Warrior said:
With water cooling the last card will be hotter, but we are talking maybe 4-5 degrees, and when you're dealing with 40-55 degrees a few more is NOTHING to a GPU. Just find the best loop that works for you and your budget.
Click to expand...


No offense but thats bullshit.

Loop order makes no difference. Just make sure that you have the reservoir in front of the pump, obviously.

Mizugori said:
I am curious about these numbers bc this is kind of the real key here. I can't find anything that suggests a 2950 would ever pull 300watts. Here is one link that shows it pulling 175-188watts at full load doing bitmining: http://www.tomshardware.com/reviews/radeon-hd-7950-review-benchmark,3207-9.html
Click to expand...

Yeah, that doesn't sound right to me either. I had three 7950s and at 1100mhz with 1.1v total system load was less than 900w including my monitor.
 
BababooeyHTJ said:
No offense but thats bullshit.

Loop order makes no difference. Just make sure that you have the reservoir in front of the pump, obviously.



Yeah, that doesn't sound right to me either. I had three 7950s and at 1100mhz with 1.1v total system load was less than 900w including my monitor.
Click to expand...

Nah the guy is right. It's a few degrees higher on the last card. As he said, it doesn't really matter though. My last two cards out of four gpu's run 3C hotter. One of them did run 6C hotter than the others but I screwed it down a little more and now it's 3C. But yeah, as everyone in this thread says, as long as the reservoir is in front of the pump, loop order doesn't matter. The water only increases temp by a few degrees over the loop.

And yeah, no way a 7950 pulls 300 watts.
 
allen5055 said:
Nah the guy is right. It's a few degrees higher on the last card.
Click to expand...

Water doesn't heat up that quickly. There are several threads about this subject. I saw on thread on OCN with engineers arguing about the exact variance in temps across a loop. IIRC the difference is something like less than half of a degree between any point in the loop.

Any difference in temps that you've seen is likely due to a mount or the gpu itself.
 
BababooeyHTJ said:
Water doesn't heat up that quickly. There are several threads about this subject. I saw on thread on OCN with engineers arguing about the exact variance in temps across a loop. IIRC the difference is something like less than half of a degree between any point in the loop.

Any difference in temps that you've seen is likely due to a mount or the gpu itself.
Click to expand...

All I can say is what I see on my own loop. Seems very suspicious that the last 2 cards, which are in parallel with each other, are exactly 3C higher than the first 2, which are in parallel each other.
 
Mizugori said:
thanks where did you get those numbers??? I have been trying to do an honest comparison but I don't know where the numbers come from aside from the litecoin hardware comparison page which doesn't really show how much power each is using. Why does everyone seem to prefer the 7950 if the 290 is clearly more bang per watt??
Click to expand...

Yea, my 290x is lucky to get 800 hash....

allen5055 said:
That's probably accurate. However, at 10 cents a kW the energy costs are insignificant compared to the increased hash rate and how much better the cards will hold their value when you resell them since they are the newest gen.
Click to expand...

10c a Kw I pay closer to 35 cetns per kw...

I think it's actually costing me money to mine.
 
Last edited:
You must log in or register to reply here.
Back
Top