Jump to content
alpenwasser

APOLLO (2 CPU LGA1366 Server | InWin PP689 | 24 Disks Capacity) - by alpenwasser [COMPL. 2014-MAY-10]

Recommended Posts

Posted · Original PosterOP

DO IT! NOAW!

As the internet cats would say: But it doesn't fits, so it can't sits. :(

But how awesome would that look... :lol:


BUILD LOGS: HELIOS - Latest Update: 2015-SEP-06 ::: ZEUS - BOTW 2013-JUN-28 ::: APOLLO - Complete: 2014-MAY-10
OTHER STUFF: Cable Lacing Tutorial ::: What Is ZFS? ::: mincss Primer ::: LSI RAID Card Flashing Tutorial
FORUM INFO: Community Standards ::: The Moderating Team ::: 10TB+ Storage Showoff Topic

Link to post
Share on other sites

Just make it fit! Use a hammer!

 

Or get a bigger sits


Andres "Bluejay" Alejandro Montefusco - The Forums Favorite Bird!!!

Top Clock: 7.889 Ghz Cooled by: Liquid Helium   

#ChocolateRAM #OatmealFans #ScratchItHarder #WorstcardBestoverclocker #CrazySexStories #SchnitzelQuest TS3 SERVER

Link to post
Share on other sites
Posted · Original PosterOP

Just make it fit! Use a hammer!

 

 

Or get a bigger sits

Hm, yes, I could saw the M/B apart and then manually wire together the exposed

traces to each other. :lol:

On another note: Still stability issues on stress testing despite the 80 mm fan

on the heat sink (which is curios as I ran BOINC Friday and yesterday without

any issues whatsoever). Very interesting... thinking.gif

Luckily the server won't ever actually have this much stress once it's deployed,

so it's probably not going to be an actual issue in the long run, but I'm still

curios about this behavior.


BUILD LOGS: HELIOS - Latest Update: 2015-SEP-06 ::: ZEUS - BOTW 2013-JUN-28 ::: APOLLO - Complete: 2014-MAY-10
OTHER STUFF: Cable Lacing Tutorial ::: What Is ZFS? ::: mincss Primer ::: LSI RAID Card Flashing Tutorial
FORUM INFO: Community Standards ::: The Moderating Team ::: 10TB+ Storage Showoff Topic

Link to post
Share on other sites

 

Luckily the server won't ever actually have this much stress once it's deployed,

so it's probably not going to be an actual issue in the long run, but I'm still

curios about this behavior.

Unstable = unstable

 

I wouldn't trust my data (especialy corporate data) to an unstable server at all...

Link to post
Share on other sites

Pro tip: If you have an old xbox 360 lying around or if you can buy a broken one for dirt cheap, there are 2 delta's in it!


Main Rig: -FX8150 -32gb Kingston HyperX BLUE -120gb Kingston HyperX SSD -1TB WD Black -ASUS R9 270 DCUII OC -Corsair 300r -Full specs on Profile


Other Devices: -One Plus One 64gb Sandstone Black -Canon T5 -Moto G -Pebble Smartwatch -Nintendo 2DS -G27 Racing Wheel


#PlugYourStuff - 720penis - 1080penis - #KilledMyWife - #LinusButtPlug - #HashtagsAreALifestyle - CAR BOUGHT: 2010 Corolla

Link to post
Share on other sites
Posted · Original PosterOP

Unstable = unstable

 

I wouldn't trust my data (especialy corporate data) to an unstable server at all...

 

True dat. However, we will (well, do, actually ;)) have backups of important

data, so it's less a problem of reliability and security and more one of availability

and convenience.

But yes, it is of course not at all optimal.

 

Pro tip: If you have an old xbox 360 lying around or if you can buy a broken one for dirt cheap, there are 2 delta's in it!

 

Thanks, nice to know. I don't have an XB, but I still have one 92 mm San Ace.

I think the problem is probably less the fan power and more that the heat

sink just can't dissipate as much heat as needed without some 3k rpm or similar.

 

Looking great so far, mate!  

Good luck with the whole build! :)

Thanks! :)

Latest error adventures: :lol:

I have carefully gone through the BIOS with the manual, disabling stuff which might

have potential to cause instabilities. More interestingly, I have found a curious

feature on the M/B itself: There's a watch dog jumper which will reset the system

upon an application hanging, and it was in fact enabled. Not sure if that might

also have caused issues, but I have disabled it for now.

Also, I have managed to get all temp sensors working. There are a lot of them on

this board, but only two show temperatures above 35 C when the system is idling.

In that state, those two display around 50~52 C. I have not been able to determine

with absolute certainty that these two are in fact the chipset (or, maybe the chipset

and the ICH10R), but it seems likely that they are.

With the 80 mm fan mounted directly onto the chipset heatsink, that's already a

pretty high temperature IMO, I do not want to know how hot that thing got without

the fan under stress... :o

I will cautiously run some more tests, see how things go. If they system keeps

crapping out on me I'll need to start doing some proper debugging (swapping CPUs

and memory modules, checking if the PSU might be defective and all that good

stuff that takes up so much time :lol: ).


BUILD LOGS: HELIOS - Latest Update: 2015-SEP-06 ::: ZEUS - BOTW 2013-JUN-28 ::: APOLLO - Complete: 2014-MAY-10
OTHER STUFF: Cable Lacing Tutorial ::: What Is ZFS? ::: mincss Primer ::: LSI RAID Card Flashing Tutorial
FORUM INFO: Community Standards ::: The Moderating Team ::: 10TB+ Storage Showoff Topic

Link to post
Share on other sites

Just mill a larger heatsink already... Take the existing design, but make the fins five times as large ;)

And 5 times longer ;D


Andres "Bluejay" Alejandro Montefusco - The Forums Favorite Bird!!!

Top Clock: 7.889 Ghz Cooled by: Liquid Helium   

#ChocolateRAM #OatmealFans #ScratchItHarder #WorstcardBestoverclocker #CrazySexStories #SchnitzelQuest TS3 SERVER

Link to post
Share on other sites
Posted · Original PosterOP

Just mill a larger heatsink already... Take the existing design, but make the fins five times as large ;)

Hehe, we shall see. :D

For now, moar testing!

And 5 times longer ;D

I'd have a beefier chipset cooler than my CPU coolers. :lol:


BUILD LOGS: HELIOS - Latest Update: 2015-SEP-06 ::: ZEUS - BOTW 2013-JUN-28 ::: APOLLO - Complete: 2014-MAY-10
OTHER STUFF: Cable Lacing Tutorial ::: What Is ZFS? ::: mincss Primer ::: LSI RAID Card Flashing Tutorial
FORUM INFO: Community Standards ::: The Moderating Team ::: 10TB+ Storage Showoff Topic

Link to post
Share on other sites
Posted · Original PosterOP

checked bios and ipmi logs?

Yup, went through BIOS with a fine-tooth comb, disabled all the stuff that I

considered to have even slight potential for causing instabilities.

IPMI log only displayed missing keyboard errors from 2010. :lol:

solder fins to heatsink.  :D

Hm... thinking.gif

I have made some progress, but I'm not declaring victory just yet:

Successfully ran mprime for 10 hours yesterday, 6 hours of blend and

4.5 hours of small FFTs, no instabilities, no warnings, no errors. :)

Will do more of that during this week.

While running mprime, the chipset gets to around 67 C, and that's

with an 80 mm fan directly mounted to the heat sink. Critical temperature

is 95 C, and TBH I think it's not entirely unrealistic that this threshold

was breached without the fan on the heat sink. Though I'm hesitant to

actually try it out and get conclusive evidence.

Also, I have noticed that the program I'm using to read out the temp

sensors seems to hang after a while. If that watch dog thing really

did work as advertised, it's not inconceivable that it would have killed

the system due to that IMO. But I'm not familiar enough with how the

watch dog actually does its job to be sure of that, so I'm still voting

for high temperatures as the primary source of my worries at the moment.

I have however disabled the watch dog, just to be sure.

So yes, yesterday went pretty well in the end, but I'm still very skeptical

and will be doing plenty more testing during this week.


BUILD LOGS: HELIOS - Latest Update: 2015-SEP-06 ::: ZEUS - BOTW 2013-JUN-28 ::: APOLLO - Complete: 2014-MAY-10
OTHER STUFF: Cable Lacing Tutorial ::: What Is ZFS? ::: mincss Primer ::: LSI RAID Card Flashing Tutorial
FORUM INFO: Community Standards ::: The Moderating Team ::: 10TB+ Storage Showoff Topic

Link to post
Share on other sites

Mount a Delta to the chipset or one of the ones that came with my one Dell system. Those little blowers haul ASS.


Laptop: Latitude E4310 w/ i5 580M, 4GB RAM, 120GB 840 EVO, and Intel 7260 wireless card + Windows 10 Pro x64

Main PC: Look at my profile you lazy skrublord

Link to post
Share on other sites
Posted · Original PosterOP

aaweeeeeeee yeah, nice man!  :D

 

Hehe, thanks!

 

Mount a Delta to the chipset or one of the ones that came with my one Dell system. Those little blowers haul ASS.

 

The 80 mm that's on there right now can go up to 4.5k rpm, I think that should

suffice (although it's running at 1.3k rpm at the moment).

 

Heatsink your heatsink! It will totally work!

Ah yes, the good ol' heatsinkception technique! :D


BUILD LOGS: HELIOS - Latest Update: 2015-SEP-06 ::: ZEUS - BOTW 2013-JUN-28 ::: APOLLO - Complete: 2014-MAY-10
OTHER STUFF: Cable Lacing Tutorial ::: What Is ZFS? ::: mincss Primer ::: LSI RAID Card Flashing Tutorial
FORUM INFO: Community Standards ::: The Moderating Team ::: 10TB+ Storage Showoff Topic

Link to post
Share on other sites

Yup, went through BIOS with a fine-tooth comb, disabled all the stuff that I

considered to have even slight potential for causing instabilities.

IPMI log only displayed missing keyboard errors from 2010. :lol:

 

I swear... that's the only thing that's logged in bios of most motherboards and the last thing I care about.

 

Hm... thinking.gif

I remember doing an FEA on heat transfer way back, and found conic shaped with larger base at heat source is optimal, based on the gradient nature of heat dissipation and material conductivity  :)

 

Also, I have noticed that the program I'm using to read out the temp

sensors seems to hang after a while. If that watch dog thing really

did work as advertised, it's not inconceivable that it would have killed

the system due to that IMO. But I'm not familiar enough with how the

watch dog actually does its job to be sure of that, so I'm still voting

for high temperatures as the primary source of my worries at the moment.

I have however disabled the watch dog, just to be sure.

So yes, yesterday went pretty well in the end, but I'm still very skeptical

and will be doing plenty more testing during this week.

What do your IPMI sensors tell you... they should not hang since it's independent of the OS.

My Rigs (past and present)

Link to post
Share on other sites
Posted · Original PosterOP

I swear... that's the only thing that's logged in bios of most motherboards and the last thing I care about.

Ooops, seems I looked at the wrong log *facepalm*

Logging into the system via IPMI and looking at the log I get a few chassis

intrusion alarms (obviously) and a few CPU overheating alarms. I must say

that is very curious, seeing as my CPU temp reading from the OS never went

above 37 C, and more importantly: The CPU fans never started spinning up,

it just shut off. Since the CPU fans are regulated according to CPU temps

I would assume that they would have spun up to higher rpms if the CPUs had

actually been overheating, so TBH I'm taking this with at least one grain

of salt.

Also, I just don't believe that one of the CPUs (once it was CPU1 and the

other time CPU2) suddenly overheated. If I'd mounted a heat sink incorrectly

maybe, but with temps remaining stable for so long (and always below 40 C

when I checked) I just don't see how they could suddenly spike enough to

make the system panic this hard.

Plus, those are 40 W CPUs, and I'm running some pretty decent tower coolers

with some very decent fans on them. I just don't buy this.

I remember doing an FEA on heat transfer way back, and found conic shaped with larger base at heat source is optimal, based on the gradient nature of heat dissipation and material conductivity  :) .

That actually makes sense if I think about it.

 

What do your IPMI sensors tell you... they should not hang since it's independent of the OS.

OK, this is interesting. Getting all sensors via the OS gives me something different

than looking at the sensors via the IPMI View software.

Reading the temps via OS gives me those two 60~70 C results, the CPUs, the fan rpms, a few

voltage readings and some negative temp readings.

Reading the temps via IPMI View gives me the CPU temps as "low" (no actual number given, but

this is consistent with my expectations and the OS temp output), as well as the system temp

(~40 C) and the six dimm modules, which are all hovering around 40 C or slightly below.

However, on a positive note, I have by now run the following mprime cycles:

  • Blend: 6 hrs 20 mins,
  • Small FFT: 4 hrs 37 mins,
  • Blend: 22 hrs 26 mins,
  • Large FFT: 5 hrs 58 mins
So far 0 errors, 0 warnings, 0 crashes since last running BOINC on Sunday morning.

So the combo of the new chipset fan and my conservative BIOS settings seem to work

(at least so far, I still don't fully trust matters yet...).

Thanks for your help btw. :)

PS: I quite like this IPMI thing, wish there was something like that for desktop

boards.


BUILD LOGS: HELIOS - Latest Update: 2015-SEP-06 ::: ZEUS - BOTW 2013-JUN-28 ::: APOLLO - Complete: 2014-MAY-10
OTHER STUFF: Cable Lacing Tutorial ::: What Is ZFS? ::: mincss Primer ::: LSI RAID Card Flashing Tutorial
FORUM INFO: Community Standards ::: The Moderating Team ::: 10TB+ Storage Showoff Topic

Link to post
Share on other sites

... and a few CPU overheating alarms. I must say

that is very curious, seeing as my CPU temp reading from the OS never went

above 37 C, and more importantly: The CPU fans never started spinning up,

it just shut off. Since the CPU fans are regulated according to CPU temps

I would assume that they would have spun up to higher rpms if the CPUs had

actually been overheating, so TBH I'm taking this with at least one grain

of salt.

 

 

I wonder if the cpu monitor the chipset temps?  If so, it might be getting warning from CPU, so log thinks it's CPU overheat.  hrm... just a guess...but too lazy to find out now  :P .

 

 

 

Reading the temps via IPMI View gives me the CPU temps as "low" (no actual number given, 

Odd, since your board is also IPMI 2 like mine. Do you have the latest IPMI firmware? I get the temp like this:

 

post-7162-0-23264300-1384910621.jpg

 

 

 

However, on a positive note, I have by now run the following mprime cycles:

 

Are you using the San Ace fans on the chipset?  Whoever designed the heatsink for the PCH must expect fans to be blowing like jet engines already to keep it cool at that point.  :lol:

 

 

 

 

PS: I quite like this IPMI thing, wish there was something like that for desktop

boards.

 

 

Funny, b/c Nuvoton makes those chips, and they are in like all types of motherboards (sans the network and kvm feature).  


My Rigs (past and present)

Link to post
Share on other sites
Posted · Original PosterOP

I wonder if the cpu monitor the chipset temps?  If so, it might be getting warning from CPU, so log thinks it's CPU overheat.  hrm... just a guess...but too lazy to find out now  :P .

Hm, not completely unrealistic, but I have not been able to find any info on this.

However, IIRC (can't check at the moment, in lecture) I have had only two or three

of these events logged, and I've had more emergency shutdowns (something like five

or six), so the number of emergency stops and CPU overheating events logged doesn't

match up as far as I can tell.

But, the timestamps are more or less consistent between the two. Very curios overall.

 

Odd, since your board is also IPMI 2 like mine. Do you have the latest IPMI firmware? I get the temp like this:

I'll need to check the IPMI firmware when I get home, I think it's something like

v 1.3-something. I doubt it's the newest one.

The output I get looks actually very similar to yours. I do not have the Peripheral

temp sensor, but I also have System Temp (~40 C), alont with six DIMM temps and

two CPU temps (the CPU temps being reported as low instead of medium ).

Looking at your screenshot my results look OK to me.

 

I think the discrepancies between my IPMI sensor listing and the output of sensors

from Linux might just be that IPMI has one more layer of abstraction between the raw

sensor data and its GUI, not displaying non-essential data and obfuscating the CPU temps

behind those low, medium and high labels. The output sensors produces seems

to display all possible ports the monitoring chipsets have, even those which have not been

used on this board. That's just a wild guess though.

Are you using the San Ace fans on the chipset?

Yes, I have the 80x25 fan mounted to the chipset heatsink, running at ~1,300 rpm

It can go up to 4,500 rpm, but it's a PWM fan and its rpm is unfortunately coupled

to the CPU temps instead of the chipset temps, at least as far as I can tell.

Naturally, since the CPU temps remain below 40 C, the fans never spin up higher.

Since mounting that fan I've had one crash right at the start. After that I went

through the BIOS and the available jumpers and made the most conservative settings

which still made sense.

No instabilities since then (will check when I get back home, running another mprime

torture test at the moment).

 

Whoever designed the heatsink for the PCH must expect fans to be blowing like jet engines already to keep it cool at that point. :lol:

Yes, that's what I've been thinking. This M/B was definitely designed with the

idea of a massive amount of air being pushed through the chassis, which of course

makes sense considering it's a 2CPU server board.


BUILD LOGS: HELIOS - Latest Update: 2015-SEP-06 ::: ZEUS - BOTW 2013-JUN-28 ::: APOLLO - Complete: 2014-MAY-10
OTHER STUFF: Cable Lacing Tutorial ::: What Is ZFS? ::: mincss Primer ::: LSI RAID Card Flashing Tutorial
FORUM INFO: Community Standards ::: The Moderating Team ::: 10TB+ Storage Showoff Topic

Link to post
Share on other sites

The 80 mm that's on there right now can go up to 4.5k rpm, I think that should

suffice (although it's running at 1.3k rpm at the moment).

The replacement I have goes up to about 6.7k RPM. The other one it had went up to about 7k RPM. It kind of sounds like a mini jet engine. And a car turbo.


Laptop: Latitude E4310 w/ i5 580M, 4GB RAM, 120GB 840 EVO, and Intel 7260 wireless card + Windows 10 Pro x64

Main PC: Look at my profile you lazy skrublord

Link to post
Share on other sites

But, the timestamps are more or less consistent between the two. Very curios overall.

Check the time in the IPMI... it's not synced with system clock so it might a bit off (most likely due to how IT management can vary in locations).  I couldn't get the NTP setting to work before, but after I just upgraded my IPMI firmware it finally works! using pool.ntp.org.

 

 

Looking at your screenshot my results look OK to me.

*facepalm*... for some reason I kept thinking that was cpu temperature. 

 

 

I'll need to check the IPMI firmware when I get home, I think it's something like

v 1.3-something. I doubt it's the newest one.

 Great thing is that you can update while system is up and running .  Use the web interface and uncheck "retain settings".  

 

 

I think the discrepancies between my IPMI sensor listing and the output of sensors

from Linux might just be that IPMI has one more layer of abstraction between the raw

sensor data and its GUI, not displaying non-essential data and obfuscating the CPU temps

behind those lowmedium and high labels. The output sensors produces seems

to display all possible ports the monitoring chipsets have, even those which have not been

used on this board. That's just a wild guess though.

 Looks like your right, IPMI just reads from a subset.  I guess it's more for IT management and not tweaking purposes.  It was easy to check if system is running ok.

 

Looking at HWinfo64, there's a separate figure for IPMI and the Nuvonton chip (left part of screenshot).

post-7162-0-67653200-1384977828_thumb.jp

 

Yes, I have the 80x25 fan mounted to the chipset heatsink, running at ~1,300 rpm

It can go up to 4,500 rpm, but it's a PWM fan and its rpm is unfortunately coupled

to the CPU temps instead of the chipset temps, at least as far as I can tell.

Naturally, since the CPU temps remain below 40 C, the fans never spin up higher.

I have Sunon 100+ CFM 80mm but too much motor/bearing noise.  I'm curious of how well the 120's you ordered will perform.

 

There's a fan control in the "IPMI devices" tab... though it turns up everything (turn it up when your out of room  :lol: )

 

 

The replacement I have goes up to about 6.7k RPM. The other one it had went up to about 7k RPM. It kind of sounds like a mini jet engine. And a car turbo.

 

Nice.  What fans are they?  Sounds like these?

 


My Rigs (past and present)

Link to post
Share on other sites

Nice.  What fans are they?  Sounds like these?

Not sure what they are (I tried peeling off the Dell replacement sticker and I ripped off the fan's info). And they do sound like that (more or less). Just add in a higher pitched sound, like a car turbo, and then it would be perfect.


Laptop: Latitude E4310 w/ i5 580M, 4GB RAM, 120GB 840 EVO, and Intel 7260 wireless card + Windows 10 Pro x64

Main PC: Look at my profile you lazy skrublord

Link to post
Share on other sites
Posted · Original PosterOP

The replacement I have goes up to about 6.7k RPM. The other one it had went up to about 7k RPM. It kind of sounds like a mini jet engine. And a car turbo.

 

Haha, you could build a hovering computer with that! :D

 

Check the time in the IPMI... it's not synced with system clock so it might a bit off (most likely due to how IT management can vary in locations).  I couldn't get the NTP setting to work before, but after I just upgraded my IPMI firmware it finally works! using pool.ntp.org.

 

Yeah, the time on the system is not yet correct. But taking that into account the

times of logged events match up with my expectations, what does not match up are

number of events logged, there should be about double the amount of what is in the

log. Ah well, computers... :lol:

 

Great thing is that you can update while system is up and running .  Use the web interface and uncheck "retain settings".

Good to know. :)

I'll definitely consider doing a firmware upgrade, but at the moment the IPMI

seems to be working as it should, so I'm a bit hesitant to tinker with a working

part of the system.

 

Looks like your right, IPMI just reads from a subset.  I guess it's more for IT management and not tweaking purposes.  It was easy to check if system is running ok.

That sounds very plausible.

 

Looking at HWinfo64, there's a separate figure for IPMI and the Nuvonton chip (left part of screenshot).

That actually looks quite similar to the output from sensors on Linux from

a dataset perspective, thanks! :)

I have Sunon 100+ CFM 80mm but too much motor/bearing noise.  I'm curious of how well the 120's you ordered will perform.

Haha, 100 CFM from an 80 mm fan! :lol: 

There's a fan control in the "IPMI devices" tab... though it turns up everything (turn it up when your out of room  :lol: )

 

Hm, I don't have that. Maybe in newer firmware versions? Although turning up

the fan speeds would not be all that optimal. Despite being in its own room,

the machine would still be in our apartment, so at some point noise would

still start to bother people. Plus, the room in which it will most likely

be placed is the workshop in our apartment (no worries, it's not a "dirty"

workshop so to speak, it's more for electronics work and for dad to repair

small stuff on accordions etc., stuff that makes a proper mess gets done in

the basement), and if all the fans are turned up working in there would not

be very pleasant.

Again, thanks for the help! :)


BUILD LOGS: HELIOS - Latest Update: 2015-SEP-06 ::: ZEUS - BOTW 2013-JUN-28 ::: APOLLO - Complete: 2014-MAY-10
OTHER STUFF: Cable Lacing Tutorial ::: What Is ZFS? ::: mincss Primer ::: LSI RAID Card Flashing Tutorial
FORUM INFO: Community Standards ::: The Moderating Team ::: 10TB+ Storage Showoff Topic

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


  • Recently Browsing   0 members

    No registered users viewing this page.


×