My Boss Isn't Convinced about ECC Memory for a File Server

[LordMastodon]

A bit of backstory: I work for a local tech repair shop, and we are essentially contracted by a local school to do their IT work. My boss is a really great guy and a genius when it comes to fixing pre-builts and laptops, but he really doesn't know much at all about custom building, picking parts, etc.

 

We've recently been asked by the aforementioned local school to make them a new file server. We were given a grant by the government to build it, but the drawback was that we could only buy from certain retailers, so that meant we had to get a "custom pre-built", or a prebuilt where I can configure most things but there's a heavily limited selection of parts to choose from. Even though it's limited, there's still enough of a selection that my boss needed help configuring the server, so I stepped in. I essentially picked out all the parts and sent my final list to my boss so he could do the final buying. After a week, I had a chance to look at the final invoice, and I noticed some of my suggestions had been ignored, but most of all, my ECC memory suggestion had been ignored. I brought this up with my boss, and I explained to him the concept and why it's important in mission-critical servers like the file server we were making, and why bit-flipping could potentially be really bad in this kind of computer, but he (confusingly) told me, "Don't worry - it won't happen". The conversation kind of ended there because I had absolutely no clue what to say about that. So I'm pretty much asking you guys what I say to him. He seems convinced that this is a really minor problem, not something to worry about at all. For context, I'm pretty sure nothing happened to their last server, which they had had for 3 years, but I can't confirm whether they had ECC or normal RAM, although this may be one of the things driving my boss to say that nothing will happen since nothing happened to the last server, despite him not knowing whether it had ECC or normal memory. Advice?

[PlayStation 2]

Eh, if it's just a small file server there isn't much reason to dump cash on ECC memory.

Now considering that you think that shit could possibly go down, try convincing him that the extra cash is worth it. 

[79wjd]

How much extra are you talking about? And is the total cost still within the budget of the grant? 

 

You could take the approach -- while the possibility isn't high, the possibility does exist, and it would be best for our name to try to avoid as many potential issues as we possibly can. Afterall, if something went wrong and they came to the conclusion that the lack of ECC memory was at fault, then your company could be blamed and they may or may not choose to do business with your company again. 

[zMeul]

if the memory doesn't fail, the disks might - and if you don't have back-up off site, it doesn't matter either way

[LordMastodon]

4 minutes ago, Dan Castellaneta said:

Eh, if it's just a small file server there isn't much reason to dump cash on ECC memory.

Now considering that you think that shit could possibly go down, try convincing him that the extra cash is worth it. 

 

I mean it's not really worth it to tell him anything about the current server, because it's already been bought and is on its way, but for future servers I want him to know. It's not really a "small" file server, it's the only one for a not-too-small school, and I want to make sure that nothing happens. It's honestly not even that much cash, and we'd definitely still be underbudget if we went for it.

4 minutes ago, djdwosk97 said:

How much extra are you talking about? And is the total cost still within the budget of the grant? 

 

You could take the approach -- while the possibility isn't high, the possibility does exist, and it would be best for our name to try to avoid as many potential issues as we possibly can. Afterall, if something went wrong and they came to the conclusion that the lack of ECC memory was at fault, then your company could be blamed and they may or may not choose to do business with your company again. 

 

 

Yeah, we'd still be within the budget, but strangely, that's not the issue that my boss has been citing. He's simply been saying that it won't happen, and not providing any evidence or counterarguments to support his claim. I don't know that the school would stop doing business with us, and my boss could find a way to deflect it if something happens, but I'm really just trying to make sure it doesn't happen, and I'm not even sure what to say to him.

2 minutes ago, zMeul said:

if the memory doesn't fail, the disks might - and if you don't have back-up off site, it doesn't matter either way

Precisely, and I'm really not sure we have any sort of backup for this thing (another point I tried to bring up with them)...

 

[2FA]

To be honest, the chance of bit flipping is extremely low. 9/10 a flipped bit happened on a DIMM that has done it before so it's pretty much just your usual chance of bad RAM. What OS is it going to be running as well file system?

[NumLock21]

Looks like your boss pockets the money while, you're the one that's going to deal with the issues. This is why HP and Dell will only let you select ECC, where you want it or not on their servers.

[79wjd]

@LordMastodon I understand that it's already been done and that you can't undo the purchase and whatever. But, what is the price difference between the two systems? And what difference does it make to your boss which system he went with if it's within the grant's budget anyway -- what happens to the extra money from the grant? 

 

Also, I can tell you, a lot of bosses are VERY stubborn and are convinced they know best. My boss is/was a really smart guy and basically ran the entire division of the company, but he was convinced that all employees needed 1tb HDDs and i7-6700s when all they doing is basic excel/email work -- where an i3+120gb SSD would be FAR superior. 

[LordMastodon]

1 minute ago, DeadEyePsycho said:

To be honest, the chance of bit flipping is extremely low. 9/10 a flipped bit happened on a DIMM that has done it before so it's pretty much just your usual chance of bad RAM. What OS is it going to be running as well file system?

 

 

It's going to be running Windows Server 2012. The thing is, I know that the chance is low, but I want to eliminate the chance completely.

[zMeul]

4 minutes ago, LordMastodon said:

I'm really not sure we have any sort of backup for this thing (another point I tried to bring up with them)...

a cheap NAS that powers on at set invervals and software on main rig that takes snapshots of the data and sends it to the NAS at specific times

[Tosa]

I think of bit flipping like cancer; Relatively rare, but the consequenses can be catastrophic. You'll most likely have a kernel panic(BSOD), crashing applications and/or data corruption. Worst case scenario, it'll flip some privileges from "false" to "true", which, you know, isn't too great if the students of the school have access to the server.

[Dash Lambda]

So he's given absolutely no explanation as to why he would choose against ECC?

I mean, safety features aren't there because something will happen, they're there because something is possible and bad.

[GDRRiley]

even my MC server has ECC. Common if it is important data and you have the option to "JUST DO IT"

[NumLock21]

So the shop you work at got picked to build the local school a file server? Most school will do bids and then see which one is the cheapest. Your shop will be responsible for the file server for as long as it's running. Tell your boss, this file server is not for home use where it's access by 1 or 2 people at home. It's for a school, where it's will be access by hundreds of people at the same time, and ECC eliminates, bsod related to memory.

Won't listen, then let him take care of it.

[DXMember]

usually you don't need ECC below 32GB

and for a local schools file server I doubt you need anything more than 8GB RMA honestly

put most of the money in multiple fast NICs and redundant storage, maybe an SSD for cache if it's used by pupils as well

 

would really like to see the full spec sheet though

[DXMember]

50 minutes ago, LordMastodon said:

my boss could find a way to deflect it if something happens, but I'm really just trying to make sure it doesn't happen, and I'm not even sure what to say to him.

either find something to say or find a different job/boss because otherwise nothing will change

doesn't sound like you're happy with the situation and it's a not a healthy environment for you to be in

[Dash Lambda]

51 minutes ago, DXMember said:

either find something to say or find a different job/boss because otherwise nothing will change

doesn't sound like you're happy with the situation and it's a not a healthy environment for you to be in

That's a tad drastic.

[DXMember]

9 hours ago, Dash Lambda said:

That's a tad drastic.

something has to change in that relationship

[leadeater]

12 hours ago, djdwosk97 said:

How much extra are you talking about? And is the total cost still within the budget of the grant? 

 

You could take the approach -- while the possibility isn't high, the possibility does exist, and it would be best for our name to try to avoid as many potential issues as we possibly can. Afterall, if something went wrong and they came to the conclusion that the lack of ECC memory was at fault, then your company could be blamed and they may or may not choose to do business with your company again. 

Completely agree with this, the cost of ECC ram now days is so low it should be a non issue. Servers should always use ECC ram, home servers are a different story but even then I still have the same stance on this.

12 hours ago, LordMastodon said:

I mean it's not really worth it to tell him anything about the current server, because it's already been bought and is on its way, but for future servers I want him to know. It's not really a "small" file server, it's the only one for a not-too-small school, and I want to make sure that nothing happens. It's honestly not even that much cash, and we'd definitely still be underbudget if we went for it.

 

Yeah, we'd still be within the budget, but strangely, that's not the issue that my boss has been citing. He's simply been saying that it won't happen, and not providing any evidence or counterarguments to support his claim. I don't know that the school would stop doing business with us, and my boss could find a way to deflect it if something happens, but I'm really just trying to make sure it doesn't happen, and I'm not even sure what to say to him.

 

Precisely, and I'm really not sure we have any sort of backup for this thing (another point I tried to bring up with them)...

It may seem counter intuitive but servers have a higher chance of getting ram bit errors than a standard desktop. One of the reasons for this is there are more devices in a server: RAID cards, NICs, remote monitoring (IPMI). There's just more points in the system for something to go wrong, more firmware chips involved. They are just more complex in general than a desktop.

 

Ram errors are actually more common than is generally believed since they can be very hard to conclusively point to for a system issue or crash, sometimes they can happen with no consequence at all (less likely for a server).

 

The "size" of the server to me is irrelevant, the risk is the same, the impact is the same. Whether it's a file server containing 1TB for 10 users to 100TB for 1000 users, you still have a broken server with a pissed off client. Never treat clients any less or more based on size, they all deserve equal care and consideration.

 

For backup I would do two things. On the data disk that contains the files enable Previous Versions/Volume Shadow Copies, 8am 12pm 6pm I find are good for schools. Also create a dedicated partition for the Volume Shadow Copy space and let Windows manage this space so you can get as many versions as possible for the amount of disk space you are willing to give it.

 

Second setup Windows Backup to backup to a NAS, use an iSCSI mounted disk for this so it appears to the OS as a physically attached disk. The reason for this is that if you use a network share on the NAS Windows only supports 1 backup copy point not multiple like it can on a local/physical disk.

[Zodiark1593]

16 hours ago, DXMember said:

something has to change in that relationship

A sudden job change is too drastic to make hasty moves, even if the boss is threatening apocalypse. I would make the recommendation to keep eyes open for any opportunities though. 

 

As for the situation at hand, evaluate the worst case scenario for the server. If the worst case scenario involves the school suing your company into oblivion for lost data (or at least, heavy financial damage to one or both parties) , then you want to take every step that is financially feasible to mitigate such risks. 

 

I'm not a server builder myself, but a file server involving client's data that would be loathed to be missed, I would spec out ECC RAM without hesitation. 

[alex75871]

Sounds like your boss ordered a desktop instead of a server lol

[dalekphalm]

On 2016-08-20 at 4:40 PM, LordMastodon said:

A bit of backstory: I work for a local tech repair shop, and we are essentially contracted by a local school to do their IT work. My boss is a really great guy and a genius when it comes to fixing pre-builts and laptops, but he really doesn't know much at all about custom building, picking parts, etc.

 

We've recently been asked by the aforementioned local school to make them a new file server. We were given a grant by the government to build it, but the drawback was that we could only buy from certain retailers, so that meant we had to get a "custom pre-built", or a prebuilt where I can configure most things but there's a heavily limited selection of parts to choose from. Even though it's limited, there's still enough of a selection that my boss needed help configuring the server, so I stepped in. I essentially picked out all the parts and sent my final list to my boss so he could do the final buying. After a week, I had a chance to look at the final invoice, and I noticed some of my suggestions had been ignored, but most of all, my ECC memory suggestion had been ignored. I brought this up with my boss, and I explained to him the concept and why it's important in mission-critical servers like the file server we were making, and why bit-flipping could potentially be really bad in this kind of computer, but he (confusingly) told me, "Don't worry - it won't happen". The conversation kind of ended there because I had absolutely no clue what to say about that. So I'm pretty much asking you guys what I say to him. He seems convinced that this is a really minor problem, not something to worry about at all. For context, I'm pretty sure nothing happened to their last server, which they had had for 3 years, but I can't confirm whether they had ECC or normal RAM, although this may be one of the things driving my boss to say that nothing will happen since nothing happened to the last server, despite him not knowing whether it had ECC or normal memory. Advice?

Just out of curiosity, was there a reason you couldn't order from Dell or HP or IBM and get a server directly from them?

 

Or configure one through a local reseller? As others have mentioned, most of the big enterprise vendors won't even sell a server without ECC RAM, nor allow you to configure one without it.

 

Honestly, it seems like your boss has no idea what he's talking about when it comes to Enterprise networking and Servers.

 

Where's the new backup solution? Are they re-using the old backup system? What was the old backup system?

 

Either you're missing details, or your boss is flying by the seat of his pants. Either way - not good.

[Rison]

Ok, i've been reading this forum since well, it started and have not ever made an account to post. 

 

This one struck a personal chord.

 

Your boss is an idiot. I know this because I used to work for one. Someone who thinks they know everything, resistant to change and won't trust your opinion. 

Leave. Find another boss. 

People don't leave their work, they leave bosses - and you have one of the bad ones. Find someone who will accept new input, be adaptive to new technologies and supports employee growth and knowledge. Personally I would be frustrated and bitter in that scenario. I personally moved on and have been more happy ever since (my current position, I decide the hardware / software and new installation, setups and configuration)

I don't mean run into his office and quit, but quietly look for alternative work. Give the two weeks notice and be professional and respectful when leaving, it may help down the road.

 

It's a frickin industry standard to use ECC in a file server. I've been doing this for over 20 years professionally, and my last job had a boss similar to what you describe. 

It's his mistake to make, and perhaps he'll "get away with it" and their won't be any consequences.. but i've personally seen bit switch due to ram writing crap to the hard drive. Then the backups get sent this bad data.. then, then, then.. just hope you're not involved if this happens. 

[Smite]

Your boss is the one whom assumes responsibility. You have presented the options and your opinion, nothing more to do. When you become the boss you can make the decisions and accept the responsibilities. A word of advice, "local repair shops" are A-Typical just like best buy know how to sell crap but know very little of the industry. Also like most have stated current server hardware configurations will only allow ecc ram generally in good principle should stick with it. I had a boss once that would give a complete 1 minute deer in headlights stair when I said something technical or asked something technical, highlighting she didn't know shit. When I asked for a 10k raise and didn't receive it, I had no quams about applying else where and getting more money. To each their own. 

 

[dalekphalm]

26 minutes ago, Rison said:

Ok, i've been reading this forum since well, it started and have not ever made an account to post. 

 

This one struck a personal chord.

 

Your boss is an idiot. I know this because I used to work for one. Someone who thinks they know everything, resistant to change and won't trust your opinion. 

Leave. Find another boss. 

People don't leave their work, they leave bosses - and you have one of the bad ones. Find someone who will accept new input, be adaptive to new technologies and supports employee growth and knowledge. Personally I would be frustrated and bitter in that scenario. I personally moved on and have been more happy ever since (my current position, I decide the hardware / software and new installation, setups and configuration)

I don't mean run into his office and quit, but quietly look for alternative work. Give the two weeks notice and be professional and respectful when leaving, it may help down the road.

 

It's a frickin industry standard to use ECC in a file server. I've been doing this for over 20 years professionally, and my last job had a boss similar to what you describe. 

It's his mistake to make, and perhaps he'll "get away with it" and their won't be any consequences.. but i've personally seen bit switch due to ram writing crap to the hard drive. Then the backups get sent this bad data.. then, then, then.. just hope you're not involved if this happens. 

Well hell, even from an Ethical standpoint, even if you can get away with it without consequences... there WILL be consequences for the School. That's incredibly irresponsible to leave the School in a vulnerable position like that, with no ECC memory, and also (possibly) no backup solution?

 

@LordMastodon are you guys fully configuring the Server for them as well, or simply delivering the hardware and letting whatever internal school IT do the config?