File and CCTV server

[Fidycent]

tl;dr - Is it a bad idea to have surveillance and file server on the same server? (Separate drives)

 

 

I'm not going to pretend like i know a lot about servers, i know the basics, i guess.

 

So a family company wants a local file server, and asked me for help, I don't work at the company, but sometimes help out with IT solutions, they also do not have surveillance, but are thinking about investing in that too, I thought, since neither of these processes use a lot of resources, other than storage, that this could be a combined server.

 

File server use:

Storage (duhh)

Backups (duhh)

User access based on permission

Live file editing (6 employees max, almost never more than 3)

 

Surveillance use:

Probably under 8 IP cameras, some recording 24/7, some movement based (maybe all 24/7, depends on the storage use)

 

What i thought of:

2 SSD's in RAID for the previously mentioned live access

2 HDD's in RAID for backup and larger file storage (if necessary, mostly its just pdfs, office documents)

2 HDD's in RAID for the surveillance

Entry level Xeon for that ECC goodness, and really just a prebuilt server when thinking about rest of the hardware, surely the people at dell and stuff know better than me building my own stuff

Was thinking about cheaping out on Windows 10 instead of Windows Server, but im thinking its not that good of an idea and we should stick with windows server.

 

So when we went to the store that provides everything IT related for them, the guy working the counter said he has never heard of having the same server run surveillance and file storage, and said its not a good idea, as previously mentioned im not too experienced with servers, but I don't really see the problem with it, so I'm here to confirm that or deny it?

[leadeater]

As long as you have enough CPU power and separate disk arrays yes you can do this. If you can go dual NIC and dedicate 1 of the NICs to the IP camera traffic.

[chiller15]

How much file server storage do you estimate they require? How much CCTV storage do you estimate they require? Knowing the answers to these would greatly benefit your choices of RAID and backup.

 

For RAID, I'd personally recommend using three or more separate hard drives for each in either a RAID5 or RAID6 configuration. This would provide you with better protection against drive failure.

 

As for backups - The rule of thumb is to have your data in three locations; live data on the live servers, backup storage (such as NAS, an off-site copy such as in the cloud or on another NAS/HDD stored in another physical location. One problem that I can see is backing up the CCTV footage. Depending on how much capacity it uses, it may take a long time to backup that data.

[Fidycent]

19 minutes ago, chiller15 said:

How much file server storage do you estimate they require? How much CCTV storage do you estimate they require? Knowing the answers to these would greatly benefit your choices of RAID and backup.

 

For RAID, I'd personally recommend using three or more separate hard drives for each in either a RAID5 or RAID6 configuration. This would provide you with better protection against drive failure.

 

As for backups - The rule of thumb is to have your data in three locations; live data on the live servers, backup storage (such as NAS, an off-site copy such as in the cloud or on another NAS/HDD stored in another physical location. One problem that I can see is backing up the CCTV footage. Depending on how much capacity it uses, it may take a long time to backup that data.

Sorry, probably should have mentioned that

 

The company is specialized in forestery machinery, so mostly it would be the usual business stuff any company has to deal with, along with parts catalogs etc, some larger files like videos for presentations etc, but nothing too major, I can't imagine their whole file "inventory" being more than 300gb currently, most of it being data that is not edited at all, so the backups wouldnt fill up quickly (probably like the SSDs being 500gb each, and backups around eh, 3tb?)

 

As for CCTV, They don't want to keep records of the footage for years on end, a month most likely would be fine, so at decent settings 6tb should be enough.

 

This is something that needs to be considered yes, a NAS offsite, at either of the owners houses should be installed at some point. As for off-siting the CCTV, probably not possible in thier case, as the only internet connection available wont be able to handle uploading that much data to somewhere.

 

As for RAID, may i ask why more than 2 drives per array? Especially since there is an off-site backup, ofcourse both drives can fail, but what are the odds of that? And with there bieng off-site periodic backups, I dont think it would be such a problem.

[chiller15]

2 minutes ago, Fidycent said:

Sorry, probably should have mentioned that

No problem.

2 minutes ago, Fidycent said:

The company is specialized in forestery machinery, so mostly it would be the usual business stuff any company has to deal with, along with parts catalogs etc, some larger files like videos for presentations etc, but nothing too major, I can't imagine their whole file "inventory" being more than 300gb currently, most of it being data that is not edited at all, so the backups wouldnt fill up quickly

In that case I'd recommend 3 or 4 1TB disks in RAID5. Don't forget that with RAID, you're going to lose some total capacity for it's parity storage.

2 minutes ago, Fidycent said:

As for CCTV, They don't want to keep records of the footage for years on end, a month most likely would be fine, so at decent settings 6tb should be enough.

This is where the costs will come in as you will need a fair amount of disks to accommodate this level or storage. So something like 3 or 4 3TB drives would be good enough for that I would assume (I haven't done any calculations, you'd need to check first!)

2 minutes ago, Fidycent said:

This is something that needs to be considered yes, a NAS offsite, at either of the owners houses should be installed at some point. As for off-siting the CCTV, probably not possible in thier case, as the only internet connection available wont be able to handle uploading that much data to somewhere.

With an off-site NAS, the other thing you need to consider is how you'll transport the data from the workplace to the off-site locaiton. So you'd need to account for external HDDs or something similar to transport the data. You don't want to take the NAS to the workplace to do it. Should a fire break out or something whilst both are there, you're losing your live data, on-site backup NAS data and off-site backup NAS data.

2 minutes ago, Fidycent said:

As for RAID, may i ask why more than 2 drives per array? Especially since there is an off-site backup, ofcourse both drives can fail, but what are the odds of that? And with there bieng off-site periodic backups, I dont think it would be such a problem.

Whilst the odds are low, it does happen. I've lost two drives at the same time in my work's 12-disk RAID5 array before.

If you have two disks, you're limited to either RAID0 or RAID1, neither of which are recommended in a production environment.

Moving to three disks would allow you to use RAID5. However with three disks, you can only afford for one to die and hope you can order, receive, replace and the RAID array to re-build itself before another one dies.

Moving to four disks would allow you to use either RAID5 or RAID6, where you can lose two disks at the same time and the array can still re-built itself. So if one disk dies, you at least have some leeway in getting a replacement sorted, even if a second disk dies during the process.

 

Another point of failure can be the RAID controller itself. I've heard (but not experienced first hand) that RAID controllers can die or something happens that it needs replacing/re-building and you need to use the exact same controller as you cannot transfer RAID arrays between different RAID controllers. If you use software RAID, such as within Windows, then this can be done...but if it's the controller, you can't. Note: I would recommend checking this information, because this is just going from what I've been told/seen in the past...so it may be wrong. Maybe someone on here can correct me. 

 

I know I've gone a bit overboard and it sounds like overkill, but with storage, you're always playing with excess, because the worst can and sometimes does happen. Plus you want to cover your own back. If you just use two disks in RAID1 and something happens - the company will come back to you. 

[Fidycent]

2 minutes ago, chiller15 said:

Whilst the odds are low, it does happen. I've lost two drives at the same time in my work's 12-disk RAID5 array before.

Well, you see, whilst, yes as you mentioned, it does happen, in your situation, there were 12 points of failure,  if you were to have only 2 drives, theres only 2 points of failure. So the odds of it happening in a 2 drive configuration are much lower (obviously). In your config 16% (ish) of the drives have to fail, in a 2 drive config its 50% of the drives. The important files will be the ones also kept on the SSD's, so theres really 4 drives that need to fail for it to be completely lost, yes, you might lose some progress if  BOTH SSDs crash during the workday, but, at least at the beginning, It's going to be a massive upgrade, since currently backups are pretty much not a thing at all.

4 minutes ago, chiller15 said:

With an off-site NAS, the other thing you need to consider is how you'll transport the data from the workplace to the off-site locaiton. So you'd need to account for external HDDs or something similar to transport the data. You don't want to take the NAS to the workplace to do it. Should a fire break out or something whilst both are there, you're losing your live data, on-site backup NAS data and off-site backup NAS data.

The server will have remote access enabled, so that won't be a problem.

 

11 minutes ago, chiller15 said:

I know I've gone a bit overboard and it sounds like overkill, but with storage, you're always playing with excess, because the worst can and sometimes does happen. Plus you want to cover your own back. If you just use two disks in RAID1 and something happens - the company will come back to you. 

I am helping a Family company, im not getting paid for this, ofcourse, if their budget allows, id go as overboard as you, but, its a small company, and shelling out 5k$ for a server is not possible at this point, later upgrades might allow for bigger arrays and bigger budgets.

[chiller15]

3 minutes ago, Fidycent said:

Well, you see, whilst, yes as you mentioned, it does happen, in your situation, there were 12 points of failure,  if you were to have only 2 drives, theres only 2 points of failure. So the odds of it happening in a 2 drive configuration are much lower (obviously). In your config 16% (ish) of the drives have to fail, in a 2 drive config its 50% of the drives. The important files will be the ones also kept on the SSD's, so theres really 4 drives that need to fail for it to be completely lost, yes, you might lose some progress if  BOTH SSDs crash during the workday, but, at least at the beginning, It's going to be a massive upgrade, since currently backups are pretty much not a thing at all.

Valid point, I was just saying that it is possible, even if unlikely.

3 minutes ago, Fidycent said:

The server will have remote access enabled, so that won't be a problem.

So you plan on transferring the backups via remote access? You'll encounter the same issues you explained about backing up to the cloud.

3 minutes ago, Fidycent said:

I am helping a Family company, im not getting paid for this, ofcourse, if their budget allows, id go as overboard as you, but, its a small company, and shelling out 5k$ for a server is not possible at this point, later upgrades might allow for bigger arrays and bigger budgets.

Completely understandable. As I mentioned, I was going overboard, but it's always good to have as much information as possible before you embark on the project!

[Fidycent]

1 minute ago, chiller15 said:

Valid point, I was just saying that it is possible, even if unlikely.

Yeah, I understood that, just pointing out my point of view

1 minute ago, chiller15 said:

So you plan on transferring the backups via remote access? You'll encounter the same issues you explained about backing up to the cloud.

The connection is not that bad, might have bashed it too much, its a 4g LTE connection, and during the nighttime (another discussion to be had, but during peak times the connection is awful) the connection is solid, 20mbps up should be avaiable for atleast 12 hours a day, i just didnt want to stress it constantly by having the CCTV footage uploaded aswell

1 minute ago, chiller15 said:

Completely understandable. As I mentioned, I was going overboard, but it's always good to have as much information as possible before you embark on the project!

Yeah, I understood that aswell, just, again, noting down my point of view, and yeah, its a very good idea to get as much info as possible regarding the starting config and the chances to upgrade.

[seagate_surfer]

There are hard drives that were designed specifically to work in these types of environments, I will paste the link to the specifications of Seagate SkyHawk in case you are interested: https://www.seagate.com/internal-hard-drives/hdd/skyhawk/

[leadeater]

11 hours ago, chiller15 said:

Moving to four disks would allow you to use either RAID5 or RAID6, where you can lose two disks at the same time and the array can still re-built itself. So if one disk dies, you at least have some leeway in getting a replacement sorted, even if a second disk dies during the process.

Just want to make sure it's clear that the 2 disk failure is for the RAID 6 configuration only, that or doesn't make it all that clear. Just being careful . Small disk count array, around that 3 or 4 disk mark and RAID 5 I'd go for 2 larger disks in RAID 1. Both can only survive a single disk failing and the more disks the more likely to fail, the rebuild is also longer on RAID 5. At least in a worst case with RAID 1 data recovery is also simpler and cheaper and you have 2 possible sources for that recovery versus 1 for the RAID 5.

 

This advice doesn't really apply here because of the low camera count but it's generally recommended to store live footage on a RAID 1 or 10 then use archiving to move the footage after a set amount of time to a RAID 6 array, software like Milestone does this very well and it's free for 8 cameras which is nice. Parity RAIDs aren't that good for sustained writes and when you have a lot of cameras you can hit flashcache exhaustion that may lead to dropped frames.

 

11 hours ago, chiller15 said:

I've heard (but not experienced first hand) that RAID controllers can die or something happens that it needs replacing/re-building and you need to use the exact same controller as you cannot transfer RAID arrays between different RAID controllers.

You can replace pretty much any LSI RAID card with any other LSI RAID card and import the array, the only cases where there is restrictions on that is if you have enabled advanced features like SSD cache then you have to replace with another RAID card with that feature upgrade present. I avoid that type of configuration because of that, it's a huge risk.

 

11 hours ago, Fidycent said:

The connection is not that bad, might have bashed it too much, its a 4g LTE connection, and during the nighttime (another discussion to be had, but during peak times the connection is awful) the connection is solid, 20mbps up should be avaiable for atleast 12 hours a day, i just didnt want to stress it constantly by having the CCTV footage uploaded aswell

Is there a data cap on that or other bandwidth charges, those 4G LTE connections can be wildly, wildly expensive. For those situations of small servers in remote locations without good internet access a set of external HDDs rotated weekly works well. It's not great as if you lose both the server and the connected HDD for that week your earliest recovery point is whenever the swap over happened.

 

Some better options can be to use cloud storage from the get go like Office 365, G Suite or anything similar and work directly off that. Most of them use client local caching if you need better performance. Have a look in to that type of option and compare the cost of buying hardware versus a paid subscription service. There are CCTV options like this too but that 4G connection is a bit of a problem and if you have to buy a server/nas for that anyway you lose a lot of the cost benefit of those cloud options.

 

As a side warning when it comes to small businesses like that once you do something for them like this you basically get tied to them for life so go with a solution that requires the least support from you, that you feel as the least long term risk. That's where cloud options win out so much for me, after setup it's not my problem and I'm not providing lots of free support or bearing any risks from hardware failures.

[Fidycent]

On 3/23/2019 at 2:27 AM, leadeater said:

This advice doesn't really apply here because of the low camera count but it's generally recommended to store live footage on a RAID 1 or 10 then use archiving to move the footage after a set amount of time to a RAID 6 array, software like Milestone does this very well and it's free for 8 cameras which is nice. Parity RAIDs aren't that good for sustained writes and when you have a lot of cameras you can hit flashcache exhaustion that may lead to dropped frames.

Exactly why we thought about not getting more than 8, ive used milestone before and its super simple to use and setup. Plus 8 cameras should be enough until the company expands.

On 3/23/2019 at 2:27 AM, leadeater said:

Is there a data cap on that or other bandwidth charges, those 4G LTE connections can be wildly, wildly expensive. For those situations of small servers in remote locations without good internet access a set of external HDDs rotated weekly works well. It's not great as if you lose both the server and the connected HDD for that week your earliest recovery point is whenever the swap over happened.

Thank baby jesus - no, its a flat 20 euros for unlimited data, as I said it may act up sometimes, but since no proper connection can be achieved, it will do the job

 

On 3/23/2019 at 2:27 AM, leadeater said:

As a side warning when it comes to small businesses like that once you do something for them like this you basically get tied to them for life so go with a solution that requires the least support from you, that you feel as the least long term risk. That's where cloud options win out so much for me, after setup it's not my problem and I'm not providing lots of free support or bearing any risks from hardware failures.

As I said before, its a family company, I might get hired by them later on the line, but currently im just helping out, as they are mechanics they sometimes fix my car and along the lines.

[Fidycent]

Not wanting to spam much more, ill just post here.

 

@leadeater @chiller15

 

So it seems like a good starter server would be the Dell T130 

 

The G3900 imo would be a bad choice, not enough resources for what i want to do.

The E3-1220 v5 seems like a good choice, however if you chip in the extra 150$ ish, you get the 16gb ram (that id upgrade to anyhow), plus you get the E3-1230 v6 which has hyper threading would be a better choice.

 

As for raid controllers - do I actually need one? Or would the Windows RAID options be plenty for my use case.

 

For software - Windows Server is VERY Expensive, paying almost the same amount of money for software as I would pay for the server is quite ridiculous, the server would be accessed by 6 people at most, for a good while it wouldnt breach the 25 user mark - Windows Server Essentials should be a good choice, right?

 

Thanks in advance.

[Mikensan]

3 hours ago, Fidycent said:

Not wanting to spam much more, ill just post here.

 

@leadeater @chiller15

 

So it seems like a good starter server would be the Dell T130 

 

The G3900 imo would be a bad choice, not enough resources for what i want to do.

The E3-1220 v5 seems like a good choice, however if you chip in the extra 150$ ish, you get the 16gb ram (that id upgrade to anyhow), plus you get the E3-1230 v6 which has hyper threading would be a better choice.

 

As for raid controllers - do I actually need one? Or would the Windows RAID options be plenty for my use case.

 

For software - Windows Server is VERY Expensive, paying almost the same amount of money for software as I would pay for the server is quite ridiculous, the server would be accessed by 6 people at most, for a good while it wouldnt breach the 25 user mark - Windows Server Essentials should be a good choice, right?

 

Thanks in advance.

I can't imedately find a difference in the T130 and T30, however if you live in the states slickdeals.net offers the T30 pretty frequently on sale.

https://slickdeals.net/f/12850774-dell-poweredge-t30-mini-server-intel-xeon-e3-1225-1tb-hdd-299-free-shipping?src=SiteSearchV2_SearchBarV2Algo1

 

 

Windows Storage spaces would probably be enough for what you're doing. If you only ever use a single gigabit connection then you're definitely fine.

 

Windows Server should be considered if you want centralized authentication or some other various Windows service - majority of which honestly can be hosted on linux (including AD actually lol). The number of people accessing it from a SMB/File share doesn't effect what license you buy. Well I guess technically there are user CALs but I don't think anybody under 1,000 employees adhers to it.

 

Otherwise you're probably fine just using Windows Pro (I think has a couple more "Features" for Windows Storage Spaces) or even Ubuntu honestly. Ubuntu + KVM + mdadm and you have a very capable solution.

[Fidycent]

1 hour ago, Mikensan said:

I can't imedately find a difference in the T130 and T30, however if you live in the states slickdeals.net offers the T30 pretty frequently on sale.

https://slickdeals.net/f/12850774-dell-poweredge-t30-mini-server-intel-xeon-e3-1225-1tb-hdd-299-free-shipping?src=SiteSearchV2_SearchBarV2Algo1

 

 

Windows Storage spaces would probably be enough for what you're doing. If you only ever use a single gigabit connection then you're definitely fine.

 

Windows Server should be considered if you want centralized authentication or some other various Windows service - majority of which honestly can be hosted on linux (including AD actually lol). The number of people accessing it from a SMB/File share doesn't effect what license you buy. Well I guess technically there are user CALs but I don't think anybody under 1,000 employees adhers to it.

 

Otherwise you're probably fine just using Windows Pro (I think has a couple more "Features" for Windows Storage Spaces) or even Ubuntu honestly. Ubuntu + KVM + mdadm and you have a very capable solution.

Not from the states, sadly.

 

Id like to use (as suggsted by @leadeater) another NIC for the CCTV system, will that be an issue?

 

As for the OS, id like to stick with Windows, as my CCTV software runs on that, a requirement is simultaneous remote access with "tiered" user access, is that possible with Windows 10 Pro?

[leadeater]

1 hour ago, Fidycent said:

As for the OS, id like to stick with Windows, as my CCTV software runs on that, a requirement is simultaneous remote access with "tiered" user access, is that possible with Windows 10 Pro?

Not really, that sort of remote access requires Windows Server usually. Windows 10 only allows single user remote access and will disconnect any other users if someone else connects. If you're not using RDP and some other form of remote access, like Milestone remote console that would work.

 

6 hours ago, Fidycent said:

Windows Server Essentials should be a good choice, right?

Yes that will work for you.