BIG Surveillance Setup for Warehouse

[LasleyMediaKentucky]

Hi all! 

 

So I'm doing my second network camera installation job/network setup... this time, quite a bit bigger than the last one that I had tons of problems with because I used lower-end hardware...

 

This setup is for a new office space that is huge... 3 floors and tons of room. when the place was built, it has a convenient way to route cables throughout the floors via 4 pieces of 6-inch conduit that go straight up through the floors, each leading into a "server closet" that is fairly roomy for a full size rack in each closet (not needed on 2nd and 3rd floor).

 

I've been planning hardware out for this for a while now. Here is the current hardware.

 

Racks - 48U server rack on first floor, 24u racks on others

Servers - For camera servers, we're using 3 custom servers. (one for each floor in rack) They are all identical. Each will be pulling around 36 cameras.

- Intel Core i7 6950X 10-Core 20-thread CPU

- 32GB RipJaws DDR4-2400

- ARCTIC Freezer i30 CPU cooler

- MSI X99A Plus Motherboard

- GTX 750Ti FTW (to help with GPU decoding and video playback, also since X99 requires a GPU) 

- Crucial 240GB SSD

- CHASSIS: Rosewill RSV-L4500 4U 

- Intel PRO/1000 dual port gigabit NIC

- EVGA 650W PSU 

Main NAS (For video ONLY) - Custom Server

- Intel Core i7 6700K 4-core 8-thread CPU

- 16GB RipJaws DDR4-2400

- Arctic Freezer i30 CPU cooler

- Asus Z170A Motherboard 

- No video card (it's a NAS...) 

- 120GB boot SSD (FreeNAS) 

- 12 x 4TB WD Red HDD's 

- CHASSIS - Rosewill RSV-L4412 

- Intel PRO/1000 dual port gigabit NIC

- EVGA 650W PSU

Employee File Storage Server - Custom

- Same as Main NAS, only running Windows Server 2016

 

INDOOR CAMERAS - AXIS M3045-V (93 of them) 

OUTDOOR CAMERAS - AXIS M2025-LE (17 of them) 

Network Switches -

- For PoE Cameras: NETGEAR ProSAFE GS752TP 48-port (3x) 

- For computer data connections: NETGEAR ProSAFE GS748T 48-Port Gigabit switch (3x) 

 

There is too much to fully explain but I'd like to hear your feedback on the selection of hardware at the moment. Nothing has been purchased yet so I can still finagle with hardware. We will be using Milestone xProtect Professional, although I sincerely HATE Milestone simply because of the lack of contact with them and expensive as hell. But there's too many cameras to really use any other software.

 

Please post your opinions! 

 

 

 

 

 

 

 

 

[ttam]

Your builds look more like workstation builds. This is going to be a huge pain in the ass when things break, and things are going to break.

 

You would be better off picking up like a xeon 2640, for a LOT less of a cost. And going for actual power supplies made for being run 24/7..

 

Just don't do these work station builds for rack mounted servers..

[brwainer]

6 minutes ago, LasleyMediaKentucky said:

Hi all! 

 

So I'm doing my second network camera installation job/network setup... this time, quite a bit bigger than the last one that I had tons of problems with because I used lower-end hardware...

 

This setup is for a new office space that is huge... 3 floors and tons of room. when the place was built, it has a convenient way to route cables throughout the floors via 4 pieces of 6-inch conduit that go straight up through the floors, each leading into a "server closet" that is fairly roomy for a full size rack in each closet (not needed on 2nd and 3rd floor).

 

I've been planning hardware out for this for a while now. Here is the current hardware.

 

Racks - 48U server rack on first floor, 24u racks on others

Servers - For camera servers, we're using 3 custom servers. (one for each floor in rack) They are all identical. Each will be pulling around 36 cameras.

- Intel Core i7 6950X 10-Core 20-thread CPU

- 32GB RipJaws DDR4-2400

- ARCTIC Freezer i30 CPU cooler

- MSI X99A Plus Motherboard

- GTX 750Ti FTW (to help with GPU decoding and video playback, also since X99 requires a GPU) 

- Crucial 240GB SSD

- CHASSIS: Rosewill RSV-L4500 4U 

- Intel PRO/1000 dual port gigabit NIC

- EVGA 650W PSU 

Main NAS (For video ONLY) - Custom Server

- Intel Core i7 6700K 4-core 8-thread CPU

- 16GB RipJaws DDR4-2400

- Arctic Freezer i30 CPU cooler

- Asus Z170A Motherboard 

- No video card (it's a NAS...) 

- 120GB boot SSD (FreeNAS) 

- 12 x 4TB WD Red HDD's 

- CHASSIS - Rosewill RSV-L4412 

- Intel PRO/1000 dual port gigabit NIC

- EVGA 650W PSU

Employee File Storage Server - Custom

- Same as Main NAS, only running Windows Server 2016

 

INDOOR CAMERAS - AXIS M3045-V (93 of them) 

OUTDOOR CAMERAS - AXIS M2025-LE (17 of them) 

Network Switches -

- For PoE Cameras: NETGEAR ProSAFE GS752TP 48-port (3x) 

- For computer data connections: NETGEAR ProSAFE GS748T 48-Port Gigabit switch (3x) 

 

There is too much to fully explain but I'd like to hear your feedback on the selection of hardware at the moment. Nothing has been purchased yet so I can still finagle with hardware. We will be using Milestone xProtect Professional, although I sincerely HATE Milestone simply because of the lack of contact with them and expensive as hell. But there's too many cameras to really use any other software.

 

Please post your opinions! 

 

 

 

 

 

 

 

 

You say GPU for decoding, but does your software support that?

 

Axis cameras are great (we are an Axis partner) but make sure you set up a monitoring system that checks both pings and HTTP(S) responses. On recent firmwares Axis moved to a new HTTP(S) server which has been a huge improvement, but we still see occasional crashes, and when that happens the camera will need to be power cycled (since there is no way to tell them to reboot once the webserver crashes!)

[SCHISCHKA]

53 minutes ago, LasleyMediaKentucky said:

Please post your opinions! 

nice hardware. i hope you comeback and benchmarks before putting into service.

why ssd for the cameras and not surveillance drives?

why i7 and not xeon?

im wondering if ecc ram would be useful in video recording? i know its good for databases but i think video recording can tolerate a bit of error

no redundant power supplies and UPS? Many years ago at a telco i had the pleasure of seeing a pretty extreme basement full of lead/acid batteries & a generator the size of a bus; this building had the most comprehensive security than any other building i have worked in

[brwainer]

1 hour ago, SCHISCHKA said:

nice hardware. i hope you comeback and benchmarks before putting into service.

why ssd for the cameras and not surveillance drives?

why i7 and not xeon?

im wondering if ecc ram would be useful in video recording? i know its good for databases but i think video recording can tolerate a bit of error

no redundant power supplies and UPS? Many years ago at a telco i had the pleasure of seeing a pretty extreme basement full of lead/acid batteries & a generator the size of a bus; this building had the most comprehensive security than any other building i have worked in

I think he intends to use the 3 video servers just for processing the video, and the storage would all be on the NAS.

[SCHISCHKA]

28 minutes ago, brwainer said:

I think he intends to use the 3 video servers just for processing the video, and the storage would all be on the NAS.

is processing not done by the cameras themselves? surely they can output in the format and resolution you specify?

[brwainer]

1 hour ago, SCHISCHKA said:

is processing not done by the cameras themselves? surely they can output in the format and resolution you specify?

The issue actually is doing the motion detection, it's a lot of work to do video analysis. The Axis cameras he's using are capable of doing the motion detection themselves and notifying the server to record, but even then the server needs to maintain the live video feed from the cameras (at least that is how it works when you use ZoneMinder, I don't know if Milestone is different)

[Lurick]

I see two problems with the networking gear:

1) I really hope those switches can handle the sheer volume of traffic that will be passed over them. There is going to be a ton of multicast traffic that will need to be handled and processed by the switches and sent over uplinks to the servers. If the cameras only stream 10Mbps of data each you're going to need roughly 4 to 5 uplinks off each switch to the servers at gigabit rates.

 

2) The switches do NOT have enough PoE budget to power all those cameras. The switch can only supply 384W of PoE power and the way PoE works (if I remember right) is a set amount is budgeted per port based on the requirements of the device. Those cameras require 802.3af which is roughly 15.4 watts of power or 24 cameras per switch that can draw power leaving the other 24 cameras without power.

 

I would suggest looking into something that can supply more power per port and that has 10Gbit uplink ports to handle all the traffic.

[LasleyMediaKentucky]

8 hours ago, ttam said:

Your builds look more like workstation builds. This is going to be a huge pain in the ass when things break, and things are going to break.

 

You would be better off picking up like a xeon 2640, for a LOT less of a cost. And going for actual power supplies made for being run 24/7..

 

Just don't do these work station builds for rack mounted servers..

I chose the 6950X because compared to it's Xeon equivalent, it's quite a bit cheaper. And Yes I am probably going to put a mini redundant PSU in there. Although, one of my servers at work (I own a business) uses an EVGA PSU and pulls about 450-500 watts at all times and it's been going strong for 2 years now. 

[LasleyMediaKentucky]

7 hours ago, SCHISCHKA said:

nice hardware. i hope you comeback and benchmarks before putting into service.

why ssd for the cameras and not surveillance drives?

why i7 and not xeon?

im wondering if ecc ram would be useful in video recording? i know its good for databases but i think video recording can tolerate a bit of error

no redundant power supplies and UPS? Many years ago at a telco i had the pleasure of seeing a pretty extreme basement full of lead/acid batteries & a generator the size of a bus; this building had the most comprehensive security than any other building i have worked in

Because when you RAID 12 drives, read/write speeds are probably going to be higher than regular Gigabit speeds of each client. Plus the equivalent amount of storage in SSD's would cost a fortune. 

 

i7 6950x equivalent of a xeon is a lot more expensive. ECC is not super necessary here. 

 

I forgot the UPS's in the hardware. We're using CyberPower PR3000LCDRTXL2U 3000VA rackmount units.

6 hours ago, brwainer said:

I think he intends to use the 3 video servers just for processing the video, and the storage would all be on the NAS.

Correct. 

5 hours ago, SCHISCHKA said:

is processing not done by the cameras themselves? surely they can output in the format and resolution you specify?

No, we're using H.264 to save bandwidth since there are so many cameras.

 

4 hours ago, brwainer said:

The issue actually is doing the motion detection, it's a lot of work to do video analysis. The Axis cameras he's using are capable of doing the motion detection themselves and notifying the server to record, but even then the server needs to maintain the live video feed from the cameras (at least that is how it works when you use ZoneMinder, I don't know if Milestone is different)

Correct. Motion detection uses a ton of CPU power and on my last job, I underpowered it for iSpy. But BlueIris is beast! Very CPU friendly and never freezes up. 

 

1 hour ago, Lurick said:

I see two problems with the networking gear:

1) I really hope those switches can handle the sheer volume of traffic that will be passed over them. There is going to be a ton of multicast traffic that will need to be handled and processed by the switches and sent over uplinks to the servers. If the cameras only stream 10Mbps of data each you're going to need roughly 4 to 5 uplinks off each switch to the servers at gigabit rates.

 

2) The switches do NOT have enough PoE budget to power all those cameras. The switch can only supply 384W of PoE power and the way PoE works (if I remember right) is a set amount is budgeted per port based on the requirements of the device. Those cameras require 802.3af which is roughly 15.4 watts of power or 24 cameras per switch that can draw power leaving the other 24 cameras without power.

 

I would suggest looking into something that can supply more power per port and that has 10Gbit uplink ports to handle all the traffic.

Each camera, H.264, uses around 2Mbps per. I should be okay with a gigabit off of each. 

I'll look into that. We may use something like Avaya switches which can handle a lot more power. 

 

 

[Lurick]

16 minutes ago, LasleyMediaKentucky said:

-snip-

Yah, in that case you should be good in terms of bandwidth although it might not be a bad idea if you can find some decently priced 10Gbit NICs and switches just incase, but if they only use 2 or even 5 Mbps of bandwidth then gigabit or dual gigabit will be plenty. I would say make sure the switches can handle at least 240+ multicast routes, just for looking into the future as well.

[LasleyMediaKentucky]

32 minutes ago, Lurick said:

Yah, in that case you should be good in terms of bandwidth although it might not be a bad idea if you can find some decently priced 10Gbit NICs and switches just incase, but if they only use 2 or even 5 Mbps of bandwidth then gigabit or dual gigabit will be plenty. I would say make sure the switches can handle at least 240+ multicast routes, just for looking into the future as well.

Agreed. I am by no means a professional or a certified network tech, but I do have loads of experience in networking and computers. That's why I was hired to do this job. That's also why I come here for my questions lol

[brwainer]

1 hour ago, LasleyMediaKentucky said:

Correct. Motion detection uses a ton of CPU power and on my last job, I underpowered it for iSpy. But BlueIris is beast! Very CPU friendly and never freezes up. 

I am confused though - you have upgraded to cameras that are able to do the motion detection in-camera (compared to the last job) but you are still going to do the motion detection on the servers? The Axis Motion Detection 2 was very good, and I don't have much experience with the Axis Motion Detection 3 that these have but I imagine it is no worse that version 2.

 

2 hours ago, Lurick said:

2) The switches do NOT have enough PoE budget to power all those cameras. The switch can only supply 384W of PoE power and the way PoE works (if I remember right) is a set amount is budgeted per port based on the requirements of the device. Those cameras require 802.3af which is roughly 15.4 watts of power or 24 cameras per switch that can draw power leaving the other 24 cameras without power.

802.3af can supply up to 15.4W, but that doesn't mean the cameras actually use it all. The data sheets say max 3.0W http://www.axis.com/files/datasheet/ds_m3045v_1539634_en_1607.pdf and max 6.9W http://www.axis.com/files/datasheet/ds_m2025le_1587734_en_1609.pdf

[Lurick]

7 minutes ago, brwainer said:

I am confused though - you have upgraded to cameras that are able to do the motion detection in-camera (compared to the last job) but you are still going to do the motion detection on the servers? The Axis Motion Detection 2 was very good, and I don't have much experience with the Axis Motion Detection 3 that these have but I imagine it is no worse that version 2.

 

802.3af can supply up to 15.4W, but that doesn't mean the cameras actually use it all. The data sheets say max 3.0W http://www.axis.com/files/datasheet/ds_m3045v_1539634_en_1607.pdf and max 6.9W http://www.axis.com/files/datasheet/ds_m2025le_1587734_en_1609.pdf

Ah, I missed that in the data sheet earlier, I just saw they required 802.3AF so I made some assumptions. In that case the switches would work since 48*7=336 which is right about the limit for what the switch can supply for PoE. In that case it would more be a matter of preference/tolerance and if any other PoE devices will be hooked up that will need additional power.

[brwainer]

33 minutes ago, Lurick said:

Ah, I missed that in the data sheet earlier, I just saw they required 802.3AF so I made some assumptions. In that case the switches would work since 48*7=336 which is right about the limit for what the switch can supply for PoE. In that case it would more be a matter of preference/tolerance and if any other PoE devices will be hooked up that will need additional power.

He only will have 19 of the 6.9W cameras, so even if they are all on one switch, the budget will be about 2/3 used, which is a comfortable amount of usage.

[LasleyMediaKentucky]

4 hours ago, brwainer said:

I am confused though - you have upgraded to cameras that are able to do the motion detection in-camera (compared to the last job) but you are still going to do the motion detection on the servers? The Axis Motion Detection 2 was very good, and I don't have much experience with the Axis Motion Detection 3 that these have but I imagine it is no worse that version 2.

 

802.3af can supply up to 15.4W, but that doesn't mean the cameras actually use it all. The data sheets say max 3.0W http://www.axis.com/files/datasheet/ds_m3045v_1539634_en_1607.pdf and max 6.9W http://www.axis.com/files/datasheet/ds_m2025le_1587734_en_1609.pdf

My bad, yes we're using the motion detection on those. We are still going with a high-level CPU to just stay on the safe side and future upgrading. 

[Remix]

You'd be better off using a Xeon or server-grade CPU despite the cost difference. Server equipment is designed to run for ages, without needing frequent restarts. If not on the other machines, at least on the NAS.

[leadeater]

I would do a price compare between your current proposed camera servers and a system using dual Intel E5-2620 v4. With this you get 6 more cores, lower average power draw, ecc (just a bonus) and parts rated for 24/7 use.

 

Edit:

Dual Intel E5-2630 v4 may also be close to the same price as the 6950X but with 10 more cores.

[leadeater]

17 hours ago, brwainer said:

Axis cameras are great (we are an Axis partner) but make sure you set up a monitoring system that checks both pings and HTTP(S) responses. On recent firmwares Axis moved to a new HTTP(S) server which has been a huge improvement, but we still see occasional crashes, and when that happens the camera will need to be power cycled (since there is no way to tell them to reboot once the webserver crashes!)

This is why I much prefer midspan PoE injectors with network management. You can remotely power down a port etc.

 

Also high power PoE switches fail at a higher rate and cost proportionally too much compared to their non PoE counterpart and a rackmount midspan injector. Down side is you need more rack space and more cables.

 

http://www.microsemi.com/products/poe-systems/pd-9000g-high-power-family

http://www.midspans.com/pages/30w/POE806U-24AT_24_port_SNMP_midspan.php

 

http://www.newegg.com/Product/Product.aspx?Item=N82E16833996067

[leadeater]

17 hours ago, LasleyMediaKentucky said:

We will be using Milestone xProtect Professional, although I sincerely HATE Milestone simply because of the lack of contact with them and expensive as hell. But there's too many cameras to really use any other software.

If your not happy with Milestone check out Genetec, their software supports that amount of cameras and also supports GPU accelerated decoding.

 

https://www.genetec.com/solutions/all-products/omnicast

[ttam]

11 hours ago, LasleyMediaKentucky said:

I chose the 6950X because compared to it's Xeon equivalent, it's quite a bit cheaper. And Yes I am probably going to put a mini redundant PSU in there. Although, one of my servers at work (I own a business) uses an EVGA PSU and pulls about 450-500 watts at all times and it's been going strong for 2 years now. 

This is an incredibly rookie thing to reply with. Just skip the workstation builds and actually put to use having a rack mounted server, and put the saved cash into a nice UPS for the rack. These kind of systems don't need to be rocking the high of the high end workstations..

 

I too own a business, or well 11 right now including a company that deals with installing camera systems with servers that range from $100,000 to $3,000,000

[LasleyMediaKentucky]

On 1/1/2017 at 8:53 PM, ttam said:

I too own a business, or well 11 right now including a company that deals with installing camera systems with servers that range from $100,000 to $3,000,000

Dayyummmm...