CRITICAL Camera Surveillance Problem

[LasleyMediaKentucky]

Hello all, 

 

So I have posted about this topic before and was to no avail. I am constructing a surveillance server/system at a daycare and I have had nothing but problems since day one. 

 

I am using 24 HOSAFE 2MD4P RTSP cameras, 11 Reo-link RLC-410 outdoor cameras, and a Dell Powerconnect 3448p PoE switch. 

 

The server specs are as listed below. 

 

CPU - Intel Core i7 6800K @ 4GHz 

RAM - 16GB Ripjaws DDR4-2400 

Motherboard - Gigabyte X99 UD3P

SSD - 120GB Adata 

HDD - (2x) Hitachi 2TB NAS drives (RAID-1) 

GPU - Quadro FX 1800 (x99 requires a GPU) 

Network Card - Intel PRO-1000 dual port gigabit nic

PSU - EVGA 500w 

OS - Windows Server 2012 R2 

Current VMS - iSpy / HD IPC PRO (CMS 2.0) 

 

CPU maxes out to 100% in iSpy with all cameras at 2fps (yes, TWO). RAM usage is about 7GB (a lot for iSpy) 

 

I just need this system to work. I'm afraid that one of these days I could end up costing this place thousands of dollars when they don't have video evidence of something. I don't care what it takes at this point, it just NEEDS TO WORK... I've tried everything that I know to fix it and the same thing just keeps happening over and over again. 

 

I want to use Milesone Xprotect SO badly, but it won't accept RTSP streams from any of my cameras (rtsp://admin:@xxx.xxx.xxx.xxx/live0.264), and it won't even recognize them all... 

 

Someone please help. I don't want to be sued for this and I am beyond frustrated at this point. It's been going on for three months and nobody has answers. 

[tlink]

11 minutes ago, LasleyMediaKentucky said:

Hello all, 

 

So I have posted about this topic before and was to no avail. I am constructing a surveillance server/system at a daycare and I have had nothing but problems since day one. 

 

I am using 24 HOSAFE 2MD4P RTSP cameras, 11 Reo-link RLC-410 outdoor cameras, and a Dell Powerconnect 3448p PoE switch. 

 

The server specs are as listed below. 

 

CPU - Intel Core i7 6800K @ 4GHz 

RAM - 16GB Ripjaws DDR4-2400 

Motherboard - Gigabyte X99 UD3P

SSD - 120GB Adata 

HDD - (2x) Hitachi 2TB NAS drives (RAID-1) 

GPU - Quadro FX 1800 (x99 requires a GPU) 

Network Card - Intel PRO-1000 dual port gigabit nic

PSU - EVGA 500w 

OS - Windows Server 2012 R2 

Current VMS - iSpy / HD IPC PRO (CMS 2.0) 

 

CPU maxes out to 100% in iSpy with all cameras at 2fps (yes, TWO). RAM usage is about 7GB (a lot for iSpy) 

 

I just need this system to work. I'm afraid that one of these days I could end up costing this place thousands of dollars when they don't have video evidence of something. I don't care what it takes at this point, it just NEEDS TO WORK... I've tried everything that I know to fix it and the same thing just keeps happening over and over again. 

 

I want to use Milesone Xprotect SO badly, but it won't accept RTSP streams from any of my cameras (rtsp://admin:@xxx.xxx.xxx.xxx/live0.264), and it won't even recognize them all... 

 

Someone please help. I don't want to be sued for this and I am beyond frustrated at this point. It's been going on for three months and nobody has answers. 

why did you go with an x99 platform and an i7 6800k cpu? why not a 12 core xeon or something? Ispy strikes me as the kind of program that uses multiple cores efficiently. you're trying to hook up A LOT of live feeds and encoding.. 35 HD live feeds seems like an awfull lot to me to run even on 8 threads. but i dunno, i never tried anything like it but that would be my theory.. 

[LasleyMediaKentucky]

3 minutes ago, tlink said:

why did you go with an x99 platform and an i7 6800k cpu? why not a 12 core xeon or something? Ispy strikes me as the kind of program that uses multiple cores efficiently. you're trying to hook up A LOT of live feeds and encoding.. 35 HD live feeds seems like an awfull lot to me to run even on 8 threads. but i dunno, i never tried anything like it but that would be my theory.. 

Low budget and honestly, I thought for sure a 6800K with 6 cores and 12 threads would be enough. The 6700k we tried was a joke. 

[brwainer]

Clearly, if your CPU usage is 100%, you didn't spec out a fast enough CPU for this software, and either need a more powerful CPU ir more efficient software. As for how fast a CPU you need, I can't really help you there as I have no experience with iSpy. Are you doing 24/7 capturing (less CPU but lots of HDD activity) or is iSpy doing motion detection (crapton of CPU usage)? If you are doibg motion detection, is there a way to have the cameras do it instead? There are two ways this works, we use Axis and Cisco cameras where the camera notifies the server when there is motion so that the server records - the server is always watching all the video streams and is just waiting for the signal. The other way it works is that the camera internally captures the video clips then uploads them via FTP or another file protocol to the server - this is the absolute lowest method in terms of server CPU usage, but you don't inherently have a way to see all the video streams, and you have to be more diligent about monitoring cameras for outages.

 

Again, I have no experience with iSpy, but have you made sure there is no bottleneck on your hard drives? If you hard drives are more than 50% active, you can end up with frames having to be stored in memory here and there. It's the same concept as how packet loss increases exponentially as traffic leaving a port on a switch tends towards 100%, because random timing overloads the queues.

 

Another thought, if your cameras are currently streaming using MP4, which is relatively CPU intensive, try swapping the stream to MJPEG or MP2 or whatever else it supports. That will increase your network activity, but should lower your CPU usage.

 

EDIT: BTW, all of our surveillance contracts say that we provide a best effort service, and we aren't liable if a particular recording is unavailable. Cameras crash and need to be rebooted sometimes, and you wouldn't believe how many times the property wants a recording during a camera outage that maybe lasted 3-4 hours (the time between when it went down and when we power cycled it). We can show them that they have thousands of hours of recordings (all of which are motion detection, not empty time), just not the specific one they want, because the universe is cruel. If a property is concerned about not missing things, like the room with their safe or the computer lab, etc, we just double up on the cameras for that spot.

[Falconevo]

iSpy is awful and has some serious flaws in the software regarding efficiency of CPU utilisation.  What features of iSpy do you need to keep?  May be able to suggest more efficient alternatives.

[beavo451]

iSpy is your main problem.

 

Blue Iris is pretty cheap and has a trial period if you want to give that a shot. Milestone has free trial too but I'm pretty sure it'll be a lot more expensive than Blue Iris for the number of cameras.

[LasleyMediaKentucky]

On 12/24/2016 at 8:08 PM, beavo451 said:

iSpy is your main problem.

 

Blue Iris is pretty cheap and has a trial period if you want to give that a shot. Milestone has free trial too but I'm pretty sure it'll be a lot more expensive than Blue Iris for the number of cameras.

Blue Iris is amazing. I tried all of our cameras on the 6700K, and even though it keeps the CPU at 100%, all streams are fluent at 5fps and NO screen tearing or blur! But, just to be safe, I am using the 6800k machine. All cameras are on motion detection with WatchDog set up... It's amazing! I should have tried it sooner.

 

Also, Milestone is BS right now. I was told I'd be contacted by the territorial manager within a couple days and I never heard a word for 3 weeks. Plus I've spent too much time setting blue Iris up. 

[LasleyMediaKentucky]

On 12/24/2016 at 9:23 AM, Falconevo said:

iSpy is awful and has some serious flaws in the software regarding efficiency of CPU utilisation.  What features of iSpy do you need to keep?  May be able to suggest more efficient alternatives.

Blue Iris turned out to be AMAZING!

[LasleyMediaKentucky]

On 12/24/2016 at 0:28 AM, brwainer said:

Clearly, if your CPU usage is 100%, you didn't spec out a fast enough CPU for this software, and either need a more powerful CPU ir more efficient software. As for how fast a CPU you need, I can't really help you there as I have no experience with iSpy. Are you doing 24/7 capturing (less CPU but lots of HDD activity) or is iSpy doing motion detection (crapton of CPU usage)? If you are doibg motion detection, is there a way to have the cameras do it instead? There are two ways this works, we use Axis and Cisco cameras where the camera notifies the server when there is motion so that the server records - the server is always watching all the video streams and is just waiting for the signal. The other way it works is that the camera internally captures the video clips then uploads them via FTP or another file protocol to the server - this is the absolute lowest method in terms of server CPU usage, but you don't inherently have a way to see all the video streams, and you have to be more diligent about monitoring cameras for outages.

 

Again, I have no experience with iSpy, but have you made sure there is no bottleneck on your hard drives? If you hard drives are more than 50% active, you can end up with frames having to be stored in memory here and there. It's the same concept as how packet loss increases exponentially as traffic leaving a port on a switch tends towards 100%, because random timing overloads the queues.

 

Another thought, if your cameras are currently streaming using MP4, which is relatively CPU intensive, try swapping the stream to MJPEG or MP2 or whatever else it supports. That will increase your network activity, but should lower your CPU usage.

 

EDIT: BTW, all of our surveillance contracts say that we provide a best effort service, and we aren't liable if a particular recording is unavailable. Cameras crash and need to be rebooted sometimes, and you wouldn't believe how many times the property wants a recording during a camera outage that maybe lasted 3-4 hours (the time between when it went down and when we power cycled it). We can show them that they have thousands of hours of recordings (all of which are motion detection, not empty time), just not the specific one they want, because the universe is cruel. If a property is concerned about not missing things, like the room with their safe or the computer lab, etc, we just double up on the cameras for that spot.

Thank you for the info for future reference! I didn't know cameras did that. Ours are on the cheap but have great image quality - HOSAFE 2MD4P indoors and Reo-Link RLC-410 outdoors.

 

ALSO... Found out the servers and cameras were on two different IP ranges... cams were on 192.168.137.xxx and servers were on 192.168.254.xxx lol. All are working now!