DJI Europe looking for a Server

[Ragsan]

Hey LTT Team, Hey Community,

 

My name is Fred, and I work at the DJI Europe Studio. 

 

We have a Problem and looking for a Solution.

 

Our current Situation: 

 

2x new iMac Pro's (the best ones we could get)

2x Dustbins (Mac Pros) 2,7 GHz 12-Core Intel Xeon E5, 64 GB 1866 MHz DDR3 ECC, AMD FirePro D700 6144 MB

Software: Adobe Premiere CC, Davinci Resolve

Several Pegasus 2 R6 Raids 

 

Currently everyone has their own Raid where they're working on. We wanna change that, we're looking for a Server where we can all work on at the same time. Easy and Fast.

 

Our first attempt:

We had a company that installed us a Facilis Terablock Server which was controlled by a Windows Computer and connected to our Macs via 10Gbit Ethernet (and a Thunderbolt Adapter for the Dustbins).

The Read&Write Speed is pretty good, the Problem is we cant playback ProRes422 Footage in Realtime without dropping Frames. The company said this is a Problem of a Mac running Premiere beeing connected to a Windows Server.

 

So our Question:

Is there a Solution? I mean it has to be possible beeing able to edit on 4 Mac Clients on a Server with the native Footage (ProRes422/4444,H264 and so on) or am I wrong? 

 

Our conditions:

-4 Mac Clients working at the same time

-Using the native ProRes422/4444 Footage (sometimes h264 or cDNG)

-Simple to use (like our Pegasus Raids for example)

 

Whats for example with the "Storinator" from 45Drives which Linus installed at several Youtube Studios?

The Problem is, I think they all used FinalCut on a Mac and not Adobe Premiere.

 

Please Help....

Fred

[Lord Nicoll]

16 minutes ago, DJI-Fred said:

Hey LTT Team, Hey Community,

 

My name is Fred, and I work at the DJI Europe Studio. 

 

We have a Problem and looking for a Solution.

 

Our current Situation: 

 

2x new iMac Pro's (the best ones we could get)

2x Dustbins (Mac Pros) 2,7 GHz 12-Core Intel Xeon E5, 64 GB 1866 MHz DDR3 ECC, AMD FirePro D700 6144 MB

Software: Adobe Premiere CC, Davinci Resolve

Several Pegasus 2 R6 Raids 

 

Currently everyone has their own Raid where they're working on. We wanna change that, we're looking for a Server where we can all work on at the same time. Easy and Fast.

 

Our first attempt:

We had a company that installed us a Facilis Terablock Server which was controlled by a Windows Computer and connected to our Macs via 10Gbit Ethernet (and a Thunderbolt Adapter for the Dustbins).

The Read&Write Speed is pretty good, the Problem is we cant playback ProRes422 Footage in Realtime without dropping Frames. The company said this is a Problem of a Mac running Premiere beeing connected to a Windows Server.

 

So our Question:

Is there a Solution? I mean it has to be possible beeing able to edit on 4 Mac Clients on a Server with the native Footage (ProRes422/4444,H264 and so on) or am I wrong? 

 

Our conditions:

-4 Mac Clients working at the same time

-Using the native ProRes422/4444 Footage (sometimes h264 or cDNG)

-Simple to use (like our Pegasus Raids for example)

 

Whats for example with the "Storinator" from 45Drives which Linus installed at several Youtube Studios?

The Problem is, I think they all used FinalCut on a Mac and not Adobe Premiere.

 

Please Help....

Fred

Given your business, you're gonna want a SANs not a NAS, a SANs makes it's RAID arrays available to clients instead of just the files. They appear as if they're a drive in your computer, a networked drive if you will, but multiple people can access it.  In terms of costs, it can be quite expensive to do it right with brand new equipment, you'll also want a 10GBbps network (as in the entire network, not just host and client, I seem some installations where they use 1Gbps switches on 10Gbps ports) for best results on large files. Can you tell me a bit more about the space required, existing network and any other needs?

I'm also not sure how Mac's respond to SANs, but if it's like Linux it should be fine.

[AntiTrust]

Im sure if you flew Linus out and let him shoot a video he would solve your problem... just saying... gl tho

[Lord Nicoll]

Thinking more about it, Linux would probably work a lot nicer than a Windowes hosts with Mac Clients. 

[Ragsan]

6 minutes ago, Lord Nicoll said:

Given your business, you're gonna want a SANs not a NAS, a SANs makes it's RAID arrays available to clients instead of just the files. They appear as if they're a drive in your computer, a networked drive if you will, but multiple people can access it.  In terms of costs, it can be quite expensive to do it right with brand new equipment, you'll also want a 10GBbps network for best results on large files. Can you tell me a bit more about the space required, existing network and any other needs?

I'm also not sure how Mac's respond to SANs, but if it's like Linux it should be fine.

Yeah the Terablock System von Facilis we can also set to different "modes" so it will show up as a normal Harddrive and Stuff like that, but I guess thats not what you're talking about right? 

 

Costs are only a secondary concern, the Terablock Solution was like 90k so also not cheap. 

 

We have an existing Network here and a cooled Serverroom, and as far as I know, the Network was fast enough, so the general Speed of the Network and Server is not the Issue at the Moment, its a wierd Situation.

 

Space would be something like 200TB. Best would be Raid5 (that was Raid5 right?) or something like that so we have an internal backup). Other than that we dont have any other needs. It just have to be fast, easy to work with (without mounting and unmounting virtual drives for different Projects) and we have to be able to playback our native Footage (at least ProRes and H264).

 

Thanks for your help

[Ragsan]

7 minutes ago, AntiTrust said:

Im sure if you flew Linus out and let him shoot a video he would solve your problem... just saying... gl tho

 let him fly all the way from Canada to Germany just to let him install a Server and let him film our "messy" studio?

Sounds like a good Idea! And I would even ask him/them, but I dont know how and where. (also I dont know if we are interessting enough for that

[Lord Nicoll]

13 minutes ago, DJI-Fred said:

Yeah the Terablock System von Facilis we can also set to different "modes" so it will show up as a normal Harddrive and Stuff like that, but I guess thats not what you're talking about right? 

 

Costs are only a secondary concern, the Terablock Solution was like 90k so also not cheap. 

 

We have an existing Network here and a cooled Serverroom, and as far as I know, the Network was fast enough, so the general Speed of the Network and Server is not the Issue at the Moment, its a wierd Situation.

 

Space would be something like 200TB. Best would be Raid5 (that was Raid5 right?) or something like that so we have an internal backup). Other than that we dont have any other needs. It just have to be fast, easy to work with (without mounting and unmounting virtual drives for different Projects) and we have to be able to playback our native Footage (at least ProRes and H264).

 

Thanks for your help

That sounds like a SANs with extra flexibility, honestly since you have the hardware obviously, maybe get your system admin to reconfigure the Windows systems with either Linux or BSD, Mac is largely based on BSD so I'm like 70% sure it would work good. RAID 5 (not 6, oops) is depreciated largely, RAID 6 and RAID 60 are much better options, but require more drives. Not that cost if your worry as liekly you have enough drives and might only need an extra few if even, the RAID controller might need upgrade though if they don't support RAID 6. I have no idea why they went with Windows for a Mac environment, that usually causes issues. This largely isn't my best domain for advice, while I do personally own a few servers with a network SANs and webpages, I took me a long time and wasn't the easiest for me. 

[Lord Nicoll]

This is RAID 60 with double parity. I'm not 100% sure if that's ideal, maybe if someone else who uses RAID more could chime in? 

[Jay Deah]

modern SANs use a concept called distributed RAID anyway, so your raid level only really means the level of protection you have. ie Raid5 1 drive failure Raid6 2 drive failures. the way it distributes the extents across drives is it effectively stripes it across every disk in the array, so its great for performance.

 

anyway, on the SAN from you'll probably better off with a UNIFIED device, which is a SAN that does both file level and block level storage. a normal SAN will only do block level so you still need a client like a windows server to act as the file server for cients to connect to unless you want raw block drives non-shared for each user (you wont).

 

first thing's first... what is your budget

 

second thing's second, what are your requirements for performance? do you know the bitrate of the videos you'll want to stream?

 

with modern protocols like SMB3 and decent networking (even if its 1gbe) the normal "bottleneck" is the mechanical drives themselves

[leadeater]

16 hours ago, DJI-Fred said:

We had a company that installed us a Facilis Terablock Server which was controlled by a Windows Computer and connected to our Macs via 10Gbit Ethernet (and a Thunderbolt Adapter for the Dustbins).

The Read&Write Speed is pretty good, the Problem is we cant playback ProRes422 Footage in Realtime without dropping Frames. The company said this is a Problem of a Mac running Premiere beeing connected to a Windows Server.

That's not a good enough response from the company. Did you engage with them or an official partner during purchase? If so they have an obligation to work through the issue and get a working solution going because they would have been contracted to sell you a working solution not a half working one.

 

You'll also need to do a bit of analysis of why you are getting dropped frames, find the break point. On the Windows server use performance monitor to gather data over the day (log to file) and collect metrics like CPU, RAM, volume disks latency, volume disk queue, each disk latency (avg disk sec/Read & avg disk sec/Write)

 

Can you tell me any more information about the server itself? Is it using a hardware RAID card or Windows Storage Spaces?

 

Do you get dropped frames when only a single person is utilizing the server?

 

How are you mounting the storage on the Macs? SMB share, NFS or iSCSI?

[leadeater]

15 hours ago, DJI-Fred said:

 let him fly all the way from Canada to Germany just to let him install a Server and let him film our "messy" studio?

Sounds like a good Idea! And I would even ask him/them, but I dont know how and where. (also I dont know if we are interessting enough for that

@LinusTech got your superman cape ready? Or just some email help .

 

@DJI-Fred Hopefully the above will net some help. You could also just email him.

[leadeater]

Quote

Facilis systems are not NAS, although we have the simplicity of connection common to network storage. Our shared file system is unique in the market, delivering video and audio data over Fibre Channel and Ethernet connections, through a custom protocol that avoids the overhead and latencies of network storage systems. Developed in 2009, the Shared File System is compatible with all major operating systems at the same time, on the same volume and with the same permissions structure. When avoiding the complexities of OS-based network file systems, our method of data sharing is simple, and interoperable with more applications than any NAS-based system. It's in use at thousands of facilities worldwide, and helping to produce some of the most compelling content in the business.

https://www.facilis.com/products-shared-storage/

 

Well going by that sales pitch it shouldn't even matter what the OS's are.

[Ragsan]

8 hours ago, leadeater said:

That's not a good enough response from the company. Did you engage with them or an official partner during purchase? If so they have an obligation to work through the issue and get a working solution going because they would have been contracted to sell you a working solution not a half working one.

Yes the Company did a lot of testing we even gave Facilis a Remote Access to the System to check everything but nobody could find the real Issue. Performance (Copy/Write Speed) was very good, but the ProRes Files were constantly dropping frames after a few seconds of playback (felt a bit like a Cache Overflow or something) it worked for a few seconds but then the playback completely stucked. 

The IT Company told us that its probably a Problem with Premiere (on a OSX System) trying to work on a Windows Server and they offered us a full refund.

 

19 hours ago, Jay Deah said:

first thing's first... what is your budget

We dont really have a budget, the System (that doesnt work) was about 90k. But as long as it works flawless and is future proof it doesnt matter.

 

19 hours ago, Jay Deah said:

what are your requirements for performance? do you know the bitrate of the videos you'll want to stream?

500-1000Mbit/s (but the pure Data Performance doesnt seem to be a Problem) 

We want to be able to have 4 Systems to edit  at the same with the native ProRes Files 

8 hours ago, leadeater said:

Can you tell me any more information about the server itself? Is it using a hardware RAID card or Windows Storage Spaces?

See attached Pictures  (i kow a little bit about Server Stuff, but not that much sorry^^)

 

8 hours ago, leadeater said:

 

Do you get dropped frames when only a single person is utilizing the server?

Yes it doesnt matter if only one person is editing or 4 and copying files at the same time. Like is said, the pure Datarate Perfomance was good.

 

8 hours ago, leadeater said:

How are you mounting the storage on the Macs? SMB share, NFS or iSCSI?

Sorry I dont know (I dont even know the difference between those modes) I only know that we are connected via Ethernet to that 10G Switch to the Server, every Client has its own IP Adress (we have 2 different IP Groups 192.168.0.xxx and 192.168.1.xxx) and we're using the Facilis Client Console to connect (mount/unmount) the different Volumes on the Server.

 

@LinusTech Please Halp!  

[leadeater]

3 minutes ago, DJI-Fred said:

Yes the Company did a lot of testing we even gave Facilis a Remote Access to the System to check everything but nobody could find the real Issue. Performance (Copy/Write Speed) was very good, but the ProRes Files were constantly dropping frames after a few seconds of playback (felt a bit like a Cache Overflow or something) it worked for a few seconds but then the playback completely stucked. 

The IT Company told us that its probably a Problem with Premiere (on a OSX System) trying to work on a Windows Server and they offered us a full refund.

Thanks for the extra info, don't know if you have the time to try it but is it possible to confirm that it's an OS issue by installing a trial copy of Windows and prove it's working in that scenario.

 

5 minutes ago, DJI-Fred said:

See attached Pictures  (i kow a little bit about Server Stuff, but not that much sorry^^)

Doesn't quite have the information I was after but I'm very happy to see you included in your purchase a backup system.

 

7 minutes ago, DJI-Fred said:

Yes it doesnt matter if only one person is editing or 4 and copying files at the same time. Like is said, the pure Datarate Perfomance was good.

Sounds a bit like network card issue or configuration, increase the transmit and receive buffers to the maximum possible on the server and clients.

 

That's the hard thing about network based storage, it can be very difficult to diagnose problems as there are so many points where the problem could be or a combination of.

[Ragsan]

2 minutes ago, leadeater said:

Thanks for the extra info, don't know if you have the time to try it but is it possible to confirm that it's an OS issue by installing a trial copy of Windows and prove it's working in that scenario.

No, I have to thank YOU for the Work you're doing here  Nothing to thank me about. Cant install Windows on the MacPro I'm working on but we have a Macbook Pro with a Windows Bootcamp (or Virtual Windows) on it. Might try that one out, good Idea. 

 

4 minutes ago, leadeater said:

That's the hard thing about network based storage, it can be very difficult to diagnose problems as there are so many points where the problem could be or a combination of.

Yeah our small Raids are definitely easier to handle, plug it in and its working

 

That is what our Studio in LA is doing:

Quote

1. We use an old mac pro tower which we have configured as a server.
2. Our storage is comprised of 16 bay MAXX DIGITAL RAID arrays set up in raid-6. (we currently have 2 of these arrays 84tb a piece)
3. We use a 10g switch and heavy copper CAT6A cable from the switch to our workstations. ****I cannot stress the importance of NOT using fiber*** Though we have the ability to run fiber in our system we choose to run copper. The ONLY reason to ever run fiber is if your server is over 300 feet from your workstation. If you run fiber, you better have a full time IT guy. Trust me.
4. Our workstations are Mac Pro Trashcans mounted in Sonnet Xmac Units. The Sonnet enclosure allows us to employ ATTO 10G network cards by adding PIC slots to our Macs. (Nowadays you could just work off of an iMac Pro, which has industry leading 10g connectivity built in)

 

So as far as I know now, we should exchange the Server we currently have (the company offered us a full refund) and change it to something Linux or BSD based right? 

[leadeater]

23 minutes ago, DJI-Fred said:

So as far as I know now, we should exchange the Server we currently have (the company offered us a full refund) and change it to something Linux or BSD based right? 

If it comes to that then I guess you have no choice. I always tried to keep configurations standard across customers so if there is a known working configuration in the company then I would align as closely to that as possible.

 

I forget which storage server our media design department went with, I'll try and dig out the quotes so see who it was. We were looking at ProMAX Systems but I don't think that was the final choice.

[leadeater]

@DJI-Fred

Our media design department went with two SNS EVO storage systems, one for video storage and one for audio. I don't know too much about how they are using it but it's supporting at least one lab of 30 iMacs, we're a University btw and they are in another city than I'm in. Being education they'll be using multiple different applications but I'm sure they use Premiere, we are licensed for it site wide for the whole Adobe Suite.

 

http://www.studionetworksolutions.com/evo/

[Ragsan]

31 minutes ago, leadeater said:

f it comes to that then I guess you have no choice. I always tried to keep configurations standard across customers so if there is a known working configuration in the company then I would align as closely to that as possible.

They said they unfortunately don't have a solution right now, so we're thinking about switching the company or we try to get informations ourself so that the IT Company can inform themselves about that system. 

[leadeater]

2 minutes ago, DJI-Fred said:

They said they unfortunately don't have a solution right now, so we're thinking about switching the company or we try to get informations ourself so that the IT Company can inform themselves about that system. 

I meant something that works within DJI currently, copying something that you know works and you can speak to people using it is a good way to ensure you get something you know works.

[Ragsan]

5 minutes ago, leadeater said:

I meant something that works within DJI currently, copying something that you know works and you can speak to people using it is a good way to ensure you get something you know works.

Oh yeah sure, but you cant really compare our Europe Studio to the one in LA, they have an extra Person for Data Management and IT Stuff, his only Job is to keep their system running and to transcode the incoming footage. But it gives us a good Idea in what direction we could go.

[Jay Deah]

I would definitely return that and look into a decent storage solution from a tier 1 vendor. All the big boys offer unified storage SANs that will easily meet your requirements and be <100k.

 

the kind of performance you’re talking about I could get out of my IBM v7000 at near enough idle! and that was well under 100k that kit looks like a company charging big bucks for sh*t to people who don’t know better.

[leadeater]

23 minutes ago, Jay Deah said:

I would definitely return that and look into a decent storage solution from a tier 1 vendor. All the big boys offer unified storage SANs that will easily meet your requirements and be <100k.

 

the kind of performance you’re talking about I could get out of my IBM v7000 at near enough idle! and that was well under 100k that kit looks like a company charging big bucks for sh*t to people who don’t know better.

I would be careful with assumptions like that when it comes to video editing, most of that software is not well written for network hosted storage and odd things can crop up with little reason as to why.

 

In the mentioned campus before our main storage there is a Netapp FAS2552 dual controller and had similar issues as being described here, personally I was 100% sure it was just not enough HDDs for the required performance load but it was easier, safer and cheaper to offload that work to a dedicated storage company that deals with video production with written performance guarantees in actual video editing programs.

 

Our larger campuses have 6 8040 controller heads and a ton more HDDs and SSDs but that wasn't where the clients were and there's no way video editing was going to work on storage across different cities . Would have liked to test it though on the bigger storage arrays, not that I really needed to since I know Weta Digital use Netapp and they have far far higher performance requirements.

[lacion]

if money is not a big problem here, I can recommend an SNS EVO i used to work in a big studio and they work with all sort of assets on the VFX/Editing side they work with 4k/8k with different tools and they had no issue using their solution.

 

they have solutions up to  50Gb Ethernet.

 

the team i used to work with used to deliver things for netflix The EVO DPX system was able to handle real-time playback of uncompressed 4K media and finishing in 16-bit EXR/DPX from a single shared storage system without using Fibre Channel.

 

this was people colour grading in Resolve, I'm guessing for workflows in premiere it will a lot simpler and cheaper as you don't actually require the fully uncompress sequences.

[Ragsan]

1 hour ago, lacion said:

if money is not a big problem here, I can recommend an SNS EVO i used to work in a big studio and they work with all sort of assets on the VFX/Editing side they work with 4k/8k with different tools and they had no issue using their solution.

 

What OS where they using? Windows or Mac? 

[lacion]

7 minutes ago, DJI-Fred said:

What OS where they using? Windows or Mac? 

it depended on the part of the process, some of the editors were on mac, some of the VFX team was on Linux, the resolve users were on windows. SNS mention their solution is totally OS independent. 

 

the nice thing about them is that their support is great and if you don't have anyone inhouse to deal with the SAN they will handle it for you.