Jump to content

Help! Storage workflow Nightmare, Cloud backup

Ecto

Hello everyone, I am new here in this forum, I am looking for some guidance, to be pointed into the right direction in what I can do in the present and the future.

I am by no means an expert but I understand what it will take to get to a point of fast workflow

 

I work with a team of Video and Photo editors who are tired of the current workflow.

We have large data demands that 120/mbs External hard drives are just killing time and workflow. 

I would like to implement a process in the future for Editing over a networking using our existing NAZ and LACIE drives, or upgrading to something like EVO System.

 

Right now we have an ingest area, which is backed up to back-blaze b2.

We pull data from the ingest area via external usb, upload or edit from external on our editing stations.

Moving from editing station via external usb to ingest area finally to upload new files.

It's slow, time consuming and we need a better solution for our workflow.

 

If there is an option to Edit a Video From Home and updating the main Project file over the web, that would be amazing as well! 

 

We currently have 80TB of local storage via LACIE 

and other NAZ systems that are sitting around the office that are not being used do to issues i need to dig into.

 

We have MAC OS and Windows users 

Setting up a network to edit from would be Ideal

Budget is negotiable 

 

Any help would be greatly appreciated!

 

Link to comment
Share on other sites

Link to post
Share on other sites

Well right off the bat look into 10Gbit networking. I don't know if your MACs will be able to utilize that but the PCs could have a 10Gbit NIC installed and from there a SFP+ switch can be bought new for ~$500 or you can finds RJ-45 10Gbit retired enterprise switches on eBay for about half that but the RJ-45 NICs for the PCs generally cost more.

 

This would help a lot depending on how big the drive pool is in the server. You could also look into Link Aggregation so multiple people can hit the server at once and have a better experience. You can try adding a cache to the pool be it a SATA SSD or PCI_e SSD. All of this would make a world of a difference.

 

I can't give a recommendation on your other current procedures though.

Link to comment
Share on other sites

Link to post
Share on other sites

Our media studies departments use ProMax NAS's which seems to do the job well at a reasonable price, https://www.promax.com/. Their primary business is video production and media creation and they have detailed specs on how many streams each device can support per media format and bit rate etc.

 

I think there is around 5 of those NAS's now around 100+ TB each, we don't have much to do with them but I know they are happy with them and do what they need. They have around 30-60 Mac Pros editing off them.

 

QNAP are also highly rated for this as well, stick to their higher end offerings like TVS-1282T3.

Link to comment
Share on other sites

Link to post
Share on other sites

1 minute ago, Windows7ge said:

Well right off the bat look into 10Gbit networking. I don't know if your MACs will be able to utilize that but the PCs could have a 10Gbit NIC installed and from there a SFP+ switch can be bought new for ~$500 or you can finds RJ-45 10Gbit retired enterprise switches on eBay for about half that but the RJ-45 NICs for the PCs generally cost more.

 

This would help a lot depending on how big the drive pool is in the server. You could also look into Link Aggregation so multiple people can hit the server at once and have a better experience. You can try adding a cache to the pool be it a SATA SSD or PCI_e SSD. All of this would make a world of a difference.

 

I can't give a recommendation on your other current procedures though.

I am looking into 10gbit networking for our new office location which is within 8 months from now. Thank you!

All of our iMacs are new and have thunderblot 3 and we iMac Pro users as well. I have plans to equip all iMacs that need it. 

Windows Editing will need a 10Gbit network update agreed.

 

No need for recommendations on current procedures, just future solutions to get away from the slow workflow we are possessed with now. 

 

Link to comment
Share on other sites

Link to post
Share on other sites

8 minutes ago, Ecto said:

I am looking into 10gbit networking for our new office location which is within 8 months from now. Thank you!

All of our iMacs are new and have thunderblot 3 and we iMac Pro users as well. I have plans to equip all iMacs that need it. 

Windows Editing will need a 10Gbit network update agreed.

 

No need for recommendations on current procedures, just future solutions to get away from the slow workflow we are possessed with now. 

 

I do know there are TB to 10Gbit copper adapters. One was featured on LTT. If you went the route of 10Gbit copper ethernet Cat6a cable would be most desirable. Shielded if you deal with a lot of power cables mixed up in your Ethernet. SFP+ adapters can be had pretty cheaply. software wise they behave identically to Ethernet they're just drastically different physically.

 

I can offer some hardware suggestions if you're interested. It'd be a mix of hardware I use personally and some hardware I want to try buying myself.

Link to comment
Share on other sites

Link to post
Share on other sites

10 minutes ago, leadeater said:

Our media studies departments use ProMax NAS's which seems to do the job well at a reasonable pricehttps://www.promax.com/. Their primary business is video production and media creation and they have detailed specs on how many streams each device can support per media format and bit rate etc.

 

I think there is around 5 of those NAS's now around 100+ TB each, we don't have much to do with them but I know they are happy with them and do what they need. They have around 30-60 Mac Pros editing off them.

ProMax Looks amazing and I really like their interface thank you for sharing! 
We have already invested In storage solutions for in house and we want to keep everything centralized. 

As our team continues to grow, in the future we may be looking into ProMax or other solutions like them. 
For now we would like to focus on Local Storage and media sharing solutions. 

 

Thank you ! 

Link to comment
Share on other sites

Link to post
Share on other sites

2 minutes ago, Windows7ge said:

I do know there are TB to 10Gbit copper adapters. One was featured on LTT. If you went the route of 10Gbit copper ethernet Cat6a cable would be most desirable. Shielded if you deal with a lot of power cables mixed up in your Ethernet. SFP+ adapters can be had pretty cheaply. software wise they behave identically to Ethernet they're just drastically different physically.

 

I can offer some hardware suggestions if you're interested. It'd be a mix of hardware I use personally and some hardware I want to try buying myself.

Yes indeed I am interested in hearing hardware suggestions please ! 

We have network switch,  I need to identify if it is capable of 10Gbit or if we need to upgrade as well. 

Link to comment
Share on other sites

Link to post
Share on other sites

Just now, Ecto said:

Yes indeed I am interested in hearing hardware suggestions please ! 

We have network switch,  I need to identify if it is capable of 10Gbit or if we need to upgrade as well. 

Well as I mentioned you can look into Cat6a UTP (Unshielded Twisted Pair) or the STP (Shielded Twisted Pair) if it's worth it will depend on how long the cable runs need to be and how electrically noising the environment is.

 

If you want to use Ethernet you can look into the Intel X540-1T or 2T. They're quite popular and work pretty well. If you want to go SFP+ most of my NICs are used or kind of bizzare brands so I can't confidently recommend them but for the server if you're open to setting up Link Aggregation you can consider something like this. If you don't need 40Gbit now it'd be something the company could grow into. You could also opt for two 2-port cards which would allow for fault tolerance.

 

Transceiver wise I use these.

If you wanted to convert a SFP+ port to 10Gbit Ethernet: adapter

For fiberoptic cable take these with a grain of salt. They work for me but may not for you (depending on cable distance) also their outer jacketing doesn't comply with my local fire-code standards (OFNR) you would probably want (if not require) LSZH.

 

This would be a solid 10Gbit SFP+ switch: Ubiquiti Networks ES-16-XG-US understand though that you'd only want to use this for switching. For routing it might start to suffer.

Link to comment
Share on other sites

Link to post
Share on other sites

The network switch and network cards are not going to help fix 120mb/s - 1gb network can handle 110-120mb/s already. You need to address the storage before the network.

 

If the point of the ingest station is for it to simply upload to backblaze, I would just install the backblaze agent at every workstation and skip the ingest process. Doubling the work for no real benefit.

 

Of the 80TB how much is actually used? It will help plan a storage solution if you plan on having a proper NAS vs a bunch of external drives.

Link to comment
Share on other sites

Link to post
Share on other sites

19 hours ago, Windows7ge said:

Well as I mentioned you can look into Cat6a UTP (Unshielded Twisted Pair) or the STP (Shielded Twisted Pair) if it's worth it will depend on how long the cable runs need to be and how electrically noising the environment is.

 

If you want to use Ethernet you can look into the Intel X540-1T or 2T. They're quite popular and work pretty well. If you want to go SFP+ most of my NICs are used or kind of bizzare brands so I can't confidently recommend them but for the server if you're open to setting up Link Aggregation you can consider something like this. If you don't need 40Gbit now it'd be something the company could grow into. You could also opt for two 2-port cards which would allow for fault tolerance.

 

Transceiver wise I use these.

If you wanted to convert a SFP+ port to 10Gbit Ethernet: adapter

For fiberoptic cable take these with a grain of salt. They work for me but may not for you (depending on cable distance) also their outer jacketing doesn't comply with my local fire-code standards (OFNR) you would probably want (if not require) LSZH.

 

This would be a solid 10Gbit SFP+ switch: Ubiquiti Networks ES-16-XG-US understand though that you'd only want to use this for switching. For routing it might start to suffer.

All of this recommended hardware looks great! 

Thank you for sharing, I have some research to do for sure on everything :D

Link to comment
Share on other sites

Link to post
Share on other sites

On 6/6/2019 at 1:46 PM, Mikensan said:

The network switch and network cards are not going to help fix 120mb/s - 1gb network can handle 110-120mb/s already. You need to address the storage before the network.

 

If the point of the ingest station is for it to simply upload to backblaze, I would just install the backblaze agent at every workstation and skip the ingest process. Doubling the work for no real benefit.

 

Of the 80TB how much is actually used? It will help plan a storage solution if you plan on having a proper NAS vs a bunch of external drives.

Everyone is working with all kinds of cards from micro SD right up to C-Fast even RedMag time to time. Therefore to keep down on costs and improve efficiency

We only need One point of ingest, helps organize everything and then we don't have to buy extra Media readers for each work station!

 

We Ingest media from that day, sort it on the LACIE, and then workers grab the data once assigned a project from the LACIE to a external.

The point of the Ingest in the future is to have a dedicated spot to unload all data and have it sent sent to a NAS while a copy is made local on the Ingest, then working from NAS over a network.

 

Before I came along, data migration and backup was happening. Data copied from Nas with an external, unloaded to a 6 bay LACIE. This process is time consuming and not necessary what so ever. I have discovered the Ingest station is hooked up to a LACIE 80Tb via firewire. 42Tb are being used currently on the LACIE and keeps adding up.

Old Archived files plus new files are being sorted on the LACIE then uploaded to the cloud. (This should only be uploading from the LACIE locally from the ingest once new files are added) I want to step away from doing that all together and just go back to using the NAS because it it's accessed via network cable, I don't think having 4 video editors accessing a LACIE over the network that's connected via thunderbolt to a mac will keep up. 

 

I will get back to you all with what we have for hardware, when I step in the office so I can paint a bigger picture of what we really have! 

 

I appreciate everyone's help here. Please be patient as I am new to figuring out how to setup a NAS and Configuration for Editing over a network. 

Link to comment
Share on other sites

Link to post
Share on other sites

4 hours ago, Ecto said:

Everyone is working with all kinds of cards from micro SD right up to C-Fast even RedMag time to time. Therefore to keep down on costs and improve efficiency

We only need One point of ingest, helps organize everything and then we don't have to buy extra Media readers for each work station!

 

We Ingest media from that day, sort it on the LACIE, and then workers grab the data once assigned a project from the LACIE to a external.

The point of the Ingest in the future is to have a dedicated spot to unload all data and have it sent sent to a NAZ while a copy is made local on the Ingest, then working from NAZ over a network.

 

Before I came along, data migration and backup was happening. Data copied from Naz with an external, unloaded to a 6 bay LACIE. This process is time consuming and not necessary what so ever. I have discovered the Ingest station is hooked up to a LACIE 80Tb via firewire. 42Tb are being used currently on the LACIE and keeps adding up.

Old Archived files plus new files are being sorted on the LACIE then uploaded to the cloud. (This should only be uploading from the LACIE locally from the ingest once new files are added) I want to step away from doing that all together and just go back to using the NAZ because it it's accessed via network cable, I don't think having 4 video editors accessing a LACIE over the network that's connected via thunderbolt to a mac will keep up. 

 

I will get back to you all with what we have for hardware, when I step in the office so I can paint a bigger picture of what we really have! 

 

I appreciate everyones help here. Please be patient as I am new to figuring out how to setup a NAZ and Configuration for Editing over a network. 

Do you have a model for this LACIE? I've only heard of their external drives, did not know they offer NAS solutions.

Link to comment
Share on other sites

Link to post
Share on other sites

On 6/7/2019 at 1:30 PM, Mikensan said:

Do you have a model for this LACIE? I've only heard of their external drives, did not know they offer NAS solutions.

LACIIE model is STFK60000400. LACIE does not support NAS 

 

We have a QNAP TVS-872XT 8-Bay NAS Enclosure, it's what I want to work towards editing straight from.

Link to comment
Share on other sites

Link to post
Share on other sites

On 6/7/2019 at 1:30 PM, Mikensan said:

Do you have a model for this LACIE? I've only heard of their external drives, did not know they offer NAS solutions.

Looks like it’s a desktop Thunderbolt DAS. 

 

1 hour ago, Ecto said:

LACIIE model is STFK60000400. LACIE does not support NAS 

 

We have a QNAP TVS-872XT 8-Bay NAS Enclosure, it's what I want to work towards editing straight from.

Is the LACIE connected to a Computer?

For Sale: Meraki Bundle

 

iPhone Xr 128 GB Product Red - HP Spectre x360 13" (i5 - 8 GB RAM - 256 GB SSD) - HP ZBook 15v G5 15" (i7-8850H - 16 GB RAM - 512 GB SSD - NVIDIA Quadro P600)

 

Link to comment
Share on other sites

Link to post
Share on other sites

On 6/8/2019 at 11:14 PM, Ecto said:

LACIIE model is STFK60000400. LACIE does not support NAS 

 

We have a QNAP TVS-872XT 8-Bay NAS Enclosure, it's what I want to work towards editing straight from.

Ah I see, since you already have the QNAP then you might as well use it. It appears to have thunderbolt support so it ~should~ be plug and play. Those NASs typically run Linux which should should handle NTFS / EXT just fine assuming your LACIE is in one of those formats.

 

If you didn't have the QNAP I would just advise to use the ingest station as a NAS.

 

 

Link to comment
Share on other sites

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×