1PB ssd San

[tt2468]

You heard correct. I am doing a semi-secret project that includes 1PB of ssd storage. I need to pick out a 1u server that has 2 pcie x16 slots. One for a 40Gb dual port qsfp+ adapter, and one for the secret sauce of the project. I was looking at using a Dell R430 with 64GB ecc ram and an e5-2620v3. Any thoughts? It should be able to handle 80+Gbps. Those are the only details I can share. (and no, this is not for the nsa.)

[Enderman]

are you sponsored by bill gates or something

[tt2468]

1 minute ago, Enderman said:

are you sponsored by bill gates or something

Lol no. I might be able to reveal more details later on in the project.

[themctipers]

1PB!?!??!
Are you sure thats not a typo..

[Guest]

Pick up the phone and get in contact with your local presales for HP, Dell, Lenovo or IBM. This is quite clearly for work and an expensive project - no offense but you'd be stupid to go start picking around with consumer grade parts that don't include onsite same day or next day replacement.

[tt2468]

1 minute ago, Windspeed36 said:

Pick up the phone and get in contact with your local presales for HP, Dell, Lenovo or IBM. This is quite clearly for work and an expensive project - no offense but you'd be stupid to go start picking around with consumer grade parts that don't include onsite same day or next day replacement.

We plan to produce tons of devices. I just want advice from here to try to remove some of my personal biases.

[TheKDub]

Damn, that's gonna be a lot of drives.

Samsung's 15TB SSD is the largest one in the world right now isn't it? if so, you're gonna need about 68 of em.

[tt2468]

1 minute ago, TheKDub said:

Damn, that's gonna be a lot of drives.

Samsung's 15TB SSD is the largest one in the world right now isn't it? if so, you're gonna need about 68 of em.

The company I work wit actually beat that record years ago. I just need to take care of the processing part of things.

[leadeater]

17 minutes ago, Kyle Manning said:

You heard correct. I am doing a semi-secret project that includes 1PB of ssd storage. I need to pick out a 1u server that has 2 pcie x16 slots. One for a 40Gb dual port qsfp+ adapter, and one for the secret sauce of the project. I was looking at using a Dell R430 with 64GB ecc ram and an e5-2620v3. Any thoughts? It should be able to handle 80+Gbps. Those are the only details I can share. (and no, this is not for the nsa.)

Hope the networking side of things is going to have RDMA support else E5-2620v3 CPU's aren't going to be enough to push that amount of throughput or that many IOPs.

 

The Commvault servers at work are all 2x E5-2667 64GB RAM with two dual port 10Gb NICs (4 ports total). We have 7 of these and push them very hard, and that's with client side dedup turned on. Granted Commvault usage profile is not that close to what you are going to be doing but hopefully useful information

 

A better example would be our Netapp 8060 filers. Each node has dual E5-2658 with 64GB RAM with 4 Nodes per C-Mode cluster and we have 3 of these.

[FirstArmada]

Huh.........

[FirstArmada]

32 minutes ago, Windspeed36 said:

Pick up the phone and get in contact with your local presales for HP, Dell, Lenovo or IBM. This is quite clearly for work and an expensive project - no offense but you'd be stupid to go start picking around with consumer grade parts that don't include onsite same day or next day replacement.

Says he was born in 2001 

 

Your fucking 15.......

 

i am at a loss for words

 

 

 

[tt2468]

2 minutes ago, FirstArmada said:

Says he was born in 2001 

 

Your fucking 15.......

 

i am at a loss for words

 

 

 

I dont sit around smoking weed...

[FirstArmada]

1 minute ago, Kyle Manning said:

I dont sit around smoking weed...

I see my facade was great i'm actually 14 and was never arrested

 

Clearly the dox info you got wasnt very reliable 

[ozziestig]

2 minutes ago, FirstArmada said:

Says he was born in 2001 

 

Your fucking 15.......

 

i am at a loss for words

 

 

 

Dude does anyone really put their age on the internet if they wish to be anonymous, hell I could be a wizard with a cat.

[FirstArmada]

Just now, ozziestig said:

Dude does anyone really put their age on the internet if they wish to be anonymous, hell I could be a wizard with a cat.

I know the kid i worked with him last year 

[tt2468]

1 minute ago, ozziestig said:

Dude does anyone really put their age on the internet if they wish to be anonymous, hell I could be a wizard with a cat.

Maybe I am a wizard with a cat... Anyway, this is getting off topic.

[brwainer]

I agree with the prior post that even if this is for R&D, picking out hardware based on a forum is silly. Definitely talk to some presales engineers

 

EDIT: I mean picking out hardware with this type of system requirements.

 

EDIT2: I am very interested in seeing more details about this project as you are allowed to share them.

[Blake]

2 hours ago, Kyle Manning said:

You heard correct. I am doing a semi-secret project that includes 1PB of ssd storage. I need to pick out a 1u server that has 2 pcie x16 slots. One for a 40Gb dual port qsfp+ adapter, and one for the secret sauce of the project. I was looking at using a Dell R430 with 64GB ecc ram and an e5-2620v3. Any thoughts? It should be able to handle 80+Gbps. Those are the only details I can share. (and no, this is not for the nsa.)

 

I read that as 1PB of SSD Storage in a 1U case.

As others have said assuming 15TB SSDs, you are not going to fit ~67 of them into a 1U rack mount.

 

Also, you should be looking at SANs, not NAS not DAS, not internal Storage.

HPE 3PAR StoreServ - Have used can confirm very good.

Dell Storage PS6610 Series Arrays Looks like it will also fit the bill.

 

Also keep in mind that if your having 1PB of data, your going to need DR, as there is no way your going to be able to back that system up using traditional means (tape).

[Stefan1024]

The SAN will be kept busy with calculating checksumms and splitting the data to the huge drive array. Do you have RAID cards for the heavy lifting? Otherwise you need a lot of CPU power. As the project is already super expensive I would go with the E5-2687W V3 for the prototype. Than you can measure the CPU load and get a cheaper one for the mass production.

[brwainer]

6 minutes ago, Blake said:

 

I read that as 1PB of SSD Storage in a 1U case.

As others have said assuming 15TB SSDs, you are not going to fit ~67 of them into a 1U rack mount.

 

Also, you should be looking at SANs, not NAS not DAS, not internal Storage.

HPE 3PAR StoreServ - Have used can confirm very good.

Dell Storage PS6610 Series Arrays Looks like it will also fit the bill.

 

Also keep in mind that if your having 1PB of data, your going to need DR, as there is no way your going to be able to back that system up using traditional means (tape).

I read the first post as meaning this system is just the entry point (one of several ideally of course) to the SAN, seeing as he has a PCIe 16x card as "secret sauce" in his requirements. But the rest of your points are valid in either case.

[.:MARK:.]

Wrong forum, servethehome would be better, but even then it's still not quite right.

[KuJoe]

I agree with some of the others who already said to contact HP/Dell/IBM/Cisco (I recommend Cisco) to get a quote for a server that meets your needs. No offense to the forums but for such a high level project I would rather get suggestions and answers from somebody who can be held accountable if the hardware doesn't work out.

[tt2468]

20 hours ago, Blake said:

 

I read that as 1PB of SSD Storage in a 1U case.

As others have said assuming 15TB SSDs, you are not going to fit ~67 of them into a 1U rack mount.

 

Also, you should be looking at SANs, not NAS not DAS, not internal Storage.

HPE 3PAR StoreServ - Have used can confirm very good.

Dell Storage PS6610 Series Arrays Looks like it will also fit the bill.

 

Also keep in mind that if your having 1PB of data, your going to need DR, as there is no way your going to be able to back that system up using traditional means (tape).

Lmao I dont need no raid cards. The company is what is making the ssds. We got everything covered. What I was truely wondering is how much cpu power should be used for zfs and samba sharing.

[brwainer]

5 minutes ago, Kyle Manning said:

Lmao I dont need no raid cards. The company is what is making the ssds. We got everything covered. What I was truely wondering is how much cpu power should be used for zfs and samba sharing.

Woah woah woah zfs with 1pb? I'm not sure that's within the bounds of a single CPU system.

[leadeater]

15 minutes ago, Kyle Manning said:

Lmao I dont need no raid cards. The company is what is making the ssds. We got everything covered. What I was truely wondering is how much cpu power should be used for zfs and samba sharing.

See my first post around CPU resources etc. I would very much doubt dual E5-2620v3 CPU's would be enough to handle the load you want to put on them, unless there is a very significant amount of offloading to the NICs or that special sauce PCI-E device you mentioned, but even then SAMBA SMB isn't exactly efficient in CPU usage and zfs with dedup would monster the CPU's.

 

During backup our Netapp filers with a total of 8x E5-2658 in each C-Mode cluster sit around 60%-80% CPU usage with a total throughput of around 20Gbps+. These filers serve iSCSI luns to SQL/Exchange servers, NFS mounts to VMware and SMB shares.