10Gbit PFSense Router

[Aelita Sophie]

For a startup ISP company we are in the need of a 10Gbit capable PFSense Router. Our current line is 2 Gbit up/down, but will probably upgraded to 10Gbit within a year.

My biggest concern is the picking of the right hardware for this job. According to some articles the older L5620 Xeon processor would suffice with the correct 10Gbit network cards, but I am not that experienced with PFSense. (Used it only a few times so far for smaller business who only have 1Gbit)

Going from the article any newer Xeon would suffice (as the performance difference between current gen Xeons and that 5 generations older one is apparently huge)

Space usage is a thing though. I would like this to fit within 1U. The dept of the 1U isnt a big concern.

We will be setting up 2 networks from this PFSense based router. 1 100mbit network for our IPMI connections. Which will be only be accessible remotely with a VPN (Preferable already in PFSense, So Advanced Encryption Standard would be a MUST. And the mentioned 10GBit network. 

Now pricing is an issue as well. I only got a small budget allocated from my boss for this. It's debateable, but only if it makes a LOT of sense.

Current budget is set at roughly €500 without the 10Gbit cards.

[TapfererToaster]

I would look into a system like this: Supermicro SYS-5018D-FN8T

https://www.amazon.com/Supermicro-SuperServer-5018D-FN8T-Rackmount-10GbE/dp/B01LXUATHB/ref=sr_1_1?ie=UTF8&qid=1502672642&sr=8-1&keywords=5018D-FN8T

 

It would only need RAM and a SSD and is good to go. Price is over the 500$, but it already got 2x 10Gb SFP+ Ports, 6x 1GbE RJ45 and the seperate IPMI port.

 

It SHOULD handle the 10Gbit traffic fine, but if you later use heavy encryption on multiple interfaces or w/e, there is a hardware solution for this.

Doubt it will be necessary in the 10Gbit segment, but if you upgrade you might want to look into Intel QuickAssist cards. i read somewhere freebsd recently got driver for those.

[Mornincupofhate]

You're starting up your own ISP?

Do your customers a favor and buy some dedicated routing hardware from Cisco. Like ones that will actually be able to sustain 10Gbps.

[GDRRiley]

11 minutes ago, Mornincupofhate said:

You're starting up your own ISP?

Do your customers a favor and buy some dedicated routing hardware from Cisco. Like ones that will actually be able to sustain 10Gbps.

cisco gear is way more than their budget allows 

[GDRRiley]

you could grab a box like this a add a quad core xeon, ram and boot drive. 

https://www.newegg.com/Product/Product.aspx?Item=9SIA9AX62K0105

[Guest]

7 hours ago, Aelita Sophie said:

For a startup ISP company we are in the need of a 10Gbit capable PFSense Router. Our current line is 2 Gbit up/down, but will probably upgraded to 10Gbit within a year.

My biggest concern is the picking of the right hardware for this job. According to some articles the older L5620 Xeon processor would suffice with the correct 10Gbit network cards, but I am not that experienced with PFSense. (Used it only a few times so far for smaller business who only have 1Gbit)

Going from the article any newer Xeon would suffice (as the performance difference between current gen Xeons and that 5 generations older one is apparently huge)

Space usage is a thing though. I would like this to fit within 1U. The dept of the 1U isnt a big concern.

We will be setting up 2 networks from this PFSense based router. 1 100mbit network for our IPMI connections. Which will be only be accessible remotely with a VPN (Preferable already in PFSense, So Advanced Encryption Standard would be a MUST. And the mentioned 10GBit network. 

Now pricing is an issue as well. I only got a small budget allocated from my boss for this. It's debateable, but only if it makes a LOT of sense.

Current budget is set at roughly €500 without the 10Gbit cards.

What're you actually using this for?

[Sir Asvald]

6 hours ago, Mornincupofhate said:

You're starting up your own ISP?

Do your customers a favor and buy some dedicated routing hardware from Cisco. Like ones that will actually be able to sustain 10Gbps.

Do you realise the Most Cisco ISR routers are expensive. 

[Sir Asvald]

10 hours ago, Aelita Sophie said:

For a startup ISP company we are in the need of a 10Gbit capable PFSense Router. Our current line is 2 Gbit up/down, but will probably upgraded to 10Gbit within a year.

My biggest concern is the picking of the right hardware for this job. According to some articles the older L5620 Xeon processor would suffice with the correct 10Gbit network cards, but I am not that experienced with PFSense. (Used it only a few times so far for smaller business who only have 1Gbit)

Going from the article any newer Xeon would suffice (as the performance difference between current gen Xeons and that 5 generations older one is apparently huge)

Space usage is a thing though. I would like this to fit within 1U. The dept of the 1U isnt a big concern.

We will be setting up 2 networks from this PFSense based router. 1 100mbit network for our IPMI connections. Which will be only be accessible remotely with a VPN (Preferable already in PFSense, So Advanced Encryption Standard would be a MUST. And the mentioned 10GBit network. 

Now pricing is an issue as well. I only got a small budget allocated from my boss for this. It's debateable, but only if it makes a LOT of sense.

Current budget is set at roughly €500 without the 10Gbit cards.

How much people are you serving? How much inbound and outbound traffic? Look at some Dell rack servers that support dual or quad socket CPUs. You're going to need it. 

 

 

[Aelita Sophie]

8 hours ago, TapfererToaster said:

I would look into a system like this: Supermicro SYS-5018D-FN8T

https://www.amazon.com/Supermicro-SuperServer-5018D-FN8T-Rackmount-10GbE/dp/B01LXUATHB/ref=sr_1_1?ie=UTF8&qid=1502672642&sr=8-1&keywords=5018D-FN8T

 

It would only need RAM and a SSD and is good to go. Price is over the 500$, but it already got 2x 10Gb SFP+ Ports, 6x 1GbE RJ45 and the seperate IPMI port.

 

It SHOULD handle the 10Gbit traffic fine, but if you later use heavy encryption on multiple interfaces or w/e, there is a hardware solution for this.

Doubt it will be necessary in the 10Gbit segment, but if you upgrade you might want to look into Intel QuickAssist cards. i read somewhere freebsd recently got driver for those.

 

6 hours ago, GDRRiley said:

you could grab a box like this a add a quad core xeon, ram and boot drive. 

https://www.newegg.com/Product/Product.aspx?Item=9SIA9AX62K0105

The Quad Core in the Super Micro Superserver doesn't seem to powerful, if I have to be honest.  You sure that would be enough?

 

6 hours ago, Mornincupofhate said:

You're starting up your own ISP?

Do your customers a favor and buy some dedicated routing hardware from Cisco. Like ones that will actually be able to sustain 10Gbps.

I do not think that it is absolutely necessary. With ISP I mean we will server Internet Services. Which would mainly be (web/application)hosting, dedicated server rent or leases, colocation services and VPS hosting.

 

3 hours ago, Windspeed36 said:

What're you actually using this for?

It will mainly be used to serve IP's to certain segments of our network within the rack unit. (47U) We will be replicating this in every rack.

 

10 minutes ago, Abdul201588 said:

How much people are you serving? How much inbound and outbound traffic? Look at some Dell rack servers that support dual or quad socket CPUs. You're going to need it. 

 

 

It would be hard to tell, as our business model is focused on small business and bigger. If my calculations are somewhat correct, there will be about 10 colocation servers max per rack, 1 or 2 Webservers per rack and about 200 VPSs per rack. We will probably NEVER saturate the 10GBit line. But the 2x 1Gbit lane will definitely be saturated. Every rack would be having their own 10Gbit connection (in the near future) and thus their own 1U PFSense Router. All encryption algoritmes will be done on the other servers in the datarack itself. So this would only act as a firewall and IP distribution. Though it just has to be able to achieve 10Gbit connection speeds. (Or somewhat in the neighborhood. ie 9Gbit would be fine 2)

[Sir Asvald]

3 minutes ago, Aelita Sophie said:

It would be hard to tell, as our business model is focused on small business and bigger. If my calculations are somewhat correct, there will be about 10 colocation servers max per rack, 1 or 2 Webservers per rack and about 200 VPSs per rack. We will probably NEVER saturate the 10GBit line. But the 2x 1Gbit lane will definitely be saturated. Every rack would be having their own 10Gbit connection (in the near future) and thus their own 1U PFSense Router. All encryption algoritmes will be done on the other servers in the datarack itself. So this would only act as a firewall and IP distribution. Though it just has to be able to achieve 10Gbit connection speeds. (Or somewhat in the neighborhood. ie 9Gbit would be fine 2)

So, 10 servers per rack. Maximum of 2 webservers, and 200 VPSs? WOW

 

You're going to need a lot of CPU power and a lot of memory to route all traffic. 

 

You said you've only got $500 to spend? 

[Aelita Sophie]

Just now, Abdul201588 said:

So, 10 servers per rack. Maximum of 2 webservers, and 200 VPSs? WOW

 

You're going to need a lot of CPU power and a lot of memory to route all traffic. 

 

You said you've only got $500 to spend? 

Yeap that is the budget I've got from my boss. (Though it is €500, so that would be $600 roughly) Though the VPSs will be limited to 200mbit burst, with datacaps. Same goes with the other servers, they will be capped on contract. The colocation will have a 1Gbit up/down Link but only a 2mbit 95% a month (for example). This all depends on how much the customer thinks he/she will need for their business. Bigger business will be housed in separate racks with their own hardware.

[Sir Asvald]

8 minutes ago, Aelita Sophie said:

Yeap that is the budget I've got from my boss. (Though it is €500, so that would be $600 roughly) Though the VPSs will be limited to 200mbit burst, with datacaps. Same goes with the other servers, they will be capped on contract. The colocation will have a 1Gbit up/down Link but only a 2mbit 95% a month (for example). This all depends on how much the customer thinks he/she will need for their business. Bigger business will be housed in separate racks with their own hardware.

Not be rude, does your boss have any networking knowledge? 

 

I'd look for a server with 2 CPUs and some good amount of ram. Like a Dell rack server. Install the 10Gbit card in there and use it as a router. 

 

 

[Aelita Sophie]

1 minute ago, Abdul201588 said:

Not be rude, does your boss have any networking knowledge? 

 

I'd look for a server with 2 CPUs and some good amount of ram. Like a Dell rack server. Install the 10Gbit card in there and use it as router. 

 

 

Well the 10gbit capability is not for right now. So we would be fine with a server for just 1 or 2Gbit, but it needs to be upgradeable for 10Gbit within probably this year. So for example, if we can achieve that by adding an extra CPU or swapping the CPU with a more powerful one. Add some ram and 10Gbit NICS then that would be exactly what we want.

[Sir Asvald]

3 minutes ago, Aelita Sophie said:

Well the 10gbit capability is not for right now. So we would be fine with a server for just 1 or 2Gbit, but it needs to be upgradeable for 10Gbit within probably this year. So for example, if we can achieve that by adding an extra CPU or swapping the CPU with a more powerful one. Add some ram and 10Gbit NICS then that would be exactly what we want.

Sure thing... But, I'm going to say this, 600 Euros is not going to get you anything good. Servers start at 1000+ if you're looking for 2 socket CPUs.  

 

[Aelita Sophie]

Just now, Abdul201588 said:

Sure thing... But, I'm going to say this, 600 Euros is not going to get you anything good. Servers start at 1000+ if you're looking for 2 socket CPUs.  

 

Well I either need to get more budget from my boss, or we will be replacing the entire machine when its necessary. In the past week we've been investing €10.000s for new servers. (SuperMicro SuperServers X10's for example) Hence the reason he wants a somewhat "cheap" routing solution to get everything started. I was trying to get something that would last a while. But if that can't be done, then it can't be done. Then we simply get "whats available now" and swap it out later. We can also reuse the hardware for other purposes down the line.

[Sir Asvald]

2 minutes ago, Aelita Sophie said:

Well I either need to get more budget from my boss, or we will be replacing the entire machine when its necessary. In the past week we've been investing €10.000s for new servers. (SuperMicro SuperServers X10's for example) Hence the reason he wants a somewhat "cheap" routing solution to get everything started. I was trying to get something that would last a while. But if that can't be done, then it can't be done. Then we simply get "whats available now" and swap it out later. We can also reuse the hardware for other purposes down the line.

Then go for something cheap. You've said that you don't need 10Gbit right now. So go for a 1 socket CPU server 

[Lurick]

2 hours ago, Abdul201588 said:

Do you realise the Most Cisco ISR routers are expensive. 

Nah, Cisco ISRs are for branchs and small sites.

In this instance they would need ASR 9000 or 9900s along with some very good switching gear.

$500 could get a couple pictures of some ASR 9000's to put in place of the gear

 

 

 

OP, you're going to need much more routing capacity. Not sure what your switching situation looks like but that might need some considerations as well. I would talk to your boss and tell him that $600 isn't going to get you what you need, or at least it isn't going to get you anything in terms of a lasting solution. Then, I would call around to your local Cisco, Ubiquity, and other sales teams, and see what they can offer in terms of 10Gbit routing and switching gear. I know most companies offer financing options too, if you can't afford the upfront costs. I'm not going to be the Cisco sales guy, just going to give you some pointers  

If you need Cisco contacts or help, shoot me a PM.

 

 

Edit:

PFsense is a good starting option but I'm wondering if it will provide all your needs in terms of firewall and routing options.

[Sir Asvald]

3 minutes ago, Lurick said:

Nah, Cisco ISRs are for branchs and small sites.

In this instance they would need ASR 9000 or 9900s along with some very good switching gear.

$500 could get a couple pictures of some ASR 9000's to put in place of the gear

I knew I there was something else.  

[Lurick]

7 minutes ago, Abdul201588 said:

I knew I there was something else.  

There is the ASR 900 and 1000 lines as well which provide a good middle ground  

[Aelita Sophie]

26 minutes ago, Lurick said:

Nah, Cisco ISRs are for branchs and small sites.

In this instance they would need ASR 9000 or 9900s along with some very good switching gear.

$500 could get a couple pictures of some ASR 9000's to put in place of the gear

 

 

 

OP, you're going to need much more routing capacity. Not sure what your switching situation looks like but that might need some considerations as well. I would talk to your boss and tell him that $600 isn't going to get you what you need, or at least it isn't going to get you anything in terms of a lasting solution. Then, I would call around to your local Cisco, Ubiquity, and other sales teams, and see what they can offer in terms of 10Gbit routing and switching gear. I know most companies offer financing options too, if you can't afford the upfront costs. I'm not going to be the Cisco sales guy, just going to give you some pointers  

If you need Cisco contacts or help, shoot me a PM.

 

 

Edit:

PFsense is a good starting option but I'm wondering if it will provide all your needs in terms of firewall and routing options.

Well currently we don't really have a baseline of how much capacity we actually will need. Because simple bandwidth speeds is not the only story of course. But right now we need something that is "affordable", very customize able and will do the job. With some upgrade path for the future. The switching situation will be using mostly Netgear XS716E or similar. All cables will be Cat6A or Cat7 SF/UTP cables. The VPN-Enabled network for the IPMI will be a simple 48 port 100mbit "dumb" switch, this will be secured by SSL certificates etc.

In terms of firewall, we are only looking for a "works" solution. Our customers either need to have their own security done well, or can rent/buy additional firewall equipment for their needs. So it will only be employed for "good-enough" for most use cases. As for routing goes, we are not looking to route multiple 47u dataracks. This solution will ONLY route 1 datarack. We currently also only own 1 Datarack. As mentioned before its a startup company.

Though cisco contacts would be a great help. Do you have any contacts in either Europe or even better BeNeLux (Belgium Netherlands Luxemborg, specificaly Netherlands). I happen to know there is Cisco HQ in Brussels, been there once, back when I was still a student.

[Lurick]

8 minutes ago, Aelita Sophie said:

Well currently we don't really have a baseline of how much capacity we actually will need. Because simple bandwidth speeds is not the only story of course. But right now we need something that is "affordable", very customize able and will do the job. With some upgrade path for the future. The switching situation will be using mostly Netgear XS716E or similar. All cables will be Cat6A or Cat7 SF/UTP cables. The VPN-Enabled network for the IPMI will be a simple 48 port 100mbit "dumb" switch, this will be secured by SSL certificates etc.

In terms of firewall, we are only looking for a "works" solution. Our customers either need to have their own security done well, or can rent/buy additional firewall equipment for their needs. So it will only be employed for "good-enough" for most use cases. As for routing goes, we are not looking to route multiple 47u dataracks. This solution will ONLY route 1 datarack. We currently also only own 1 Datarack. As mentioned before its a startup company.

Though cisco contacts would be a great help. Do you have any contacts in either Europe or even better BeNeLux (Belgium Netherlands Luxemborg, specificaly Netherlands). I happen to know there is Cisco HQ in Brussels, been there once, back when I was still a student.

Gotcha, wasn't sure how big everything was right now in terms of space and offerings from you to the customers. I was thinking at least 5 to 10 racks but if you've only got a single rack or two then a PFSense router would be a good start and putting security on the customer is a good start as well. Then when you expand in the future you can look to more wholesale solutions and whatnot.

Look at see what you can find on eBay and the likes in terms of Dell R620 servers.

 

I know some regional sales people who've got contacts all over the world, I'll sync up with them later today or tomorrow and try to get some names for the area  

[Aelita Sophie]

2 minutes ago, Lurick said:

Gotcha, wasn't sure how big everything was right now in terms of space and offerings from you to the customers. I was thinking at least 5 to 10 racks but if you've only got a single rack or two then a PFSense router would be a good start and putting security on the customer is a good start as well. Then when you expand in the future you can look to more wholesale solutions and whatnot.

Look at see what you can find on eBay and the likes in terms of Dell R620 servers.

 

I know some regional sales people who've got contacts all over the world, I'll sync up with them later today or tomorrow and try to get some names for the area  

Oh very much appreciated! Thanks a bunch!

[Lurick]

2 minutes ago, Aelita Sophie said:

Oh very much appreciated! Thanks a bunch!

Something like this should be a good starting point:

http://www.benl.ebay.be/itm/Dell-R620-4B-3x-PCI-8-Core-2-40GHz-E5-2609-8GB-RAM-iDRAC7-FM487-No-2-5-HDD-/382079126133

 

You'll need some more RAM obviously but maybe 32GB or 64GB of ECC DDR3 RAM coupled with that would work well.

[Guest]

 

2 hours ago, Aelita Sophie said:

 

It will mainly be used to serve IP's to certain segments of our network within the rack unit. (47U) We will be replicating this in every rack.

 

It would be hard to tell, as our business model is focused on small business and bigger. If my calculations are somewhat correct, there will be about 10 colocation servers max per rack, 1 or 2 Webservers per rack and about 200 VPSs per rack. We will probably NEVER saturate the 10GBit line. But the 2x 1Gbit lane will definitely be saturated. Every rack would be having their own 10Gbit connection (in the near future) and thus their own 1U PFSense Router. All encryption algoritmes will be done on the other servers in the datarack itself. So this would only act as a firewall and IP distribution. Though it just has to be able to achieve 10Gbit connection speeds. (Or somewhat in the neighborhood. ie 9Gbit would be fine 2)

 

So I'm going to preface this by saying I'm by no means a networking guy; I've done the course material for CCNA but haven't sat the exams. I do however work as a level 2/3 technician for an MSP including solution design. One of the solutions I recently had to put together was involving the design of our own VMware IaaS platform and the biggest tip I can give you is to make the SLA your number one priority when designing this. 

 

What guarantee are are you giving customers in terms of uptime?

 

Is that guarantee financially backed?

 

If you're aiming to achieve anything higher than 99.9% you need to throw your ideas of PFSense out the window and get equipment designed for the job. The last thing you want to be doing is scratching your head at 2am on Christmas morning trying to come up with a fix for a failed makeshift router. 

 

One of my other questions for you is involving your total collocation; how many racks had you provisioned? If you're needing more than 2 x 48RU to provision production and DR for 200 virtual machines, you've got an issue. 

 

Colo; especially power and cross connects add up fast and eat away the margin.  

[Aelita Sophie]

1 minute ago, Windspeed36 said:

 

 

So I'm going to preface this by saying I'm by no means a networking guy; I've done the course material for CCNA but haven't sat the exams. I do however work as a level 2/3 technician for an MSP including solution design. One of the solutions I recently had to put together was involving the design of our own VMware IaaS platform and the biggest tip I can give you is to make the SLA your number one priority when designing this. 

 

What guarantee are are you giving customers in terms of uptime?

 

Is that guarantee financially backed?

 

If you're aiming to achieve anything higher than 99.9% you need to throw your ideas of PFSense out the window and get equipment designed for the job. The last thing you want to be doing is scratching your head at 2am on Christmas morning trying to come up with a fix for a failed makeshift router. 

 

One of my other questions for you is involving your total collocation; how many racks had you provisioned? If you're needing more than 2 x 48RU to provision production and DR for 200 virtual machines, you've got an issue. 

 

Colo; especially power and cross connects add up fast and eat away the margin.  

SLA will only be given on payment base. If a customer requests a SLA, we will definitely put them on separate hardware. But more on that I can't answer just yet, as my boss hasnt been to clear about pricing and services.

 

We are not aiming 99.9% just yet. Though working on christmas isn't a problem. Our datacenter offers technicians for "rent" when we are unable to do the work ourselves, this is 24/7. They have replacement hardware in stock, and so would we have in the near future. Currently we only are filling 1 rack of 47U (1U is predefined used by the datacenter to deliver the power)

 

Power isn't an issue, currently we will be getting 2 Different power lines on 32 Amps 220volts. (Which has Datacenter provided UPSes and Diesel Generators, though we will install PSU's ourselves just to be sure)