Jump to content

Question regarding IP Transit

Hello, currently we are renting some servers at OVH, the service isn't bad at all with the only downside of them modifying the kernel.

 

Since the servers are being rented I cannot just "upgrade" what I want (disk space for example). So we're starting to see different options.

 

One of the options (and the best I saw so far) was to rent a half rack unit in a data center. This data center is located few Km from our office, so It seems to be a good option.

 

They gave us some options which got me kinda lost, and this is why I'm asking here in the forums.

 

First, the difference between tier2 and tier4 data centers.

Their tier 2 data center costs 350€ /month for half a rack. This comes with 1kW metered power.

Their tier 4 data center costs 490€ /month for half a rack. This comes with 1kW metered power.

Regarding the network speed they only state:

"Connectivity: Redundant fibre optic loops ensure Internet connectivity"

From other packages which are rack units their network speed goes over 2Gbps.

 

What is the difference between Tier2 and Tier4? I saw it as to do with uptime at some point. But is it worth it? paying 140€+ monthly?

 

Then we have the second problem which are the IPs.

I contacted them to know how it would work since we are doing virtualization and we would need around 24 IPs (for now).

Then they told me that we would need an IP transit.

50 Mbit/s IP Transit:  100 EUR / month excl. VAT.

 

I'm totally new with IP Transit and i'm kinda lost. How will it work to add 24 IPs in this case?

 

In case we move forward it will be a big investiseement on which we will move from 150€ /monthly to over 450€ /monthly if we take the cheapest. Without counting the costs of the servers. So I want to make sure everything goes as planned.

 

They also told us that if we take the half rack, we will get badges and alarm security codes so we can access the data center 24/7 to provide server maintenance.

 

Hopefully someone can provide some help. Thank you.

 

Link to comment
Share on other sites

Link to post
Share on other sites

17 minutes ago, Diogo Jesus said:

What is the difference between Tier2 and Tier4? I saw it as to do with uptime at some point. But is it worth it? paying 140€+ monthly?

Think of it as the effort gone in to redundancy of design of the facility and all things related to, what things and combinations of things have to go wrong/fail to cause an outage. This can be things like power sources (more than 1 grid supplier from different utility feeds) plus generator or building fire suppression. The higher tier facilities are built on top of ground isolators for earth quakes and the building is technically buildings within buildings and each data hall is completely isolated so fire cannot spread from one to another.

 

It's all risk mitigation, insurance and comes down to how much you are willing to pay. However two geo-separated tier 2 facilities is better than 1 tier 4, as long as everything is replicated between the two and you have working HA and/or DR.

 

https://www.colocationamerica.com/data-center/tier-standards-overview.htm

 

17 minutes ago, Diogo Jesus said:

Then we have the second problem which are the IPs.

I contacted them to know how it would work since we are doing virtualization and we would need around 24 IPs (for now).

Then they told me that we would need an IP transit.

50 Mbit/s IP Transit:  100 EUR / month excl. VAT.

 

I'm totally new with IP Transit and i'm kinda lost. How will it work to add 24 IPs in this case?

@Lurick is likely best to help out here.

 

I would ask if they have other options, do you actually need all your servers to have public IP addresses? Would a security appliance/firewall doing NAT with 1 or 2 public IP addresses suffice and have all your servers using private IPs?

 

Do you currently own your own IP range/ASN? Do you, or anyone else, have any BGP experience at all?

 

https://www.telstraglobal.com/uk/insights/blogs/blog/peering-vs-transit

Link to comment
Share on other sites

Link to post
Share on other sites

49 minutes ago, leadeater said:

However two geo-separated tier 2 facilities is better than 1 tier 4, as long as everything is replicated between the two and you have working HA and/or DR. 

They only have 1 facility tier2 and 1 facility tier4. Both have fire security and earthquakes in our country are not usual.

 

50 minutes ago, leadeater said:

do you actually need all your servers to have public IP addresses? Would a security appliance/firewall doing NAT with 1 or 2 public IP addresses suffice and have all your servers using private IPs?

We run multiple applications for different needs, For example 1 app related to our back office team will have a different public address than an app for our clients. We do this way to provide security while using different virtualizations.

 

One of the IPs for example is used for AS2 messaging layer which we need to run the business. That IP is shared between partners and it cannot be down. So if we suffer a DDoS attack while the attacker used a public address related to the clients portal. It won't affect our message communication. (This is just an example).

 

We are running proxmox for this matter and inside these virtualizations we are running all kind of services, from web pages, to CDN, to mail servers, VPNs and so on. When we first designed the server architecture we decided it would be better to run under different IP addresses. I'm not sure if it was the best idea tho.

 

58 minutes ago, leadeater said:

Do you currently own your own IP range/ASN? Do you, or anyone else, have any BGP experience at all? 

No, We use OVH IP Fail Over which are attached to our server.

Also neither do I or any of my colleagues have BGP experience(I might do some research on that).

 

Just as background, I'm a junior sysadmin and the only sysadmin in the company. That's why I prefer to ask it here where people have a lot more knowledge than me. Also the company is a startup which means that financially it's not the greatest. This is why I'm wondering if it's worth this upgrade as I need to convince the boss in the next meetings to provide better server infrastructure.

 

Link to comment
Share on other sites

Link to post
Share on other sites

32 minutes ago, Diogo Jesus said:

-snip-

Based on the information you gave I'm leaning towards not renting rack space. It's a nice option but I would do that only if the provider you are looking at has leased networking options and handles that for you and just assigns you an IP range/subnet to use. I'm not exactly sure what network services they are offering but it sounds to me like layer 1/2 only and you need to control everything above that which means your on IP space and IP transit and/or peering. I only base this off that fact you are being told you need to pay for IP transit.

 

Pretty much you want a similar network setup to what you have now with OVH where they do it all for you otherwise you'll have to start getting in to more specialist networking areas of skill sets.

 

I don't have too much experience using co-location services as we run our own facilities so hopefully when @Lurick comes on line he can help out better.

 

I would point out that using different IP addresses like you describe would mostly only help out if you have a DDoS mitigation service that will change the routing to move the traffic to go through their filtering platform then send it on to you. Without that any and all traffic destined to the same point, regardless of IP, will saturate either the link bandwidth or the network processing capability of the equipment and you'll have service interruption.

 

Personally I would look at the application layer design and higher level infrastructure and see if there are any improvements you can make there, and also enable the usage of any provider or platform so what you run it on doesn't matter. See if there are cost savings by using an exist CDN for example or not hosting your own mail services. Realistically I can't help much here because all the required information to actually do that is your company knowledge but I think you get the gist of what I'm saying.

 

Running your own hardware can seem cheaper but there are many costs that you might not know about or extra time and resourcing requirements.

Link to comment
Share on other sites

Link to post
Share on other sites

Thank you for the support @leadeater 

Our main concern about this idea of running our own hardware is due to get "unlimited disk space". The problem at OVH (SoYouStart) is that they cannot provide more disk space in case we need it.

 

If we need more disk space we must rent a new server.

We're not very AWS or AZURE friendly as well so I don't think that will be an option to run our services.

 

Our mail services are running under a VM server inside our main server and it's doing the job perfectly. All our DNS are correctly setup giving us the possibility to run as many mailboxes as we need. When I got hired one of the main purposes was to run our own mail server so we can create or delete mailboxes when we want and when we need without extra charges. For our CDN we created a simple NGinx web server storing all website images and so on.

 

The only downside of our current setup is that we're only able to run the main server under 250Mbps. Which is not the best. Right now we are running 2 different physical servers running over 20 VMs.

Link to comment
Share on other sites

Link to post
Share on other sites

29 minutes ago, Diogo Jesus said:

Our main concern about this idea of running our own hardware is due to get "unlimited disk space". The problem at OVH (SoYouStart) is that they cannot provide more disk space in case we need it.

Do you have a way you can utilize external data outside of OVH or other OVH options? Where are you thinking you might hit storage limitations?

 

For example none of the data for any of our Linux VMs are kept in the VM/virtual disk, we use NFS mounts (many reasons for this). Is something like that possible for you, or moving some data to S3 type storage maybe some run yourselves like Gluster or Ceph.

Link to comment
Share on other sites

Link to post
Share on other sites

4 minutes ago, leadeater said:

Where are you thinking you might hit storage limitations?

With our AS2 and SQL servers. We run master slave and currently have 4 servers MySQL and PostgreSQL (8 knowing they are in master slave).

1 minute ago, leadeater said:

Do you have a way you can utilize external data outside of OVH or other OVH options? Where are you thinking you might hit storage limitations?

They have a NFS mount storage option. Which we are using for proxmox daily backups.

3 minutes ago, leadeater said:

or moving some data to S3 type storage maybe some run yourselves like Gluster or Ceph.

Didn't had time to go further on this option but i'll do some research for sure.

Thank you

Link to comment
Share on other sites

Link to post
Share on other sites

1 minute ago, Diogo Jesus said:

SQL servers. We run master slave and currently have 4 servers MySQL and PostgreSQL (8 knowing they are in master slave).

 

1 minute ago, Diogo Jesus said:

Didn't had time to go further on this option but i'll do some research for sure.

 

S3 storage isn't an option for that workload type, just to save you some time. Databases prefer block storage, can live on file storage but absolutely not on object storage.

Link to comment
Share on other sites

Link to post
Share on other sites

1 minute ago, leadeater said:

Databases prefer block storage, can live on file storage but absolutely not on object storage.

I'm aware of this in case we use it it would be to store only the AS2 Messages which are taking a lot of disk usage at the moment. Each received message is saved and its values stored in the different database (it depends on the message type).

 

Maybe the S3 would be enough for that matter. Thank you

Link to comment
Share on other sites

Link to post
Share on other sites

For the IP transit part I think they're saying you'll need to pay just for the block of public addresses and the uplink speed. They likely have a large block of address space reserved with an upstream ISP and then lease those addresses and the uplink speed for a price but when I hear IP transit I usually think of you peering across their network to another network, kind of like with MPLS (or a GRE tunnel even) but there are different types of transit and peering, it's just been a while for me dealing with some of that. Is there a chance you can link to their website by chance @Diogo Jesus ? I just want to see if they mention if the transit price is per IP or per block and check on some other stuff.

Current Network Layout:

Current Build Log/PC:

Prior Build Log/PC:

Link to comment
Share on other sites

Link to post
Share on other sites

16 minutes ago, Lurick said:

Is there a chance you can link to their website by chance

for sure their website is https://www.vo.lu/

17 minutes ago, Lurick said:

I just want to see if they mention if the transit price is per IP or per block and check on some other stuff.

They didn't provide this info trough their website but by mail.  And they didn't state if it was per IP or per block.

The first time we asked them about the price for 8 IPs which they answered 30 EUR /month.

Then I asked if was there anyway to get more than 8 IPs because we would need around 24 IPs which they answered "If you need more than 8 IP addresses you will need a Transit IP specific for your server 50 Mbit/s IP Transit:  100 EUR / month excl. VAT."

 

 

 

Link to comment
Share on other sites

Link to post
Share on other sites

If your goal is DDoS mitigation I would pay a service like Cloudflare and save costs on an IP block.

 

If you're trying to divide/segment corporate network access and customer access by IP address yet they both go to the same environment it is a moot point. It would be better to layer the networks internally behind a single public IP. Most would have a firewall/security appliance in front of each section. So you'd have a perimeter firewall, 1 firewall in front of "corporate" and 1 in front of customer services.

 

Ultimately I can see the need for 2 IP addresses at most, but hiding behind different IP addresses won't add much security unfortunately.

 

Unrelated.. just started watching a show called Patriot which takes place in Luxembourg. Thought to myself, I've never heard of that city before but I've never been to France either. Now I see you are from Luxembourg I'm totally floored lol.

 

Is it true there's a lot of Brazilians there?

Link to comment
Share on other sites

Link to post
Share on other sites

31 minutes ago, Mikensan said:

If your goal is DDoS mitigation I would pay a service like Cloudflare and save costs on an IP block.

We've been reading about this as well.

 

32 minutes ago, Mikensan said:

Ultimately I can see the need for 2 IP addresses at most, but hiding behind different IP addresses won't add much security unfortunately

Guess that I need to do more research on this. Usually people advise to separate web servers from mail servers so they don't get the same IP. I could get you a full list of servers + IPs so you can get an idea on the architecture behind it. Because we run a lot of different services. In case of DDoS I can simply low the network speed under the affected VM. So even if that service keeps busy and down, it won't affect the back office system.

 

34 minutes ago, Mikensan said:

Unrelated.. just started watching a show called Patriot which takes place in Luxembourg. Thought to myself, I've never heard of that city before but I've never been to France either. Now I see you are from Luxembourg I'm totally floored lol

Never thought there were showing taking place in Luxembourg. Small country a lot of money and sad life. Guess we can't get both things at the time haha.

 

36 minutes ago, Mikensan said:

Is it true there's a lot of Brazilians there? 

Not really, there's a really good mix of nationalities.

There are a lot of Portuguese people tho. There are around 15% of the whole population. Knowing that there are around 50% of foreign people living in Luxembourg. Means that 25% of the population are actually Portuguese (like myself hehe)

Link to comment
Share on other sites

Link to post
Share on other sites

Ooooo interesting, the show made it sound like Brazilians made up a lot of the labor force - I suppose it was mostly for theatrics since they wanted some Brazilian martial arts in the series lol. The show focused on Luxembourg because it was a neutral spot for countries to do business that otherwise should not be doing business (In this case, the U.S. and Iran). How true it is who knows but those people lol.

 

Don't forget DDoS isn't exactly the same as a flood attack (overwhelming your bandwidth). It could be targeted at your service (Exchange as you said) and crippling just that service. This wouldn't impact your other services if it were targeted. With a 1gb pipe at a datacenter that likely has BGP, it would take a very serious effort to overwhelm / flood the bandwidth.

 

There are a lot of concepts of how a network should be and you have to do what makes the most sense for your budget. Your management should be looking at the cost of the risk vs the yearly cost of the service (risk management). In that risk evaluation is to consider how likely you are for a targeted attack - is the company high profile enough or contain valuable enough data that somebody would want to do this? DDoS's are typically not random as malware / viruses and the like, eats up too much resources to just attack the world. You also consider how fast you could recover and how much business would be lost during that time. This is where a disaster recovery plan is helpful, where if it happens you're not running around trying to figure out what to do.

 

IPv4 blocks are only getting more expensive as time goes on, and I feel like that money could better fortify your network than a separate IP would. Normally you do separate your internet facing services from your internal services. This is typically done by sectioning/segmenting/zones - familiar term would be to have a "DMZ" for any internet facing services. However you don't put the service directly, you put some form of proxy or "front end". Exchange for example, you would put an edge server in your DMZ. Or for web servers, you would use a reverse proxy (though the web servers themselves shouldn't be on the same network as credential/authentication servers like domain controllers).

 

As somebody from Portugal living in France your english is very good by the way. Better than a lot of people who were born and raised here in the states lol.

Link to comment
Share on other sites

Link to post
Share on other sites

9 minutes ago, Mikensan said:

Don't forget DDoS isn't exactly the same as a flood attack (overwhelming your bandwidth). It could be targeted at your service (Exchange as you said) and crippling just that service. This wouldn't impact your other services if it were targeted. With a 1gb pipe at a datacenter that likely has BGP, it would take a very serious effort to overwhelm / flood the bandwidth.

 

There are a lot of concepts of how a network should be and you have to do what makes the most sense for your budget. Your management should be looking at the cost of the risk vs the yearly cost of the service (risk management). In that risk evaluation is to consider how likely you are for a targeted attack - is the company high profile enough or contain valuable enough data that somebody would want to do this? DDoS's are typically not random as malware / viruses and the like, eats up too much resources to just attack the world. You also consider how fast you could recover and how much business would be lost during that time. This is where a disaster recovery plan is helpful, where if it happens you're not running around trying to figure out what to do.

 

IPv4 blocks are only getting more expensive as time goes on, and I feel like that money could better fortify your network than a separate IP would. Normally you do separate your internet facing services from your internal services. This is typically done by sectioning/segmenting/zones - familiar term would be to have a "DMZ" for any internet facing services. However you don't put the service directly, you put some form of proxy or "front end". Exchange for example, you would put an edge server in your DMZ. Or for web servers, you would use a reverse proxy (though the web servers themselves shouldn't be on the same network as credential/authentication servers like domain controllers).

Guess that I need to do a lot of more research then. Our main problem is that since we are a startup and I'm just a junior sysadmin (1year and 8 months of experience) and also the only sysadmin here, I'm facing some "dumb" questions which no one in the company can actually answer them. I'll do some readings about this during this week and maybe try to re-architecture the whole VM/networks. 

 

14 minutes ago, Mikensan said:

As somebody from Portugal living in France your english is very good by the way. Better than a lot of people who were born and raised here in the states lol.

Portuguese people usually are good in english (better than the French) but don't forget that Luxembourg is an independent country and not a village or city from France :D

Link to comment
Share on other sites

Link to post
Share on other sites

That's definitely a lot to put on your shoulders, I hope they pay you well! I wouldn't re-invent the wheel as I'm sure there's a lot of work for you to do. Certainly draw it out to get a firm understanding and identify ways to improve. Plan it out and don't try to do everything in one weekend, else you will go grey early lol.

 

I love the planning and designing phase personally, and always feel free to tag me here for any advice. I'm not as experienced as others and I have my own opinions on a lot of things (as most sys admins do lol) but love shooting the breeze / brainstorming. 

 

I've worked from small to medium sized (100 users - 5,000 users) companies and so far none of them have had more than 2 IPs. A few have had bonded connections and other solutions for redundancy but that's about it. Larger companies is an entirely different ball game and my little brain can't work that hard lol. Now all that said, I've never worked for a company that is a provider of IT services to customers. Any web-based services we had were meant for employees only with maybe a website for the retail side.

 

I neither realized that Luxembourg is an independent country (like Hong Kong I guess?) nor that Portguguese people spoke english commonly. I may have to add Portugal to list of places to visit. So far it's Spain / Italy / Germany. I had one friend from portugal and he grew the manliest of beards by age 19, and another friend who just retired to go live there.

Link to comment
Share on other sites

Link to post
Share on other sites

19 hours ago, Mikensan said:

I've worked from small to medium sized (100 users - 5,000 users) companies and so far none of them have had more than 2 IPs. A few have had bonded connections and other solutions for redundancy but that's about it. Larger companies is an entirely different ball game and my little brain can't work that hard lol. Now all that said, I've never worked for a company that is a provider of IT services to customers. Any web-based services we had were meant for employees only with maybe a website for the retail side.

Previous job same situation for me, many public IPs just weren't required as reverse proxy did the job and also provided a layer of security/filtering. Anything else was just VIP/DNAT to the internal server IP from the firewall. Around 4 to 6 public IPs was generally the most.

 

Currently where I am now we have two /16 public IP blocks but that's super legacy when all the education institutions where given those huge blocks back in the 90's before anyone realized that was a stupid idea, but we have them now. For a long time all devices, literally all of them and that includes desktops and laptops, were assigned public IP addresses and firewalls were used to protect them. This is much like IPv6 is designed to be used, what else do we use all those IPv4 address for though?

 

Now were are moving all our non internet facing servers to private IPv4 addresses and placing them in dedicated network zones with firewalls between each zone etc, little overboard but it's to meet security audit requirements etc.

 

On the topic of DDoS I don't think we have ever been a target, or at least not of one of actual significance that required us to care. Flooding roughly 8 10Gbps links would be hard though so who knows, maybe we have been targeted and they failed.

Link to comment
Share on other sites

Link to post
Share on other sites

Hey sorry for the late reply but been busy lately with other projects.

 

On 4/29/2019 at 7:15 PM, Mikensan said:

Certainly draw it out to get a firm understanding and identify ways to improve

The company owners have no idea of my work, we are a small IT team of 3 (2 devs 1 sysadmin) and then there's the rest of the company. Only the IT knows what I actually do. For the rest no one has idea of my actual work (not even the company owner unfortunately).

 

On 4/29/2019 at 7:15 PM, Mikensan said:

I've worked from small to medium sized (100 users - 5,000 users) companies and so far none of them have had more than 2 IPs

 

On 4/30/2019 at 2:19 AM, leadeater said:

Previous job same situation for me, many public IPs just weren't required as reverse proxy did the job and also provided a layer of security/filtering. Anything else was just VIP/DNAT to the internal server IP from the firewall. Around 4 to 6 public IPs was generally the most. 

 

The thing is that our servers are at OVH, so we don't have anything internal that we could use local network. That's the main issue and that's why we're using that many IPs.

 

On 4/29/2019 at 7:15 PM, Mikensan said:

I neither realized that Luxembourg is an independent country (like Hong Kong I guess?) nor that Portguguese people spoke english commonly. I may have to add Portugal to list of places to visit. So far it's Spain / Italy / Germany. I had one friend from portugal and he grew the manliest of beards by age 19, and another friend who just retired to go live there.

Portugal is a really nice country indeed. You should add it to your list you will not regret it. Luxembourg isn't like Hong Kong but it's a real independent country, just like France or Germany or Portugal, the only difference is that its really small hehe.

 

 

Thank you for your tips. I will do some brainstorm on this once I come back from Holidays, as I am traveling to Las Vegas next week (first time in US btw haha)

Link to comment
Share on other sites

Link to post
Share on other sites

8 hours ago, Diogo Jesus said:

Hey sorry for the late reply but been busy lately with other projects.

 

The company owners have no idea of my work, we are a small IT team of 3 (2 devs 1 sysadmin) and then there's the rest of the company. Only the IT knows what I actually do. For the rest no one has idea of my actual work (not even the company owner unfortunately).

 

 

 

The thing is that our servers are at OVH, so we don't have anything internal that we could use local network. That's the main issue and that's why we're using that many IPs.

 

Portugal is a really nice country indeed. You should add it to your list you will not regret it. Luxembourg isn't like Hong Kong but it's a real independent country, just like France or Germany or Portugal, the only difference is that its really small hehe.

 

 

Thank you for your tips. I will do some brainstorm on this once I come back from Holidays, as I am traveling to Las Vegas next week (first time in US btw haha)

 

 

My list has been amended to include Portugal! That is one hell of a first place to visit in the U.S. lol. I've never heard anyone complain, definitely try to catch some shows (cirque du soleil) while there. I've yet to go but based off my friends who have went, it too is on my list of places to visit domestically.

Link to comment
Share on other sites

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×