What can cause maintenance down times?

[IAmAndre]

Hi,

 

I'm not sure if I'm posting in the right section. I'm a programmer and web developer more particularly, but in my 3 years of experience I've never been in a situation where a website has to be down for maintenance purposes. What could be causing that? If it's to add more features, I usually have a development/test server, and it just take a few minutes/seconds to migrate the files/data to the production server. If it's due to package updates, again you could still use a backup server momentarily, or use a cached version of the website until the process is completed. So I'd like to know why websites have to be down for a few hours for maintenance purposes.

 

Thanks

[KuJoe]

Database maintenance, you don't want anything writing to the database when you are working on it and some upgrades can take hours. There are a lot of other reasons like migrations and such but most of the things I can think of are database related.

[IAmAndre]

2 minutes ago, KuJoe said:

Database maintenance, you don't want anything writing to the database when you are working on it and some upgrades can take hours. There are a lot of other reasons like migrations and such but most of the things I can think of are database related.

OK can you elaborate on that? Why wouldn't you clone the database, work on it, then, I don't know, somehow update it with the changes that just happened?

[Mr.Meerkat]

3 minutes ago, IAmAndre said:

OK can you elaborate on that? Why wouldn't you clone the database, work on it, then, I don't know, somehow update it with the changes that just happened?

That would be extremely expensive if there's a large database involved considering that some databases could have multiple terabytes or even petabytes of data  

[KuJoe]

Just now, IAmAndre said:

OK can you elaborate on that? Why wouldn't you clone the database, work on it, then, I don't know, somehow update it with the changes that just happened?

If your database is hundreds of millions of records or is tens of gigabytes big it's faster to do an in place update/upgrade than to clone it and if you have thousands of changes a second you don't want to merge the changes afterwards, that is a pain a and would take a lot of manual intervention. Taking the website offline can save you hours in manual work, in the past I've migrated 10+ terabyte database servers before and it was no small task (multiple teams involved, lots of DBAs, lots of planning, all in a 6 hour window).

[IAmAndre]

1 minute ago, Mr.Meerkat said:

That would be extremely expensive if there's a large database involved considering that some databases could have multiple terabytes or even petabytes of data  

Why would it be expensive? You could just log all the SQL queries then run them again on the new database. That's something that just came to my mind but there are probably better solutions out there.

[IAmAndre]

2 minutes ago, KuJoe said:

If your database is hundreds of millions of records or is tens of gigabytes big it's faster to do an in place update/upgrade than to clone it and if you have thousands of changes a second you don't want to merge the changes afterwards, that is a pain a and would take a lot of manual intervention. Taking the website offline can save you hours in manual work, in the past I've migrated 10+ terabyte database servers before and it was no small task (multiple teams involved, lots of DBAs, lots of planning, all in a 6 hour window).

I (kind of) get that, but what about logging the queries then reexecute them on the new database?

[KuJoe]

19 minutes ago, IAmAndre said:

I (kind of) get that, but what about logging the queries then reexecute them on the new database?

If the database and web server has to be offline then there is no way to do that. If you want to spend tens of thousands of dollars it's possible, but companies would rather have a few hours of downtime outside of business hours than spend money on a temporary solution. Excluding licenses, our DBAs spend a lot of money on servers and storage (luckily the bandwidth is free because replicating terabytes of data in real-time to multiple data centers is pricey), they would hate to have to spend another $15k of their budget on a server for 1 week.

[vorticalbox]

33 minutes ago, IAmAndre said:

Why would it be expensive? You could just log all the SQL queries then run them again on the new database. That's something that just came to my mind but there are probably better solutions out there.

what use would having SQL log be? You would need two copies of the database in order to run them and as pointed they can be very very large and mostl of the time spread across multiple servers.

 

Amazon does not have one database it has hundreds maybe more spread across it's data centre.

[DevBlox]

A website has as much potential downtime as is it is complicated. A simple website can easily survive even a hosting service restart, because it takes less than a second, while a bigger project might be more dangerous to restart, causing all sorts of business losses along the way.

 

If a system is complicated there may be more ways than one to experience downtime. What I'm trying to say, only a part of the system may experience downtime at one point, while, let's say, the front facing system is fine (ever seen some message of downtime on some big site that still seems to function? Chances are, they are doing maintenance in their backend or something not nice is going on there, some features may not work, while others will work just fine). It's not just the database that can mean downtime, some systems have their data travel many servers to from backend to frontend, with all sorts of processing happening along the way, some can have nothing to do with the database at all. It's all to to with the architecture of a system. With such large systems, there are also cunning ways of avoiding downtime, dealing with large load and so on.

 

Usually that means more than just pushing to master and immediately testing it on the site. I understand if some things may be harder to comprehend without witnessing, it takes some time to learn :). Though that's a giant topic/topics that would take hours to explain in depth, seriously, you could write a whole book about that stuff, i bet people have, I did not read them though, I learned all that in the field, so I don't know where to point you to :).