Google Cloud accidentally deletes $125 billion Australian pension fund’s online account due to ‘unprecedented misconfiguration’

[LAwLz]

2 minutes ago, jagdtigger said:

Yeah sure, if you exclude gdrive loosing several months of data and older "happenings" i cant remember off the top of my head..........
Companies go for cloud thinking the provider will take care their end of the bargain comptently, which so far AFAIK every bigtime provider proven to not do. We are back in the old days, if you want it done properly do it yourself.

The problem is that 90% of people who think they "do things properly", don't. They also lack the knowledge to even realize what they are doing wrong.

 

 

 

 

17 hours ago, leadeater said:

But of the two on-prem is more correctly and better operated than Cloud in ways that matter and in ways that have and do effect people.

I strongly disagree. I think it is foolish to say one is "more correctly and better operated" because it entirely depends on how you use it.

It's like saying screwdrivers are better operated than hammers. 

[jagdtigger]

17 minutes ago, LAwLz said:

The problem is that 90% of people who think they "do things properly", don't. They also lack the knowledge to even realize what they are doing wrong.

That is an educational issue and plagues both cloud and on-prem, and id say it hits cloud harder than on-prem. Automation is a finicky thing and it derails quite easily........

[leadeater]

6 hours ago, LAwLz said:

The problem is that 90% of people who think they "do things properly", don't. They also lack the knowledge to even realize what they are doing wrong.

I don't quite think it is that high and a lot of it comes from only being able to do the best they can with the resources and funding given.

 

Also there is a systemic issue of consulting companies that come in that know next to knowing about the business and insist that migrations aren't as complex and difficult as the employees say, that cloud is always cheaper (always) and when challenged go to managers after meeting and say the team is just being negative and obstructive only trying to protect their jobs when the reality is nobody in IT cares about any of that and it's not difficult to change from administering XYZ on-prem system to ABC Cloud system. It's not like anyone massive panics about switching from old FC arrays from vendor A to FCoE/NFS/iSCSI arrays from Vendor B that is entirely different to configure and operate. Which this rhetoric stems from Microsoft and AWS to undermine people that aren't "onboard"  and "Cloud, Cloud Cloud" evangelists. 

 

One has to be careful when saying people don't know properly in case the one saying it is in fact the one that doesn't know.

 

Twice now we've had an "independent" (not of ITS selection) external company come in and audit our infrastructure, install software on all our servers to collect usage data and then them generate a list of servers and their mapped Azure instanced size and "cost".  After this meetings are called and the very first thing they insist on as the foundation of all discussions and meetings is that hosting VMs (IaaS) in the Cloud is ALWAYS cheaper than on-prem. Epidemic of morons who then go on to manipulate the data in very easily proven ways to try and fit their narrative. It's gone nowhere twice because funnily enough we actually do know how to evaluate solution cost, complexities and requirements and make sensible recommendations.

 

It's also quite insulting to be told we aren't doing enough in the Cloud and we need to rethink how we do things when for one we have more than 600TB of data in Microsoft 365 alone, not including anything from our Azure and AWS tenants, and then when we want to and try to do any significant project to migrate something we think makes sense the funding and project approval never happens.

 

I am very critical of Microsoft and AWS because I consider them hostile actors that undermine employees that actually try and do what is best for the business and actually have the understanding of it and a stake in it's success.

 

6 hours ago, LAwLz said:

I strongly disagree. I think it is foolish to say one is "more correctly and better operated" because it entirely depends on how you use it.

It's like saying screwdrivers are better operated than hammers. 

That's fine you can disagree. while I also hope you are aware of the absolute glut of Cloud configuration issues that expose massive amounts of data, cyber attacks more easily taking over entire infrastructure and also deleting it. I could go on but this is not a how you use it issue, this is not a critique of how to best utilize Cloud for the right purpose. Even when used for the right things Cloud has over the recent history more instances of substandard operations leading to actual major issues.

 

Just the number of exposed publicly accessible S3 buckets with sensitive data even right now is is pretty condemning of just how badly Cloud resources are operated. 

[SansVarnic]

Cloud storage is one thing but entrusting a superannuation fund to this as the primary is absolute asinine.

Such things should be kept on-prem not relying on a 3rd party cloud, off site back is all right but entrusting google cloud as your primary or secondary is ... I haven't the proper word for how fail this failure is. 

I don't know this company but whoever heads the board of this fund needs to be investigated for gross negligence, if I were an investor I would be demanding it.

 

 

[Thaldor]

Alphabet when when someone tried to recover YouTube or Google Cloud account data:

"Sorry, your data has been erased, no can do."

 

Alphabet when it's literally anything else, even regarding YouTube and Google Cloud:

[SansVarnic]

I haven't forgotten when Google claimed that a certain small business's data (I've forgotten the name) belonged to google because the data was kept on google servers. Almost ruined the business when they couldn't access their data.

[wanderingfool2]

15 hours ago, SansVarnic said:

Cloud storage is one thing but entrusting a superannuation fund to this as the primary is absolute asinine.

Such things should be kept on-prem not relying on a 3rd party cloud, off site back is all right but entrusting google cloud as your primary or secondary is ... I haven't the proper word for how fail this failure is. 

I don't know this company but whoever heads the board of this fund needs to be investigated for gross negligence, if I were an investor I would be demanding it.

 

 

Not really.

 

They had backups, actually they appear to have followed backup strategies quite well in this case.  They had Google Cloud as the primary service, followed up with a back-up secondary geolocation with Google and finally followed up with a 3rd party provider as well.

 

Having seen large corporations, many do not follow the actual practices this well.

 

Relying on Google can also be beneficial to an extent, such as having SLA's in place guaranteeing certain uptime with penalties if it's not hit.  Google being the size they are can likely offer a better overall reliability than the company themselves.

 

I do tend to like local over cloud services, but it's not like it's a terrible thing like gross negligence.

 

e.g. Twitter as an example, back when Mudge reported on it...if they had lost a few critical datacenters they would have lost Twitter itself pretty much [and didn't really verify backups].

 

Managing your own equipment has it's own risks vs rewards.  So really when it comes to the cloud it's more about does it fit the need for cost; and what contingencies are in place.

[leadeater]

3 hours ago, wanderingfool2 said:

Relying on Google can also be beneficial to an extent, such as having SLA's in place guaranteeing certain uptime with penalties if it's not hit.  Google being the size they are can likely offer a better overall reliability than the company themselves.

Not really, the size of this would have warranted infrastructure with the same or better actual up time to that of services hosted in GCP. Uptime reporting of public clouds are rather obfuscated because they want to report on it in the best light for themselves. For example you could have a SaaS instance for Fortinet Cloud firewall for S2S VPN to your on-prem network and there could be an issue of that service confined to your specific firewall instances that cause an outage or one way traffic flow (still an outage) and it's these issues which are not counted in publicized uptime metrics for public Cloud providers so those stats aren't a realistic representation of service uptime hosted with these providers and you can also be left with little insight and ability to diagnose issues like that other than to log a support case or destroy the instance and recreate, find that it work now and have to shrug and move on without any explanation as to why or how it broke in the first place.

 

Microsoft has violated their Office 365 SLA with us many times and we've only ever been compensated once, none of these providers are going to just pay out on SLA breaches without fighting it first and they have the upper hand because they have their own contradictory monitoring data and terms in their SLA's which confine outages to specific ways and types meaning your outage could be excluded.

 

For a huge amount of businesses these providers offer infrastructure services with a foundation of design and protection not feasible, but not in this instance. They have more virtual machines than where I work and we have multiple datacenters with redundant A+B power feeds and generators and VMware SRM DR failover with NetApp underlying storage and can bring everything back online within a few minutes with 30 minute recovery points. I doubt we have the IT funding that UniSuper does too.

[mr moose]

On 5/20/2024 at 7:16 AM, SansVarnic said:

I haven't forgotten when Google claimed that a certain small business's data (I've forgotten the name) belonged to google because the data was kept on google servers. Almost ruined the business when they couldn't access their data.

Google don't care a twat about any other business but their own.  The number of times their search results claim a business is permanently closed when it isn't, or is not open when it is is astounding, not too mention the plethora of fake reviews.   I like the idea of giving people useful data like independent 3rd party reviews and access to open hours, but they have to be accurate.