Google Cloud accidentally deletes $125 billion Australian pension fund’s online account due to ‘unprecedented misconfiguration’

[scissors]

Summary

Unisuper is a superannuation fund (think pension fund) that manages retirement savings for members of Australia’s higher education and research sector. With 620,000 members and $125 billion in assets, they’re Australia’s 13^th largest by number of members and 7^th largest superannuation fund by assets managed. For the last week, Unisuper members have been unable to access their accounts due to what turns out to have been the wholesale deletion of the fund’s Google Cloud account. The fund had to recover their data from a backup at a different provider, as the deletion affected two redundant instances stored on Google’s service. Account access is slowly being restored. It isn’t clear at this point what exactly happened to cause this incident, other than that Unisuper and Google Cloud have issued a joint statement to the effect that it was due to an “unprecedented sequence of events.” Unisuper had migrated a large proportion of its operations to Google Cloud Platforms in 2023.

 

Quotes

Quote

 “More than half a million UniSuper fund members went a week with no access to their superannuation accounts after a “one-of-a-kind” Google Cloud “misconfiguration” led to the financial services provider’s private cloud account being deleted, Google and UniSuper have revealed.”

 

“The UniSuper CEO, Peter Chun, wrote to the fund’s 620,000 members on Wednesday night, explaining the outage was not the result of a cyber-attack, and no personal data had been exposed as a result of the outage. Chun pinpointed Google’s cloud service as the issue.”

 

“Google Cloud CEO, Thomas Kurian has confirmed that the disruption arose from an unprecedented sequence of events whereby an inadvertent misconfiguration during provisioning of UniSuper’s Private Cloud services ultimately resulted in the deletion of UniSuper’s Private Cloud subscription,” the pair said.”

 

“While UniSuper normally has duplication in place in two geographies, to ensure that if one service goes down or is lost then it can be easily restored, because the fund’s cloud subscription was deleted, it caused the deletion across both geographies. UniSuper was able to eventually restore services because the fund had backups in place with another provider.”

 

“UniSuper has approximately $125bn in funds under management.”

 

My thoughts

At least they had backups outside Google? Superannuation in Australia is a highly regulated industry, providing a lot of incentive to do things properly. What would have happened in a less regulated environment? It certainly drives home the idea that “backups” stored in the same infrastructure as the primary data aren’t actually backups. It remains to be seen who has what share of the blame, but it's certainly an indication of how vulnerable this kind of data can be without the proper processes in place to secure it.

 

Sources

https://www.theguardian.com/australia-news/article/2024/may/09/unisuper-google-cloud-issue-account-access

https://www.datacenterdynamics.com/en/news/unisuper-private-cloud-outage-caused-by-google-cloud-issues/

[soldier_ph]

That's what they get for trusting Google, of all people, with their Money.

[GoStormPlays]

1 minute ago, soldier_ph said:

That's what they get for trusting Google, of all people, with their Money.

poor youtubers

[Fasterthannothing]

What in the world how do you screw up that badly 

[Dabombinable]

And that is why cloud services should never be relied on for anything.

[MadAnt250]

6 hours ago, soldier_ph said:

That's what they get for trusting Google, of all people, with their Money.

 

[LAwLz]

2 hours ago, Dabombinable said:

And that is why cloud services should never be relied on for anything.

Yes, because as we know on-prem setups are always 100% flawlessly configured and never have any issues...

 

 

In case someone think I am serious, a recent estimate said that around 25% of all on-prem Exchange servers were not up to date with security patches and had big security vulnerabilities in them. I suspect the number for other on-prem services is about the same. That number doesn't even count misconfigurations which probably would result in an even higher number of vulnerabilities.

People who think they do a better job than Google/Microsoft at configuring services properly are most likely naive and don't understand the complexity of the subject at hand. Not to mention the 20/20 hindsight and all that...

[leadeater]

1 hour ago, LAwLz said:

In case someone think I am serious, a recent estimate said that around 25% of all on-prem Exchange servers were not up to date with security patches and had big security vulnerabilities in them. I suspect the number for other on-prem services is about the same. That number doesn't even count misconfigurations which probably would result in an even higher number of vulnerabilities.

Security vulnerabilities really aren't the same as entire infrastructure outages. For one not all out of date systems can be exploited (not accessible), can't be exploited in a way to destroy anything or cause an outage etc.

 

Complete and utter 'oops isn't gone' it pretty rare anywhere. That said when it happens in the cloud the most common outcome is it's gone forever with no recovery possibility because managers and business owners get sucked in to thinking that Azure, AWS, GCP have all this resiliency making backup unnecessary. Thankfully not the situation here.

 

12 hours ago, scissors said:

“While UniSuper normally has duplication in place in two geographies, to ensure that if one service goes down or is lost then it can be easily restored, because the fund’s cloud subscription was deleted, it caused the deletion across both geographies. UniSuper was able to eventually restore services because the fund had backups in place with another provider.”

 

 

If you are going to or have to utilize the same Cloud service provider for backups as your production systems then you need to have your backups in a different, not linked, subscription. But it's better, like above, to have it somewhere else.

[Mark Kaine]

10 hours ago, Fasterthannothing said:

What in the world how do you screw up that badly 

 

13 hours ago, scissors said:

Google

 

.

[Mark Kaine]

4 hours ago, Dabombinable said:

And that is why cloud services should never be relied on for anything.

how did people not learn this lesson...?

 

back when it all was "new" i tested it, uploaded something... worked fine... few hours late... "gone"!  Nope, apparently i never uploaded anything to "google drive"...

 

using mega since then, works perfectly fine, has much better UI. ¯\_(ツ)_/¯ 

 

ps: i would still never use this to "keep" something important...

[wanderingfool2]

4 hours ago, leadeater said:

If you are going to or have to utilize the same Cloud service provider for backups as your production systems then you need to have your backups in a different, not linked, subscription. But it's better, like above, to have it somewhere else.

Also nice to have a good SLA...while it doesn't prevent the whoops issues, it at least gives you someone you can go after for the downtime caused.

 

Generally all this really has shown is what you said.  Backups are king, but would like to add as well, having properly tested backups is crucial as well to prevent the other "oopsies" of "our backup's don't work".

[Senzelian]

Shit happens.

 

Anyone that ever worked in IT knows exactly how this can easily happen at any time.

[porina]

20 hours ago, scissors said:

Unisuper and Google Cloud have issued a joint statement to the effect that it was due to an “unprecedented sequence of events.”

I'm getting Air Crash Investigation vibes from that line, and wonder if they'll ever disclose what sequence of events lead to this happening. This isn't one mistake, but likely a sequence of mistakes and missed opportunities to detect and correct.

[LAwLz]

9 hours ago, leadeater said:

Security vulnerabilities really aren't the same as entire infrastructure outages. For one not all out of date systems can be exploited (not accessible), can't be exploited in a way to destroy anything or cause an outage etc.

Of course. What's your point?

 

 

9 hours ago, leadeater said:

Complete and utter 'oops isn't gone' it pretty rare anywhere. That said when it happens in the cloud the most common outcome is it's gone forever with no recovery possibility because managers and business owners get sucked in to thinking that Azure, AWS, GCP have all this resiliency making backup unnecessary. Thankfully not the situation here.

That's an issue with management and IT planning, not cloud providers. 

 

 

My point is that whenever stories like these appear there are always a bunch of people who come crawling out and go "this is why the cloud sucks!" when the on-prem solutions most likely have a bunch of issues too. 

It's the whole "something has to be either good or bad" attitude I have an issue with, especially since people who fall into the "I have decided that I dislike X" camp usually refuse to see issues with the alternative and will be very quick with showing their heads to gloat when something bad happens related to "X". 

 

 

Do you agree with the person I replied to when they said cloud services should never be relied on for anything? 

 

 

 

4 hours ago, wanderingfool2 said:

Generally all this really has shown is what you said.  Backups are king, but would like to add as well, having properly tested backups is crucial as well to prevent the other "oopsies" of "our backup's don't work".

As my colleague to manages our backup service usually says, "it's not backed up until we have done a restore". 

[leadeater]

1 hour ago, LAwLz said:

Of course. What's your point?

I was questioning yours. What was your point about security vulnerabilities in a story about total loss of everything? My reply was a long form and respectful "so what?". What you gave was a false equivalency. There is no strong correlation between software vulnerabilities and hardware infrastructure maintenance and lifecycle management, or complete failures thereof.  

 

1 hour ago, LAwLz said:

My point is that whenever stories like these appear there are always a bunch of people who come crawling out and go "this is why the cloud sucks!" when the on-prem solutions most likely have a bunch of issues too.

It's extremely rare for everything to be gone when it's on-prem, it's more often recoverable compared to the cloud where data is stored specifically in a way that it cannot be recovered. And any attempts to do so would be denied because you'd have to offline the entire storage zone and try a data recovery operation which again would be pretty futile given how data is stored.

 

While people may have a bad attitude about Cloud services these do legitimately come with different risk profiles and concerns.

 

I just didn't find your point about security vulnerabilities to be all that useful or relevant because as I said, they almost never lead to this type of situation and I doubt you could point to one either. Neither is it strong evidence beyond issues with vulnerability and software patch management.

 

Security vulnerabilities aren't the same situation, misconfiguration causing service outage is not the same either. What is the same is for example cascading disk failure cause total data loss, administrative error deleting the wrong LUN or entire aggregate/pool. Thing is most enterprise storage systems have a 'recycle bin' and put all LUN, Volume and Pool/Aggregate deletes in a pending hold for 24 hours (or more) so you can revert a mistake.

 

Cloud providers also often have this capability for everything with the exception of a subscription deletion.

 

1 hour ago, LAwLz said:

Do you agree with the person I replied to when they said cloud services should never be relied on for anything? 

The way the vast majority of how it's used yes, because like I pointed out huge majority that migrate to cloud services forgo backups because of the promises of up time and geo-replication etc etc and don't read the SLA's and exceptions within them or the buried advice they don't tell you up front they put in solely to cover their ass i.e. Azure/Microsoft.

 

Microsoft on their web documentation recommend backups, they will NEVER give this advice to you during consultations to get you to move from on-prem Exchange to Office 365/Azure, never ever. Not a single time did they with us or anyone else we spoke to. We used Office 365 for 3 years without backups because of the promises made from Microsoft directly to our CIO and that argument continued and lost every time until we experienced unrecoverable data loss with someone rather important.

 

Most of the issues around "Cloud" stem from naivety due to how "new" it is unlike on-prem infrastructure which has decades of lessons learnt behind it. The first step is realizing it's just someone else's "computer"/"infrastructure" that you are paying to use, it is not "different" to what you had before that you owned. Most of the do's and don't are pretty well transferable to Cloud, at least at the basic level.

 

1 hour ago, LAwLz said:

That's an issue with management and IT planning, not cloud providers. 

Yes it is an issue with Cloud provides, they are the root source of this narrative. Cloud providers have historically been bad faith actors and bear a lot of blame for leading people in to higher risk situations that they are unaware of.

 

The long and short of it is while issues exist everywhere it is easier and more likely to lose everything in a Cloud provider. This is something few like to hear, accept and will counter argue without any real objective thought in to how and why. If you are going to use Cloud then you need to do it right, understand the differences, understand the risks, understand the benefits, actually know why you are doing it in the first place (Industry trend is not a legitimate 'why', see all the IT Magazine and article coverage on the 'Cloud Exit" wave happening recently).

[Kisai]

14 hours ago, Dabombinable said:

And that is why cloud services should never be relied on for anything.

Cloud services should only used as front end (eg CDN) services, not back end, and good god so many companies try to outsource their entire business to cloud services because it saves people/maintenance costs, but instead it buries them in their own graves if something ever fails at the cloud service provider.

 

Same with VPS servers. They are junk, do not use them for anything business critical. Store your valuable data on-premises, on physical hardware you own.

[LAwLz]

2 hours ago, leadeater said:

I was questioning yours. What was your point about security vulnerabilities in a story about total loss of everything? My reply was a long form and respectful "so what?". What you gave was a false equivalency. There is no strong correlation between software vulnerabilities and hardware infrastructure maintenance and lifecycle management, or complete failures thereof.  

My original reply about security vulnerabilities in on-prem systems was to highlight that no infrastructure, cloud or on-prem, is infallible. You're right that these vulnerabilities do not typically result in total system outages, they are indicative of a broader issue of management and maintenance that affects all types of IT infrastructure. The point of my post was to underscore that dismissing cloud solutions based on high-profile failures ignores the nuanced realities of both environments.

Highlighting the vulnerable Exchange servers was meant to challenge the dismissive attitude toward cloud solutions when they fail, by pointing out that on-prem systems also face significant challenges. Those challenges might not be the same challenges as cloud service, but they exist nonetheless and should be factored in.

 

 

Regarding the rarity of total data loss on-prem, while it's true that such events might be less common, they are not unheard of. On-prem systems can suffer from catastrophic failures due to natural disasters, physical damage, or severe human errors which might not be as recoverable as you suggest. In contrast, cloud providers often deploy extensive disaster recovery protocols across multiple geographically distributed data centers, enhancing potential resilience against such catastrophic events. That is not always feasible to do for an individual company, especially smaller ones.

 

 

 

2 hours ago, leadeater said:

Most of the issues around "Cloud" stem from naivety due to how "new" it is unlike on-prem infrastructure which has decades of lessons learnt behind it. The first step is realizing it's just someone else's "computer"/"infrastructure" that you are paying to use, it is not "different" to what you had before that you owned. Most of the do's and don't are pretty well transferable to Cloud, at least at the basic level.

I both agree and disagree with this. I think it entirely depends on what infrastructure we are talking about. "The cloud" is more than just renting CPU and memory. It also has a lot of pre-packaged services like Office 365.

If you are renting CPU, memory, storage and so on then a lot of the responsibility to configure it properly is on you as the system administrator, even though this message might not always be clear (because salespeople want to get you to buy their stuff and say everything is easier).

If we are talking about the pre-packaged services however then I would argue that, while some of the responsibility is still on the sysadmin, the challenges are new and it is a new thing compared to what others had before on-prem. Office 365 is very different from hosting everything on-prem with a Skype server, Exchange server, and so on.

 

 

3 hours ago, leadeater said:

The way the vast majority of how it's used yes, because like I pointed out huge majority that migrate to cloud services forgo backups because of the promises of up time and geo-replication etc etc and don't read the SLA's and exceptions within them or the buried advice they don't tell you up front they put in solely to cover their ass i.e. Azure/Microsoft.

 

Microsoft on their web documentation recommend backups, they will NEVER give this advice to you during consultations to get you to move from on-prem Exchange to Office 365/Azure, never ever. Not a single time did they with us or anyone else we spoke to. We used Office 365 for 3 years without backups because of the promises made from Microsoft directly to our CIO and that argument continued and lost every time until we experienced unrecoverable data loss with someone rather important.

I think it is worth mentioning that the company I work for sells Office 365 backup solutions as a service. So part of our work is educating customers about the exact thing you are describing as being lacking. That might be a reason why we have such different views on this, because I don't think (or didn't know) that this was an issue.

We are also a Microsoft partner that sell Office 365 but we never let Microsoft talk to our customers alone. I work in networking and I never let companies like Cisco or Fortinet talk to the customers I am responsible for alone either, because I do not trust marketing people. 

 

I would always advise the people who make IT-related decisions to do due diligence and not solely rely on the promises from some vendor.

While there have been instances of cloud providers failing to fully disclose risks, it’s should also be on the businesses to perform due diligence. The narrative that cloud providers are solely to blame does not fully account for the responsibility of businesses to critically assess and plan their infrastructure needs. Both providers and users share responsibility for the outcomes.

 

 

My objection to the initial response I replied to was, as I said earlier, that dismissing cloud solutions based on high-profile failures ignores the nuanced realities of both environments. Each has its place and utility, and understanding both environments' strengths and weaknesses is essential for making informed IT decisions.

 

 

And for disclosure purposes, the company I work for sell a lot of consulting time to do maintenance on on-prem solutions. So I don't think there is any conflict of interest here. We get paid regardless of how some customer decides to implement their infrastructure. In fact, we probably get paid more to help them host stuff on-prem.

 

[bcredeur97]

since pretty much every business in the US is using cloud services in one way or another, imagine we wake up one morning and MS announces that "the hackers deleted everything in an unprecedented way that we never knew could happen"

The world economy would literally crash and burn lol

There's waaaayyyyy too many eggs in one basket.


You know 100% they are relying on all their redundancies and don't actually have real cold storage backups lol

[jagdtigger]

2 hours ago, LAwLz said:

The point of my post was to underscore that dismissing cloud solutions based on high-profile failures ignores the nuanced realities of both environments.

And your post ignores that on self host it is very hard to make the same mistake. Google, MS, and the rest relies almost entirely on automated systems to handle literally everything without human oversight. And their systems lack the ability to restore accidentally deleted things as the example shows.
Sure, it is a shared responsibility model. But when the hosting provider screws up this hard, and at this point many of the so called cloud providers did, very much warrants distrust and abandonment.

[leadeater]

3 hours ago, LAwLz said:

You're right that these vulnerabilities do not typically result in total system outages, they are indicative of a broader issue of management and maintenance that affects all types of IT infrastructure.

Well I'm contesting that it does not actually correlated to issues such as this so it doesn't point to it, or much else. You can be very well on top of your infrastructure life cycle, maintenance, configuration, change control etc but simply be behind in software patches and vulnerabilities due to multiple different reasons or just not have the time to do that over other things or view them with lesser importance.

 

The existence of out of date/vulnerable systems is itself a huge and complicated topic and those who point it out the most tend to not have the most pure motives. Like for example Microsoft or a Microsoft influenced publication pointing out how all these vulnerable Exchange servers exist and it would be better to use Office 365 without any regard to the circumstances to those vulnerable system or for how long or any possible vector of attack of them at all etc. Valid points and issues can be misused, and do get misused.

 

Neither am I unfamiliar with such On-Prem disasters either. Where my father works is a hospital who suffered through years of under funding of IT budgeting and directly getting blocked by Government overnight bodies to replace critical infrastructure because "the grand plan" was Cloud first and always regardless that it wasn't affordable or funded correctly (OpEx vs CapEx) or that transitioning takes a lot of time and care. While all of this meddling, arguing and funding restrictions were going on they had a cascading disk failure that took all critical IT systems down, for days. No data was lost and the storage system was able to be repaired and brought back online.

 

Microsoft and AWS political meddling have a lot to answer for, they are callous and self-centered and only care about their bottom line. It is their influence that caused this. And who do you think took the blame?

 

3 hours ago, LAwLz said:

The point of my post was to underscore that dismissing cloud solutions based on high-profile failures ignores the nuanced realities of both environments.

Cloud services have had a higher rate of high profile issues like this than on-prem for the reasons I have pointed out. Recovery options are greatly less with Cloud compared to on-prem so you need to be making sure you are doing a better job compared to on-prem in regards to Governanance, which if it's poor on-prem is not going to be better utilizing Cloud.

 

It didn't underscore anything, security vulnerabilities are so much different issue to anything like this or even most minor outages. I'd posit most minor on-prem outages are administrative error and few of those could ever result in anything this bad.

 

3 hours ago, LAwLz said:

I think it is worth mentioning that the company I work for sells Office 365 backup solutions as a service. So part of our work is educating customers about the exact thing you are describing as being lacking. That might be a reason why we have such different views on this, because I don't think (or didn't know) that this was an issue.

Microsoft was literally telling our CIO's and others that backups were not necessary due to Office 365 geo-replication and their very high service up times and their SLA.

 

It doesn't matter what you sell, you don't matter more than what Microsoft says.

 

I am unfortunately in a position most are not, at a organization that Microsoft themselves considers a client of significant importance warranting direct influence and engagement by Microsoft executives very high up the chain. That is true of all the Universities here in my country, Microsoft and AWS are both whining and dining much to our (ITS engineers etc) protest.

 

3 hours ago, LAwLz said:

We are also a Microsoft partner that sell Office 365 but we never let Microsoft talk to our customers alone. I work in networking and I never let companies like Cisco or Fortinet talk to the customers I am responsible for alone either, because I do not trust marketing people. 

I cannot stop my CIO or Microsoft talking with each other directly, that is impossible. I cannot also stop Microsoft going even higher up the chain to the Vice Chancellor and spinning them all sorts of wonder stories that benefit Microsoft, not us.

 

You, I or anyone else can't stop anything. You can try, that is all you/we can.

 

3 hours ago, LAwLz said:

In contrast, cloud providers often deploy extensive disaster recovery protocols across multiple geographically distributed data centers, enhancing potential resilience against such catastrophic events. That is not always feasible to do for an individual company, especially smaller ones.

Which does completely nothing at all for a subscription deletion, the most common way everything is lost in the Cloud and has happened to many through configuration errors like this to cyber attacks. Companies have actually ceased operation due to cyber attack taking control of their Cloud administration and deleting the subscription which had everything including their backups in it.

 

While on-prem you will pretty much never find a single "delete everything" button or tool and will actually both delete actually everything while also doing it in a way that it cannot be recovered. This is objective fact, that's just the fundamental difference in on-prem and Cloud. Cloud automation and orchestration along with multi-tennant isolation and encryption very easily and quickly allows you to delete and lose everything, the hand giveth and the hard taketh.

 

3 hours ago, LAwLz said:

"The cloud" is more than just renting CPU and memory. It also has a lot of pre-packaged services like Office 365.

Doesn't change any of the basics of how to treat it. Fancy web portals and someone else taking care of the software maintenance doesn't actually change how it should be risk evaluated. Does you no good to have all the resiliency and multiple service endpoints when your connection out doesn't have any. While on-prem you would have had HA etc for internal calling, internal emailing, chat (SFB etc).

 

There are really basic things that just get over looked or hand waved away. These are the basics am I referring to, they do not change Cloud vs On-prem. Nobody cares more about your data than you do, nobody cares more about your service quality and up times than you do, nobody cares more than yourself. Some unfortunately have to learn that the hard way, and will only learn it the hard way.

[Dabombinable]

21 hours ago, LAwLz said:

Yes, because as we know on-prem setups are always 100% flawlessly configured and never have any issues...

 

 

In case someone think I am serious, a recent estimate said that around 25% of all on-prem Exchange servers were not up to date with security patches and had big security vulnerabilities in them. I suspect the number for other on-prem services is about the same. That number doesn't even count misconfigurations which probably would result in an even higher number of vulnerabilities.

People who think they do a better job than Google/Microsoft at configuring services properly are most likely naive and don't understand the complexity of the subject at hand. Not to mention the 20/20 hindsight and all that...

They deleted the main and backup instances, with no way to recover the deleted files at all - beyond a secondary backup on a completely different cloud service. This has nothing to do with configuration or vulnerabilities. Files can be just deleted arbitrarily without Google having any method to restore them.

[LAwLz]

7 hours ago, jagdtigger said:

And your post ignores that on self host it is very hard to make the same mistake. Google, MS, and the rest relies almost entirely on automated systems to handle literally everything without human oversight. And their systems lack the ability to restore accidentally deleted things as the example shows.
Sure, it is a shared responsibility model. But when the hosting provider screws up this hard, and at this point many of the so called cloud providers did, very much warrants distrust and abandonment.

I didn't ignore anything.

Yes, this mistake would be very difficult to make with self-hosting (and cloud hosting I might add, this has only happened once according to the article). My point was that if you are self-hosting you might avoid this risk and some others, but you also introduce other types of risks. Neither is a perfect solution and both have some strengths and weaknesses. 

"This solution has this issue so therefore we should never use it and instead use this"  is the post that ignores issues, not mine. By the way, just to be 100% clear, I am not saying cloud solutions are better than on-prem. What I am saying is that both have strengths and weaknesses and saying one is shit is reductive and a very naive view of a very complicated topic.

 

 

  

2 hours ago, Dabombinable said:

They deleted the main and backup instances, with no way to recover the deleted files at all - beyond a secondary backup on a completely different cloud service. This has nothing to do with configuration or vulnerabilities. Files can be just deleted arbitrarily without Google having any method to restore them.

Which is why you should have a proper backup system in place, and that is true for both on-prem and cloud systems.

Good thing that in this case, the pension fund did have proper backups. They had done their homework. 

 

 

 

  

7 hours ago, leadeater said:

I cannot stop my CIO or Microsoft talking with each other directly, that is impossible. I cannot also stop Microsoft going even higher up the chain to the Vice Chancellor and spinning them all sorts of wonder stories that benefit Microsoft, not us.

 

You, I or anyone else can't stop anything. You can try, that is all you/we can.

Cloud services are a tool. A tool whose effectiveness largely depends on how it is utilized. From your experiences, it sounds like much of your concern/issues stems from the way cloud services are promoted and pushed by salespeople, potentially leading to suboptimal or inappropriate use. I think it is an important distinction because it points to problems with how the tool is sold and implemented, rather than issues with the tool itself.

 

When used appropriately, cloud services can offer substantial benefits. The key lies in understanding the capabilities and limitations of cloud services and integrating them thoughtfully into an organization's overall IT strategy. This approach ensures that the cloud serves the organization’s needs effectively, rather than being led by for example Microsoft's salespeople's agenda.

 

It’s crucial for decision-makers like your CIO to critically evaluate the advice and motivations of vendors to ensure that any adopted solutions actually align with their strategic objectives and operational requirements. By doing so, organizations can leverage cloud technologies in ways that truly benefit their operations and avoid potential pitfalls highlighted by experiences like yours.

 

I often disagree with salespeople from various vendors. That is why I always insist on taking part in meetings when vendors want to talk to customers. Fortinet recently wanted to talk to a municipality I am a trusted advisor for and I told them, you are not allowed to talk to them without my presence. Because I know that what salespeople say might not align with what's best for the client. That doesn't mean I hate Fortinet and will say their products are shit. It just means I don't trust their salespeople to make the right choices for my client.

 

I might be lucky to be in a situation where the higher-ups trust me and listen to my advice over the advice of salespeople from various vendors. I completely understand that things might go really poorly if the salespeople were allowed to say and sell anything they wanted to my clients. It seems like that's where most of your frustration stems from. I don't think that is grounds to completely reject and hate the tools being sold though.

[Mark Kaine]

it's kinda similar to vpns, there are use cases, but you should never rely on it for "security" reasons...

 

web2.0 at work.  make money out of solutions (almost) nobody needs but wants lol 

[leadeater]

4 hours ago, LAwLz said:

I don't think that is grounds to completely reject and hate the tools being sold though.

I don't hate and reject any of the major Cloud provider platforms, I just know huge numbers of organizations and people are using them with a lot greater risk than they realize most often as a direct result of what these providers have told them. It doesn't matter how good these platforms are, how safely they can be used etc when the entities that own them actively and maliciously seek out customers without any regard to the harm they cause in that pursuit.

 

But of the two on-prem is more correctly and better operated than Cloud in ways that matter and in ways that have and do effect people.

 

4 hours ago, LAwLz said:

(and cloud hosting I might add, this has only happened once according to the article)

No it has not, there was a new topic even here about few years ago or so about a company, I think in either programming or a web platform it codes and delivered itself, that had it's subscription deleted and their only choice was to cease operation and not exist anymore.

 

Also when GCP say it's an isolated incident that hasn't happened before they do and can only mean in this specific way, which is true. Whatever happened likely only ever happened in this exact way this time, to them, GCP.

 

4 hours ago, LAwLz said:

It’s crucial for decision-makers like your CIO to critically evaluate the advice and motivations of vendors to ensure that any adopted solutions actually align with their strategic objectives and operational requirements. By doing so, organizations can leverage cloud technologies in ways that truly benefit their operations and avoid potential pitfalls highlighted by experiences like yours.

Impossible, we are subject to the directives of the Vice Chancellor and the Department of Internal Affairs, both directly influenced and poisoned by Microsoft. How do I know might you ask, the latter because it is the explicit directive and the former because we've had 3 different CIO's come in and at the start state that moving everything to the Cloud is not right for us and would not be cheaper to, then in 6 months do nothing other than repeatedly ask why we aren't wholesale moving everything to the Cloud and why we need all these datacenters and can't we get rid of them. Experienced CIO's, very experienced some from large multinational NGO's.

 

You simply cannot isolate out the tool/platform from the ones that own and operate it. That's not how things work in reality. Microsoft and AWS lobby at the highest levels and unlike trickledown economics their shit does flow down and does real harm. You gravely underestimate just how pervasive and persuasive these vendors are and you are absolutely kidding yourself if you think you can and will be involved in all discussions and decisioning making. I know for a fact Satya Nadella has had multiple meetings with our VC over the years and has also been in NZ on our campuses meeting our senior executives and board members, zero people from ITS involved with those ones, not even the CIO.

 

Azure/AWS, not so much GCP, are like Peanuts. Doesn't matter how good they are when peanut allergies are involved, no amount of arguing the nutritional good they have will change someone's allergic response and health risk. Yes I am aware this is not explicitly equivalent but the point is clear in the meaning and effects. 

[jagdtigger]

20 hours ago, LAwLz said:

I didn't ignore anything.

Yes, this mistake would be very difficult to make with self-hosting (and cloud hosting I might add, this has only happened once according to the article).

Yeah sure, if you exclude gdrive loosing several months of data and older "happenings" i cant remember off the top of my head..........
Companies go for cloud thinking the provider will take care their end of the bargain comptently, which so far AFAIK every bigtime provider proven to not do. We are back in the old days, if you want it done properly do it yourself.