Sorry Canada, your IT failed.

[TechyBen]

Canada's latest attempt at a unified Payroll system has imploded. It's sad to see IT projects fall apart. Either through budget problems, feature creep or lack of knowledge and understanding.

 

http://www.theregister.co.uk/2018/05/29/canada_phoenix_payroll_system_audit/

 

Quote

 

Canada's top auditor has issued a scathing postmortem report on Phoenix: the nation's disastrous attempt to overhaul a key government IT system...

 

"Phoenix executives prioritized certain aspects, such as schedule and budget, over other critical ones, such as functionality and security."...

 

What started as a CAN$310m project would balloon by a further CAN$540m, and broke down soon after it went live in 2016. That's CAN$850m total, or $650m (£500m). The Trudeau government estimated the total cost to fix the system could hit CAN$1bn by 2022.

 

So a massive extra cost just to rip it out and start again. I know it's a bit of old news, as this is just a confirmation that it failed by the auditors at this point. But it got me wondering about things like how tiny little companies can get it right or wrong, in comparison to national or global companies getting it right or wrong. I guess scale makes a massive difference.

 

So the advice to LTTs and their "IT" systems? Keep it simple, because the bigger it is, the more you can drop!

 

[edit]

I just realised... this one needs to rise from the ashes! 

[Herman Mcpootis]

 

[Taf the Ghost]

Put the jobs of the people in charge of the change on the line, then you'll find most government programs can actually operate efficiently, on-time & roughly on budget.

 

Lack of responsibility to outcomes is the biggest issue with all organizations. Lack of "skin in the game" creates very troublesome conflicts of interest, for the health of the outcome of the project.

[mr moose]

This is actually why many business avoid updating and upgrading unless they have to.    LTT are generally quick to poo poo company X because they still use XP or refuse to try Linux/swap to AMD servers/workstations,  But the reality is sometimes it only takes a few small unexpected issues to cost $100's of millions.

[TechyBen]

And I guess why LTT videos can take months to make and get ready. If LTT followed on the "just release it" (every time), then a lot of content would just be broken unedited and content lacking clips. Taking 2-3 months to get all the parts, shots and full story ready is appreciated I think.

[Teddy07]

I had a high opinion of Canada but they seem to fail in the recent years more often. I don´t want to make my post too political

[TechyBen]

2 minutes ago, Teddy07 said:

I had a high opinion of Canada but they seem to fail in the recent years more often. I don´t want to make my post too political

It's IT projects in general. They all seem to implode when getting too big, if the management side of things is not well structured or staffed with competent people.

 

That and it must be extremely difficult if your a project manager/outsourced firm and the organisation/client is asking for an "on time" delivery, and that just ain't possible.

[Ezzy-525]

Is your national payroll system just not quite up to the task?

 

Then you need Freshbooks! Whilst freshbooks doesn't actually do payroll, there's probably some kind of addon to get your Canadian rupees flowing in no time. Freshbooks!

 

For 10% off your first national trial just enter the code "LinusFixedIt" or follow the link in the video description.

 

*Sugar High by Approaching Nirvana fades in and begins to play*

[WkdPaul]

That's not news to any Canadian, that fiasco has been in the news for quite some time now ;

 

http://www.cbc.ca/news/politics/auditor-general-phoenix-refugees-1.4409978

[AshleyAshes]

Canadians: Oh look, the Americans only just now heard of Phoenix. :V

[ScratchCat]

Quote

Phoenix executives cancelled a pilot project with one department that would have assessed whether Phoenix was ready to be used government-wide.

Government project over budget and non functional? Maybe they should have adequately tested it before deployment as well as checking that it would work on a larger scale.

[leadeater]

59 minutes ago, Taf the Ghost said:

Put the jobs of the people in charge of the change on the line, then you'll find most government programs can actually operate efficiently, on-time & roughly on budget.

 

Lack of responsibility to outcomes is the biggest issue with all organizations. Lack of "skin in the game" creates very troublesome conflicts of interest, for the health of the outcome of the project.

Actually that will change little, let the technical people make the technical decisions and as a manager if you don't understand learn to shut up and run with the advice. So much time is wasted explaining stuff that simply cannot be understood without the years of technical background and training.

 

/rant

Also stop hiring business analysts for what is actually a system analyst role, whats the difference you ask? BA = Business and Processes, SA = Technical and Architectural. Please stop mixing them up and hiring the wrong people. If it's actually a SA role you need don't advertise it under a BA job listing.

/rant

 

Edit:

Fear of the responsibility and making bad decisions makes 30 minute decisions take 4 weeks, honestly seen this very thing many times.

[asus killer]

government bureaucracy, inefficiency, overall size, slow to progress, inability to deal with the simplest of budgets and contracts and for last it probably ended in a departmental feud (no we don't want it this way, say them all). It could happen in any country anywhere.

[mynameisjuan]

25 minutes ago, leadeater said:

Fear of the responsibility and making bad decisions makes 30 minute decisions take 4 weeks, honestly seen this very thing many times.

Everyday for me at work while I just end up provisioning anyway until I get tired of waiting the weeks and just bring it to the manager and just get it done. 

[Brooksie359]

1 hour ago, mr moose said:

This is actually why many business avoid updating and upgrading unless they have to.    LTT are generally quick to poo poo company X because they still use XP or refuse to try Linux/swap to AMD servers/workstations,  But the reality is sometimes it only takes a few small unexpected issues to cost $100's of millions.

Yeah it's sadly true. I know the place I work at just started testing switching to Windows 10 and there are a ton if issues that we are running into for the programs we use. It makes you think that maybe it's not so insane that companies are wary of switching to something new especially if it is untested for their use case. 

[Oggy]

I've worked with some Canadian IT in the past. All were fired due to incompetence and repeated screwups.

 

They do this thing, becoming fixated on a color or shape and begin repeating some rtard mantra. They putter around like somebody thats been up for days and are generally slow in work.

[Tribalinius]

55 minutes ago, leadeater said:

Actually that will change little, let the technical people make the technical decisions and as a manager if you don't understand learn to shut up and run with the advice. So much time is wasted explaining stuff that simply cannot be understood without the years of technical background and training.

 

/rant

Also stop hiring business analysts for what is actually a system analyst role, whats the difference you ask? BA = Business and Processes, SA = Technical and Architectural. Please stop mixing them up and hiring the wrong people. If it's actually a SA role you need don't advertise it under a BA job listing.

/rant

 

Edit:

Fear of the responsibility and making bad decisions makes 30 minute decisions take 4 weeks, honestly seen this very thing many times.

Clearly, the project was handled poorly. A lot of people had/still have issues not being paid for what, 2 years now? That software went live without any safety net and no way to revert to the previous platform. Some heads need to roll, if you are hired by the government, you are supposed to be the cream of the crop period.

If it's anything like the Quebec "Health" IT project we have here, there was a lot of external consultants / friends of friends / egos implicated in the project and somehow I feel they forgot the initial scope of the project. Of course, everyone in the project took home a huge slice of the pie and still do because they have been dragging the project for years now, i guess it's the same for Phoenix unfortunately.

I'm ashamed to see such an IT project go to waste. It gives us, IT professionals, really bad rep again. The project, if it would have been handled properly, had a really good potential. It gives me the impression that some external influencers decided to butt in the project and keep piling demands and asking for features when the analysis was done. We all had these people in our businesses doing that. Project ABC is nearing completion but Timmy from accounting decides all of a sudden that he's an IT specialist and wants feature XYZ deployed in the first version of the software.

[samcool55]

27 minutes ago, Brooksie359 said:

Yeah it's sadly true. I know the place I work at just started testing switching to Windows 10 and there are a ton if issues that we are running into for the programs we use. It makes you think that maybe it's not so insane that companies are wary of switching to something new especially if it is untested for their use case. 

The issue is that due to the way Windows 10 handles upgrades. Every 6 months you risk stuff breaking again.

Today a few systems that run W10 upgraded to 1803. All of them broke... It was an easy fix but still, it was downtime that wouldn't have happend if it updated the same way like W7 and W8.

[WkdPaul]

34 minutes ago, Tribalinius said:

I'm ashamed to see such an IT project go to waste. It gives us, IT professionals, really bad rep again.

That's what I hate about stuff like this, similar to some stuff that happened at my previous employer. Chances are, the IT guys did their job correctly and warned the heads that it wasn't ready but some top brass told them it would be fine, that they shouldn't worry so much, yeah right, heard that one before!

 

I've seen it many times where the top and middle management decides on some big project without consulting their own IT department, and then we (IT guys) take the fallout for the fuck ups they did and that they simply had no business in deciding! As I said earlier, previous employer decided to change the inventory and billing software, then told us (IT) 6 months AFTER the contract was signed ... sigh ... the live deployment happened 2 years late AND made all the customers angry, they ended up closing a few branches because of that! That's how I lost my job but found a much better one, so not all bad news I guess.

 

And if it's anything like I've seen elsewhere, the top management people that took all those decisions won't be hold accountable for any of it.

[mynameisjuan]

41 minutes ago, samcool55 said:

The issue is that due to the way Windows 10 handles upgrades. Every 6 months you risk stuff breaking again.

Today a few systems that run W10 upgraded to 1803. All of them broke... It was an easy fix but still, it was downtime that wouldn't have happend if it updated the same way like W7 and W8.

Sounds like a lot of custom programs. I used to work at a large corporation that had a lot of custom programs. I started at T1 and ended at a sys admin so I have worked with it all. On the PCs with custom programs, yeah shit would break, PCs without no problems at all. 

 

But I was a hardass and kept to security standards. I gave hard dates (due to the info we held) and if you didnt have the test PC I provisioned working. Well, sucks for you....better get working on it. 

[shermantanker]

This reminds me of the ObamaCare website rollout.

[GoodBytes]

As mentioned, this news is VERY old.

And this has nothing to do with "IT of Canada". The government (previous) wanted to modernize the payroll system for ALL departments at the SAME time. However, such massive undertaking requires collaboration between all the departments, to end up with this massive set of requirements, with the hope that no one forgets anything, which can be very complicated to include afterwards as it could change drastically the algorithms to define pays). It also requires a very high level of expertise. Sadly, in Canada, governmental contracts are signed from the lowest bidder, not a relationship between price and reputation of quality of work, or consideration of past failures, even if the government had a bad experience in the past (unless it was real bad, and now the government puts the company in a ban for a few years). Anyway, they signed with the lowest bidder, which same company had the same task at hand back in Australia (or some other country, I don't recall for sure), and it flopped equally in similar fashion.

 

Despite the hard to swallow pill, it is best for the government to pull out, and undo everything, costing a lot of money, than to stick with it, as it was done in the past, and end up paying a lot more, and still have issues.

[leadeater]

@Tribalinius @wkdpaul

Another issue that really gets to me is the way tender processes are done to select the software vendor to go with. There are a lot of things you aren't allowed to do or steps that don't get done that seem rather fundamental to enabling the selection of the best fit product.

 

Like for our new Student Management System, it was all done in secret and none of the infrastructure support team or application support team were allowed to know what the options were so we could investigate how the proposed systems would fit in to our environment, what systems it would need to interact with and how. How are you supposed to make an informed decision and pick the best product that is actually supportable when you never bother to find those things out, yes the business side of it is the most important by far but it does you no good to select a product that just will not work no matter how much shoehorning you try and do.

 

Then to top that off no one is allowed to ask any other businesses that use the proposed software what it's like, how they implemented it, what issues they encountered, nothing.

 

The above project was multiple tens of millions of dollars, not allowed to say exactly how much and was literally dropped on us over night. Not kidding the first we found out a product was selected was when they walked in and demanded 100 VMs be created immediately and the project was starting that day. 

 

Good decisions can not be made in a vacuum.

[XenosTech]

2 hours ago, mynameisjuan said:

Everyday for me at work while I just end up provisioning anyway until I get tired of waiting the weeks and just bring it to the manager and just get it done. 

I just do it and either get scolded for it or they just give a pat on the back.... Sometimes both at the same time.

[samcool55]

1 hour ago, mynameisjuan said:

Sounds like a lot of custom programs. I used to work at a large corporation that had a lot of custom programs. I started at T1 and ended at a sys admin so I have worked with it all. On the PCs with custom programs, yeah shit would break, PCs without no problems at all. 

 

But I was a hardass and kept to security standards. I gave hard dates (due to the info we held) and if you didnt have the test PC I provisioned working. Well, sucks for you....better get working on it. 

Sadly that won't do much. Most of the programs are provided by third parties and for some reason most of them have crappy support. Doesn't matter how big you are, you can't exactly demand a hard date from companies like HPE or IBM, or proper support...

 

Also, how tf did you end up in a position in a large company where you could demand hard dates? Our IT department isn't 1 person but nowhere near enough to offer actually proper support.

 

Funny detail: some issues went all the way up to the chain, to the big boss, and he actually gave a green light for multiple things that would help us, but the rest of management said no, so nothing happened...