Jump to content
Search In
  • More options...
Find results that contain...
Find results in...
TechyBen

Sorry Canada, your IT failed.

Recommended Posts

Posted · Original PosterOP

Canada's latest attempt at a unified Payroll system has imploded. It's sad to see IT projects fall apart. Either through budget problems, feature creep or lack of knowledge and understanding.

 

http://www.theregister.co.uk/2018/05/29/canada_phoenix_payroll_system_audit/

 

Quote

 

Canada's top auditor has issued a scathing postmortem report on Phoenix: the nation's disastrous attempt to overhaul a key government IT system...

 

"Phoenix executives prioritized certain aspects, such as schedule and budget, over other critical ones, such as functionality and security."...

 

What started as a CAN$310m project would balloon by a further CAN$540m, and broke down soon after it went live in 2016. That's CAN$850m total, or $650m (£500m). The Trudeau government estimated the total cost to fix the system could hit CAN$1bn by 2022.

 

So a massive extra cost just to rip it out and start again. I know it's a bit of old news, as this is just a confirmation that it failed by the auditors at this point. But it got me wondering about things like how tiny little companies can get it right or wrong, in comparison to national or global companies getting it right or wrong. I guess scale makes a massive difference.

 

So the advice to LTTs and their "IT" systems? Keep it simple, because the bigger it is, the more you can drop!

 

[edit]

I just realised... this one needs to rise from the ashes! :D

Link to post
Share on other sites

Put the jobs of the people in charge of the change on the line, then you'll find most government programs can actually operate efficiently, on-time & roughly on budget.

 

Lack of responsibility to outcomes is the biggest issue with all organizations. Lack of "skin in the game" creates very troublesome conflicts of interest, for the health of the outcome of the project.

Link to post
Share on other sites

This is actually why many business avoid updating and upgrading unless they have to.    LTT are generally quick to poo poo company X because they still use XP or refuse to try Linux/swap to AMD servers/workstations,  But the reality is sometimes it only takes a few small unexpected issues to cost $100's of millions.


QuicK and DirtY. Read the CoC it's like a guide on how not to be moron.  Also I don't have an issue with the VS series.

Sometimes I miss contractions like n't on the end of words like wouldn't, couldn't and shouldn't.    Please don't be a dick,  make allowances when reading my posts.

Link to post
Share on other sites
Posted · Original PosterOP

And I guess why LTT videos can take months to make and get ready. If LTT followed on the "just release it" (every time), then a lot of content would just be broken unedited and content lacking clips. Taking 2-3 months to get all the parts, shots and full story ready is appreciated I think.

Link to post
Share on other sites

I had a high opinion of Canada but they seem to fail in the recent years more often. I don´t want to make my post too political ;)

Link to post
Share on other sites
Posted · Original PosterOP
2 minutes ago, Teddy07 said:

I had a high opinion of Canada but they seem to fail in the recent years more often. I don´t want to make my post too political ;)

It's IT projects in general. They all seem to implode when getting too big, if the management side of things is not well structured or staffed with competent people.

 

That and it must be extremely difficult if your a project manager/outsourced firm and the organisation/client is asking for an "on time" delivery, and that just ain't possible.

Link to post
Share on other sites

Is your national payroll system just not quite up to the task?

 

Then you need Freshbooks! Whilst freshbooks doesn't actually do payroll, there's probably some kind of addon to get your Canadian rupees flowing in no time. Freshbooks!

 

For 10% off your first national trial just enter the code "LinusFixedIt" or follow the link in the video description.

 

*Sugar High by Approaching Nirvana fades in and begins to play*


DAEDALUS (2018 Refit) - Processor: AMD Ryzen 5 - 1600 @ 3.7Ghz // Cooler: Cooler Master Hyper 212 LED Turbo Black Edition // Motherboard: Asus RoG Strix B350-F Gaming // Graphics Card: Gigabyte GTX 1060 Windforce 6GB GDDR5 // Memory: 2 x 8GB DDR4 Corsair LPX Vengeance 3000Mhz // Storage: WD Green - 250GB M.2 SATA SSD (Boot Drive and Programs), SanDisk Ultra II 120GB (GTA V), WD Elements 1TB External Drive (Steam Library) // Power Supply: Cooler Master Silent Pro 700W // Case: BeQuiet Silentbase 600 with SilentWings Mk.2 Internal Fans // Peripherals: VicTop Mechanical Gaming Keyboard & VicTsing 7200 DPI Wired Gaming Mouse

 

PROMETHEUS (2018 Refit) - Processor: Intel Core i5-3470 @ 3.2Ghz // Cooler: Cooler Master 212 EVO // Motherboard: Foxconn 2ABF // Graphics Card: ATI Radeon HD 5450 (For Diagnostic Testing Only) // Memory: 2 x 4GB DDR3 Mushkin Memory // Storage: 10TB of Various Storage Drives // Power Supply: Corsair 600W // Case: Bitfenix Nova Midi Tower - Black

 

SpeedTest Results - Having Trouble Finding a Decent PSU? - Check the PSU Tier List!

Link to post
Share on other sites

That's not news to any Canadian, that fiasco has been in the news for quite some time now ;

 

http://www.cbc.ca/news/politics/auditor-general-phoenix-refugees-1.4409978


If you need help with your forum account, please use the Forum Support form !

 

VPN server guide

Introduction to Mechanical Keyboard

Spoiler

My Gaming Rig - Motherboard: MSI Z370-A PRO CPU: i7-8700 RAM: 32GB DDR4 2400(4x8GB) GPU: Gigabyte GTX 1060 3GB OS SSD: 240GB Avexir E100 Storage: 2x 1TB Seagate PSU: Seasonic G650 OS: Windows 10 Pro 64bits Monitor: Acer 21in G205H + Lenovo 21in

 

Link to post
Share on other sites

Its probably just a case of the money ending up somewhere very different from its intended purpose

Oh I'm sure the people will be fine with where their tax money is going


Awareness is key. Never enough, even in the face of futility. Speak the truth as if you may never get to say it again. This world is full of ugly. Change it they say. The only way is to reveal the ugly. To change the truth you must first acknowledge it. Never pretend it isn't there. Never bend the knee.

 

Please quote my post in your reply, so that I will be notified and can respond to it. Thanks.

Link to post
Share on other sites
Quote

Phoenix executives cancelled a pilot project with one department that would have assessed whether Phoenix was ready to be used government-wide.

Government project over budget and non functional? Maybe they should have adequately tested it before deployment as well as checking that it would work on a larger scale.

Link to post
Share on other sites
59 minutes ago, Taf the Ghost said:

Put the jobs of the people in charge of the change on the line, then you'll find most government programs can actually operate efficiently, on-time & roughly on budget.

 

Lack of responsibility to outcomes is the biggest issue with all organizations. Lack of "skin in the game" creates very troublesome conflicts of interest, for the health of the outcome of the project.

Actually that will change little, let the technical people make the technical decisions and as a manager if you don't understand learn to shut up and run with the advice. So much time is wasted explaining stuff that simply cannot be understood without the years of technical background and training.

 

/rant

Also stop hiring business analysts for what is actually a system analyst role, whats the difference you ask? BA = Business and Processes, SA = Technical and Architectural. Please stop mixing them up and hiring the wrong people. If it's actually a SA role you need don't advertise it under a BA job listing.

/rant

 

Edit:

Fear of the responsibility and making bad decisions makes 30 minute decisions take 4 weeks, honestly seen this very thing many times.

Link to post
Share on other sites

government bureaucracy, inefficiency, overall size, slow to progress, inability to deal with the simplest of budgets and contracts and for last it probably ended in a departmental feud (no we don't want it this way, say them all). It could happen in any country anywhere.


.

Link to post
Share on other sites
25 minutes ago, leadeater said:

Fear of the responsibility and making bad decisions makes 30 minute decisions take 4 weeks, honestly seen this very thing many times.

Everyday for me at work while I just end up provisioning anyway until I get tired of waiting the weeks and just bring it to the manager and just get it done. 

Link to post
Share on other sites
1 hour ago, mr moose said:

This is actually why many business avoid updating and upgrading unless they have to.    LTT are generally quick to poo poo company X because they still use XP or refuse to try Linux/swap to AMD servers/workstations,  But the reality is sometimes it only takes a few small unexpected issues to cost $100's of millions.

Yeah it's sadly true. I know the place I work at just started testing switching to Windows 10 and there are a ton if issues that we are running into for the programs we use. It makes you think that maybe it's not so insane that companies are wary of switching to something new especially if it is untested for their use case. 

Link to post
Share on other sites

I've worked with some Canadian IT in the past. All were fired due to incompetence and repeated screwups.

 

They do this thing, becoming fixated on a color or shape and begin repeating some rtard mantra. They putter around like somebody thats been up for days and are generally slow in work.

Link to post
Share on other sites
55 minutes ago, leadeater said:

Actually that will change little, let the technical people make the technical decisions and as a manager if you don't understand learn to shut up and run with the advice. So much time is wasted explaining stuff that simply cannot be understood without the years of technical background and training.

 

/rant

Also stop hiring business analysts for what is actually a system analyst role, whats the difference you ask? BA = Business and Processes, SA = Technical and Architectural. Please stop mixing them up and hiring the wrong people. If it's actually a SA role you need don't advertise it under a BA job listing.

/rant

 

Edit:

Fear of the responsibility and making bad decisions makes 30 minute decisions take 4 weeks, honestly seen this very thing many times.

Clearly, the project was handled poorly. A lot of people had/still have issues not being paid for what, 2 years now? That software went live without any safety net and no way to revert to the previous platform. Some heads need to roll, if you are hired by the government, you are supposed to be the cream of the crop period.

If it's anything like the Quebec "Health" IT project we have here, there was a lot of external consultants / friends of friends / egos implicated in the project and somehow I feel they forgot the initial scope of the project. Of course, everyone in the project took home a huge slice of the pie and still do because they have been dragging the project for years now, i guess it's the same for Phoenix unfortunately.

I'm ashamed to see such an IT project go to waste. It gives us, IT professionals, really bad rep again. The project, if it would have been handled properly, had a really good potential. It gives me the impression that some external influencers decided to butt in the project and keep piling demands and asking for features when the analysis was done. We all had these people in our businesses doing that. Project ABC is nearing completion but Timmy from accounting decides all of a sudden that he's an IT specialist and wants feature XYZ deployed in the first version of the software.


Current Build: SD-DESK-07

 

Case: Bitfenix Prodigy // PSU: SeaSonic SS-650RM // Motherboard: P8Z77-I DELUXE // CPU: Intel Core i5 3570k // Cooler: Corsair H80i // RAM: Patriot Intel Extreme Masters 2X8GB DDR3 1600MHz // SSD: Crucial M500 240GB // Video: EVGA GeForce GTX 660Ti SC 2GB

Link to post
Share on other sites
27 minutes ago, Brooksie359 said:

Yeah it's sadly true. I know the place I work at just started testing switching to Windows 10 and there are a ton if issues that we are running into for the programs we use. It makes you think that maybe it's not so insane that companies are wary of switching to something new especially if it is untested for their use case. 

The issue is that due to the way Windows 10 handles upgrades. Every 6 months you risk stuff breaking again.

Today a few systems that run W10 upgraded to 1803. All of them broke... It was an easy fix but still, it was downtime that wouldn't have happend if it updated the same way like W7 and W8.


If you want my attention, quote meh! D: or just stick an @samcool55 in your post :3

Spying on everyone to fight against terrorism is like shooting a mosquito with a cannon

Link to post
Share on other sites
34 minutes ago, Tribalinius said:

I'm ashamed to see such an IT project go to waste. It gives us, IT professionals, really bad rep again.

That's what I hate about stuff like this, similar to some stuff that happened at my previous employer. Chances are, the IT guys did their job correctly and warned the heads that it wasn't ready but some top brass told them it would be fine, that they shouldn't worry so much, yeah right, heard that one before!

 

I've seen it many times where the top and middle management decides on some big project without consulting their own IT department, and then we (IT guys) take the fallout for the fuck ups they did and that they simply had no business in deciding! As I said earlier, previous employer decided to change the inventory and billing software, then told us (IT) 6 months AFTER the contract was signed ... sigh ... the live deployment happened 2 years late AND made all the customers angry, they ended up closing a few branches because of that! That's how I lost my job but found a much better one, so not all bad news I guess.

 

And if it's anything like I've seen elsewhere, the top management people that took all those decisions won't be hold accountable for any of it.


If you need help with your forum account, please use the Forum Support form !

 

VPN server guide

Introduction to Mechanical Keyboard

Spoiler

My Gaming Rig - Motherboard: MSI Z370-A PRO CPU: i7-8700 RAM: 32GB DDR4 2400(4x8GB) GPU: Gigabyte GTX 1060 3GB OS SSD: 240GB Avexir E100 Storage: 2x 1TB Seagate PSU: Seasonic G650 OS: Windows 10 Pro 64bits Monitor: Acer 21in G205H + Lenovo 21in

 

Link to post
Share on other sites
41 minutes ago, samcool55 said:

The issue is that due to the way Windows 10 handles upgrades. Every 6 months you risk stuff breaking again.

Today a few systems that run W10 upgraded to 1803. All of them broke... It was an easy fix but still, it was downtime that wouldn't have happend if it updated the same way like W7 and W8.

Sounds like a lot of custom programs. I used to work at a large corporation that had a lot of custom programs. I started at T1 and ended at a sys admin so I have worked with it all. On the PCs with custom programs, yeah shit would break, PCs without no problems at all. 

 

But I was a hardass and kept to security standards. I gave hard dates (due to the info we held) and if you didnt have the test PC I provisioned working. Well, sucks for you....better get working on it. 

Link to post
Share on other sites

This reminds me of the ObamaCare website rollout.


NZXT S340 ELITE  -  i7-5930K  -  ASUS X99-A/USB 3.1  -  ZOTAC GTX 1080 AMP EXTREME!  -  32GB PC4-21300  -  256GB 840 PRO + 500GB 860 EVO + (2) 2TB HDDs IN RAID 1  -  CORSAIR AX850 850W  -  KRAKIN X61  -  WINDOWS 10 PRO  -  DELL U3415W 34"  -  RAZER BLACKWIDOW CHROMA  -  LOGITECT G502 SPECTRUM  -  LOGITECH Z623 200W  -  iPHONE XS MAX 256GB

Link to post
Share on other sites

As mentioned, this news is VERY old.

And this has nothing to do with "IT of Canada". The government (previous) wanted to modernize the payroll system for ALL departments at the SAME time. However, such massive undertaking requires collaboration between all the departments, to end up with this massive set of requirements, with the hope that no one forgets anything, which can be very complicated to include afterwards as it could change drastically the algorithms to define pays). It also requires a very high level of expertise. Sadly, in Canada, governmental contracts are signed from the lowest bidder, not a relationship between price and reputation of quality of work, or consideration of past failures, even if the government had a bad experience in the past (unless it was real bad, and now the government puts the company in a ban for a few years). Anyway, they signed with the lowest bidder, which same company had the same task at hand back in Australia (or some other country, I don't recall for sure), and it flopped equally in similar fashion.

 

Despite the hard to swallow pill, it is best for the government to pull out, and undo everything, costing a lot of money, than to stick with it, as it was done in the past, and end up paying a lot more, and still have issues.

Link to post
Share on other sites

@Tribalinius @wkdpaul

Another issue that really gets to me is the way tender processes are done to select the software vendor to go with. There are a lot of things you aren't allowed to do or steps that don't get done that seem rather fundamental to enabling the selection of the best fit product.

 

Like for our new Student Management System, it was all done in secret and none of the infrastructure support team or application support team were allowed to know what the options were so we could investigate how the proposed systems would fit in to our environment, what systems it would need to interact with and how. How are you supposed to make an informed decision and pick the best product that is actually supportable when you never bother to find those things out, yes the business side of it is the most important by far but it does you no good to select a product that just will not work no matter how much shoehorning you try and do.

 

Then to top that off no one is allowed to ask any other businesses that use the proposed software what it's like, how they implemented it, what issues they encountered, nothing.

 

The above project was multiple tens of millions of dollars, not allowed to say exactly how much and was literally dropped on us over night. Not kidding the first we found out a product was selected was when they walked in and demanded 100 VMs be created immediately and the project was starting that day. 

 

Good decisions can not be made in a vacuum.

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


×