OldBoIs urgently wanted - States cry for urgent help as Pre-Y2K systems struggle to handle epidemic

[Kisai]

On 4/9/2020 at 5:36 AM, rcmaehl said:

Source:
CNN (qoute source)
Medium (media source)
Newser

 

Summary:

States such as Connecticut, Kansas, and New Jersey have been found to still be using 40+ year old COBOL based systems which are struggling to handle the large influx of requests and claims due to the pandemic
 

Media:

Quotes/Excerpts:

 

 

My Thoughts:
These are cases of If it ain't broke, don't fix it taken to the extreme and it has severely bit them in the rear. It's time for, at least the government, to upgrade to more modern systems, or task for the creation of a new programming language to replace the aging COBOL.

Eggs, Basket.

 

There is nothing wrong with COBOL. In fact there's way to fix all of this, and we've been staring it in the face for the last 10 years. Virtualize the machine. That solves problem the first problem, where running old hardware with no replacement parts can be fixed. You can probably replace an entire warehouse full of 1980's systems with one mid-tier Xeon system (though that would also be stupidly dangerous.) Just do 1:1 replacements with server tier hardware.

 

The second problem is adapting COBOL so that you can run it on the native hardware instead of the legacy hardware. The programming language was designed and used in the 60's. Believe it or not COBOL has been updated to as late as 2014, however that doesn't mean people know or want to use it. So you know what the solution there is? Cross-compile the old binaries it to another language. This is literately the kind of stuff we've been doing since the early 2000's with game console emulators. You know what the original hardware is, you know what the original language is, and you may even have the original source code. So you run something like Clang and compile to IL, and then decompile that back into C/C++/C# or whatever floats your boat. It will not be as intelligible up front like the original source code would be, but at least you're not porting the COBOL program from scratch.

 

 

 

 

[StDragon]

The thing most people fail to keep in mind is that IT (Information Technology) is a moving target; a standing wave of progress. At the forefront is bleeding edge technology with an uncertain future, which is inherently a risky gamble. In the middle is established standards of interoperability. And lastly, the EOL (End Of Life) phase where it's all being held together with bailing wire, bubblegum, and duct tape.

 

The problem with propping up old systems without training young people is that eventually the gray beards that run them will die off. Need I remind that COVID-19 hits older people really hard?! Imagine if a vast number of COBOL and Fortran folk perished in great numbers in a short window of time. A lot of that institutional knowledge goes with them into the great beyond; lost to us mortals. For the most part there's documentation, but getting replacements up to speed is not as easy as it might seem.

 

Again, nothing wrong with using older platforms so long as they're modernized, supported, and you have the talent pool available to maintain them in mission-critical environments. It's when institutions drag it on and out that catches up with you in a not so good way. And here we are....

[leadeater]

24 minutes ago, Kisai said:

There is nothing wrong with COBOL. In fact there's way to fix all of this, and we've been staring it in the face for the last 10 years. Virtualize the machine. That solves problem the first problem, where running old hardware with no replacement parts can be fixed. You can probably replace an entire warehouse full of 1980's systems with one mid-tier Xeon system (though that would also be stupidly dangerous.) Just do 1:1 replacements with server tier hardware.

Mainframes have always been 'virtual machines' way before the term virtual machine existed. Mainframes have Logical Partitions (LPAR) that run their own operating systems, mainframes can also run VMs that are more like x86 VMs too. Different but similar things but not actually a problem here.

 

It's also not like IBM and Co stopped making mainframes and migrating to newer hardware wasn't possible. Who knows how technically accurate the article is, saying 40 year old system could just mean the design and software and not the hardware, or any combination of.

 

I don't particularly think it's likely the hardware is actually 40 years old because it's unlikely it would actually still be operational for that long because parts fail and you won't have replacements for it today.

 

Who really knows, thing is old software often doesn't respond well or gain much just by throwing much faster hardware at it, sometimes it'll actually break outright because it's too fast.

[Sauron]

10 hours ago, mr moose said:

You keep saying how sure you are that these things can be done, but the reality is they aren't being done instead they are choosing to maintain them,  there is a good reason for that.

Yes, bad management of government spending.

[mr moose]

2 minutes ago, Sauron said:

Yes, bad management of government spending.

This would have to be the equivalent of putting your head in the sand, I have provided two perfectly reasonable articles on the subject explaining why it isn't bad management, both include technical reasoning.  There is also a good article from one of the bigwigs at HSBC who said if they could shift from the mainframe to cloud solutions they could save up to 75% on their IT budget, but he also said he knows nothing about the system and that that upgrade was going to cost at least 5 years worth of IT budget, carried major risks and the system is more than capable of doing the required job into the future.   That is not bad management. that is a reasoned approach. 

[Sauron]

1 minute ago, mr moose said:

This would have to be the equivalent of putting your head in the sand, I have provided two perfectly reasonable articles on the subject explaining why it isn't bad management, both include technical reasoning.  There is also a good article from one of the bigwigs at HSBC who said if they could shift from the mainframe to cloud solutions they could save up to 75% on their IT budget, but he also said he knows nothing about the system and that that upgrade was going to cost at least 5 years worth of IT budget, carried major risks and the system is more than capable of doing the required job into the future.   That is not bad management. that is a reasoned approach. 

Your articles talked about banks - for profit organizations. Not to mention that they justified why you might want a mainframe, not why you wouldn't want to update your system to perform better.

[mr moose]

1 minute ago, Sauron said:

Your articles talked about banks - for profit organizations. Not to mention that they justified why you might want a mainframe, not why you wouldn't want to update your system to perform better.

 

They explain why people do not upgrade cobol mainframes the way you say they should, period.  They used banks as the example.   You seem to be under the impression that upgrading these things is feasible,  sometimes it's merely a matter of money and time (something government agency have neither of) and sometimes it's purely a matter of being programmed into a corner, there is no way out without creating a whole new system from the ground up.  In most cases is somewhere in the middle,  and that by itself is why it is not so easily done and is not a case of "bad management".

 

 

[leadeater]

30 minutes ago, mr moose said:

There is also a good article from one of the bigwigs at HSBC who said if they could shift from the mainframe to cloud solutions they could save up to 75% on their IT budget, but he also said he knows nothing about the system

Wait, if he knows nothing about the system then how could he know it's going to be cheaper? Sounds like the same crap I here from the university senior leadership at work, making statements about how cloud is cheaper based off of???? I dunno, falling for hype and cloud salespeople, because they aren't biased now are they?

 

You never deploy to the cloud because it's cheaper, it almost never is, you do it for other reasons. Sometimes it can be cheaper but that is very case specific or it's about auto-scaling to meet demand or when load demand goes above on-prem capability that is supplemented by temporary cloud resources.

 

But it's a bank so what do I care, hope they waste as much of their money as possible like the rest of the people before them, the biggest uptick in the IT sector right now is cloud exit projects. If that isn't a giant red flag to hold off and do some proper analysis and listen to the people you employ who actually know a thing about it then all hope is lost.

[mr moose]

10 minutes ago, leadeater said:

Wait, if he knows nothing about the system then how could he know it's going to be cheaper? Sounds like the same crap I here from the university senior leadership at work, making statements about how cloud is cheaper based off of???? I dunno, falling for hype and cloud salespeople, because they aren't biased now are they?

I guess being the head of one of the worlds largest investment banks means you don't have a habit of spending money on something that doesn't have a return attached to it. Whether they personally understand that business or not,  they have advisors who do and have done the sums for them.

 

10 minutes ago, leadeater said:

You never deploy to the cloud because it's cheaper, it almost never is, you do it for other reasons. Sometimes it can be cheaper but that is very case specific or it's about auto-scaling to meet demand or when load demand goes above on-prem capability that is supplemented by temporary cloud resources.

 

10 minutes ago, leadeater said:

But it's a bank so what do I care, hope they waste as much of their money as possible like the rest of the people before them, the biggest uptick in the IT sector right now is cloud exit projects. If that isn't a giant red flag to hold off and do some proper analysis and listen to the people you employ who actually know a thing about it then all hope is lost.

They aren't deploying to the cloud and they are listening to the people they employ to do the sums,  that was the point,  even though they have crunched the numbers and supposedly the long term pay off is good, the short term cost and risks are too high.  Hence my point about it being a rational decision and not bad management.

[CarlBar]

2 hours ago, leadeater said:

Wait, if he knows nothing about the system then how could he know it's going to be cheaper? Sounds like the same crap I here from the university senior leadership at work, making statements about how cloud is cheaper based off of???? I dunno, falling for hype and cloud salespeople, because they aren't biased now are they?

 

You never deploy to the cloud because it's cheaper, it almost never is, you do it for other reasons. Sometimes it can be cheaper but that is very case specific or it's about auto-scaling to meet demand or when load demand goes above on-prem capability that is supplemented by temporary cloud resources.

 

But it's a bank so what do I care, hope they waste as much of their money as possible like the rest of the people before them, the biggest uptick in the IT sector right now is cloud exit projects. If that isn't a giant red flag to hold off and do some proper analysis and listen to the people you employ who actually know a thing about it then all hope is lost.

 

To be fair any long term not likely to go away service should have the emergency planning in place to upscale things as needed, (and i'm talking about an upscale in handling of whatever work is involved not neces serially strictly hardware and/or software). Now to be fair the US social system is far from the only thing to find it's disaster planning coming up waaaay short during this mess. I don't think anyone's disaster planning has come out looking good from this one. So i can cut them some slack there, (though the truth is it's exposed flaws in our assumptions that derive our way of doing things across the spectrum, so whilst some slack is due because everyone messed up, it dosen;t change the fact that everyone messed up, its just more understandable). The questions about the hardware, software, programming language are fun side discussions, (and i reserve the right to dive back in at a later point on those), but they're really all arguing details around a core point that they seem to have been caught particularly short, (I haven't heard of any other similar systems in a 1st world country suffering similar issues, and it even seems to be localised in the US to just some states if i understand correctly), and that suggests that they're planning for any future need to upscale in a short time period was not very good.

[leadeater]

1 hour ago, CarlBar said:

I don't think anyone's disaster planning has come out looking good from this one

Dunno I think ours has, from both my country and my workplace perspective. Benefit of being an island nation in the middle of nowhere and my work already being a leading distance learning university so it's just been a lot of do the same stuff but from home while just deploy some more VDI hosts and RDS hosts.

 

For quite a while now NZ has had good information sharing between primary care facilities and hospitals across the country through HealthLink and WebPAS and our government and Ministry of Health makes a concerted effort to ensure IT standardization across the country as best they can so everyone is singing the same song walking down the same path. Now the nitty gritty details are not quite as rosy as it sounds, nothing really is, but at least everyone is using the same software or software that interops with each other and the main health network of the country.

 

Hardware purchasing is also audited and justifications required, particularly now as the mandate right now is to use approved hosting service providers in country rather than local in hospital IT server fleets. The goal is to get everyone on a shared infrastructure to reduce costs while continuing the already happening of everyone using the same actual software platform so there isn't a Patient Management System per hospital, a Radiology Systems per hospital etc etc. For us there is no need for that kind of duplication, it makes no sense and is feasible to do away with that. 

[mr moose]

25 minutes ago, comander said:

I don't expect too many migrations until we hit the year 2038. 

At that point A LOT of old stuff will get ripped up. 

https://en.wikipedia.org/wiki/Year_2038_problem

In all likelihood this'll just mean massive government contracts to Amazon, Google or whomever who have already solved this issue and who will gladly run government servers on the cloud for the right price. 

There is a lot of product development between now and then, I predict building and running servers onsite will be cheaper than cloud based data centers with the speed hardware is being developed compared to the actual requirements.   18 years is a rather long time in tech terms.

[mr moose]

5 hours ago, comander said:

I never said it'd be cheaper or even better. 

Just that THAT is what I expect quasi-competent government officials to end up doing in around 15 years when a lack of response would get them fired come 2038. 

I was just adding my prediction for the course of all this before 2038.

 

EDIT: I mean I highly doubt they'll get to 2038 and go," oh shit, we didn't do anything about this back when we had a chance and now it's all going to fail".  It'll be more likely between now and then they'll either start training people in cobol again and continue to build mainframes with modern mitigations and compatibility layers or they will be investing in something completely different to how we do things now.  For all we know quantum might be the best answer to migrating away from cobol.

[mr moose]

16 minutes ago, comander said:

There will eventually be something so big that'll require a complete refactoring. 

 

Of course that will always be the case (even if it is likely as rare as a meteor or alien invasion), however we can only instigate mitigations to known problems.  2038 is know problem like y2k was.  We have plenty of time to work it out and it will get worked out even if it means going back to paper money and land titles.