OldBoIs urgently wanted - States cry for urgent help as Pre-Y2K systems struggle to handle epidemic

[rcmaehl]

Source:
CNN (qoute source)
Medium (media source)
Newser

 

Summary:

States such as Connecticut, Kansas, and New Jersey have been found to still be using 40+ year old COBOL based systems which are struggling to handle the large influx of requests and claims due to the pandemic
 

Media:

Quotes/Excerpts:

1 minute ago, CNN said:

You can now add COBOL programmers to the list of what several states urgently need as they battle the coronavirus pandemic. In New Jersey, Gov. Phil Murphy has put out a call for volunteers who know...COBOL because many of the state's systems still run on older mainframes. In Kansas, Gov. Laura Kelly said the state's Departments of Labor was in the process of modernizing from COBOL but then the virus interfered. "So they're operating on really old stuff," Connecticut has also admitted that it's struggling to process the large volume of unemployment claims with its "40-year-old system comprised of a COBOL mainframe and four other separate systems." "Literally, we have systems that are 40-plus-years-old," New Jersey Gov. Murphy said. "There'll be lots of postmortems and one of them on our list will be how did we get here where we literally needed COBOL programmers?"  With more than 362,000 New Jersey residents filing for unemployment in the past two weeks, the 40-year-old mainframes that process those claims are being overloaded. COBOL...is a computer programming language that was developed back in 1959, according to the National Museum of American History. "It's a programming language that was used to create a very significant percentage of business systems over the period of the 60s, 70s and even into the 80s," But over time, coders have moved away from the aging language. "The general population of COBOL programmers is generally much older than the average age of a coder," Despite a dwindling number of COBOL programmers, a 2017 report...found that there are still 220 billion lines of COBOL in use today. 43% of banking systems are built on COBOL and 95% percent of ATM...rely on COBOL code. COBOL is being used in agencies such as the Department of Veterans Affairs, Department of Justice and Social Security Administration, according to a 2016 report by the Government Accountability Office. A 2018 report by the inspector general for the Social Security Administration found that the administration maintained more than "60 million lines of COBOL" with "millions more lines of other legacy programming languages." "Governors should not have to think about computer systems during a pandemic," he said, "and we should have systems that if there are emergency situations, should not make the emergencies worse."

 

 

My Thoughts:
These are cases of If it ain't broke, don't fix it taken to the extreme and it has severely bit them in the rear. It's time for, at least the government, to upgrade to more modern systems, or task for the creation of a new programming language to replace the aging COBOL.

[Arika]

1 hour ago, rcmaehl said:

In Kansas, Gov. Laura Kelly said the state's Departments of Labor was in the process of modernizing from COBOL but then the virus interfered

Buuuuuuuulllllllllllllllllllshit. no chance in cold hell that a system this old just happened to be in the process of getting updated in the months that the infection hit.

[paddy-stone]

1 hour ago, VegetableStu said:

i'm super far removed from the process, but surely there's a need to modernise said machines when said programmers are becoming more scarce in the first place? ._.
 

i mean no point crying over spoiled milk, but from a systems standpoint that seems a little too short sighted

That's the problem though, many people in control of the budget don't care what the systems run on, and focus purely on what the cost would be to replace it... then the answer is no. It's only when something like this happens that they start to care, because they are then getting questions as to why it wasn't upgraded in the previous 40 years. And many of them will pass the buck.

[XenosTech]

32 minutes ago, Arika S said:

Buuuuuuuulllllllllllllllllllshit. no chance in cold hell that a system this old just happened to be in the process of getting updated in the months that the infection hit.

I mean it could have been in the process for the last 20+ years lol. It's the same shit with government here, at least now this new administration is taking steps to digitize parts of government slowly.

[descendency]

Every system has a shelf life. If you paid for it in 1970, you might not need to replace it in 1980 or 1990 - but if you are still using it in 2020, something is wrong. The cost for maintenance now must dwarf what it would cost to implement a modern system. 

 

That said, I think there is a bit of fairness that needs to be given to the states. The number of people applying for unemployment right now is orders of magnitude more than ever before. If you see the graph, it looks like a flat line and then a massive spike in the last 2 weeks. That would stress any system. 

 

But I still think it's laughable that COBOL programmers are what we need in 2020. 

[Dylanc1500]

So, I guess IBM ought to get some requests for the new Z series. I guess the good thing about Them investing into these systems is that there are drop in replacements that require little to no change.

[TetraSky]

Weren't your nukes also operating off those giant floppy disk?

It's like you guys are allergic to progress. I get the whole "if it ain't broke don't fix it", but come on. It's plain negligence at this point.

[Guest]

37 minutes ago, descendency said:

Every system has a shelf life. If you paid for it in 1970, you might not need to replace it in 1980 or 1990 - but if you are still using it in 2020, something is wrong. The cost for maintenance now must dwarf what it would cost to implement a modern system. 

 

That said, I think there is a bit of fairness that needs to be given to the states. The number of people applying for unemployment right now is orders of magnitude more than ever before. If you see the graph, it looks like a flat line and then a massive spike in the last 2 weeks. That would stress any system. 

 

But I still think it's laughable that COBOL programmers are what we need in 2020. 

You would be amazed at just how many still exist, and how many people are still using the code.

 

I’ve spent most of my career involved in mainframes to some extent. They really are superb at what they do. Most of the time, you just leave them running and they carry on doing their thing. Many are still running for certain tasks an order of magnitude more efficiently than their Linux or windows alternative. It is not just the processing either, it is the bandwidth used by these systems. We often see after a migration the need for a 50 fold increase in storage, and a 1000+ fold increase in network traffic. 
 

When trying to migrate an established system onto something new the hurdles are massive. Often there is code written years ago that people do not even realise is running, and essential for the task. By the time it is discovered nobody has the source code so it has to be painstakingly reverse engineered. As well as the bandwidth and scaling issues, these are often critical live systems so migration can be very difficult and risky, we have seen this in banks throughout the world.

 

One of the biggest hurdles is the user base. Consider a bank, with 20,000 users. All their terminals need upgrading, and to avoid having to re-train all those users and hence risk mistakes the new systems have to operate in the same way. Front ends have to slowly evolve over a long period to make the transition easier. All the branches need to upgrade their networks and external connectivity seamlessly in preparation. This is something that can take years. I know of one project that has been running 18years so far and is still a long way from completion. So when people comment on issues such as this they really have no idea the scale of the problems, the costs involved and the time it takes. In the initial stages when the bean counters realise it will cost hundreds of millions to migrate but a tenth of that to just upgrade the mainframe, you can guess which route they will take. While the new mainframe can easily cope, what those same bean counters often forget are the people and skimp on keeping a trained resource to run it. Natural attrition leaves just a few running everything and we get the problems seen here. 
 

There is nothing wrong with modern mainframes, as long as we modernise the code running on them and keep the staff well trained and plentiful.

[BuckGup]

1 hour ago, TetraSky said:

Weren't your nukes also operating off those giant floppy disk?

It's like you guys are allergic to progress. I get the whole "if it ain't broke don't fix it", but come on. It's plain negligence at this point.

The military continued using floppies for safety reasons after they become obsolete 

[3rrant]

4 hours ago, VegetableStu said:

i'm super far removed from the process, but surely there's a need to modernise said machines when said programmers are becoming more scarce in the first place? ._.
 

i mean no point crying over spoiled milk, but from a systems standpoint that seems a little too short sighted

Happens in every industry, private or public owned either.. Vast majority of the times, if something is not ESSENTIAL (you actively lose money and stop production entirely because of it) it is never replaced, or updated, until the before condition is met. Even there isn't anyone on the planet able to fix your system, but your system still "kinda works", it isn't gonna be replaced.

 

3 minutes ago, BuckGup said:

The military continued using floppies for safety reasons after they become obsolete 

They do however have trained staff for that purpose, they aren't likely to end up without either floppies or staff trained to fix/replace/work with them.

[Tristerin]

FTE had to be "created" for Governments not to abuse budgets.  This, is another abuse of budgets.  

 

I 100% bet they have had their offices remodeled, new vehicles, etc - if I had time Id file me an FOIA request on their departments just to prove it - but I don't have time.

[Radium_Angel]

Ex-COBOL programmer here.

 

You have *no idea* how verbose COBOL is, or for that matter, how well it scales, until you've sat down and done it for some time.

Swapping out an established mainframe, that's been running for 30+ years, is not trivial.

And anyone who has ever tried, knows this.

There are a million little "if A and B but not Q or R but sometimes L" subroutines in place in old COBOL systems, that redoing them without wrecking the entire system...

 

...well it's easier to keep the COBOL system in place, and re-hiring all the retired programmers.

 

For the record, I wouldn't touch COBOL again for under 7 figures.

But then again I wouldn't touch Assembly again for all the money in the world. Fact. You can't spend it if you're insane. 

[ZestyJalapeno]

Throw the whole damn thing away, and use that money get some full stack dudes to build you a new system using [*insert any SQL based stack that a 2-year student can make]. This cost is sunk. 

[Letgomyleghoe]

5 hours ago, TetraSky said:

Weren't your nukes also operating off those giant floppy disk?

They still are...

[atxcyclist]

6 hours ago, TetraSky said:

Weren't your nukes also operating off those giant floppy disk?

It's like you guys are allergic to progress. I get the whole "if it ain't broke don't fix it", but come on. It's plain negligence at this point.

Many older systems were built so well that they just don't need replacement yet. Spending money to replace it with possibly less durable hardware and having to re-train a whole bunch of the users on a new OS/system is expensive.

[mr moose]

 

There is a very good reason a lot of these systems are not upgraded or replace and it has  very little to do with budgets or bureaucracy.   Imagine what happens if your bank suddenly crashes? no payments, purchases, wages, records lost (maybe permanently), what about if the revenue system fails?   When old systems work, are keeping up and replacing them means not only spending insanely huge amounts of money but introducing a huge risk then the logical answer is to not upgrade.  

 

How many times have you had  service issue (game, phone, gym system) and the reason given was a recent upgrade failure?  Happens all the time. I have seen it 3 times this year already. 

[CarlBar]

5 hours ago, Radium_Angel said:

Ex-COBOL programmer here.

 

You have *no idea* how verbose COBOL is, or for that matter, how well it scales, until you've sat down and done it for some time.

Swapping out an established mainframe, that's been running for 30+ years, is not trivial.

And anyone who has ever tried, knows this.

There are a million little "if A and B but not Q or R but sometimes L" subroutines in place in old COBOL systems, that redoing them without wrecking the entire system...

 

...well it's easier to keep the COBOL system in place, and re-hiring all the retired programmers.

 

For the record, I wouldn't touch COBOL again for under 7 figures.

But then again I wouldn't touch Assembly again for all the money in the world. Fact. You can't spend it if you're insane. 

 

From my PoV thee issue here isn't so much th programming language as it is the fact that the hardware is so old. Surely if there's so many people using the stuff someone somwhere should be willing/able/allready be producing more modern drop in replacement hardware that can run the same code but with the ability to scale up by being more capable?

 

I mean thats the reason x86 has proven nigh impossible for any standard to replace. It's backwards compatible, (though ofc getting drivers for older OS's for more modern hardware might be a custom job).

[justpoet]

COBOL is everywhere, honestly, and really isn't a problem.  Most of the places that "replaced" it, did so with Java, which is soooo much worse, in so many ways, for the tasks, that it is laughable…and often still kept sections of "simple" COBOL code since it ran so much better.

 

The only real issue is that they just kept random old mainframe running it, which doesn't have modern power to handle the sheer number of simultaneous inputs that we expect today.  If they had a "if this breaks, this is our replacement path" plan in place, it is perfectly reasonable to not have updated it, since nobody would've ever expected a storm of mass entries like this.

[Maslofski]

time to ship out my mother

[Sauron]

21 hours ago, VegetableStu said:

i'm super far removed from the process, but surely there's a need to modernise said machines when said programmers are becoming more scarce in the first place? ._.
 

i mean no point crying over spoiled milk, but from a systems standpoint that seems a little too short sighted

But that would cost money and work, can't have that! Better procrastinate until it's directly responsible for unnecessary deaths or insolvencies!

[Radium_Angel]

12 hours ago, CarlBar said:

Surely if there's so many people using the stuff someone somwhere should be willing/able/allready be producing more modern drop in replacement hardware that can run the same code but with the ability to scale up by being more capable?

If it ain't broke...

There is a reason it's run on mainframes, whose uptimes are measured in *decades* you can't do that with off the shelf "cheap" gear.

And remember, that COBOL program isn't running on x86, it's bespoke systems (mips, z arch whathaveyou) soa  drop-in replacement would simply be another mainframe.

[williamcll]

I remember a news a few years back where a 30 year old air traffic computer malfunctioned because other systems got upgraded.

[Sauron]

2 hours ago, Radium_Angel said:

If it ain't broke...

There is a reason it's run on mainframes, whose uptimes are measured in *decades* you can't do that with off the shelf "cheap" gear.

And remember, that COBOL program isn't running on x86, it's bespoke systems (mips, z arch whathaveyou) soa  drop-in replacement would simply be another mainframe.

Maybe they can figure out a way to change them for something newer every few years instead of waiting 40 years for it to break and be completely unprepared to deal with it...

[mr moose]

14 hours ago, Sauron said:

Maybe they can figure out a way to change them for something newer every few years instead of waiting 40 years for it to break and be completely unprepared to deal with it...

They already have considered it.  The idea they waited for it to break insinuates there was/is a viable alternative, that I am afraid is a guess at best.  

[PlayStation 2]

Fun fact: the ATF does a whole lot of paper shit. Ever wondered why getting a suppressor takes ages? 

Welcome to the US government, where modernizing anything to more modern technology takes at least 10 years, if not longer.