[Update: Fixed] Y22K - Microsoft rings in the New Year by breaking Email Globally. Only solution? Disable Antimalware.

[rcmaehl]

Update

 

Microsoft issues a fix for Exchange Y2K22 bug that shut down company emails - The Verge

 

Summary

By storing the date as a signed INT in Exchange. Microsoft broke exchange servers globally as the year ticked over to 2022. The only current solution is to disable Antimalware services.

 

Media

 

Quotes

Quote

Microsoft Exchange servers have not been able to properly process the new date, and therefore, can't process mail. The first social report for this rolled in at 1am EST. Microsoft released a bad update, aptly named “220101001”. This was presumably a scheduled patch to allow for processing the new date, but it didn’t go as planned. “ Cannot convert “220101001” to long,” This aligns with reports from Norwegian firm Sopra Steria, who released a detailed synopsis of the cause. Microsoft Exchange servers have stopped processing mail all together because it isn't prepared to handle today’s date. The reason for this is because Microsoft is using a signed int32 for the date and with the new value of 2,201,010,001 is over the max value of the “long” int being 2,147,483,647. The most troubling part of this is the stop-gap solution. In order to resume processing of mail, sysadmins are disabling malware scanning on their exchange servers, leaving their users, and possibly the servers themselves, vulnerable to attack.

 

My thoughts

Well Happy New Year Microsoft. I hate that the solution is to disable the antimalware service, but ideally companies with have additional antimalware and quarantining for their emails. At worst case, it's a holiday, so email can stay down or stay held until a proper patch is released which I'm sure will be any time now... any second...

 

Sources

Neowin

Microsoft Community

[Windows7ge]

So it's Y2K all over again.

 

I remember reading that Microsoft or a 3rd party knew this was coming. Not necessarily to their e-mail services but that 2022 marked the end of the date counter for how the code worked and that a patch needed to go out. Guess that didn't happen in time.

[Sauron]

36 minutes ago, rcmaehl said:

storing the date as a signed INT in Exchange.

this is an incomprehensibly stupid idea. wouldn't even have been a problem if they used the last two digits for the year rather than the first two.

[leadeater]

2 hours ago, Sauron said:

this is an incomprehensibly stupid idea. wouldn't even have been a problem if they used the last two digits for the year rather than the first two.

Ah yes but in 100 years it would have been so I'm such a smart and diligent coder that I accounted for this problem in 100 years and have already made sure it would never be a problem because my code in 100 years time might be still used, because it such great coder.

 

But FYI they did only use 2 digits for the year, read the post again 

[Sauron]

12 minutes ago, leadeater said:

But FYI they did only use 2 digits for the year, read the post again 

yes, I'm saying that if they had the year as the least significant two digits this wouldn't have happened since the months only go up to 12

 

and it wouldn't have been a problem in 100 years since I'm assuming the century was already being truncated

[leadeater]

9 minutes ago, Sauron said:

yes, I'm saying that if they had the year as the least significant two digits this wouldn't have happened since the months only go up to 12

Oh right, flip the format the other way, lol doh.

 

MMYYDDHHMM

[MultiGamerClub]

Microsoft.. not suprised.. or kinda.. its 2am.

[Salv8 (sam)]

Microshit gets everyone the best presents!

 

[wanderingfool2]

6 hours ago, Sauron said:

yes, I'm saying that if they had the year as the least significant two digits this wouldn't have happened since the months only go up to 12

 

and it wouldn't have been a problem in 100 years since I'm assuming the century was already being truncated

I honestly don't get why they were even storing it as an int (and the way they were) in the first place.

 

In regards to why they wouldn't do MMYYDD...is because it messes up the simple if(val < val2) scenario.  Doing YYMMDD allows for a simple comparison to tell if a date is...although really it should have been just stored as time_t...but at the very least they should have just used an unsigned long type [or if negative dates were thought possible signed long].

 

Really glad I'm not working with MS exchange servers anymore (wouldn't like to have to explain this to the boss, who use to always think any downtime was the IT departments fault)

[Lightwreather]

Huh, think someone wanted the Apocalypse to happen a bit too early /s

[HQuan]

Oh god Y2K again, need to shutdown my computer before midnight again...

[mr moose]

In the days of major software issues on servers and shit (log4j for example), this seems pretty insignificant and easy to fix.

 

[WolframaticAlpha]

13 hours ago, rcmaehl said:

By storing the date as a signed INT in Exchange

good lord, who thought this was a good idea?

Spoiler

 

5 minutes ago, mr moose said:

log4j for example

Oh, log4j was also avoidable. I mean why would a logger even have a feature that would let you access some arbitrary URL... Using LDAP of all things. It is just beyond me

[Sauron]

6 hours ago, wanderingfool2 said:

In regards to why they wouldn't do MMYYDD...is because it messes up the simple if(val < val2) scenario.

sure, but of course this is also the reason you just shouldn't be doing this with a numeric value. you should just have a date type/class and implement the correct comparison operator. it's not like we have had this type of memory constraint since the year 2000.

[sof006]

I thought something was a bit weird, I work in IT and on the night of New Years I was working and I noticed non of my emails were sending from Outlook.

 

They all suddenly came through in one go today however. 

[wanderingfool2]

2 hours ago, Sauron said:

sure, but of course this is also the reason you just shouldn't be doing this with a numeric value. you should just have a date type/class and implement the correct comparison operator. it's not like we have had this type of memory constraint since the year 2000.

Well like I said, it really should be utilizing time_t if anything.  It doesn't really make sense converting the way they are because they are wasting a ton of numbers.  Given this involves antimalware portion, I don't know how often they call it/pass data through it...if they utilize it a lot using a class and implementing a comparison operator could add too much of a performance hit than using a straight up time_t type (which would be 64 bits on a 64 bit compile)

[strajk-]

Didn't seem to have affected our Exchange 2010 server.

I guess this depends on the locale the service is running on.

[Paul Thexton]

At a previous job I resisted management’s requests for using Exchange for our mail. They only wanted it for shared calendaring and weren’t interested in alternative solutions. We ended up with a hybrid system where specific mailboxes only were routed to Exchange. 
 

I feel weirdly vindicated by this news item. Enterprise ready my arse. 

[Paul Thexton]

1 hour ago, wanderingfool2 said:

Well like I said, it really should be utilizing time_t if anything.  It doesn't really make sense converting the way they are because they are wasting a ton of numbers.  Given this involves antimalware portion, I don't know how often they call it/pass data through it...if they utilize it a lot using a class and implementing a comparison operator could add too much of a performance hit than using a straight up time_t type (which would be 64 bits on a 64 bit compile)

Ditto on the use of time_t

 

I’ve seen many implementations that use int or long for time stamps, it’s genuinely frustrating. 

[Sauron]

3 hours ago, wanderingfool2 said:

Well like I said, it really should be utilizing time_t if anything.  It doesn't really make sense converting the way they are because they are wasting a ton of numbers.  Given this involves antimalware portion, I don't know how often they call it/pass data through it...if they utilize it a lot using a class and implementing a comparison operator could add too much of a performance hit than using a straight up time_t type (which would be 64 bits on a 64 bit compile)

If you compare the dates one datapoint at a time, at the absolute worst you can take 6x as many comparisons (in cases where only the seconds differ). It's a linear performance hit so it shouldn't be a problem on any remotely modern hardware. Of course a time_t works too if you really can't afford the computational cost for some reason; if it's not user-facing and checked thousands of times per second then it shouldn't be in a human readable format.

[NoobsWeStand]

A company I worked for sometime stored dates like this. I always thought it was weird, but I was working with code from the early 2000s, so I thought it was the norm back then. I thought maybe it was an easy way to do date comparisons.

 

I should ask some of my old co-workers if they had issues with this yet.  Lol

 

[TetraSky]

Didn't notice any downtime on my side, so I guess I don't use Microsoft's exchange servers. Was still getting emails on outlook on my Android phone and on Thunderbird on PC.

 

Why didn't they just use UINT (double the available numbers, but no negatives... which should be fine for dates).

[rcmaehl]

Update:

Quote

“The problem relates to a date check failure with the change of the new year and is not a failure of the AV engine itself,” Microsoft explains in a post on its Tech Community forum... Microsoft outlines the solution to the problem in its post, requiring administrators to either implement the fix manually or apply an automated script. The post also contains a detailed FAQ, noting that the solution “will take some time” for administrators to implement. And depending on how many emails are stuck in the queue, it may take a while for the messages to land in their intended recipients’ inboxes.

 

[Sir Asvald]

GREAT MORE STUFF THAT I NEED TO WORK ON...... THANKS MICROSHIT AND YOUR SHITTY SOFTWARE,