Backdoor in upstream xz/liblzma leading to SSH server compromise

CosmicEmotion · March 29

The guy who is responsible for this has been part of the project for 2 years.

Noone is safe and if your system is compromised already you have to reinstall and change all your credentials. I personally went with Fedora Kinoite and an ecrypted install.

The perpetrator made this commit yesterday ironically.

manikyath · March 29

26 minutes ago, CosmicEmotion said:

Noone is safe

so... "no one is safe" because of a commit that only made it into a dev build of fedora and debian?

while this is a potentially very big backdoor.. it seems that humanity had some karma saved up, and a microsoft dev of all people just happened upon the backdoor while working on a dev build.

the short of it is; if you havent updated in the past few days, there is no reason to suspect your system is affected.

igormp · March 29

1 hour ago, CosmicEmotion said:

Noone is safe and if your system is compromised already you have to reinstall and change all your credentials. I personally went with Fedora Kinoite and an ecrypted install.

Only if you're in one of the specific affected distros with that specific xz version, and also had your device's ssh port exposed to the internet. Otherwise, it's a big security issue but the implications are minor for most home users, and even servers shouldn't just have their ssh ports wide open anyway.

Biohazard777 · March 29

Just posted a status update about this, didn't see this thread.

47 minutes ago, manikyath said:

the short of it is; if you havent updated in the past few days, there is no reason to suspect your system is affected.

It is also present in 5.6.0 which was released in late February.

52 minutes ago, manikyath said:

so... "no one is safe" because of a commit that only made it into a dev build of fedora and debian?

Overblown, yeah... but it is concern for people on rolling releases, like Arch:
https://archlinux.org/
Which has this on their front page right now:

15 minutes ago, igormp said:

Otherwise, it's a big security issue but the implications are minor for most home users, and even servers shouldn't just have their ssh ports wide open anyway.

That is just the thing, I don't expect people running production servers to be on rolling releases with ssh ports exposed.
Linux has gained a lot of newbs recently which don't know any better, those people are at risk now and we should try to give them a heads up.

Ralphred · March 29

"The resulting malicious build interferes with authentication in sshd via systemd."

SysVinit chads keep winning.

On a serious note https://www.openwall.com/lists/oss-security/2024/03/29/4 is an interesting read, and for anyone wondering what to do right now I'll quote it

Quote

== Detecting if installation is vulnerable ==

Vegard Nossum wrote a script to detect if it's likely that the ssh binary on a
system is vulnerable, attached here. Thanks!

The script mentioned is https://www.openwall.com/lists/oss-security/2024/03/29/4/3 but the use of set -e means that when the grep fails it exits without telling you "probably not vulnerable", so just look for 'liblzma' in the list when you run ldd $(which sshd), if it's there get and run the script for a deeper diagnosis...

24 minutes ago, igormp said:

even servers shouldn't just have their ssh ports wide open anyway.

Meh, they should if there purpose is to allow remote access. That is what's pernicious about this backdoor, it interferes with RSA authentication, the default "hardened" approach to keep the script kiddies out.

3 minutes ago, Biohazard777 said:

those people are at risk now and we should try to give them a heads up

Indeed!

igormp · March 29

17 minutes ago, Biohazard777 said:

Overblown, yeah... but it is concern for people on rolling releases, like Arch:

It is not, as per that own post that you apparently didn't read:

Quote

From the upstream report (one

openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma.

Arch does not directly link openssh to liblzma, and thus this attack vector is not possible. You can confirm this by issuing the following command:

ldd "$(command -v sshd)"

However, out of an abundance of caution, we advise users to remove the malicious code from their system by upgrading either way. This is because other yet-to-be discovered methods to exploit the backdoor could exist.

18 minutes ago, Biohazard777 said:

Linux has gained a lot of newbs recently which don't know any better, those people are at risk now and we should try to give them a heads up.

I doubt they'd have a proper port forward from their firewall to their vulnerable system, with a specific dev distro that has been affected. Even if that's the case, they'd likely face worse issues than this possible backdoor.

16 minutes ago, Ralphred said:

Meh, they should if there purpose is to allow remote access.

Without a proper bastion or similar? Sounds amateur-ish IMO.

Since this is a recent issue that hasn't even made out in any release for the targeted distros the only possible victims are VPS users that like to have their systems on the bleeding edge.

However, if it managed to propagate to proper releases some months from now, then yeah, it'd be a really huge issue.

Biohazard777 · March 30

3 minutes ago, igormp said:

It is not, as per that own post that you apparently didn't read:

I did, don't see how it invalidates my statement that the concern is real and people should read about it.

Did you apparently not read your quote:

My point is simple: this shouldn't be downplayed, people should be informed.
On which we clearly disagree.

Ralphred · March 30

8 minutes ago, igormp said:

Without a proper bastion or similar?

How do you think they work without an open sshd port?

10 minutes ago, igormp said:

the only possible victims are VPS users that like to have their systems on the bleeding edge.

Yeah, I can see this, no one in their right mind runs anything other than "stable" on outward facing hardware.

17 minutes ago, igormp said:

then yeah, it'd be a really huge issue.

The issue right now is it was pure luck this was found, "will we find the next one?" and "did we find the last one?" are scary questions.

CosmicEmotion · March 30

This has been going on for a much longer time than one month. Some reports on Reddit say the maintainers themselves are in on it and also it's possible to inject malicious code to a compressed packaged as well given that xz was used (which means all kernel packahes and ALL packages on Linux, even Flatpak).

This is the greatest disaster that's happened to Linux and I'm not overblowning anything. I switched to Windows personally, and will be on it until all this is completely resolved.

igormp · March 30

24 minutes ago, Biohazard777 said:

My point is simple: this shouldn't be downplayed, people should be informed.
On which we clearly disagree.

Sure, let's not downplay it, but also let's avoid the other extreme of overblowing it and telling others to wipe their systems as OP did.

15 minutes ago, Ralphred said:

How do you think they work without an open sshd port?

40 minutes ago, igormp said:

Great, you invaded the bastion, good luck on the next jump. There are many possible vectors, but many are just theoretical and I doubt we will likely see any bad outcome from this since it was found in time.

15 minutes ago, Ralphred said:

The issue right now is it was pure luck this was found, "will we find the next one?" and "did we find the last one?" are scary questions.

Agreed, but being overly anxious about what may happen is not a good idea IMO. Better to just audit stuff and wait to see if anything actually appears (which likely won't).

13 minutes ago, CosmicEmotion said:

This has been going on for a much longer time than one month.

The bad actor, yes.

But the actual malicious part only has been getting in place in the last month (just look at the commits), and NO distro has released an infected package as of now in any actual release version, only compromised devices were the ones running development versions (like fedora rawhide or debian unstable).

16 minutes ago, CosmicEmotion said:

also it's possible to inject malicious code to a compressed packaged as well given that xz was used (which means all kernel packahes and ALL packages on Linux, even Flatpak).

There is NO proof or even any indication of that being the case. Don't go around fearmongering without even posting a source.

17 minutes ago, CosmicEmotion said:

This is the greatest disaster that's happened to Linux

Huh, I guess you haven't seen the whole log4j fiasco, or even a previous openssh vulnerability lol

18 minutes ago, CosmicEmotion said:

I switched to Windows personally

This just makes you sound like you're trolling tbh

CosmicEmotion · March 30

11 minutes ago, igormp said:

The bad actor, yes.

But the actual malicious part only has been getting in place in the last month (just look at the commits), and NO distro has released an infected package as of now in any actual release version, only compromised devices were the ones running development versions (like fedora rawhide or debian unstable).

Arch has released it to mainline for a month now. I know cause I used CachyOS.

Quote

There is NO proof or even any indication of that being the case. Don't go around fearmongering without even posting a source.

There is not but Linux is unreliable on the current state of things. I don't want to be unsure about the security of the OS I'm using.

Quote

Huh, I guess you haven't seen the whole log4j fiasco, or even a previous openssh vulnerability lol

Thank God no. Could you elaborate?

Quote

This just makes you sound like you're trolling tbh

How come? The potential implications of this are immense. I will switch back once all this is resolved. Until then, Windows is the best option since this is a very serious issue.

Ralphred · March 30

4 minutes ago, igormp said:

There is NO proof or even any indication of that being the case. Don't go around fearmongering without even posting a source.

23 minutes ago, CosmicEmotion said:

Agreed, devs don't run "bleeding edge" on the systems they build stable packages on, it's just inviting non-issues to present themselves.

Did Fedora devs build and package on their own "infected systems", I doubt it very much, if I build for other systems I do it on the most mature and stable system available, and all because no one wants a shitty binary or archive.

4 minutes ago, CosmicEmotion said:

There is not but Linux is unreliable on the current state of things.

Not really, the payload was only uploaded 5 weeks ago, so even if it did/does have "other functionality" (like injecting malicious code whilst de/compressing an archive) nothing produced before 23rd Feb has even the remotest chance of being compromised.

If you are concerned because you have had sshd running on an open port exposed on a public IP with RSA priv/pub key authentication enabled and use systemd then run ldd $(which sshd). If liblzma isn't in the list you have nothing to worry about.

igormp · March 30

14 minutes ago, CosmicEmotion said:

Arch has released it to mainline for a month now. I know cause I used CachyOS.

And as posted already in this thread, Arch is unaffected. I do run arch and this isn't relevant tbh. Also, if you don't have your device ssh port exposed to the web there's even less reason to worry about this.

15 minutes ago, CosmicEmotion said:

Thank God no. Could you elaborate?

Oh boy, hope you have some spare time:

https://www.ncsc.gov.uk/information/log4j-vulnerability-what-everyone-needs-to-know

https://www.trendmicro.com/en_us/what-is/apache-log4j-vulnerability.html

17 minutes ago, CosmicEmotion said:

How come? The potential implications of this are immense. I will switch back once all this is resolved. Until then, Windows is the best option since this is a very serious issue.

If you want to worry about possible implications, you're better off not using any device at all. OpenSSH (which was the target of this attack) has vulnerabilities every so often:

https://www.openssh.com/security.html

https://blog.qualys.com/vulnerabilities-threat-research/2023/12/21/ssh-attack-surface-cve-2023-48795-find-and-patch-before-the-grinch-arrives-with-cybersecurity-asset-management

https://blog.qualys.com/vulnerabilities-threat-research/2023/07/19/cve-2023-38408-remote-code-execution-in-opensshs-forwarded-ssh-agent

Vulnerabilities are inherent to software, just patch things and move on. The major scandal with this specific case is that it was made on purpose, and not by an accident or something that was overlooked, so it's a supply chain attack.

I bet you're using tons of software that likely have open CVEs going on at moment, and it should be even worse on the windows side of things.

grg994 · March 30

Summary

Over the course of the last months an obfuscated, multi-stage malicious code was embedded into versions 5.6.0 and 5.6.1 of xz/liblzma - a widely used unix (de)compression software and library - by one of the open-source maintainers of the project^[4].

Before it was discovered around 2024-03-29, the backdoor managed to get deployed to the users of Fedora Rawhide [currently Fedora 41 preview / beta] release^[2] and the Debian testing, unstable and experimental distributions^[5]. Moreover as a precautionary measures an immediate system update is also recommended for Fedora Linux 40^[2] and Arch Linux^[6] users. (Arch users had the compromised package deployed to them - but likely without the active malware - for ca. 2 weeks).

Debian stable, its derivatives (eg. Ubuntu) and Red Hat Enterprise Linux are presumed to be unaffected as they did not update to the compromised version of the xz package yet.

Internally Red Hat assigned the related CVE-2024-3094 a score of 10.0/10 under their own metrics^[3].

The malware was eventually discovered through the performance and stability issues caused as its later stage was performing binary interactions with openssh/sshd. Analysis of this later-stage of the malware targeting openssh/sshd (the most common remote command line login tool on linux) is still ongoing, but presumed to allow unauthorized remote SSH access.^[4]

Quotes

Quote

[2] "the latest versions of the “xz” tools and libraries contain malicious code that appears to be intended to allow unauthorized access. [...] Under the right circumstances this interference could potentially enable a malicious actor to break sshd authentication and gain unauthorized access to the entire system remotely."

[4] "backdoor in upstream xz/liblzma leading to ssh server compromise [...] The upstream xz repository and the xz tarballs have been backdoored. [...] Given the apparent upstream involvement [of some of the xz open-source maintainers] I have not reported an upstream bug. Subsequently I reported the issue to distros[...]. CISA was notified by a distribution."

My thoughts

IMO this is by far the worst supply chain attack in the open-source ecosystem so far. If the backdoor hadn't been discovered and would had been deplyed to millions of Debian and REd Hat linux servers a few months later - that would have been a disaster.

The culprit is likely a state sponsored hacker group, they spent months on trying to do this, mainly a lot of otherwise legitimate open-source development work to infiltrate and cover this up.

Sources

[1] https://www.phoronix.com/news/XZ-CVE-2024-3094
[2] https://www.redhat.com/en/blog/urgent-security-alert-fedora-41-and-rawhide-users
[3] https://access.redhat.com/security/cve/CVE-2024-3094
[4] https://www.openwall.com/lists/oss-security/2024/03/29/4
[5] https://lists.debian.org/debian-security-announce/2024/msg00057.html
[6] https://archlinux.org/news/the-xz-package-has-been-backdoored/

Alex Atkin UK · March 30

This gives an all new meaning to "always update your system to keep on top of the latest security flaws".

Its usually to prevent them, not install them.

Sauron · March 30

11 hours ago, manikyath said:

while this is a potentially very big backdoor.. it seems that humanity had some karma saved up, and a microsoft dev of all people just happened upon the backdoor while working on a dev build.

Microsoft has a lot of people working on various Linux related projects (and Linux itself) so this isn't that weird.

manikyath · March 30

1 hour ago, Sauron said:

Microsoft has a lot of people working on various Linux related projects (and Linux itself) so this isn't that weird.

it's not weird, it's just some mild irony that a microsoft person happened to happen upon the backdoor while working on something entirely unrelated.. as opposed to the maintainers of the affected distro's.

having slept on it, perhaps the power of this exploit is also it's weakness; it fiddled with SSH logins, so anyone doing in-depth testing on performance that just so has happened to SSH into a system with their monitoring running... they would have been likely to catch tomfoolery going on.

beyond that.. this also shows the painful reality of ecosystems like arch linux, as they essentially have no method to protect against this.

Sauron · March 30

34 minutes ago, manikyath said:

beyond that.. this also shows the painful reality of ecosystems like arch linux, as they essentially have no method to protect against this.

I disagree, packages go through validation and testing before they're thrown into the repos and this didn't just randomly occur because of a single malicious commit by a random contributor. Clearly there was long standing malicious intent behind this and collaboration from the upstream developers; hypothetically this could have been done with a "bugfix" of an older release, which could have made its way into any distribution.

Moreover Arch was not affected, so it's possible that if the backdoor had been active in the Arch build it would have been caught by the Arch maintainers - we just can't know for sure. Debian Sid and Fedora Rawhide are considered testing distributions so they don't have the same safeguards as mainline Arch - if you want that experience on Arch you need to look at the testing repo.

CosmicEmotion · March 30

I think this video explains everything pretty well.

Ralphred · March 30

2 hours ago, CosmicEmotion said:

I think this video explains everything pretty well.

The thing he doesn't even brush on is the convergence of configuration choices and system set-up required for the malicious code to even be exploited, so lets list those here:

An ssh daemon running on an open port of a public IP
Said sshd using RSA priv/pub key auth
Be using systemd
Your systemd is built with lzma support
Your openssh has been patched to link sshd to libsystemd
You have an infected liblzma

On point 1, as discussed above with @igormp, there are two types of people who do this; Those who know what they are inviting and are ready to deal with it, and those who "get what they fscking deserve".

Point 2 is pretty much everyone who's ever read a "how to ssh server" tutorial, and it does it by default*.

Point 3 is probably most people, except those who actively avoid it.

Point 4 is going to be distro/personal choice, but again most are probably built with xz compression support.

On point 5 you need to check if your distro does this, I know none I let near open public IP ports do, and neither does Arch.

Point 6 is almost no one as you'd have to be running "bleeding edge" software, most people who do this (who aren't distro testers) chose distro's that allow them to pick and choose which parts should be bleeding edge and which should be stable, based on need.

Don't get me wrong, this is serious and will have repercussions for foss moving forward, but not because "half of all linux servers are infected with malware and are security compromised" because they just aren't and, without further developments in this case, aren't going to be either.

*In sshd_config from the openssh git repo.

Nayr438 · March 30

18 hours ago, Biohazard777 said:

That is just the thing, I don't expect people running production servers to be on rolling releases with ssh ports exposed.

I use Arch in production and expose ssh for git as do many others including Arch itself, so I would say that's a poor expectation.

And even if we ignore Arch there are plenty of people who deploy exposed upstream services via containers in production.

Technically I would fall into both of these.

igormp · March 30

19 minutes ago, Nayr438 said:

I use Arch in production and expose ssh for git as do many others including Arch itself, so I would say that's a poor expectation.

Weird but ok. I guess their phrase could be better placed as "I don't expect people running production servers to be on development releases with ssh ports exposed.", since the major distros at risk were debian unstable, fedora rawhide and opensuse tumbleweed, all of which really shouldn't be used in prod.

22 minutes ago, Nayr438 said:

exposed upstream services via containers in production.

All of those with ssh and that are on the affected distros?

Don't get me wrong, if it didn't get discovered soon enough, then it'd be a real problem and there would be tons of servers affected. It'd only be a matter of time between rolling release distros to proper stable ones to be exposed to this, but as of now the reach of this issue is close to none.

Nayr438 · March 30

52 minutes ago, igormp said:

I guess their phrase could be better placed as "I don't expect people running production servers to be on development releases with ssh ports exposed.", since the major distros at risk were debian unstable, fedora rawhide and opensuse tumbleweed, all of which really shouldn't be used in prod.

That I can agree with. In this case however for it to land in Arch it would have had to land as a stable release rather than a development one, and yes I am aware Arch may have been unaffected. xz-5.6.1 was marked as stable on "2024-03-09".

55 minutes ago, igormp said:

All of those with ssh and that are on the affected distros?

SSH is used for more than just remote access to a shell. For example git as mentioned previously or something like SFTP . As to whether they were affected, I don't believe so, but it was more of a you should consider that even if someone is using say debian in prod, they may be deploying newer packages in some form. And as we have been moving forward this has been becoming more common.

And while this may have not affected any major production instances it was more a jab at the original comment of the expectation that Arch/Rolling Releases should not be run in production and SSH should not be exposed, when there are valid reasons for both.

igormp · March 30

32 minutes ago, Nayr438 said:

the expectation that Arch/Rolling Releases should not be run in production and SSH should not be exposed, when there are valid reasons for both.

While I do agree with the point that was made, I guess this kind of discussion ends up being a bit off-topic since it's more of a personal preference on how each one wants to run their servers.

33 minutes ago, Nayr438 said:

In this case however for it to land in Arch it would have had to land as a stable release rather than a development one, and yes I am aware Arch may have been unaffected. xz-5.6.1 was marked as stable on "2024-03-09".

This circles back to the idea of what I said before it this wasn't caught soon enough. Arch was not affected by a minor technicality in the exploit process, but with enough time this would be irrelevant given the amount of systems that would actually be affected.

BoomerDutch · March 31

So if i understand correctly normal installations on normal download page

On most distros pages should be fine?

Including arch with archinstall?

Kinda sucks that this happend and glad it was on time.

It is a blow to Linux

Minecraft and now linux

And sometimes i have difficulty telling differences between packages which is bleeding edge and which is not.

Woo more work for me.

Sign In

Backdoor in upstream xz/liblzma leading to SSH server compromise

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites