Backdoor in upstream xz/liblzma leading to SSH server compromise

[Nayr438]

57 minutes ago, BoomerDutch said:

So if i understand correctly normal installations on normal download page

On most distros pages should be fine?

Including arch with archinstall?

It's complicated.

The Affected package entered Arch repositories in 5.6.0 on February 24 and wasn't patched until Mar 28 in 5.6.1-2.

The malware targets libsystemd's notification daemon, so anything that depends on it or even potentially just xz/liblzma itself could be compromised.

The primary affected target however was OpenSSH, but relied on a patch to integrate with libsystemd's notification daemon, something Arch itself doesn't ship but most other distros do.

 

The primary affected distros to my knowledge are Arch, Gentoo, OpenSUSE Tumbleweed, along with various in-development/unstable builds of distros.

That isn't to say it couldn't have been shipped in containers or sneaked it's way in through third party repositories.

[BoomerDutch]

1 hour ago, Nayr438 said:

It's complicated.

The Affected package entered Arch repositories in 5.6.0 on February 24 and wasn't patched until Mar 28 in 5.6.1-2.

The malware targets libsystemd's notification daemon, so anything that depends on it or even potentially just xz/liblzma itself could be compromised.

The primary affected target however was OpenSSH, but relied on a patch to integrate with libsystemd's notification daemon, something Arch itself doesn't ship but most other distros do.

 

The primary affected distros to my knowledge are Arch, Gentoo, OpenSUSE Tumbleweed, along with various in-development/unstable builds of distros.

That isn't to say it couldn't have been shipped in containers or sneaked it's way in through third party repositories.

Yikes, i geuss i have to check all my systems while having mental breakdown.

[igormp]

32 minutes ago, BoomerDutch said:

Yikes, i geuss i have to check all my systems while having mental breakdown.

First of all, do you have any system that has ssh exposed to the internet? If so, check those first. If they're one of the systems that are really affected (fedora rawhide, opensuse tumbleweed, kali or debian unstable) then do a really good audit in those systems. Otherwise, if they still have ssh exposed but are not one of those distros, then just update xz and you should be good to go.

 

If none of the above applies, you don't really have much to worry nor need to rush anything, just update xz as you would update all of the system anyway.

[Ralphred]

56 minutes ago, igormp said:

have any system that has ssh exposed to the internet

That reminds me: There was an update late last night showing the exploit was for REC and not authentication bypass. Whilst this is moot from a "consequence" point of view it does give a springboard for log investigation (read: connections to port 22 that don't attempt any authentication).

[colonel_mortis]

Honestly I find this attack pretty terrifying - this is the second example (that we know of) of a very well implemented supply-chain attack (the first being SolarWinds), and it was only caught by chance by someone noticing that OpenSSH was being slow. It is entirely plausible that the perf regression could have gone unnoticed (or, although I don't have a deep understanding of what it was trying to do, I suspect it could also have been possible to write the payload in a way that doesn't cause such a perf regression at all), resulting in this malicious release making it out of the bleeding edge and into mainstream distributions.

 

I hope this will lead to some changes in the industry, but I don't know what those changes could be. Now that the concept has been proven, I doubt that this will be the last time something like this is attempted. It's not a trivial attack to pull off, but nor is it overly difficult as long as you have time to burn (in this case the attacker started getting a foothold 2 years ago) - it would be a great choice for nation-state attackers, but could also be pulled off by solo attackers.

 

The scariest thing to me is that this may not be the first time - for all we know, and with no way to verify, there may be other compromised libraries out there already.

[Ralphred]

2 hours ago, colonel_mortis said:

but I don't know what those changes could be

Well to circumvent this one binary blobs need to be verified and signed by multiple stakeholders, or outright excluded, and instead of pulling archives from git repos pull via git at a specific point in the repos history. If both thees practices were followed the two vectors used in this attack wouldn't have been available.

I'm sure smarter people than me have more practicable solutions though.

[colonel_mortis]

10 hours ago, Ralphred said:

Well to circumvent this one binary blobs need to be verified and signed by multiple stakeholders, or outright excluded, and instead of pulling archives from git repos pull via git at a specific point in the repos history. If both thees practices were followed the two vectors used in this attack wouldn't have been available.

I'm sure smarter people than me have more practicable solutions though.

On the first idea (multiple signatures), that assumes there's some way for one stakeholder to prove to the other that the package is legitimate, which I don't think is possible - ultimately someone (or some CI build, which would make the attack look more like the SolarWinds attack) has to generate the package, and there needs to be trust there. I believe there is some sophisticated systems that could be built involving reproducible builds and stuff that could make that work, but that is not going to be feasible for most small OSS packages. Remember, this repo was previously only maintained by one (trustworthy) person.

 

I'm not sure I understand the git comment.

[Ralphred]

6 hours ago, colonel_mortis said:

that assumes there's some way for one stakeholder to prove to the other that the package is legitimate

Binary blobs don't just appear, and you are seeing a narrow view of the term stakeholder; if you include package maintainers for effected distro's in that cohort and poll them I'm pretty sure they'd be happy to to produce and compare their "blobs" between themselves.

6 hours ago, colonel_mortis said:

I'm not sure I understand the git comment.

It's a reference to how the git repo itself was clean, but the release archives offered were different to the repo.

Yes, there are reasons for this to happen, but everywhere in the "real world" security is always usability trade off, nearly always at the cost of "upstream" work to maintain as much "usability" as possible at the point of actual use; just think of any security measure you have ever interacted with and you'll be able to see the administrative or usability cost associated with it's existence. Back onto FOSS software though, a mechanism to check that the offered archive was a facsimile of the repo would frustrate this malicious code obfuscation method.

[colonel_mortis]

10 minutes ago, Ralphred said:

Binary blobs don't just appear, and you are seeing a narrow view of the term stakeholder; if you include package maintainers for effected distro's in that cohort and poll them I'm pretty sure they'd be happy to to produce and compare their "blobs" between themselves.

This works if builds are reproducible (ie if two people building the same release will get the same bytes out), and that is a good thing to aim for, but in practice at the moment it is surprisingly common for the output to be affected by things like the versions of dependency libs you currently have installed, compiler version, current time, etc. Progress has been made towards reproducible builds in many languages, but that has taken a fair amount of work, so it seems unlikely that it can happen any time soon for everything. I hope we do start seeing reproducible builds in more critical infrastructure so that we can make progress towards this world, and maybe that's the best we can hope for, but it seems unlikely to me that a package like this, with a single maintainer that wasn't particularly motivated to work on it, would do that work.

 

It's still not a silver bullet though - this attacker already demonstrated an ability to use sock puppet accounts to achieve their goals, so in this case it would likely have only represented a minor inconvenience for them. Of course, multiple minor inconveniences can quickly add up to a significant increase in the effort required to pull off an attack like this, especially doing so undetected, so it would still be valuable.

22 minutes ago, Ralphred said:

It's a reference to how the git repo itself was clean, but the release archives offered were different to the repo.

Yes, there are reasons for this to happen, but everywhere in the "real world" security is always usability trade off, nearly always at the cost of "upstream" work to maintain as much "usability" as possible at the point of actual use; just think of any security measure you have ever interacted with and you'll be able to see the administrative or usability cost associated with it's existence. Back onto FOSS software though, a mechanism to check that the offered archive was a facsimile of the repo would frustrate this malicious code obfuscation method.

Ah I see - yes, I agree that would clearly be a good thing. Again not a silver bullet - they did still sneak multiple changes into the repo itself in plain sight to set the groundwork for this attack, and it doesn't address any binary distribution avenues where simply taring a git repo is not sufficient - but clearly a good thing. It sounds like there are good reasons for the current setup, but that is definitely something we should be moving away from.

[Ralphred]

13 minutes ago, colonel_mortis said:

It's not a silver bullet though

Nothing ever is, just another 'layer of frustration'; just keeping closing the doors if bad actors find them and lock the ones we can predict they'll try to open.

I previously posted "I'm sure smarter people than myself have more practicable solutions though.", I should have included effective in that too.

19 minutes ago, colonel_mortis said:

This works if builds are reproducible

You are being overly broad with the term "binary blob". In this case we are not talking about the output of an entire package build, which yes would require significant controls to produce the same output, but some binary test files which could be reproduced programmatically in a fairly simple controlled environment.

 

The most important thing is though, if a couple of schmoes in a tech forum can have a productive discussion about ways of thwarting similar attempts moving forward, all hope is not lost 

[igormp]

19 hours ago, Ralphred said:

Well to circumvent this one binary blobs need to be verified and signed by multiple stakeholders, or outright excluded

Excluding those is not doable IMO (see linux-firmware), and verifying/signing them wouldn't amount to much IMO, if any one saw the bad blob in this case without a proper context they wouldn't get anything meaningful and believe that it's just a proper bad test file.

19 hours ago, Ralphred said:

and instead of pulling archives from git repos pull via git at a specific point in the repos history

That I can agree with.

[10leej]

On 3/29/2024 at 5:35 PM, CosmicEmotion said:

Red Hat announcement.

 

 

The guy who is responsible for this has been part of  the project for 2 years.

 

Noone is safe and if your system is compromised already you have to reinstall and change all your credentials. I personally went with Fedora Kinoite and an ecrypted install.

 

The perpetrator made this commit yesterday ironically.

This issue might already be resolved by your Distributions package maintainers.

[wasab]

reminds me of the time when a group of university researchers deliberately committed malicious codes to the linux kernel for ah, academic purposes.

[Dat Guy]

On 3/29/2024 at 11:25 PM, manikyath said:

so... "no one is safe" because of a commit that only made it into a dev build of fedora and debian?

 

OpenBSD users are safe.