That’s a cop-out. It should be the case that even administrator access should not be abusable to implant permanent backdoors.
Anything that makes privileges escalation exploits more damaging is a real problem. I’m getting tired of how these are being dismissed as if admin access should mean that you can ignore any security issues. There are things that even admin accounts should not be able to change at the hardware level, or if they can they must be reversible in the future by another user with admin access.
> The BMC needs to be ideally on a physically isolated network, or at least a separate one that has no route from the outside nor on the machine itself.
This is good practice but it shouldn’t excuse poor security at the hardware level.
Supermicro motherboards also commonly default to having a feature that bonds the BMC network interface to one of the main NICs if you don’t plug a cable into the BMC interface. It’s common for people to be surprised that their BMC is exposed on their main network because they didn’t plug in a cable on the BMC NIC port at all.
> It should be the case that even administrator access should not be abusable to implant permanent backdoors.
It's really the "permanently" which is the design flaw. Boards should have a mechanism to recover from bad firmware, and the same mechanism is useful to recover from a bad flash.
Make the flash chip removable, or leave a JTAG. Or have a bit of actual ROM with the write lines not even connected and just enough of a firmware to be able to reflash the main one.
The EFI firmware flasher or some other method needs to be able to erase and flash the entirety of BMC/SOC/BIOS ROMs from a trusted environment not dependent on a potentially-infected APT BIOS. Perhaps the SD card slot on many Supermicro boards and/or certain USB port should be able to flash a bricked BIOS & BMC just like some of the Asrock boards can flash themselves without booting using an additional button-holding sequence.
It is removable, by desoldering. This is not uncommon and Ars's sensationalized reporting does not help
This is exactly the kind of barrier you want for something with so much power over the system, otherwise you're not much better off than where you started as physical access allows for quick swaps of chips.
Desoldering is ridiculous. It's much more likely to damage the board, requires a much less common level of skill and doesn't allow you to check the existing data or do the reset prophylactically without performing the dangerous and expensive operation.
Meanwhile it provides no meaningful resistance against physical access because someone with physical access can swap the entire board or a dozen other things.
Many Supermicro server motherboards I've seen place both the BIOS flash chip and the BMC firmware flash chip in a SOIC socket, so that the flash chip can absolutely be removed without desoldering.
I have replaced thousands of flash chips on a running server farm, the guy who did the soldering had a 100% success rate in the end. My part was not perfect, so I agree it was hard but perfectly doable.
In theory, desoldering works. But so would scrapping & replacing all your servers after any "attacker might have gained BMC access" security incident.
You might see that as a facetious comparison. But the number of orgs which actually would desolder the chips in that circumstance is very close to the number which actually would scrap and replace. And if 99% of orgs won't actually do it when needed, then a "works in theory" method of re-securing servers is real-world useless.
The switch alone does not provide security if the supply chain is compromised. I believe a malicious actor could act along this chain by setting the switch to ON and rewriting the firmware, just like they would replace a removable chip. A step in this direction has been taken by "Server Configuration Lock" (e.g. HPE) while servers are in transit
I don't disagree, but if you have admin access to the BMC you can access the console, reboot the machine into single-user mode or even into another OS entirely, and then implant any malware you want, wipe the storage, etc.
Some of the Supermicro boards don't even have a separate BMC NIC, the only choice is to bond it to one of the main NICs or sacrifice one of them to be BMC only. I try to pay attention to that now after being surprised by that once on some servers we bought.
> I don't disagree, but if you have admin access to the BMC you can access the console, reboot the machine into single-user mode or even into another OS entirely, and then implant any malware you want, wipe the storage, etc.
Yes, all of which can be reversed by another admin in the future. That is expected.
It should not be the case that getting admin access one time can result in modifying the hardware in a way that can’t be reversed by future admin, short of physically reflashing the chip on the board.
On some boards you can access can reconfigure GPIO pins of the chipset and bitbang SPI from the application processor (aka your normal x86_64 CPU) without firmware support.
> but if you have admin access to the BMC you can access the console, reboot the machine into single-user mode or even into another OS entirely, and then implant any malware you want, wipe the storage, etc.
True in the common case, but this can/should be guarded against by disk encryption and secured boot chains.
> Supermicro motherboards also commonly default to having a feature that bonds the BMC network interface to one of the main NICs if you don’t plug a cable into the BMC interface. It’s common for people to be surprised that their BMC is exposed on their main network because they didn’t plug in a cable on the BMC NIC port at all.
Even if you changed the default from "failover" in the firmware you're just one CMOS reset (for whatever reason) away from it reverting back to putting the BMC on whatever network interface. This should really be something you can jumper off.
Well they'd still have the right, just not the ability (this is actually a distinction US courts have made regarding arbitration clauses and legal recourse).
This logic doesn't hold. If I choose to dban or degauss something, I don't expect I should be able to recover it later. Admins absolutely have the option to make irreversible changes, and do this quite often.
If you are magnetically destroying hard drives as part of decommissioning, that's not really the same thing. You're not using admin access to do it, and you're not making a change that permanently applies to all future use of the device (because there is no future use).
Just because some administrative decisions are permanent and destructive doesn't mean that every operation should be made permanent or destructive.
Should every software config require buying new hardware because the initial config gets permanently flashed with an e-fuse to only allow a single write? You could even make a security argument for such a setup, but good luck getting approval for your 15th motherboard this quarter because you typo'd the config.
Also, dban and degaussing is not entirely equivalent -- from a practical perspective the equivalent is hard drive shredding (because the hardware cannot be used again in the old/non-malware config -- dban and degaussing are more like factory default resets). Do some organisations need to do this? Sure. Should we design systems with the assumption that any mistake means that the hardware is destined for the shredder? I would hope not...
Degaussing (for orgs who do it) is just as operational a task as updating. There will be an SOP that covers data storage decommissioning. It's not as frequent, but it's not any less 'normal', certainly not ad-hoc or one-off. You don't invest in a degausser "just in case".
> It’s common for people to be surprised that their BMC is exposed on their main network because they didn’t plug in a cable on the BMC NIC port at all.
Is this documented?
If it's documented, it's not an excuse for not reading documentation. You're not supposed to throw stuff (hardware or software) in production without reading the documentation.
I've made an effort but I can't find where this is truly irreversible. It just talks about usual things like reinstalling the OS not working because the malware is active before EFI. I didn't see reference in either this article or the linked one from Binarly.
I don't work with physical servers, so this is a gap in my knowledge. Isn't it the entire purpose of BMCs to allow for remote management?
So you'd definitely have to have it connected to the internet somehow, even if very indirectly, and in an independent manner (different network with no direct routes).
Of course a network can be offline. I believe that is what you describe, a network with no routes is not connected to anything else, and certainly not to the Internet?
It is common to keep admin and backup functions on separate network interfaces, on a disconnected network. You have to physically connect to the network in a secure location to use it.
No, that is not common. Management networks are almost never air gapped, they're just segregated from publicly-accessible or higher-exposure networks (DMZ, and hopefully prod). Requiring a (role-restricted) VPN connection is the most common way to control access to management networks.
Yes, you need some way to connect to the BMC. That is never going to be something like "eh just throw it on the public internet" or similar. It's going on a separate, isolated network with strict access controls.
There are others here more qualified to comment, but I do have some old rack mount enterprise servers making a racket in my basement, so I can provide some sort of half-informed opinion here--
Besides the security issue with the BMC here, it sounds like SuperMicro _also_ has absolutely insane defaults.
Every system I have has a separate, dedicated NIC for the BMC. There is an _option_ which is disabled by default to instead have it share one of the other interfaces. So the only ways you're going to "accidentally" put the BMC on an insecure network are:
1. Rebooting the server, going into the BIOS configuration, going into the options for the BMC, and explicitly telling it which other network device to use.
2. Physically accessing the server, attaching an ethernet cable to a clearly labelled BMC port, and attaching the other end to an insecure network.
From what I'm reading here, SuperMicro's default approach is apparently more "eh, just use whatever happens to be plugged in at the time". So even if you do have it running on a separate, secure interface... if someone unplugs it, it's going to switch to using whatever other connection is available!
To connect to the BMC on my servers you first need to be on the internal network already. Then you connect via wireguard to the router on the rack. Then you can connect to it and given a username/password you can log in. I would be pretty pissed if a cable got unplugged and that meant that the server decided to instead throw the BMC _on the interface connected directly to the public friggin' internet_. And this is just equipment I use for play and hosting my own random junk!
Yep. Defense in depth. Port-based VLAN at a minimum. Only total morons place their web-based and ssh remote/KVM management directly on the internet. Note that many Supermicro boards permit (as an option) to use the regular on-board NICs reducing the need to install a second or third cable for the BMC only adapter on mixed networks. Generally, one shouldn't be placing any box of any sort directly on the 'net unless it's hardened and behind sufficient network/application filtering.
Unremovable through remote access, but reliably removed by reflashing the firmware through JTAG, or whatever interface used during manufacturing to initially load the firmware.
bios flashes are typically preflashed prior to soldering to boards -- some vendors route jtag or spi contacts, but you're more likely to need a vampire/pogo clip on for the TSOP or equivalent chip, or have fun with resoldering the bios flash if you're needing to recover from this.
It's not impossible to do in the field, but you can't really count on vendors surfacing that interface usually.
I've worked with automated EEPROM/Flash programmers (earlier versions of this line: https://www.bpmmicro.com/device-programmers/automated-progra...), and used pre-programming services from distributors, like Digi-Key, but that was the exception. It's almost exclusively faster, cheaper, and easier to load firmware from a test fixture. You need to test the assembly anyway, and it's much easier to update a test procedure, when a new firmware is developed, than to update and track inventory of pre-programmed devices, especially when different firmware versions are needed for different hardware variations.
Pogo pins are only really needed for mass production, especially for reducing repetitive stress injuries. For one-off updates, if a header isn't populated, it's easy to hold an unsoldered header in place, for long enough to flash an update.
Different. In 2018 there was articles [0] talking about hacked Supermicro servers being in the wild that had (simplifying) extra ICs in their motherboard, like some short of supply chain attack.
If you expect the BMCs of other vendors to be significantly more secure than the average home router from 2010 I have several bridges you may be interested in.
The story is called "The Big Hack" and was run by Bloomberg in 2018. All the major companies named in the story conducted investigations and never found anything, but the story was never retracted.
If you've never used the BMC on a server... it is all 100% garbage. Software mostly written by embedded folks who haven't got a clue. It is absolutely garbage software on the whole (and no matter what vendor you get the board from). Go ahead and hit up the web interface then do a bit of "View Source". If you are imagining the rest of that stack is any better than my friend have I got a Beautiful Bridge in Brooklyn to sell you!
If it were me I'd assume the majority of BMC firmware out there from all vendors:
1. Is full of many many exploitable vulnerabilities
2. To the extent they patch holes it will be whack-a-mole because the economics do not permit large investments in software quality.
3. Many server owners will never install a patch anyway.
BMC software quality is low but what's the alternative? Without BMC it is more expensive to manage a fleet of servers. In a better word hardware vendors will publish specs to allow open-source BMC firmware but for some reason they resist this idea. Having only insecure BMC available a semi-separate management network (connected via a bastion host or a VPN) provides balance between cost and security.
This won't scale. Dedicated KVM needs you as an admin walking to the server, reswitching cables, walking back to the KVM console. Instead, with Out of band managament hw/sw, you spawn a dedicated ethernet and can access it from anywhere. It is a flexibility advantage on the costs of security.
There are boxes that can KVM to multiple servers at a time. You don't need to switch cables. They probably cost similar or less than BMC cards on a per-port basis. You might have to combine with some sort of network boot to set up a machine from scratch.
> They probably cost similar or less than BMC cards on a per-port basis
If you build own servers that's an option to consider but most off-the-shelf servers are sold with BMC (so you pay for it even if don't want it). May be some low end brands sell servers without BMC but if you are looking for relatively reliable hardware you'll likely get a server with BMC.
I was thinking more like just having one IP KVM per server always hooked up to a dedicated management network, basically used exactly like a BMC just with better software.
There is the open source OpenBMC software nowaydays, which is pretty good.
Unfortunately, Supermicro doesn't use it yet for most of their servers. Probably because they sell an extremely expansive license for their own software so you can use the Redfish API.
The one vendor mentioned in the comments, AMI, is switching this code base to openbmc. Also it should be noted that often this software is system specific.
And of course there's oxide.computer making motherboards with full control of the entire hardware stack running open source software specifically for customers like the Idaho National Laboratory.
As far as I can tell it's one of their most significant features, and certainly one of its most capital intensive. And I don't know how many startups land institutions like INL as their launching customer, and enough other big enterprise leads to warrant their follow up investment.
If you remove that component from their value prop, they're not that much different from Dell.
Just a quick point of clarification that while our boot architecture is very important (e.g., a service processor in lieu of a BMC, the elimination of UEFI entirely, etc.), we are quite a bit different from Dell beyond that. There are certainly many hardware-level differentiators (e.g. DC busbar-based design, blindmated networking, built-in switch, etc.) but the big differentiator is really what these things allow: entirely integrated software. The Oxide rack comes with all of the software to run elastic infrastructure (that is, the distributed system that comprises the control plane), including switch software, storage software, etc. And then (critically!) the capacity to update all of this.[0]
All of it is a far cry from the offerings of Dell/HPE/Supermicro, which rely on others to provide the software that turns the hardware into real infrastructure.
Right, my apologies for downplaying it. That you control both the hardware and the software through the entire stack has tremendous benefit, especially for this type of customer. My point was if you as a customer don't value that aspect of Oxide's pitch, which you obviously are much better at fully conveying, then you'd be seriously considering Dell/HPE/Supermicro as well.
For what it's worth the last time I had to procure a small cluster the offer Dell made was ridiculously overpriced, didn't meet the specifications I asked for, and the whole experience didn't inspire confidence in the software they were pitching either. We were a small underfunded startup so we were never gonna drop 200k on that storage solution (0.5PB HDD + ~80TB SSD), but if we did have that kind of budget we probably would have gone with one of the smaller but more focused parties there instead like maybe TrueNAS, I bet there's a beautiful market for Oxide as well in that segment.
Every modern motherboard comes pre-installed with unremoveable malware. Vulnerabilities just let different actors try to be king of the hill. The real answer is that all these persistent storage areas (eg flash chips) need to be documented and writable by the user. Any boot integrity needs to be done on top of an open environment, rather than continuing to rely on security through obscurity.
I see the point you're trying to make but it's really very orthogonal to this issue: this was an issue in a documented part of a documented feature, it's not some "deeply embedded" system management controller with no documentation, it's "the signed firmware update feature in the big obvious selling point where the server has a backdoor management interface was broken."
Most run on hardware that is not supported by seL4, or at least on hardware where it has not been validated.
Not to mention that a task manager would be needed as well as tons of other services which aren't provided out of the box, and don't share the verification provided guarantees.
My favorite supermicro facepalm will always be when you could set the IPMI encryption cipher to "none" (ipmitool -C0) and bypass actually needing any password at all. (Though I don't think this was unique to supermicro actually?)
With some server vendors, if you don't connect an ethernet cable to the BMC, it can intercept BMC-targeted traffic from the OS-connected ethernet port.
Pretty much all of them allow unrestricted access from KMS from factory, tough all of them have a way to disable it once configured, and HPE even throws shade until it's limited. KMS only works from the host itself.
isn't the simple answer to infect all.of these servers with a benign infection, a vacine if you will, that blocks further attempts to remotely flash the bios
Or two physical firmware chips: one writable, one with no write ability and is a fallback. Then a physical switch, could even be a jumper, to select the fallback. If compromised you flip the switch, boot from the clean firmware, flash the writable chip, flip switch and reboot. I am pretty sure Gigabyte offered this same setup with Dual Bios or something like that.
Gigabyte made a lot of marketing hay about it, but I think it was popular for a while. I think their version was some sort of watchdog/failover model where it would automatically load the backup BIOS, but some other firms had a secondary-BIOS jumper.
I think these days, the stub "BIOS flashback" is the trendy thing, where you can plug a flash drive into a magic slot and press a button to flash without even having a CPU installed.
This offered the same "brick-resistance" feature with the added benefit that people weren't stuck if they tried to pair an old-stock mainboard with a new CPU that wasn't supported by the original firmware release.
TBH, I'd rather they go the complete opposite direction: replace the soldered EPROM with a SD slot and a $1 MCU that reads the card and emulates a ROM chip. That could be configurable to write-protect the card, or you could just trivially swap it if you didn't trust the firmware image for any reason, while avoiding the fumbliness of modern tiny 8-pin flash chips. You could socket a big old-fashioned DIP ROM, but will people feel comfortable even trying to pry that out of a $10,000 server even with the appropriate chip puller tool?
Pulling a physical chip to upgrade the firmware would generate so many returns or RMAs that it would be dropped as a feature immediately.
These days it’s common to do firmware updates to address small issues or even support the new CPU that was launched after the motherboard was manufactured.
I could see manufacturers adding a write-protect physical switch for those who want it. Make it opt-in and toggleable.
AMD have recently changed the firmware loading signature verification method to apply cpu microcode that uses the on-motherboard tooling.
Using the method you talk about would mean that this kind of update wouldnt be possible, 99% of users would never toggle with a switch to update firmware.
This would be a huge burden in the server world too, to unrack flip switch, update, revert switch re-install.
I assume you mean specifically motherboard firmware updates, because firmware updates are actually pretty common, for most server grade motherboards vendors ship updates about every other month[1].
Some motherboards just have a physical jumper that prevents BIOS flashing. This happens infrequently enough as to warrant it for one server, or 10 servers, or maybe 100 servers. Likely unpractical for 1000 servers though.
If they can put the jumper on the exterior it might be feasible, if its inside its out of the question if you have to unrack the chassis to change. Rolling in a server lift for an 8u thats half full of copper is not a nice process
OpenCompute (OCP) Caliptra is an effort by hyperscalers, AMD and others to enforce a platform root of trust with OSS firmware and open silicon, mandating dual signature by server OEM and hyperscaler customer. The platform RoT is responsible for validating device firmware and OS boot, https://www.youtube.com/watch?v=p9PlCm4tLb8&t=2764s
> Often we see.. great security.. compromised by other great ideas for mgmt and other things.. starts to weaken its security posture.. want to keep Caliptra very clean [via OSS firmware transparency]
Not at all, that piece described a supply chain attack replacing a component with a look-alike part analyzing tens to hundreds of gigabits if data, in a form factor so small that it wouldn't be physically possible without semiconducting fabricating processes years in advance of what existed at the time.
What this article is describing is something far more likely— a firmware attack that doesn't require specialized hardware.
FWIW, the claim in the Bloomberg article was never validated, no one pulled a Supermicro server that had the supposed components. Zero proof since the story was published in 2018 that it wasn't nonsense.
Yeah, it was a pretty bizarre story, given it's the kind of thing that definitely happens (though probably through more straightforward just tampering with the firmware), that it was very specific and yet all the details failed to add up.
Including Amazon, Meta or whomever else was mentioned in that article, all saying "What? That never happened". There's little reason to cover up the discovery of something like this, from an end-user perspective.
Wasn’t the implant supposedly (illogically) implanting custom BMC firmware? This actually always struck me as a somewhat unbelievable part of the story: why install a hardware implant when you could just clip the flash chip and implant something without a physical trace?
The Bloomberg article specifically mentioned the attack occurring through the addition of extra hardware:
“In early 2018, two security companies that I advise were briefed by the FBI’s counterintelligence division investigating this discovery of added malicious chips on Supermicro’s motherboards,” said Mike Janke, a former Navy SEAL who co-founded DataTribe, a venture capital firm. “These two companies were subsequently involved in the government investigation, where they used advanced hardware forensics on the actual tampered Supermicro boards to validate the existence of the added malicious chips.”
Flash can always be reflashed (you'd lose the implant if the customer does any firmware update) but a separate implant chip can remain infected forever.
That’s fair - the first thing I’d put in my implant firmware would be a fake firmware updater, but I suppose trading guaranteed persistence for physical detection would be a reasonable tradeoff in some places.
> with a look-alike part analyzing tens to hundreds of gigabits if data, in a form factor so small that it wouldn't be physically possible without semiconducting fabricating processes years in advance of what existed at the time.
I'm not sure where you got that idea. The article describes a tiny microcontroller, attached to the read pins of the BMC's boot flash, flipping a few bits in transit from the flash ROM to the BMC SoC as the BMC boots. This is not only practically possible, it's very similar to the technique used to hack the original Xbox and by many console mod chips. And is sufficient to boot the BMC in a vulnerable state for the next chain of an attack.
Nothing about the exploit claimed in the article was impossible or even novel.
That said, I'm not aware of any physical boards found to have the compromised hardware outside of those Bloomberg claim to have witnessed.
People like to criticize secure boot around here, but it prevents these kinds of infections (provided of course there are no vulnerabilities in the implementation of secure boot).
Yes, in theory it is possible to prevent these kinds of infections without resorting to secure boot (e.g., by insisting that all the suppliers of components of the motherboard start designing components that cannot be pwned) but so far all the computers you have actually been able to buy that are immune to these kinds of infections achieve that immunity with secure-boot technology.
Can you please explain how Secure Boot helps at all to mitigate this type of attack? I don’t see how it makes it harder to execute the attack, and I don’t see how it has any particular effect on the capabilities of the attack once executed. To the contrary, a BMC compromise of this sort seems like it should be able to arbitrarily override secure boot settings.
It seems to me that, in this situation, secure boot’s only role is to provide a false sense of security, which could make recovery from the attack less likely.
In contrast, verified boot might somewhat mitigate the damage from being able to use the BMC to write arbitrary data to the SPI flash chip. Emphasis on might — at best I expect that it would require an attacker to be a bit more creative in how they design their exploit payload.
> Can you please explain how Secure Boot helps at all to mitigate this type of attack?
Secure boot can include the hash of the firmware, computed by the root-of-trust that can't be tampered with by this attack. So the exploit will make the keys stored in the TPM inaccessible.
This will make the tampering conspicuous, at least.
I agree in general; PCRs provide some basic degree of protection against this. Unfortunately, the position these management controllers are in often grants memory access, which renders all of the boot measurement type security methods useless. Even if it doesn't, there's also the notion that an attacker will replace the firmware from the very start with one that fakes the PCR hashes which are sent to the TPM. Unfortunately, this isn't really very hard with most UEFI implementations.
I have never seen someone make a distinction between Secure Boot and boot verification on UEFI/x86, although I suppose it’s a valid one? I suspect the parent post is referring to Secure Boot in the colloquial sense it’s usually used for on x86: validation of the UEFI boot binary using Secure Boot keys followed by OS level trust chaining, usually accomplished by entangling PCR state with disk encryption in some way.
And I think this would deliver a slight level of protection from the BMC: tampering with the firmware image or key enrollment / secure boot state _should_ both break the UEFI root of trust and alter the PCR state and break everything downstream. Of course, all UEFI implementations are holier than Swiss cheese and there are probably a lot of ways to use the BMC to dump or modify memory post-boot anyway, but it does do something.
You mean the same Secure Boot that I mean. This is a software mechanism (almost universally implemented as an unholy mixture of ordinary code in firmware and a pile of immensely complex SMM code that, IMO, is entirely unnecessary if the hardware or the BMC gives even the slightest bit of help).
And Secure Boot is implemented in, and configured by, the firmware that the BMC can overwrite at its whim while entirely bypassing all the fancy CPU-hardware and SMM protections that are supposed to prevent writing arbitrary data to it.
To the extent that a mechanism not controlled by firmware will detect such an attack and extend PCRs accordingly before executing a single instruction from the compromised flash chip, it might partially mitigate attacks. But that part isn’t Secure Boot.
Oh, yes, we 100% agree on this, the true root of trust for firmware execution exists before and independently of “secure boot,” and therefore, often not at all (and “secure boot” is a terrible name).
How does secure boot help? If you control the BMC, you can enroll whatever keys you want.
The BMC usually has full access to system memory as well, so if you can get the timing right, you could replace the secure boot verified image with your own after verification.
Also, re: BusinessWeek, hey look a hardware backdoor installed on servers. Pretty sure IPMI vulnerability fits the bill for most of what was described.
"If a potential attacker already has administrative access to the BMC..."
Then you've already lost.
The BMC needs to be ideally on a physically isolated network, or at least a separate one that has no route from the outside nor on the machine itself.
That’s a cop-out. It should be the case that even administrator access should not be abusable to implant permanent backdoors.
Anything that makes privileges escalation exploits more damaging is a real problem. I’m getting tired of how these are being dismissed as if admin access should mean that you can ignore any security issues. There are things that even admin accounts should not be able to change at the hardware level, or if they can they must be reversible in the future by another user with admin access.
> The BMC needs to be ideally on a physically isolated network, or at least a separate one that has no route from the outside nor on the machine itself.
This is good practice but it shouldn’t excuse poor security at the hardware level.
Supermicro motherboards also commonly default to having a feature that bonds the BMC network interface to one of the main NICs if you don’t plug a cable into the BMC interface. It’s common for people to be surprised that their BMC is exposed on their main network because they didn’t plug in a cable on the BMC NIC port at all.
> It should be the case that even administrator access should not be abusable to implant permanent backdoors.
It's really the "permanently" which is the design flaw. Boards should have a mechanism to recover from bad firmware, and the same mechanism is useful to recover from a bad flash.
Make the flash chip removable, or leave a JTAG. Or have a bit of actual ROM with the write lines not even connected and just enough of a firmware to be able to reflash the main one.
The EFI firmware flasher or some other method needs to be able to erase and flash the entirety of BMC/SOC/BIOS ROMs from a trusted environment not dependent on a potentially-infected APT BIOS. Perhaps the SD card slot on many Supermicro boards and/or certain USB port should be able to flash a bricked BIOS & BMC just like some of the Asrock boards can flash themselves without booting using an additional button-holding sequence.
It is removable, by desoldering. This is not uncommon and Ars's sensationalized reporting does not help
This is exactly the kind of barrier you want for something with so much power over the system, otherwise you're not much better off than where you started as physical access allows for quick swaps of chips.
Desoldering is ridiculous. It's much more likely to damage the board, requires a much less common level of skill and doesn't allow you to check the existing data or do the reset prophylactically without performing the dangerous and expensive operation.
Meanwhile it provides no meaningful resistance against physical access because someone with physical access can swap the entire board or a dozen other things.
Many Supermicro server motherboards I've seen place both the BIOS flash chip and the BMC firmware flash chip in a SOIC socket, so that the flash chip can absolutely be removed without desoldering.
I have replaced thousands of flash chips on a running server farm, the guy who did the soldering had a 100% success rate in the end. My part was not perfect, so I agree it was hard but perfectly doable.
Afaik the SuperMicro still uses non-BGA flash chips that can be accessed with a vampire clamp without desoldering.
If you want to avoid quick swaps so it takes slightly longer to compromise, that's fair. But that means you should go with the "actual ROM" option.
And if you need to desolder to remove the malicious code, it's pretty reasonable to call it unremovable.
Other than hobbyists and maybe some high security environments, no IT department desolders components from servers.
How many times have you removed a chip from a motherboard by desoldering?
It’s not common in modern IT, and the only time I do it myself is in the course of preserving vintage hardware
In theory, desoldering works. But so would scrapping & replacing all your servers after any "attacker might have gained BMC access" security incident.
You might see that as a facetious comparison. But the number of orgs which actually would desolder the chips in that circumstance is very close to the number which actually would scrap and replace. And if 99% of orgs won't actually do it when needed, then a "works in theory" method of re-securing servers is real-world useless.
Or, you know, a $0.02 write protect switch on the motherboard.
The switch alone does not provide security if the supply chain is compromised. I believe a malicious actor could act along this chain by setting the switch to ON and rewriting the firmware, just like they would replace a removable chip. A step in this direction has been taken by "Server Configuration Lock" (e.g. HPE) while servers are in transit
its not about supply chain compromise. its about device compromise.
I don't disagree, but if you have admin access to the BMC you can access the console, reboot the machine into single-user mode or even into another OS entirely, and then implant any malware you want, wipe the storage, etc.
Some of the Supermicro boards don't even have a separate BMC NIC, the only choice is to bond it to one of the main NICs or sacrifice one of them to be BMC only. I try to pay attention to that now after being surprised by that once on some servers we bought.
> I don't disagree, but if you have admin access to the BMC you can access the console, reboot the machine into single-user mode or even into another OS entirely, and then implant any malware you want, wipe the storage, etc.
Yes, all of which can be reversed by another admin in the future. That is expected.
It should not be the case that getting admin access one time can result in modifying the hardware in a way that can’t be reversed by future admin, short of physically reflashing the chip on the board.
That is believe it or not true for nearly every computer on the planet.
If you have admin on windows you can flash the bios on regular motherboards with firmware that refuses to change.
> If you have admin on windows you can flash the bios on regular motherboards with firmware that refuses to change.
The vendors even sell this as downgrade prevention!
Huh? I don’t understand what you are getting at. Every PC I’ve had uses a very simple protocol for bit banging new firmware.
Which only worked because the existing firmware let's it.
Flashing the EEPROM doesn’t involve the firmware.
Who do you think bit bangs the EEPROM?
On some boards you can access can reconfigure GPIO pins of the chipset and bitbang SPI from the application processor (aka your normal x86_64 CPU) without firmware support.
Isn't the firmware ultimately in charge of those pins, and able to block access to your OS if it chooses to?
As I said, with literally every desktop PC I have ever owned I have updated the BIOS this way. So me, I guess.
You are not typing the bits in with your fingers.
What chip are you using to bit bang? Is that chip directly or indirectly controlled by the firmware? Usually it is.
This is a design flaw in itself.
It makes it easy to brick a system.
> but if you have admin access to the BMC you can access the console, reboot the machine into single-user mode or even into another OS entirely, and then implant any malware you want, wipe the storage, etc.
True in the common case, but this can/should be guarded against by disk encryption and secured boot chains.
> Supermicro motherboards also commonly default to having a feature that bonds the BMC network interface to one of the main NICs if you don’t plug a cable into the BMC interface. It’s common for people to be surprised that their BMC is exposed on their main network because they didn’t plug in a cable on the BMC NIC port at all.
Even if you changed the default from "failover" in the firmware you're just one CMOS reset (for whatever reason) away from it reverting back to putting the BMC on whatever network interface. This should really be something you can jumper off.
Both, If you need to upgrade the firmware often, have a switch on the front of the case, like everything physical access needs to be the barrier.
It should be the case that even administrator access should not be abusable
If administrator access is equivalent to ownership, then I strongly disagree.
Even as an owner, you should not be able to arbitrarily restrict the rights of future owners.
No more hole sawing my old hard drives for me, lest I restrict rights of future owners to use the drives as storage devices.
Well they'd still have the right, just not the ability (this is actually a distinction US courts have made regarding arbitration clauses and legal recourse).
Unless you got it to saw itself via administrator access you've drifted off the intent of this conversation.
Unfortunately the existence of things like efuses and OTP makes that very difficult.
As an administrator, you generally expect to be able to change your mind at some point.
Flashing data? Fine.
Permanent? Not so much.
This logic doesn't hold. If I choose to dban or degauss something, I don't expect I should be able to recover it later. Admins absolutely have the option to make irreversible changes, and do this quite often.
What's irreversible about dban?
If you are magnetically destroying hard drives as part of decommissioning, that's not really the same thing. You're not using admin access to do it, and you're not making a change that permanently applies to all future use of the device (because there is no future use).
> What's irreversible about dban?
If you do it right, the data erasure.
Just because some administrative decisions are permanent and destructive doesn't mean that every operation should be made permanent or destructive.
Should every software config require buying new hardware because the initial config gets permanently flashed with an e-fuse to only allow a single write? You could even make a security argument for such a setup, but good luck getting approval for your 15th motherboard this quarter because you typo'd the config.
Also, dban and degaussing is not entirely equivalent -- from a practical perspective the equivalent is hard drive shredding (because the hardware cannot be used again in the old/non-malware config -- dban and degaussing are more like factory default resets). Do some organisations need to do this? Sure. Should we design systems with the assumption that any mistake means that the hardware is destined for the shredder? I would hope not...
Right. But degaussing isn't a "general" decision. Updating most systems is.
Degaussing (for orgs who do it) is just as operational a task as updating. There will be an SOP that covers data storage decommissioning. It's not as frequent, but it's not any less 'normal', certainly not ad-hoc or one-off. You don't invest in a degausser "just in case".
No, that's not what I meant. I was referring to tasks you might perform regularly on the same machine.
You don't go ahead and erase the same disk once a week. Decommissioning isn't something that occurs for the same project, once a month.
Its not the same operational process.
> It’s common for people to be surprised that their BMC is exposed on their main network because they didn’t plug in a cable on the BMC NIC port at all.
Is this documented?
If it's documented, it's not an excuse for not reading documentation. You're not supposed to throw stuff (hardware or software) in production without reading the documentation.
I've made an effort but I can't find where this is truly irreversible. It just talks about usual things like reinstalling the OS not working because the malware is active before EFI. I didn't see reference in either this article or the linked one from Binarly.
I dont agree. It should be possible to flash another BMC software such as OpenBMC for example.
That's a an unimaginative assertion for this risk.
How do you track the chain of custody of your servers? Do you sample them at random to ensure they aren't compromised?
Bloomberg never backed away from their story about Chinese implants in Supermicro servers. Perhaps this is why?
I don't work with physical servers, so this is a gap in my knowledge. Isn't it the entire purpose of BMCs to allow for remote management?
So you'd definitely have to have it connected to the internet somehow, even if very indirectly, and in an independent manner (different network with no direct routes).
Of course a network can be offline. I believe that is what you describe, a network with no routes is not connected to anything else, and certainly not to the Internet?
It is common to keep admin and backup functions on separate network interfaces, on a disconnected network. You have to physically connect to the network in a secure location to use it.
No, that is not common. Management networks are almost never air gapped, they're just segregated from publicly-accessible or higher-exposure networks (DMZ, and hopefully prod). Requiring a (role-restricted) VPN connection is the most common way to control access to management networks.
Yes, you need some way to connect to the BMC. That is never going to be something like "eh just throw it on the public internet" or similar. It's going on a separate, isolated network with strict access controls.
There are others here more qualified to comment, but I do have some old rack mount enterprise servers making a racket in my basement, so I can provide some sort of half-informed opinion here--
Besides the security issue with the BMC here, it sounds like SuperMicro _also_ has absolutely insane defaults.
Every system I have has a separate, dedicated NIC for the BMC. There is an _option_ which is disabled by default to instead have it share one of the other interfaces. So the only ways you're going to "accidentally" put the BMC on an insecure network are:
1. Rebooting the server, going into the BIOS configuration, going into the options for the BMC, and explicitly telling it which other network device to use.
2. Physically accessing the server, attaching an ethernet cable to a clearly labelled BMC port, and attaching the other end to an insecure network.
From what I'm reading here, SuperMicro's default approach is apparently more "eh, just use whatever happens to be plugged in at the time". So even if you do have it running on a separate, secure interface... if someone unplugs it, it's going to switch to using whatever other connection is available!
To connect to the BMC on my servers you first need to be on the internal network already. Then you connect via wireguard to the router on the rack. Then you can connect to it and given a username/password you can log in. I would be pretty pissed if a cable got unplugged and that meant that the server decided to instead throw the BMC _on the interface connected directly to the public friggin' internet_. And this is just equipment I use for play and hosting my own random junk!
Yep. Defense in depth. Port-based VLAN at a minimum. Only total morons place their web-based and ssh remote/KVM management directly on the internet. Note that many Supermicro boards permit (as an option) to use the regular on-board NICs reducing the need to install a second or third cable for the BMC only adapter on mixed networks. Generally, one shouldn't be placing any box of any sort directly on the 'net unless it's hardened and behind sufficient network/application filtering.
This means that any sysadmin could plant a backdoor for later use.
and/or someone "renting" a bare metal server from such a provider too.
> rent a server
> infect it with unremovable backdoor
> stop paying server so company rents server to somebody else
> ????
> profit!
I own more than one used SuperMicro servers.
Unremovable through remote access, but reliably removed by reflashing the firmware through JTAG, or whatever interface used during manufacturing to initially load the firmware.
bios flashes are typically preflashed prior to soldering to boards -- some vendors route jtag or spi contacts, but you're more likely to need a vampire/pogo clip on for the TSOP or equivalent chip, or have fun with resoldering the bios flash if you're needing to recover from this.
It's not impossible to do in the field, but you can't really count on vendors surfacing that interface usually.
I've worked with automated EEPROM/Flash programmers (earlier versions of this line: https://www.bpmmicro.com/device-programmers/automated-progra...), and used pre-programming services from distributors, like Digi-Key, but that was the exception. It's almost exclusively faster, cheaper, and easier to load firmware from a test fixture. You need to test the assembly anyway, and it's much easier to update a test procedure, when a new firmware is developed, than to update and track inventory of pre-programmed devices, especially when different firmware versions are needed for different hardware variations.
Pogo pins are only really needed for mass production, especially for reducing repetitive stress injuries. For one-off updates, if a header isn't populated, it's easy to hold an unsoldered header in place, for long enough to flash an update.
Do any server boards still socket the BIOS flash?
Yes ASRock Rack has socketed bios and BMC flash. At least on their Ampere motherboards
Wouldn’t matter if the BMC can infect the BIOS flash…
This is a real problem with some laptops. You can't always get to the chip contacts.
It bugs me that it is near impossible to purchase a server without a backdoor present.
[dead]
Am I having a deja vu or wasn't this discussed earlier like 5-6 years ago?
Yup definitely. It was about SMCI too. It's not just you: at least two of us are having deja vu.
Different. In 2018 there was articles [0] talking about hacked Supermicro servers being in the wild that had (simplifying) extra ICs in their motherboard, like some short of supply chain attack.
But AFAIK, tangible evidence never surfaced. [1]
--
I remember it too, but vaguely remember evidence of compromised boards
If you expect the BMCs of other vendors to be significantly more secure than the average home router from 2010 I have several bridges you may be interested in.
The story is called "The Big Hack" and was run by Bloomberg in 2018. All the major companies named in the story conducted investigations and never found anything, but the story was never retracted.
https://www.bloomberg.com/2018-the-big-hack
wonder who got short-options on SMCI in 2018 :)
If you've never used the BMC on a server... it is all 100% garbage. Software mostly written by embedded folks who haven't got a clue. It is absolutely garbage software on the whole (and no matter what vendor you get the board from). Go ahead and hit up the web interface then do a bit of "View Source". If you are imagining the rest of that stack is any better than my friend have I got a Beautiful Bridge in Brooklyn to sell you!
If it were me I'd assume the majority of BMC firmware out there from all vendors: 1. Is full of many many exploitable vulnerabilities 2. To the extent they patch holes it will be whack-a-mole because the economics do not permit large investments in software quality. 3. Many server owners will never install a patch anyway.
BMC software quality is low but what's the alternative? Without BMC it is more expensive to manage a fleet of servers. In a better word hardware vendors will publish specs to allow open-source BMC firmware but for some reason they resist this idea. Having only insecure BMC available a semi-separate management network (connected via a bastion host or a VPN) provides balance between cost and security.
> BMC software quality is low but what's the alternative?
Dedicated KVM devices?
This won't scale. Dedicated KVM needs you as an admin walking to the server, reswitching cables, walking back to the KVM console. Instead, with Out of band managament hw/sw, you spawn a dedicated ethernet and can access it from anywhere. It is a flexibility advantage on the costs of security.
There are boxes that can KVM to multiple servers at a time. You don't need to switch cables. They probably cost similar or less than BMC cards on a per-port basis. You might have to combine with some sort of network boot to set up a machine from scratch.
> They probably cost similar or less than BMC cards on a per-port basis
If you build own servers that's an option to consider but most off-the-shelf servers are sold with BMC (so you pay for it even if don't want it). May be some low end brands sell servers without BMC but if you are looking for relatively reliable hardware you'll likely get a server with BMC.
I was thinking more like just having one IP KVM per server always hooked up to a dedicated management network, basically used exactly like a BMC just with better software.
There is the open source OpenBMC software nowaydays, which is pretty good.
Unfortunately, Supermicro doesn't use it yet for most of their servers. Probably because they sell an extremely expansive license for their own software so you can use the Redfish API.
The one vendor mentioned in the comments, AMI, is switching this code base to openbmc. Also it should be noted that often this software is system specific.
Do we know if this is also the case for other systems that use Aspeed/ami BMCs, or if the key pair in question is exclusive to SM?
Yes it is.
Supermicro is one of the only vendors that tries to prevent this attack at all through RoT.
Other vendors you can flash whatever unsigned firmware you want. It’s very useful for adding in microcode for intel engineering samples, or malware…
This is not true. Almost all firmware is signed by every vendor, and there are standards from Intel and amd on implementation of code signing.
Look up Intel pfr.
Signed ≠ enforced.
At least for 4677 Intel stuff, gigabyte & HP and others let you modify the firmware and flash it.
HPE at least makes you flip a DIP switch, otherwise it complains loudly and halts.
Something to note is that there are special BMC-less motherboards that are made for organisations like the NSA.
That should tell you everything you need to know about the security risks involved.
And of course there's oxide.computer making motherboards with full control of the entire hardware stack running open source software specifically for customers like the Idaho National Laboratory.
Oxide’s efforts are commendable, but I don’t know what compliance regime their customer adheres to.
In other words, it’s a neat feature, but maybe not one many customers actually request.
As far as I can tell it's one of their most significant features, and certainly one of its most capital intensive. And I don't know how many startups land institutions like INL as their launching customer, and enough other big enterprise leads to warrant their follow up investment.
If you remove that component from their value prop, they're not that much different from Dell.
Just a quick point of clarification that while our boot architecture is very important (e.g., a service processor in lieu of a BMC, the elimination of UEFI entirely, etc.), we are quite a bit different from Dell beyond that. There are certainly many hardware-level differentiators (e.g. DC busbar-based design, blindmated networking, built-in switch, etc.) but the big differentiator is really what these things allow: entirely integrated software. The Oxide rack comes with all of the software to run elastic infrastructure (that is, the distributed system that comprises the control plane), including switch software, storage software, etc. And then (critically!) the capacity to update all of this.[0]
All of it is a far cry from the offerings of Dell/HPE/Supermicro, which rely on others to provide the software that turns the hardware into real infrastructure.
[0] https://oxide.computer/blog/systems-software-in-the-large
Right, my apologies for downplaying it. That you control both the hardware and the software through the entire stack has tremendous benefit, especially for this type of customer. My point was if you as a customer don't value that aspect of Oxide's pitch, which you obviously are much better at fully conveying, then you'd be seriously considering Dell/HPE/Supermicro as well.
For what it's worth the last time I had to procure a small cluster the offer Dell made was ridiculously overpriced, didn't meet the specifications I asked for, and the whole experience didn't inspire confidence in the software they were pitching either. We were a small underfunded startup so we were never gonna drop 200k on that storage solution (0.5PB HDD + ~80TB SSD), but if we did have that kind of budget we probably would have gone with one of the smaller but more focused parties there instead like maybe TrueNAS, I bet there's a beautiful market for Oxide as well in that segment.
The ability to securely update each component is quite an achievement
Every modern motherboard comes pre-installed with unremoveable malware. Vulnerabilities just let different actors try to be king of the hill. The real answer is that all these persistent storage areas (eg flash chips) need to be documented and writable by the user. Any boot integrity needs to be done on top of an open environment, rather than continuing to rely on security through obscurity.
I see the point you're trying to make but it's really very orthogonal to this issue: this was an issue in a documented part of a documented feature, it's not some "deeply embedded" system management controller with no documentation, it's "the signed firmware update feature in the big obvious selling point where the server has a backdoor management interface was broken."
Why the BMC is not designed around seL4, I cannot comprehend.
Two main reasons I can think of:
Most current BMC platforms are older than seL4
Most run on hardware that is not supported by seL4, or at least on hardware where it has not been validated.
Not to mention that a task manager would be needed as well as tons of other services which aren't provided out of the box, and don't share the verification provided guarantees.
Because people dont want to pay? Openbmc clearly has difficulty getting widespread support as nothing is documented.
My favorite supermicro facepalm will always be when you could set the IPMI encryption cipher to "none" (ipmitool -C0) and bypass actually needing any password at all. (Though I don't think this was unique to supermicro actually?)
Dell also had this problem, you still needed to provide a password, it simply didn’t check the password.
With some server vendors, if you don't connect an ethernet cable to the BMC, it can intercept BMC-targeted traffic from the OS-connected ethernet port.
Pretty much all of them allow unrestricted access from KMS from factory, tough all of them have a way to disable it once configured, and HPE even throws shade until it's limited. KMS only works from the host itself.
isn't the simple answer to infect all.of these servers with a benign infection, a vacine if you will, that blocks further attempts to remotely flash the bios
Supermicro just can't catch a break!
Business idea: eprom based firmware
Or two physical firmware chips: one writable, one with no write ability and is a fallback. Then a physical switch, could even be a jumper, to select the fallback. If compromised you flip the switch, boot from the clean firmware, flash the writable chip, flip switch and reboot. I am pretty sure Gigabyte offered this same setup with Dual Bios or something like that.
Gigabyte made a lot of marketing hay about it, but I think it was popular for a while. I think their version was some sort of watchdog/failover model where it would automatically load the backup BIOS, but some other firms had a secondary-BIOS jumper.
I think these days, the stub "BIOS flashback" is the trendy thing, where you can plug a flash drive into a magic slot and press a button to flash without even having a CPU installed.
This offered the same "brick-resistance" feature with the added benefit that people weren't stuck if they tried to pair an old-stock mainboard with a new CPU that wasn't supported by the original firmware release.
TBH, I'd rather they go the complete opposite direction: replace the soldered EPROM with a SD slot and a $1 MCU that reads the card and emulates a ROM chip. That could be configurable to write-protect the card, or you could just trivially swap it if you didn't trust the firmware image for any reason, while avoiding the fumbliness of modern tiny 8-pin flash chips. You could socket a big old-fashioned DIP ROM, but will people feel comfortable even trying to pry that out of a $10,000 server even with the appropriate chip puller tool?
Pulling a physical chip to upgrade the firmware would generate so many returns or RMAs that it would be dropped as a feature immediately.
These days it’s common to do firmware updates to address small issues or even support the new CPU that was launched after the motherboard was manufactured.
I could see manufacturers adding a write-protect physical switch for those who want it. Make it opt-in and toggleable.
AMD have recently changed the firmware loading signature verification method to apply cpu microcode that uses the on-motherboard tooling.
Using the method you talk about would mean that this kind of update wouldnt be possible, 99% of users would never toggle with a switch to update firmware.
This would be a huge burden in the server world too, to unrack flip switch, update, revert switch re-install.
I assume you mean specifically motherboard firmware updates, because firmware updates are actually pretty common, for most server grade motherboards vendors ship updates about every other month[1].
1. https://git.kernel.org/pub/scm/linux/kernel/git/firmware/lin...
And put the EPROM in a socket, like it's 1987?
Some motherboards just have a physical jumper that prevents BIOS flashing. This happens infrequently enough as to warrant it for one server, or 10 servers, or maybe 100 servers. Likely unpractical for 1000 servers though.
Baseboard management is switching to easily swapped modules for exactly this reason: https://antmicro.com/platforms/dc-scm-open-source-bmc-platfo...
https://www.servethehome.com/the-ocp-dc-scm-hff-is-taking-ov...
If they can put the jumper on the exterior it might be feasible, if its inside its out of the question if you have to unrack the chassis to change. Rolling in a server lift for an 8u thats half full of copper is not a nice process
The next idea, a second oob management for the first oob managemen. A BMC for the BMC. It only does updates and maybe credential management.
Make this one simple enough and add an EPROM for it. Effectively a security chip for the oob. Extra points for secure enclave-like verified boot.
OpenCompute (OCP) Caliptra is an effort by hyperscalers, AMD and others to enforce a platform root of trust with OSS firmware and open silicon, mandating dual signature by server OEM and hyperscaler customer. The platform RoT is responsible for validating device firmware and OS boot, https://www.youtube.com/watch?v=p9PlCm4tLb8&t=2764s
> Often we see.. great security.. compromised by other great ideas for mgmt and other things.. starts to weaken its security posture.. want to keep Caliptra very clean [via OSS firmware transparency]
The security chip for the BMC is called root of trust.
I installed my own physical jumpers on my paytv receivers in the late 90s/early 2000s…
crazy
NSA has entered the chat.
This is true for nearly all computers.
Regular desktop motherboards can be flashed from windows.
Yawnnnn.
Do nearly all computers really have a signature bypass flaw?
They don't have signatures.
HP begs to differ
Is this related to controversial Bloomberg 2021 piece about China hacking Supermicro servers?
https://www.bloomberg.com/features/2021-supermicro/
Not at all, that piece described a supply chain attack replacing a component with a look-alike part analyzing tens to hundreds of gigabits if data, in a form factor so small that it wouldn't be physically possible without semiconducting fabricating processes years in advance of what existed at the time.
What this article is describing is something far more likely— a firmware attack that doesn't require specialized hardware.
FWIW, the claim in the Bloomberg article was never validated, no one pulled a Supermicro server that had the supposed components. Zero proof since the story was published in 2018 that it wasn't nonsense.
Yeah, it was a pretty bizarre story, given it's the kind of thing that definitely happens (though probably through more straightforward just tampering with the firmware), that it was very specific and yet all the details failed to add up.
Including Amazon, Meta or whomever else was mentioned in that article, all saying "What? That never happened". There's little reason to cover up the discovery of something like this, from an end-user perspective.
Wasn’t the implant supposedly (illogically) implanting custom BMC firmware? This actually always struck me as a somewhat unbelievable part of the story: why install a hardware implant when you could just clip the flash chip and implant something without a physical trace?
The Bloomberg article specifically mentioned the attack occurring through the addition of extra hardware:
Flash can always be reflashed (you'd lose the implant if the customer does any firmware update) but a separate implant chip can remain infected forever.
That’s fair - the first thing I’d put in my implant firmware would be a fake firmware updater, but I suppose trading guaranteed persistence for physical detection would be a reasonable tradeoff in some places.
> with a look-alike part analyzing tens to hundreds of gigabits if data, in a form factor so small that it wouldn't be physically possible without semiconducting fabricating processes years in advance of what existed at the time.
I'm not sure where you got that idea. The article describes a tiny microcontroller, attached to the read pins of the BMC's boot flash, flipping a few bits in transit from the flash ROM to the BMC SoC as the BMC boots. This is not only practically possible, it's very similar to the technique used to hack the original Xbox and by many console mod chips. And is sufficient to boot the BMC in a vulnerable state for the next chain of an attack.
Nothing about the exploit claimed in the article was impossible or even novel.
That said, I'm not aware of any physical boards found to have the compromised hardware outside of those Bloomberg claim to have witnessed.
People like to criticize secure boot around here, but it prevents these kinds of infections (provided of course there are no vulnerabilities in the implementation of secure boot).
Yes, in theory it is possible to prevent these kinds of infections without resorting to secure boot (e.g., by insisting that all the suppliers of components of the motherboard start designing components that cannot be pwned) but so far all the computers you have actually been able to buy that are immune to these kinds of infections achieve that immunity with secure-boot technology.
Can you please explain how Secure Boot helps at all to mitigate this type of attack? I don’t see how it makes it harder to execute the attack, and I don’t see how it has any particular effect on the capabilities of the attack once executed. To the contrary, a BMC compromise of this sort seems like it should be able to arbitrarily override secure boot settings.
It seems to me that, in this situation, secure boot’s only role is to provide a false sense of security, which could make recovery from the attack less likely.
In contrast, verified boot might somewhat mitigate the damage from being able to use the BMC to write arbitrary data to the SPI flash chip. Emphasis on might — at best I expect that it would require an attacker to be a bit more creative in how they design their exploit payload.
> Can you please explain how Secure Boot helps at all to mitigate this type of attack?
Secure boot can include the hash of the firmware, computed by the root-of-trust that can't be tampered with by this attack. So the exploit will make the keys stored in the TPM inaccessible.
This will make the tampering conspicuous, at least.
I agree in general; PCRs provide some basic degree of protection against this. Unfortunately, the position these management controllers are in often grants memory access, which renders all of the boot measurement type security methods useless. Even if it doesn't, there's also the notion that an attacker will replace the firmware from the very start with one that fakes the PCR hashes which are sent to the TPM. Unfortunately, this isn't really very hard with most UEFI implementations.
I have never seen someone make a distinction between Secure Boot and boot verification on UEFI/x86, although I suppose it’s a valid one? I suspect the parent post is referring to Secure Boot in the colloquial sense it’s usually used for on x86: validation of the UEFI boot binary using Secure Boot keys followed by OS level trust chaining, usually accomplished by entangling PCR state with disk encryption in some way.
And I think this would deliver a slight level of protection from the BMC: tampering with the firmware image or key enrollment / secure boot state _should_ both break the UEFI root of trust and alter the PCR state and break everything downstream. Of course, all UEFI implementations are holier than Swiss cheese and there are probably a lot of ways to use the BMC to dump or modify memory post-boot anyway, but it does do something.
You mean the same Secure Boot that I mean. This is a software mechanism (almost universally implemented as an unholy mixture of ordinary code in firmware and a pile of immensely complex SMM code that, IMO, is entirely unnecessary if the hardware or the BMC gives even the slightest bit of help).
And Secure Boot is implemented in, and configured by, the firmware that the BMC can overwrite at its whim while entirely bypassing all the fancy CPU-hardware and SMM protections that are supposed to prevent writing arbitrary data to it.
To the extent that a mechanism not controlled by firmware will detect such an attack and extend PCRs accordingly before executing a single instruction from the compromised flash chip, it might partially mitigate attacks. But that part isn’t Secure Boot.
Oh, yes, we 100% agree on this, the true root of trust for firmware execution exists before and independently of “secure boot,” and therefore, often not at all (and “secure boot” is a terrible name).
How does secure boot help? If you control the BMC, you can enroll whatever keys you want.
The BMC usually has full access to system memory as well, so if you can get the timing right, you could replace the secure boot verified image with your own after verification.
Also, re: BusinessWeek, hey look a hardware backdoor installed on servers. Pretty sure IPMI vulnerability fits the bill for most of what was described.