- Have NFTables checks for MSS, TCP SYN TTL, Window sizes, CIDR blocks permitted on the public ssh instances (throw away nodes) and behind a VPN on more important roles. If I have a need to keep netfilter disabled then I put sshd behind L4 haproxy with ACL's.
The article is not about abandoning ssh (client) keys but protecting them from exfiltration:
> This post documents how I’m using various secure enclaves (YubiKey, TPM, Apple Secure Enclave) to hold/encrypt SSH key material and achieve the goal of not having any unprotected key material living on-disk.
Could you help me out and state the goal of your practices as simply as this? I can guess, but I'm really not sure, especially when you get into nftables checks for window sizes.
My method reduces the threat of exfiltrated keys when someone or something manages to bypass the aforementioned methods without the need to purchase anything or use anything proprietary.
- One must connect from the correct address and with the correct TCP options.
So for example I could give you one of my ssh private keys and it would be entirely useless unless you were on my home internet connection using a device that emits my TCP options. (Or this could be a corporate SNAT pool or corporate VPN pool.).
Even the most clever MitM (man in the middle) attacks become significantly harder to perform. Leaks happen, I reduce the blast radius of the damage.
So if an attacker compromises your private keys, they’re already on your computer. If they’re on your computer, what stops them from connecting to your server from your computer? It would have the exact same TCP options, same IP address, same everything.
If I were an attacker that had an agent or RAT on your PC and got your private key, I’d first try to connect to your server on my computer. That would fail, but the very next thing I would do is use my RAT or agent to connect to the server via your own PC, because I would assume you had some sort of IP filtering in place.
If you store your keys in a trusted enclave like a Yubikey, TPM, etc it stops this cold (configured correctly).
So if an attacker compromises your private keys, they’re already on your computer.
They could be but I would not automatically assume that. Many stealers upload keys and do not install a RAT and there is no guarantee the RAT will function even if they manage to install one. If we want to go down every what if path I could make everyone want to power off their data-center and go home. If I showed everyone what SSH can do without any RAT's many would uninstall it. Involves multiplexing and one phishing victim to bypass MFA, HSM's, vaults, bastions, and does not leave a trace by design. No need to sneak past anti-virus.
Most orgs will not buy or implement an HSM so I am offering a soft middle ground that may suffice for some and may not even remotely be an option for others. Some orgs have no choice but to buy an HSM. Been there, done that. It can be a massive daunting task depending on what checkboxes are not yet checked and what requirements B2B financial customers are demanding. One of them required 6 different colored keys that get distributed to different org leaders and have to be put in the HSM to make any changes. They put the fear of diety into everyone about losing keys. One was lost. Thats when you find out that crazy expensive device is just hype.
In security threat modeling, you only factor in things that you control when determining exploitability or risk. So for instance, when you say “a RAT might not work right” or “attackers don’t always use RATs” those aren’t things that you control, those are things that an attacker controls.
One of the principles of threat modeling is that you must assume an attacker is working actively to maximize their odds of success. Saying something like “well it would be annoying for an attacker to do this” is practically universally rejected by cybersecurity professionals especially in the age of AI agents.
I disagree that I am posing a bunch of “what-ifs”. The threat I am modeling here is how companies get compromised all of the time. Living off the land attacks as I’ve described is how companies large and small get hacked.
Furthermore, you do not need an expensive HSM. Most computers nowadays have secure enclaves (TPM) that can be used for this purpose.
Everyone is free to choose what options work best for their needs and budgets. I would never attempt to inhibit anyone's choices. I use what has worked well for me and stifled red-team members.
Yes and you can change the protocol field in your TCP stack to run all connections over ip protocol 200 too, but it doesn’t add any security to your authentication system over and above your private key. What you are doing is the same thing; lots of extra too-clever-by-half complexity that doesn’t actually benefit you.
Key agreement protocols make sniffing the wire for credentials useless. Two seconds in wireshark and your hyper secure tcp options are toast.
I do not agree. I have used this exact method to detect all manor of shenanigans within VPS environments, dedicated server environments and corporate environments. Attackers stumble, fall and make a lot of noise trying to untangle my simple mitigations. I will keep it.
Correct. Thankfully none of what I am doing is security through obscurity. It's actual security with a subtle scent of obscurity. If it was just obfuscation and you knew something, then you could do that something from somewhere else which you can not.
That's not security by obscurity, that's the Swiss cheese approach; which people attracted by purity frown upon but people attracted by practical solutions can live by.
Did you create your own process for monitoring/alerting on the checksum?
Yes, I had an LLM generate a Rust binary that performs only one function, takes up a tiny amount of memory and is hard to kill. If it is terminated it makes a lot of noise on the way out.
> If it is terminated it makes a lot of noise on the way out.
How do you achieve this? A process hit with halt or sigkill has no knowledge of either (it happens in the kernel). Is this some two way multi-process watchdog type setup, that detects it based on sequential terminations? Or maybe relies on some system service manager not first being disabled?
Hey I'm in the middle of doing a very similar thing, though I didn't go quite as far as OP. I'm slowly deprecating my old RSA SSH keys and replacing them with a combination of Yubikey, Password Manager, and alternative SSH auth:
1. -sk keys on Yubikeys (with touch enabled) are my primary keys, used for signing and always installed on every host I touch/etc. I initially planned on using the PIV slots on my Yubi's to store the SSH keys instead of the built-in FIDO2 setup, but I hated having to manage multiple moving parts (like getting pscd running on all my clients). The only main benefit of using the PIV slots was being able to set the "cache" touch policy: the YK would cache a touch for up to 15 seconds so doing something like "signing multiple commits during a rebase" would only take one touch instead of num_commits. The side benefit would have been getting to re-use the same SSH key across primary and backup Yubis. The simplicity of the FIDO setup won out and I decided to treat getting annoyed at pressing my Yubikey a bunch of times as a sign I should be doing some sort of alternative setup like a dedicated key. Most prominently, I've now properly setup my Ansible configs with dedicated deploy keys kept as secrets instead of reusing my personal key.
2. I use 1Pass and Bitwarden (work/personal) to store a secondary SSH key that's only used for basic Git auth or other low-risk tasks. This SSH key is loaded from the Password manager directly into ssh-agent via stdin so it never touches my disk; I use fnox to make this backend agnostic.
3. Work uses Teleport, personal uses Tailscale so I rely on those SSH auth most of the time. Nothing much to say here except that Tailscale ssh is really convenient :) .
I've also setup PAM login and sudo on my machines, which brings me back to the old Sun Ray/Smartcard days. None of this is really to prevent security breaches/dedicated attackers or anything, though it certainly makes me harder to compromise. I just wanted to make absolutely sure I couldn't accidentally upload an SSH private key somewhere.
Running very similar setups. PAM handles FIDO / security key based auth on my local systems for root users and sudo commands, as well as LUKS decryption.
I have a handful of security keys, so I have a helper script to drop them into new hosts and another helper script on my systems to import keys straight to ssh-agent once detected.
Has worked very well and I don't think I'll ever go back to a "password manager as SSH agent" again, but it's probably the best alternative backup. I still have Bitwarden holding some backup keypairs for this reason, but haven't needed to use them in months.
1. This method only works (securely) for non-resident keys.
2. hence requiring having pk stub copied manually to every machine
3. its passphrase is prone to brute-force
While piv based ssh keys solve those 3 problems (with one caveat, left for a careful reader :)
Your 2nd one is non-resident, the secret key material is encrypted in that sk file. But it is encrypted with a master key that is only inside your yk. So you need both your sk handle file and the yk to use it. This has the benefit of not consuming the limited number of memory slots in a yk.
In the first one the handle file that's created, can be recovered from the yk key itself, that's what resident is.
Having the only practical difference between the two is that, with the first one, you (or someone else) only needs the yk to use the credential. But in the latter case, one needs the yk and the handle file.
Resident keys have their "metadata" stored on the yubikey, in an exportable way. This means using the same ssh-key on a different computer is as easy as plugging in the yubikey and typing ssh-keygen -K.
With a non-resident key you’d have to copy the "privatekey", which contains the metadata needed to use the key, to the other computer first.
The downside with resident keys is that someone stealing the yubikey can see what resident keys are on it, and then possibly use those keys (if they aren’t protected). Newer firmware comes with ways to protect against this, but you’d have to enable it.
The first one sets "no-touch-required" and not the second one, is that expected?
Anyway, resident key means that you can extract the public key from the yubikey. Say someone steals your Yubikey Fido2 PIN and your Yubikey, they have everything.
Non-resident key means that the attacker needs to get that "useless" file you describe, too. And if you lose it, then you can't use the key anymore.
Another difference is that a resident key takes space on the Yubikey, so you can't have infinitely many. Whereas you can have infinitely many "useless" files.
I'd use resident keys if it wasn't for the requirement to prepend `no-touch-required` server side, services like GitHub and BitBucket don't support it.
I don't understand why the server should enforce this...
I experimented with a YubiKey-only GPG key, with a software backup stored on some thumbdrive. The backup thumbdrive was soon lost, and the key on the Yubikey is now probably expired.
I can't imagine storing my SSH keys on a TPM that might reset them after changing some sticks of RAM changes the hardware measurements or some firmware bug triggers the self-erasure....
Great article! I discovered ssh-tpm-agent last year and was surprised that all my ThinkPads have hardware TPM, even one from 2018! I've switched over to using it for everything and there are no more private keys on my devices.
I also have a Yubikey but it's been a few years since I tried it for ssh keys. I use it as a regular security key. Maybe I should try it for ssh keys as it sounds like the tooling has improved.
How do you break back into your digital life though? You go travelling to a foreign place, get mugged, you don't have your wallet phone or keys. All you have is the grace of a strange letting you use their laptop. How do you get back in?
I wanted to warn people that I had a Yubkey usb-c nano MELT because I had a Thinkpad in an awkward location with poor ventilation for slightly too long. Not sure what kind of plastic it uses but it melts very easily and took the entire device with it.
Given how incredibly important they can be, having an operating range that can be exceeded by ambient temperatures in some (very populated!) regions of the world is a design flaw.
iPhones are only designed to work from 0-35C, which is more narrow than the yubikeys, and way more important and way more widely used, so I think it will probably be okay.
> You will notice though that the private keyfile held on disk uses a .tpm extension this time. This is where an important caveat of the TPM method comes in – Unlike the YubiKey approach, the private key material is actually being stored on disk here. This is because TPM devices do not natively support enrolling SSH keys. So what is happening is that the private key material is being stored on disk in encrypted form, and the TPM is being used to decrypt it.
While this is true, you can create static hierarchy keys that would not resolve into separate key files for the TPM. The issue here is that anyone with access to your TPM can recreate the keys, unless you set a owner password.
Yubikeys (or something like keys in the Secure Enclave of a Mac or iPhone) are themselves not accessible by software, you can only ask the hardware to perform operations with them (e.g. decrypt or sign something).
This means someone with access to your filesystem can’t just copy the key off.
Of course a passphrase is another protective layer. But passphrases can be keylogged, or the decrypted key (or passphrase, depending) in the memory of an ssh agent can be can potentially be read out by an attacker.
With separate hardware, the assumption is that the hardware is much much harder to attack, so the keys stay on there and only on there.
If someone compromises your computer, they can copy your SSH keys and their passphrase and then, well, use them. And you may never know it.
If someone steals your Yubikey without compromising your computer, they cannot use it without the passphrase.
If someone compromises your computer to copy your passphrase, AND steals your Yubikey, then they can use it. But you will know that you are not in possession of your Yubikey anymore, and you can do something about it.
You can require physical user interaction to use the key. If the mossad is after you, this doesn't really help you because they'll just replace the ssh binary and intercept one of your requests, but against a less motivated attacker, it makes it harder to pivot from one machine into others.
My employer gives us yubikeys. The first one I had randomly stopped working after a year. I wouldn't use it for anything where pairing a new key is difficult if you cannot use the old one anymore.
> My employer gives us yubikeys. The first one I had randomly stopped working after a year. I wouldn't use it for anything where pairing a new key is difficult if you cannot use the old one anymore.
I really don't see how you can blame Yubikey for this one, your employer is entirely to blame for only supplying you with one Yubikey.
Your hardware fault is unfortunate, I've been using Yubikeys for many years without a single issue. But ultimately that is beyond the point here .... come on dude, you work in IT, surely by now you know you don't put all your eggs in one IT basket ?
Not following fundamental best-practice and then complaining when things break is really a "move along, nothing to see here" moment.
The solution to the second is to have 2-5 devices, each with their own key. I have a nano in each laptop, one backup one at each house, and a floating one on my keyring.
First: Always buy two (or n+1). Enroll both. Store one somewhere safe.
Anecdotally I have keys from 2013 that are still working fine when I test them, although they've been superseded by newer models.
To my knowledge no one has done the Consumer Reports / Mythbusters style of testing, buying many Yubikeys over time (to get different hardware batches) and subjecting them to various tests...
There's also an alternative way to using ssh-tpm-agent - the tpm2-pkcs11 stack provides it, here's a guide for setting it up: https://jade.fyi/blog/tpm-ssh/
Not related to the author, but using that setup personally. Similar considerations as in the OPs TPM notes.
My sticking point with these has always been SSH agent support: the last time I looked, I couldn't use my Yubikey to authenticate the second hop in an A->B->C sequence with a jump box in the middle. Is that resolved now? This blog doesn't address that use case.
Except that's not really the network path that's going on, right? If I am on A and connect to B and then want to use the network to transfer a bunch of data from B to C over an ssh channel between them, using ProxyJump brings it all back to the bastion box. And if A is in the US and B & C are in Europe connected with a 40 GB LAN, using proxyjump will bring the connection from B->C into Europe->US->Europe. How do I do this with my credentials existing on A being used to log in to B and then the same credentials being used to connect from B to C while only having the one network connection is B->C and not to my bastion host?
If you're using -sk keys (i.e. setup via FIDO2 on the Yubikey like the article, not PIV) yes you can forward the keys loaded into your agent like normal, with ssh -A.
PIV keys can also be forwarded through an agent. The only thing to be aware of is that the agent process will effectively cache your PIN if you do this.
ssh-add -s <pkcs11.so>
To mitigate that you can add -c to the ssh-add command. That will make the agent prompt for confirmation (on the host with the PIV key) every time something asks to use it. Doesn't quite get you the same security as just not caching the PIN to start with but it's OK.
I like the idea of using my MacBook's Secure Enclave for ssh keys but what do I do if I develop on a remote VPS (purely because I'd rather have the AI accidentally rm -rf / on a VPS than my own computer)?
You can use agent forwarding (ssh -A) to make your local hardware-backed key available on the VPS.
If you're using tmux on the VPS this can get a little tricky, because by default your tmux panes will inherit your initial SSH client's SSH_AUTH_SOCK; when you re-attach tmux in a different SSH session you'll no longer be able to authenticate with the local YubiKey. But I wrote a shell script that hooks into tmux to keep SSH_AUTH_SOCK working correctly with this kind of setup: https://github.com/mshroyer/socklink
There is also a middle ground - it's possible to sign SSH certificate using yubikey as CA. By controlling expiration and set of principals, the cert you get on your machine is scope restricted, and quickly expires. Or you can load it into an ssh-agent and drop it immediately.
That's kind of the opposite of the approach presented here, though. With hardware-resident, non-exportable keys, you can know that no matter what you do, key hashes to some physical device are 1:1.
Whether you want or don't want that property is a different story.
is this for new machine installs , or periodic (daily , weekly) access?
I thought private keys were meant to be secret (not copied) and the public keys meant to be installed onto the target host.
so if you have a desktop, laptop etc, you would generate the secret once per endpoint and then copy the public keys onto targets as needed. AT scale this could be done by ldap / cloud mgmt.
Do yubikeys suffer from flash memory bit rot? If so, it seems it would best have several backups that rotate once a month to being the primary so that the memory gets used?
In my anecdotal experience Yubikeys tend to physically deteriorate long before this happens. I still have an early YubiKey 5 that must be 8 years old at this point that I use daily and it works fine.
Last time I checked, it was possible to update the firmware of Nitrokeys. Which seems to completely defeat the purpose of the device?
They started selling the Nitrokey 3 long (like years) before the software had the main features they claimed they had, so I understand why they would want to be able to push updates. But again in terms of security, the first thing I want from a security key is that a compromised computer cannot reflash it...
> Last time I checked, it was possible to update the firmware of Nitrokeys. Which seems to completely defeat the purpose of the device?
Yeah. Precisely my problem with Nitrokey too.
If you can flash it, then so will a bad-guy one day. Not if, but when. You really can't say never anymore these days.
The flashing process is also annoying and fragile, I bought a couple of Nitrokeys for random testing and generally found myself haivng to repeat the process a couple of times before it finally worked.
Also, IIRC not all Nitrokeys have a secure enclave, a bunch just store secrets in software.
Well, as soon as they compromise your computer. Which is the same security as just having the private SSH key as a file on your computer. Except that if you have a Nitrokey, you may trust it more than a file on your computer, which actually makes it even worse.
for most use cases they're functionally the same. the biggest difference for me is that nitrokey doesn't support doing hmac-sha1 challenge, so i can't use the nitrokey with keepassxc.
i recently bought three yubikeys: a 5C nano to sit in my PC, a 5C NFC on my keychain and another 5C NFC as a backup. currently only using them as SSH keys to my servers. probably overkill but all the recent cybersecurity issues got me a little spooked.
i can't lie, the thought of removing authenticator/authy 2FA from my critical accounts is a little daunting.
I add YubiKeys as backup everywhere I can. Like I have one YubiKey that always stays at home in case I lose my laptop and phone so I can get access back to some stuff sooner than later.
Yes, it's really nice. The fingerprint scanner on my Thinkpad is terrible (even when enrolling a fingerprint multiple times, etc.), whereas the YubiKey Bio work much better. So I use it as an alternative screen unlock.
Speaking of which I do something cool with my main desktop (an AMD 7700X, but whatever). It's running Debian and there's zero executable with the suid bit set. There's no sudo command installed. Using "su" to su to root is technically possible but it's like a 40 characters long password, meant to never be entered (and I never ever entered it using su).
The only way I can get root (short of using a local privilege exploitation exploit to gain root) is by logging, using a Yubikey, over ssh, from a dedicated laptop that has nothing but a SSH client. So the Yubikey is plugged into the laptop.
That dedicated laptop has no wifi drivers and is on a dedicated LAN and that dedicated LAN only has two machines on it: my desktop and that laptop.
The laptop serves no purpose other than: "log in as root on the desktop".
Of course both that laptop and the desktop have firewalls.
This means, yes, I allow log in as root using SSH. But password logins are disabled and the only way to log in as root using SSH is to use the Yubikey. And the firewall only allow SSH attempts to port 22 from the MAC address, IP, and dedicated LAN that corresponds to the laptop.
In addition to that my main desktop has multiple user accounts: but none of those users are allowed to su or sudo.
Now, I know, I could use Qubes or whatever.
But what I explained above is how I do it: even myself I cannot get root on my own desktop without my laptop + Yubikey (I'd need to find and enter the 40 chars long root password).
When I really need to do something as root (and once becoming root becomes extremely hard, suddenly you realize how rare are the occasions when you really need to be root), I open my laptop and boot it (and because that laptop has no GUI and basically nothing besides a SSH client, it boots in three seconds or so).
Haters gonna hate but that's been working totally fine for me since years and years.
Doesn't Yubico have some GUIs for the Yubikeys? I don't know, I prefer the CLI :-). But I remember seeing something like that.
To answer your question, a CLI is a lot easier to write than a GUI. So the GUI would be the next step after the CLI, if there was enough demand. Now I'm guessing that people who care enough about security to try to handle it properly are anyway people who can use the CLI.
Sorry, but how so? I use FIDO2 and PKCS #11 keys in Qubes OS. PKCS#11 I cheated slightly by assigning access to the one VM that needs that key via sys-usb, but for all the other AppVMs I use the qubes-ctap-proxy and OpenSSH FIDO2 keys. It's pretty easy and can even be locked down via ACLs.
It's true that TPM 2.0 is out the window, but FIDO keys are cheap, convenient, and secure.
Ah, I see. I don't use disposable VMs much, and when I do I'm not using them as SSH clients, so I had no idea. I would push back on heavy use of disposable VMs as the only way to meaningfully use Qubes OS though—although handy, they're not my main motivator for dailying it.
If you're using userspace software to decrypt the keys into memory then it's all just security theatre. Any sufficiently motivated actor, ot a comprised package in the software chain, can capture your private key in the clear. Especially if using ssh-agent that keeps it resident in memory 'for convenience'.
Proper HSM integration with SSH requires challenges signed by the HSM without the private key ever being emitted.
Just keep your SSH keys on disk, one per remote host, with a different encryption password for each target host. If you're worried about threats that can defeat that then they'll just come to your door and beat the passwords out of you.
Only because you clearly don't understand how it works. What's interesting is that you seem to understand half of it... or maybe you don't, and you just repeat stuff you don't understand either?
> If you're using userspace software to decrypt the keys into memory
That's exactly why you don't do it. A Yubikey or a secure enclave create the private key and emit the public key.
I will stick with ssh keys, except that I:
- Put the public authorized keys in /etc/ssh/keys/%u immutable, monitored for checksum changes.
- Have CIDR restrictions in said public keys.
- Have NFTables checks for MSS, TCP SYN TTL, Window sizes, CIDR blocks permitted on the public ssh instances (throw away nodes) and behind a VPN on more important roles. If I have a need to keep netfilter disabled then I put sshd behind L4 haproxy with ACL's.The article is not about abandoning ssh (client) keys but protecting them from exfiltration:
> This post documents how I’m using various secure enclaves (YubiKey, TPM, Apple Secure Enclave) to hold/encrypt SSH key material and achieve the goal of not having any unprotected key material living on-disk.
Could you help me out and state the goal of your practices as simply as this? I can guess, but I'm really not sure, especially when you get into nftables checks for window sizes.
My method reduces the threat of exfiltrated keys when someone or something manages to bypass the aforementioned methods without the need to purchase anything or use anything proprietary.
- One must connect from the correct address and with the correct TCP options.
So for example I could give you one of my ssh private keys and it would be entirely useless unless you were on my home internet connection using a device that emits my TCP options. (Or this could be a corporate SNAT pool or corporate VPN pool.).
Even the most clever MitM (man in the middle) attacks become significantly harder to perform. Leaks happen, I reduce the blast radius of the damage.
So if an attacker compromises your private keys, they’re already on your computer. If they’re on your computer, what stops them from connecting to your server from your computer? It would have the exact same TCP options, same IP address, same everything.
If I were an attacker that had an agent or RAT on your PC and got your private key, I’d first try to connect to your server on my computer. That would fail, but the very next thing I would do is use my RAT or agent to connect to the server via your own PC, because I would assume you had some sort of IP filtering in place.
If you store your keys in a trusted enclave like a Yubikey, TPM, etc it stops this cold (configured correctly).
So if an attacker compromises your private keys, they’re already on your computer.
They could be but I would not automatically assume that. Many stealers upload keys and do not install a RAT and there is no guarantee the RAT will function even if they manage to install one. If we want to go down every what if path I could make everyone want to power off their data-center and go home. If I showed everyone what SSH can do without any RAT's many would uninstall it. Involves multiplexing and one phishing victim to bypass MFA, HSM's, vaults, bastions, and does not leave a trace by design. No need to sneak past anti-virus.
Most orgs will not buy or implement an HSM so I am offering a soft middle ground that may suffice for some and may not even remotely be an option for others. Some orgs have no choice but to buy an HSM. Been there, done that. It can be a massive daunting task depending on what checkboxes are not yet checked and what requirements B2B financial customers are demanding. One of them required 6 different colored keys that get distributed to different org leaders and have to be put in the HSM to make any changes. They put the fear of diety into everyone about losing keys. One was lost. Thats when you find out that crazy expensive device is just hype.
In security threat modeling, you only factor in things that you control when determining exploitability or risk. So for instance, when you say “a RAT might not work right” or “attackers don’t always use RATs” those aren’t things that you control, those are things that an attacker controls.
One of the principles of threat modeling is that you must assume an attacker is working actively to maximize their odds of success. Saying something like “well it would be annoying for an attacker to do this” is practically universally rejected by cybersecurity professionals especially in the age of AI agents.
I disagree that I am posing a bunch of “what-ifs”. The threat I am modeling here is how companies get compromised all of the time. Living off the land attacks as I’ve described is how companies large and small get hacked.
Furthermore, you do not need an expensive HSM. Most computers nowadays have secure enclaves (TPM) that can be used for this purpose.
Why bother with the key? Anyone who has the right address and TCP options is obviously legit.
Just protect your keys in HSMs and all of this TCP mumbo-jumbo trying to pretend to be a secondary weak password becomes entirely unnecessary.
Everyone is free to choose what options work best for their needs and budgets. I would never attempt to inhibit anyone's choices. I use what has worked well for me and stifled red-team members.
Yes and you can change the protocol field in your TCP stack to run all connections over ip protocol 200 too, but it doesn’t add any security to your authentication system over and above your private key. What you are doing is the same thing; lots of extra too-clever-by-half complexity that doesn’t actually benefit you.
Key agreement protocols make sniffing the wire for credentials useless. Two seconds in wireshark and your hyper secure tcp options are toast.
I do not agree. I have used this exact method to detect all manor of shenanigans within VPS environments, dedicated server environments and corporate environments. Attackers stumble, fall and make a lot of noise trying to untangle my simple mitigations. I will keep it.
"Security by obscurity isn't."
"Security by obscurity isn't."
Correct. Thankfully none of what I am doing is security through obscurity. It's actual security with a subtle scent of obscurity. If it was just obfuscation and you knew something, then you could do that something from somewhere else which you can not.
That's not security by obscurity, that's the Swiss cheese approach; which people attracted by purity frown upon but people attracted by practical solutions can live by.
> Put the public authorized keys in /etc/ssh/keys/%u immutable, monitored for checksum changes.
Did you create your own process for monitoring/alerting on the checksum?
Did you create your own process for monitoring/alerting on the checksum?
Yes, I had an LLM generate a Rust binary that performs only one function, takes up a tiny amount of memory and is hard to kill. If it is terminated it makes a lot of noise on the way out.
> If it is terminated it makes a lot of noise on the way out.
How do you achieve this? A process hit with halt or sigkill has no knowledge of either (it happens in the kernel). Is this some two way multi-process watchdog type setup, that detects it based on sequential terminations? Or maybe relies on some system service manager not first being disabled?
Or maybe relies on some system service manager not first being disabled?
You got it. Turtles, all the way down. Syslog, all the way up.
You fear someone will steal your yubikey and steal the pin?
I think there is an xkcd for that.
Hey I'm in the middle of doing a very similar thing, though I didn't go quite as far as OP. I'm slowly deprecating my old RSA SSH keys and replacing them with a combination of Yubikey, Password Manager, and alternative SSH auth:
1. -sk keys on Yubikeys (with touch enabled) are my primary keys, used for signing and always installed on every host I touch/etc. I initially planned on using the PIV slots on my Yubi's to store the SSH keys instead of the built-in FIDO2 setup, but I hated having to manage multiple moving parts (like getting pscd running on all my clients). The only main benefit of using the PIV slots was being able to set the "cache" touch policy: the YK would cache a touch for up to 15 seconds so doing something like "signing multiple commits during a rebase" would only take one touch instead of num_commits. The side benefit would have been getting to re-use the same SSH key across primary and backup Yubis. The simplicity of the FIDO setup won out and I decided to treat getting annoyed at pressing my Yubikey a bunch of times as a sign I should be doing some sort of alternative setup like a dedicated key. Most prominently, I've now properly setup my Ansible configs with dedicated deploy keys kept as secrets instead of reusing my personal key.
2. I use 1Pass and Bitwarden (work/personal) to store a secondary SSH key that's only used for basic Git auth or other low-risk tasks. This SSH key is loaded from the Password manager directly into ssh-agent via stdin so it never touches my disk; I use fnox to make this backend agnostic.
3. Work uses Teleport, personal uses Tailscale so I rely on those SSH auth most of the time. Nothing much to say here except that Tailscale ssh is really convenient :) .
I've also setup PAM login and sudo on my machines, which brings me back to the old Sun Ray/Smartcard days. None of this is really to prevent security breaches/dedicated attackers or anything, though it certainly makes me harder to compromise. I just wanted to make absolutely sure I couldn't accidentally upload an SSH private key somewhere.
Running very similar setups. PAM handles FIDO / security key based auth on my local systems for root users and sudo commands, as well as LUKS decryption.
I have a handful of security keys, so I have a helper script to drop them into new hosts and another helper script on my systems to import keys straight to ssh-agent once detected.
Has worked very well and I don't think I'll ever go back to a "password manager as SSH agent" again, but it's probably the best alternative backup. I still have Bitwarden holding some backup keypairs for this reason, but haven't needed to use them in months.
YubiKeys with proof-of-presence for SSH auth are great, but I still use the YubiKey's OpenPGP slot for this because my preferred behavior is:
1. Require a PIN followed by touch on the first use after inserting the YubiKey
2. For every following authentication while the key remains connected, only require touch
The extra GnuPG setup needed to make this work is unfortunate, but AFAIK this still isn't possible with fido -sk keys on the YubiKey.
You get this exact workflow if you use a passphrase on the -sk key, and add the key to your ssh-agent.
Edit: well, maybe not exact. With -sk keys you’d have to type in a passphrase instead of a pin. But that passphrase could be a pin.
You are not supposed to use passphrases with -sk keys.
Not exactly.
1. This method only works (securely) for non-resident keys. 2. hence requiring having pk stub copied manually to every machine 3. its passphrase is prone to brute-force
While piv based ssh keys solve those 3 problems (with one caveat, left for a careful reader :)
Huh, in that case it's possible I did this wrong the last time I tried it, or maybe something's changed.
Thanks for the tip! I'll try that this evening, I'd love to move to FIDO keys if possible.
Really nice article!
They mention twice that Yubikeys are expensive, but for SSH their cheaper FIDO2 keys work:
https://www.yubico.com/product/security-key-series/security-...
What is the practical/security difference between
`ssh-keygen -t ed25519-sk -O resident -O no-touch-required -O application="ssh:nameyourkey" -f ~/.ssh/id_ed25519-sk`
and
`ssh-keygen -t ed25519-sk -f ~/.ssh/id_ed25519-sk`
?
The latter is what I always do. It creates a file that is useless without my yubikey being present, but is it not actually resident on the key?
Your 2nd one is non-resident, the secret key material is encrypted in that sk file. But it is encrypted with a master key that is only inside your yk. So you need both your sk handle file and the yk to use it. This has the benefit of not consuming the limited number of memory slots in a yk.
In the first one the handle file that's created, can be recovered from the yk key itself, that's what resident is.
Having the only practical difference between the two is that, with the first one, you (or someone else) only needs the yk to use the credential. But in the latter case, one needs the yk and the handle file.
Resident keys have their "metadata" stored on the yubikey, in an exportable way. This means using the same ssh-key on a different computer is as easy as plugging in the yubikey and typing ssh-keygen -K. With a non-resident key you’d have to copy the "privatekey", which contains the metadata needed to use the key, to the other computer first.
The downside with resident keys is that someone stealing the yubikey can see what resident keys are on it, and then possibly use those keys (if they aren’t protected). Newer firmware comes with ways to protect against this, but you’d have to enable it.
[1] has more details
1: https://developers.yubico.com/SSH/Securing_SSH_with_FIDO2.ht...
The first one sets "no-touch-required" and not the second one, is that expected?
Anyway, resident key means that you can extract the public key from the yubikey. Say someone steals your Yubikey Fido2 PIN and your Yubikey, they have everything.
Non-resident key means that the attacker needs to get that "useless" file you describe, too. And if you lose it, then you can't use the key anymore.
Another difference is that a resident key takes space on the Yubikey, so you can't have infinitely many. Whereas you can have infinitely many "useless" files.
I'd use resident keys if it wasn't for the requirement to prepend `no-touch-required` server side, services like GitHub and BitBucket don't support it.
I don't understand why the server should enforce this...
I use a resident key with touch with GitHub.
What about one that doesn't require touch? Do you need to prepend 'no-touch-required' server-side?
Yes, GP must have outdated info, I enrolled a FIDO key in Github last week.
Bitbucket, however, does not accept them.
I experimented with a YubiKey-only GPG key, with a software backup stored on some thumbdrive. The backup thumbdrive was soon lost, and the key on the Yubikey is now probably expired.
I can't imagine storing my SSH keys on a TPM that might reset them after changing some sticks of RAM changes the hardware measurements or some firmware bug triggers the self-erasure....
Well, you use multiple keys.
If you have just one Yubikey and you lose it, then you lose your SSH access.
But that's true as well if you have just one SSH key on your computer and you lose your computer, right?
One nice thing about using a YubiKey GPG key for SSH is that I don't have to generate new keys for every device.
I have offline key on two different devices, haven't lost one yet over the last decade!
Great article! I discovered ssh-tpm-agent last year and was surprised that all my ThinkPads have hardware TPM, even one from 2018! I've switched over to using it for everything and there are no more private keys on my devices.
I also have a Yubikey but it's been a few years since I tried it for ssh keys. I use it as a regular security key. Maybe I should try it for ssh keys as it sounds like the tooling has improved.
>I've switched over to using it for everything and there are no more private keys on my devices.
What happens when you replace the ThinkPad?
Best practice is to not share keys between devices. If you have multiple devices, you should have multiple keys.
Interesting, will take a look
How do you break back into your digital life though? You go travelling to a foreign place, get mugged, you don't have your wallet phone or keys. All you have is the grace of a strange letting you use their laptop. How do you get back in?
You fly back.
I wanted to warn people that I had a Yubkey usb-c nano MELT because I had a Thinkpad in an awkward location with poor ventilation for slightly too long. Not sure what kind of plastic it uses but it melts very easily and took the entire device with it.
> All YubiKeys have an operational temperature range between 0°C to 40°C (32°F to 104°F).
> All YubiKeys have a storage temperature range between -20°C to 85°C (-4°F to 185°F)
https://docs.yubico.com/hardware/yubikey/datasheet/_static/Y...
Given how incredibly important they can be, having an operating range that can be exceeded by ambient temperatures in some (very populated!) regions of the world is a design flaw.
iPhones are only designed to work from 0-35C, which is more narrow than the yubikeys, and way more important and way more widely used, so I think it will probably be okay.
Are iphones reported to melt into a goop that detroys surrounding electronics in a ~100 degree room like the yubikey apparently is though?
> > All YubiKeys have a storage temperature range between -20°C to 85°C (-4°F to 185°F)
It doesn’t sound like the yubikey is designed to melt into a goop in ~100 degree room either?
> All YubiKeys have an operational temperature range between 0°C to 40°C (32°F to 104°F).
Maybe they should work on that, otherwise Yubikeys won't be operating during the summer soon...
Also, winter.
So, I can't use it outside about a quarter of the year? Terrible
> You will notice though that the private keyfile held on disk uses a .tpm extension this time. This is where an important caveat of the TPM method comes in – Unlike the YubiKey approach, the private key material is actually being stored on disk here. This is because TPM devices do not natively support enrolling SSH keys. So what is happening is that the private key material is being stored on disk in encrypted form, and the TPM is being used to decrypt it.
While this is true, you can create static hierarchy keys that would not resolve into separate key files for the TPM. The issue here is that anyone with access to your TPM can recreate the keys, unless you set a owner password.
https://man.archlinux.org/man/ssh-tpm-agent.1#Hierarchy_keys
What is the benefit of using yubikeys instead of ssh keys (ie a plain text file)? My main concern is losing the device or damaging it.
Yubikeys (or something like keys in the Secure Enclave of a Mac or iPhone) are themselves not accessible by software, you can only ask the hardware to perform operations with them (e.g. decrypt or sign something).
This means someone with access to your filesystem can’t just copy the key off.
Of course a passphrase is another protective layer. But passphrases can be keylogged, or the decrypted key (or passphrase, depending) in the memory of an ssh agent can be can potentially be read out by an attacker.
With separate hardware, the assumption is that the hardware is much much harder to attack, so the keys stay on there and only on there.
If someone compromises your computer, they can copy your SSH keys and their passphrase and then, well, use them. And you may never know it.
If someone steals your Yubikey without compromising your computer, they cannot use it without the passphrase.
If someone compromises your computer to copy your passphrase, AND steals your Yubikey, then they can use it. But you will know that you are not in possession of your Yubikey anymore, and you can do something about it.
You can require physical user interaction to use the key. If the mossad is after you, this doesn't really help you because they'll just replace the ssh binary and intercept one of your requests, but against a less motivated attacker, it makes it harder to pivot from one machine into others.
My employer gives us yubikeys. The first one I had randomly stopped working after a year. I wouldn't use it for anything where pairing a new key is difficult if you cannot use the old one anymore.
> My employer gives us yubikeys. The first one I had randomly stopped working after a year. I wouldn't use it for anything where pairing a new key is difficult if you cannot use the old one anymore.
I really don't see how you can blame Yubikey for this one, your employer is entirely to blame for only supplying you with one Yubikey.
Your hardware fault is unfortunate, I've been using Yubikeys for many years without a single issue. But ultimately that is beyond the point here .... come on dude, you work in IT, surely by now you know you don't put all your eggs in one IT basket ?
Not following fundamental best-practice and then complaining when things break is really a "move along, nothing to see here" moment.
The solution to the second is to have 2-5 devices, each with their own key. I have a nano in each laptop, one backup one at each house, and a floating one on my keyring.
What is the longevity and migration/upgrade story for a yubikey?
I always worry what do you do when your hardware fails
> I always worry what do you do when your hardware fails
There is only one answer to your question: It is irrelevant.
Yubikeys are not that expensive, and anybody who depends on Yubikeys should have minimum two of them.
This isn't a Yubikey thing. Its a basic IT equipment redundancy 101 ... if its important, minimum N+1 it.
The same reason you don't put all your eggs on a single disk.
First: Always buy two (or n+1). Enroll both. Store one somewhere safe.
Anecdotally I have keys from 2013 that are still working fine when I test them, although they've been superseded by newer models.
To my knowledge no one has done the Consumer Reports / Mythbusters style of testing, buying many Yubikeys over time (to get different hardware batches) and subjecting them to various tests...
I used the same one for work every day from 2018 to 2025.
There's also an alternative way to using ssh-tpm-agent - the tpm2-pkcs11 stack provides it, here's a guide for setting it up: https://jade.fyi/blog/tpm-ssh/ Not related to the author, but using that setup personally. Similar considerations as in the OPs TPM notes.
My sticking point with these has always been SSH agent support: the last time I looked, I couldn't use my Yubikey to authenticate the second hop in an A->B->C sequence with a jump box in the middle. Is that resolved now? This blog doesn't address that use case.
If you use ProxyJump (-J) you don't forward the agent off of host A and it works with completely arbitrary chains (A->B->C->D->E->F->...)
Except that's not really the network path that's going on, right? If I am on A and connect to B and then want to use the network to transfer a bunch of data from B to C over an ssh channel between them, using ProxyJump brings it all back to the bastion box. And if A is in the US and B & C are in Europe connected with a 40 GB LAN, using proxyjump will bring the connection from B->C into Europe->US->Europe. How do I do this with my credentials existing on A being used to log in to B and then the same credentials being used to connect from B to C while only having the one network connection is B->C and not to my bastion host?
If you're using -sk keys (i.e. setup via FIDO2 on the Yubikey like the article, not PIV) yes you can forward the keys loaded into your agent like normal, with ssh -A.
PIV keys can also be forwarded through an agent. The only thing to be aware of is that the agent process will effectively cache your PIN if you do this.
ssh-add -s <pkcs11.so>
To mitigate that you can add -c to the ssh-add command. That will make the agent prompt for confirmation (on the host with the PIV key) every time something asks to use it. Doesn't quite get you the same security as just not caching the PIN to start with but it's OK.
Thanks to both you and the other person who replied. I appreciate the education.
I like the idea of using my MacBook's Secure Enclave for ssh keys but what do I do if I develop on a remote VPS (purely because I'd rather have the AI accidentally rm -rf / on a VPS than my own computer)?
You can use agent forwarding (ssh -A) to make your local hardware-backed key available on the VPS.
If you're using tmux on the VPS this can get a little tricky, because by default your tmux panes will inherit your initial SSH client's SSH_AUTH_SOCK; when you re-attach tmux in a different SSH session you'll no longer be able to authenticate with the local YubiKey. But I wrote a shell script that hooks into tmux to keep SSH_AUTH_SOCK working correctly with this kind of setup: https://github.com/mshroyer/socklink
Brilliant, I just ran into this problem a couple weeks ago. Thanks!
There is also a middle ground - it's possible to sign SSH certificate using yubikey as CA. By controlling expiration and set of principals, the cert you get on your machine is scope restricted, and quickly expires. Or you can load it into an ssh-agent and drop it immediately.
As a guy prone to losing and damaging hardware, this seems like a fantastic HOWTO for locking yourself permanently out of everything.
Way too much work for my taste. I use the bitwarden CLI to fetch/store the key. Opted in for ed25519 on day 1. Works great.
That's kind of the opposite of the approach presented here, though. With hardware-resident, non-exportable keys, you can know that no matter what you do, key hashes to some physical device are 1:1.
Whether you want or don't want that property is a different story.
is this for new machine installs , or periodic (daily , weekly) access?
I thought private keys were meant to be secret (not copied) and the public keys meant to be installed onto the target host.
so if you have a desktop, laptop etc, you would generate the secret once per endpoint and then copy the public keys onto targets as needed. AT scale this could be done by ldap / cloud mgmt.
Unfortunately, YubiKeys and TPM don't work under WSL2 without complex and potentially unsafe workarounds.
Do yubikeys suffer from flash memory bit rot? If so, it seems it would best have several backups that rotate once a month to being the primary so that the memory gets used?
In my anecdotal experience Yubikeys tend to physically deteriorate long before this happens. I still have an early YubiKey 5 that must be 8 years old at this point that I use daily and it works fine.
How are those compared to nitrokeys? I know an IT guy that prefers those, I think he mentioned those are open hardware?
Last time I checked, it was possible to update the firmware of Nitrokeys. Which seems to completely defeat the purpose of the device?
They started selling the Nitrokey 3 long (like years) before the software had the main features they claimed they had, so I understand why they would want to be able to push updates. But again in terms of security, the first thing I want from a security key is that a compromised computer cannot reflash it...
> Last time I checked, it was possible to update the firmware of Nitrokeys. Which seems to completely defeat the purpose of the device?
Yeah. Precisely my problem with Nitrokey too.
If you can flash it, then so will a bad-guy one day. Not if, but when. You really can't say never anymore these days.
The flashing process is also annoying and fragile, I bought a couple of Nitrokeys for random testing and generally found myself haivng to repeat the process a couple of times before it finally worked.
Also, IIRC not all Nitrokeys have a secure enclave, a bunch just store secrets in software.
> Not if, but when
Well, as soon as they compromise your computer. Which is the same security as just having the private SSH key as a file on your computer. Except that if you have a Nitrokey, you may trust it more than a file on your computer, which actually makes it even worse.
Interested also in pragmatic feedback about Nitrokeys, oss sw and hw is quite a compelling advantage over yubikeys
for most use cases they're functionally the same. the biggest difference for me is that nitrokey doesn't support doing hmac-sha1 challenge, so i can't use the nitrokey with keepassxc.
fido, gpg, piv all work perfectly
Many people have issues with nitro key as a business.
Something about stealing an iPhone jailbreak years ago?
wrong nitrokey
the ios thing was in 2009. the company that produces nitrokey security hardware was founded in 2015
i recently bought three yubikeys: a 5C nano to sit in my PC, a 5C NFC on my keychain and another 5C NFC as a backup. currently only using them as SSH keys to my servers. probably overkill but all the recent cybersecurity issues got me a little spooked.
i can't lie, the thought of removing authenticator/authy 2FA from my critical accounts is a little daunting.
Kerberos is such a beautiful thing.
I have YubiKeys but frankly, I'm not a big fan, at least not for daily use on websites. SSH keys seems like a good use case for it tbh.
For now, I'm hosting my own Vaultwarden and I pushed all my ssh keys to that. Anyone feel strongly against that? It stores their passphrase as well.
I use passkeys on the device where possible.
I add YubiKeys as backup everywhere I can. Like I have one YubiKey that always stays at home in case I lose my laptop and phone so I can get access back to some stuff sooner than later.
What does an ssh key passphrase accomplish if stored with the key?
Can yubikeys be used for logging onto Linux consoles?
Yes, it's really nice. The fingerprint scanner on my Thinkpad is terrible (even when enrolling a fingerprint multiple times, etc.), whereas the YubiKey Bio work much better. So I use it as an alternative screen unlock.
Yes, via PAM. I use the rssh module myself.
Nice thanks, I'm getting one to play with.
It's great, I'm completely done with passwords in Linux.
Do you have a backup yubi key? So it's a good idea to buy 2?
Thanks for the info, I have ubiqeys but i did not where to start!
You can also store static passwords on yubikey which requires no direct hardware access. Using a prefix to protect them a little more
Speaking of which I do something cool with my main desktop (an AMD 7700X, but whatever). It's running Debian and there's zero executable with the suid bit set. There's no sudo command installed. Using "su" to su to root is technically possible but it's like a 40 characters long password, meant to never be entered (and I never ever entered it using su).
The only way I can get root (short of using a local privilege exploitation exploit to gain root) is by logging, using a Yubikey, over ssh, from a dedicated laptop that has nothing but a SSH client. So the Yubikey is plugged into the laptop.
That dedicated laptop has no wifi drivers and is on a dedicated LAN and that dedicated LAN only has two machines on it: my desktop and that laptop.
The laptop serves no purpose other than: "log in as root on the desktop".
Of course both that laptop and the desktop have firewalls.
This means, yes, I allow log in as root using SSH. But password logins are disabled and the only way to log in as root using SSH is to use the Yubikey. And the firewall only allow SSH attempts to port 22 from the MAC address, IP, and dedicated LAN that corresponds to the laptop.
In addition to that my main desktop has multiple user accounts: but none of those users are allowed to su or sudo.
Now, I know, I could use Qubes or whatever.
But what I explained above is how I do it: even myself I cannot get root on my own desktop without my laptop + Yubikey (I'd need to find and enter the 40 chars long root password).
When I really need to do something as root (and once becoming root becomes extremely hard, suddenly you realize how rare are the occasions when you really need to be root), I open my laptop and boot it (and because that laptop has no GUI and basically nothing besides a SSH client, it boots in three seconds or so).
Haters gonna hate but that's been working totally fine for me since years and years.
Why not use:
1. Virtual terminals? The answer I can think of is usb hid attacks.
2. Serial Consoles? My answer the is encrypted connections on the wire.
Any other reasons than mines?
all the commandline magic makes me shy away. why is security always combinde with lines over lines of comamndline
Doesn't Yubico have some GUIs for the Yubikeys? I don't know, I prefer the CLI :-). But I remember seeing something like that.
To answer your question, a CLI is a lot easier to write than a GUI. So the GUI would be the next step after the CLI, if there was enough demand. Now I'm guessing that people who care enough about security to try to handle it properly are anyway people who can use the CLI.
Unfortunately hardware resident keys are basically useless under qubes.
I'd rather have qubes than hardware resident keys.
Sorry, but how so? I use FIDO2 and PKCS #11 keys in Qubes OS. PKCS#11 I cheated slightly by assigning access to the one VM that needs that key via sys-usb, but for all the other AppVMs I use the qubes-ctap-proxy and OpenSSH FIDO2 keys. It's pretty easy and can even be locked down via ACLs.
It's true that TPM 2.0 is out the window, but FIDO keys are cheap, convenient, and secure.
You can't use disposable qubes and resident keys.
So you have to put way more things into a single qube than you otherwise would want to.
The design is basically incompatible with using qubes in a meaningfully useful way.
Ah, I see. I don't use disposable VMs much, and when I do I'm not using them as SSH clients, so I had no idea. I would push back on heavy use of disposable VMs as the only way to meaningfully use Qubes OS though—although handy, they're not my main motivator for dailying it.
This is all, frankly, nonsense.
If you're using userspace software to decrypt the keys into memory then it's all just security theatre. Any sufficiently motivated actor, ot a comprised package in the software chain, can capture your private key in the clear. Especially if using ssh-agent that keeps it resident in memory 'for convenience'.
Proper HSM integration with SSH requires challenges signed by the HSM without the private key ever being emitted.
Just keep your SSH keys on disk, one per remote host, with a different encryption password for each target host. If you're worried about threats that can defeat that then they'll just come to your door and beat the passwords out of you.
At least for the Yubikey option, the whole point is that the private key does not leave the HSM.
> This is all, frankly, nonsense.
Only because you clearly don't understand how it works. What's interesting is that you seem to understand half of it... or maybe you don't, and you just repeat stuff you don't understand either?
> If you're using userspace software to decrypt the keys into memory
That's exactly why you don't do it. A Yubikey or a secure enclave create the private key and emit the public key.