> Obviously forking go’s crypto library is a little scary, and I’m gonna have to do some thinking about how to maintain my little patch in a safe way
This should really be upstreamed as an option on the ssh library. Its good to default to sending chaff in untrusted environments, but there are plenty of places where we might as well save the bandwidth
I come from a world (yesteryear) where a computer had 1KB of RAM (ZX80). I've used links with modems rocking 1200 bps (1200 bits per second). I recall US Robotics modems getting to speeds of 56K - well that was mostly a fib worse than MS doing QA these days. Ooh I could chat with some bloke from Novell on Compuserve.
In 1994ish I was asked to look into this fancy new world wide web thing on the internet. I was working at a UK military college as an IT bod, I was 24. I had a Windows 3.1 PC. I telnetted into a local VAX, then onto the X25 PAD. I used JANET to get to somewhere in the US (NIST) and from there to Switzerland to where this www thing started off. I was using telnet and WAIS and Gopher and then I was apparently using something called "www".
I described this www thing as "a bit wank", which shows what a visionary I am!
Fellow old here, I had several 56k baud modems but even my USR (the best of the bunch) never got more than half way to 56k throughput. Took forever to download shit over BBS...
The real analog copper lines were kind of limited to approx 28K - more or less the nyquist limit. However, the lines at the time were increasingly replaced with digital 64Kbit lines that sampled the analog tone. So, the 56k standard aligned itself to the actual sample times, and that allowed it to reach a 56k bps rate (some time/error tolerance still eats away at your bandwidth)
If you never got more than 24-28k, you likely still had an analog line.
56k was also unidirectional, you had to have special hardware on the other side to send at 56k downstream. The upstream was 33.6kbps I think, and that was in ideal conditions.
The special hardware was actually just a DSP at the ISP end. The big difference was before 56k modems, we had multiple analog lines coming into the ISP. We had to upgrade to digital service (DS1 or ISDN PRI) and break out the 64k digital channels to separate DSPs.
The economical way to do that was integrated RAS systems like the Livingston Portmaster, Cisco 5x00 seriers, or Ascend Max. Those would take the aggregated digital line, break out the channels, hold multiple DSPs on multiple boards, and have an Ethernet (or sometimes another DS1 or DS3 for more direct uplink) with all those parts communicating inside the same chassis. In theory, though, you could break out the line in one piece of hardware and then have a bunch of firmware modems.
The asymmetry of 56k standards was 2:1, so if you got a 56k6 link (the best you could get in theory IIRC) your upload rate would be ~28k3. In my expereience the best you would get in real world use was ~48k (so 48kbpd down, 24kbps up), and 42k (so 21k up) was the most I could guarantee would be stable (baring in mind “unstable” meant the link might completely drop randomly, not that there would be a blip here-or-there and all would be well again PDQ afterwards) for a significant length of time.
To get 33k6 up (or even just 28k8 - some ISPs had banks of modems that supported one the 56k6 standards but would not support more than 28k8 symmetric) you needed to force your modem to connect using the older symmetric standards.
Yeah 28k sounds more closer to what I got when things were going well. I also forget if they were tracking in lower case 'k' (x1000) or upper case 'K' (x1024) units/s which obviously has an effect as well.
The lower case "k" vs upper case "K" is an abomination. The official notation is lower case "k" for 1000 and lower case "ki" for 1024. It's an abomination too, but it's the correct abomination.
That's a newer representation, mostly because storage companies always (mis)represented their storage... I don't think any ISPs really misrepresent k/K in kilo bits/bytes
* 56k baud modems but even my USR (the best of the bunch) never got more than half way to 56k throughput*
56k modem standards were asymmetric, the upload rate being half that of the download. In my experience (UK based, calling UK ISPs) 42kbps was usually what I saw, though 46 or even 48k was stable¹ for a while sometimes.
But 42k down was 21k up, so if I was planning to upload anything much I'd set my modem to pretend it as a 36k6 unit: that was more stable and up to that speed things were symmetric (so I got 36k6 up as well as down, better than 24k/23k/21k). I could reliably get a 36k6 link, and it would generally stay up as long as I needed it to.
--------
[1] sometimes a 48k link would last many minutes then die randomly, forcing my modem to hold back to 42k resulted in much more stable connections
Even then, it required specialized hardware on the ISP side to connect above 33.6kbps at all, and almost never reliably so. I remember telling most of my friends just to get/stick with the 33.6k options. Especially considering the overhead a lot of those higher modems took, most of which were "winmodems" that used a fair amount of CPU overhead insstead of an actual COM/Serial port. It was kind of wild.
Yep. Though I found 42k reliable and a useful boost over 36k6 (14%) if I was planning on downloading something big¹. If you had a 56k capable modem and had a less than ideal line, it was important to force it to 36k6 because failure to connect using the enhanced protocol would usually result in fallback all the way to 28k8 (assuming, of course, that your line wasn't too noisy for even 36k6 to be stable).
I always avoided WinModems, in part because I used Linux a lot, and recommended friends/family do the same. “but it was cheaper!” was a regular refrain when one didn't work well, and I pulled out the good ol' “I told you so”.
In case anyone else is curious, since this is something I was always confused about until I looked it up just now:
"Baud rate" refers to the symbol rate, that is the number of pulses of the analog signal per second. A signal that has two voltage states can convey two bits of information per symbol.
"Bit rate" refers to the amount of digital data conveyed. If there are two states per symbol, then the baud rate and bit rate are equivalent. 56K modems used 7 bits per symbol, so the bit rate was 7x the baud rate.
Not sure about your last point but in serial comms there are start and stop bits and sometimes parity. We generally used 8 data bits with no parity so in effect there are 10 bits per character including the stop and start bits. That pretty much matched up with file transfer speeds achieved using one of the good protocols that used sliding windows to remove latency. To calculate expected speed just divide baud by 10 to covert from bits per second to characters per second then there is a little efficiency loss due to protocol overhead. This is direct without modems once you introduce those the speed could be variable.
Yes, except that in modern infra i.e. WiFi 6 is 1024-QAM, which is to say there are 1024 states per symbol, so you can transfer up to 10bits per symbol.
Yes, because at that time, a modem didn't actually talk to a modem over a switched analog line. Instead, line cards digitized the analog phone signal, the digital stream was then routed through the telecom network, and the converted back to analog. So the analog path was actually two short segments. The line cards digitized at 8kHz (enough for 4kHz analog bandwidth), using a logarithmic mapping (u-law? a-law?), and they managed to get 7 bits reliably through the two conversions.
ISDN essentially moved that line card into the consumer's phone. So ISDN "modems" talked directly digital, and got to 64kbit/s.
An ISDN BRI (basic copper) actually had 2 64kbps b channels, for pots dialup as an ISP you typically had a PRI with 23 b, and 1 d channel.
56k only allowed one ad/da from provider to customer.
When I was troubleshooting clients, the problem was almost always on the customer side of the demarc with old two line or insane star junctions being the primary source.
You didn’t even get 33k on analog switches, but at least US West and GTE had isdn capable switches backed by at least DS# by the time the commercial internet took off. Lata tariffs in the US killed BRIs for the most part.
T1 CAS was still around but in channel CID etc… didn’t really work for their needs.
33.6k still depended on DS# backhaul, but you could be pots on both sides, 56k depended on only one analog conversion.
As someone that started with 300/300 and went via 1200/75 to 9600 etc - I don't believe conflating signalling changes with bps is an indication of physical or temporal proximity.
Oh, I got the implication, but I think it was such a common mistake back then, that I don't think it's age-related now - it's a bit of a trope, to assume baud and bps mean the same thing, and people tend to prefer to use a more technical term even when it's not fully understood. Hence we are where we are with terms like decimate, myriad, nubile, detox etc, forcefully redefined by common (mis)usage. I need a cup of tea, clearly.
Anyway, I didn't think my throw-away comment would engender such a large response. I guess we're not the only olds around here!
No, just that confusing the two was ubiquitous at the time 14.4k, 28k, and 56k modems were the standard.
Like it was more common than confusing Kbps and KBps.
I mean, the 3.5" floppy disk could store 1.44 MB... and by that people meant the capacity was 1,474,560 bytes = 1.44 * 1024 * 1000. Accuracy and consistency in terminology has never been particularly important to marketing and advertising, except marketing and advertising is exactly where most laypersons first learn technical terms.
I started out with a 2400 baud US Robotics modem with my "ISP" being my local university to surf gopher and BBS. When both baud rates and bits per second were being marketed side by side I kinda lost the thread tbh. Using different bases for storage vs transmission rates didn't help.
PoiZoN BBS Sysop chiming in. I ran the BBS on a free phone line I found in my childhood bedroom. I alerted the phone company and a tech spent a day trying to untangle it, but gave up at the end of his shift. He even stopped by to tell me it wouldn’t be fixed.
I didn’t know the phone number, so I bought a Caller ID box, hooked it to my home line, and phoned home. It wasn’t long before every BBS in town had a listing for it.
I had to wait til I was old enough to get a phone line in my own name before running a BBS. And also til I had a modem that would auto-answer, which was not a given back then!
But I confess my first question for a working but unassigned phone line would be: who gets the bill for long distance calls?
I had access to no-cost long distance calling through other administrative oversights, but they were a bit more effort to maintain! :)
And I felt privileged because the configuration for my TI-99/4A Terminal Emulator (which I believe was called Terminal Emulator) had options for 110 or 300 baud, and I felt lucky to be able to use the "fast" one. :)
My first modem (you always remember your first) had no carrier detection (and no Hayes commands, and no speaker...), so I would dial the number manually, then flip a switch when I heard the remote end pick up and send carrier to get the synchronization started.
Ha, same! On a TRS-80 Color, nonetheless. But I think I used four times, because no one else in the country had a BBS at the time (small city in Latin America).
It took a couple of years until it would catch on, and by then 1200 and 2400 bps were already the norm - thankfully!
Same year, I tried this cool new "Mosaic" software and thought it was a cool proof of concept, but there was no way this web thing could ever displace gopher
It sort of already is. This behavior is only applied to sessions with a TTY and then the client can disable it, which is a sensible default. This specific use case is tripping it up obviously since the server knows ahead of time that the connection is not important enough to obfuscate and this isn't a typical terminal session, but in almost any other scenario there is no way to make that determination and the client expects its ObscureKeystrokeTiming to be honored.
What's a concrete threat model here? If you're sending data to an ssh server, you already need to trust that it's handling your input responsibly. What's the scenario where it's fine that the client doesn't know if the server is using pastebin for backing up session dumps, but it's problematic that the server tells the client that it's not accepting a certain timing obfuscation technique?
The behavior exists to prevent a 3rd party from inferring keystrokes from active terminal sessions, which is surprisingly easy, particularly with knowledge about the user's typing speed, keyboard type, etc. The old CIA TEMPEST stuff used to make good guesses at keystrokes from the timing of AC power circuit draws for typewriters and real terminals. Someone with a laser and a nearby window can measure the vibrations in the glass from the sound of a keyboard. The problem is real and has been an OPSEC sort of consideration for a long time.
The client and server themselves obviously know the contents of the communications anyway, but the client option (and default behavior) expects this protection against someone that can capture network traffic in between. If there was some server side option they'd probably also want to include some sort of warning message that the option was requested but not honored, etc.
To clarify the point in the other reply -- imagine it sent one packet per keystroke. Now anyone sitting on the network gets a rough measurement of the delay between your keystrokes. If you are entering a password for something (perhaps not the initial auth) it can guess how many characters it is and turns out there are some systemic patterns in how that relates to the keys pressed -- eg letters typed with the same finger have longer delays between them. Given the redundancy in most text and especially structured input that's a serious security threat.
It's not just the pointless chaff, the SSH protocol is inherently very chatty, and SFTP even more so. The solution, for a high-performance game, is don't use SSH. Either run it over Wireguard or grab some standard crypto library and encrypt the packets yourself. You'll probably make a few minor mistakes but unless the other player is the NSA it'll be good enough.
For that matter, why does it need to be encrypted at all? What's the threat model?
If there really is a genuine need to encrypt and low latency is critical, consider using a stream cipher mode like AES-CTR to pregenerate keystream at times when the CPU is lightly loaded. Then when you need to encrypt (say) 128 bytes you peel off that many bytes of keystream and encrypt at close to zero cost. Just remember to also MAC the encrypted data, since AES-CTR provides zero integrity protection.
Serious question, why not just use websockets? AFAIK, it's effectively a TLS socket with a little bit of handshake overhead when starting.
I'm literally working on a web interface I want to use for classic BBS door play... currently working on a DOS era EGA interface, and intend to do similar for PETSCII/Comodore64/128 play as well. I've got a couple rendering bugs to explore for ansis submitted that messed up in the viewer test mode.
It's been an opportunity to play with AI dev as well... spent as much time getting the scrollback working how I want as it took on the general rendering.
Websockets is just another layer on top of TLS, so you've got the size explosion and complexity/latency of TLS and then another layer on top of that. The OP hasn't provided additional info on what the requirements are but if it's a realtime game then they'll probably be "as close to zero latency and size increase as possible (compared to unencrypted messaging)", which websockets over TLS isn't.
Yes, but I wouldn't be surprised if the change is rejected. The crypto library is very opinionated, you're also not allowed to configure the order of TLS cipher suites, for example.
That is a completely valid threat model analysis, though? "Just hope no bad guy ever gets into the safe" is rather the entire point of a safe. If you have a safe, in which you use the contents of the safe daily, does it make sense to lock everything inside the safe in 100 smaller safes in some kind of nesting doll scheme? Whatever marginal increase in security you might get by doing so is invalidated by the fact that you lose all utility of being able to use the things in the safe, and we already know that overburdensome security is counterproductive because if something is so secure that it becomes impossible to use, those security measures just get bypassed completely in the name of using the thing. At some level of security you have to have the freedom to use the thing you're securing. Anything that could keep a bad guy from doing anything ever would also keep the good guy, ie. you, from doing anything ever.
Perhaps figuratively? I manage several servers where the majority of (LDAP) accounts have no special privileges at all. They get their data in the directories and can launch processes as their user, that's...pretty much it.
Though the upstream comment is gone and I am perhaps missing some important context here.
When the question is "how do I communicate securely with a third party," there's nothing you can do if the third party in question gets possessed by a demon and turns evil. (Which is what happens if an attacker has root.)
Random sysadmins who have access to your server have the permissions to steal whatever is communicated between third parties unrelated to this sysadmin.
Just because some random outsourced nightshift dude has the permissions to do "sudo systemctl restart" shouldn't mean he gets to read all the secret credentials the service uses.
As it is now, the dude has full unfettered access to all credentials of all services on that machine.
Threats exist in both trusted and untrusted environments though.
This feels like a really niche use case for SSH. Exposing this more broadly could lead to set-it-and-forget-it scenarios and ultimately make someone less secure.
+1... Given how much SSH is used for computer-to-computer communication it seems like there really should be a way to disable this when it isn't necessary.
In practice I've never felt this was an issue. But I can see how with extremely low bandwidth devices it might be, for instance LoRa over a 40 km link into some embedded device.
Once upon a time I worked on a project where we SSH'd into a satellite for debugging and updates via your standard electronics hobbiest-tier 915mhz radio. Performance was not great but it worked and was cheap.
I haven't heard much about the ArduCopter (and ArduPilot) projects for a decade, are those projects still at it? I used to run a quadroter I made myself a while back until I crashed it in a tree and decided to find cheaper hobbies...
Well at least crashing drones into trees has never been cheaper hahaha. So it's super easy to get into nowadays, especially if it's just to play around with flight systems instead of going for pure performance.
Because TCP interprets packet loss as congestion and slows down. If you're already on a slow, lossy wireless link, bandwidth can rapidly fall below the usability threshold. After decades of DARPA attending IETF meetings to find solutions for this exact problem [turns out there were a lot of V4 connections over microwave links in Iraq] there are somewhat standard ways of setting options on sockets to tell the OS to consider packet loss as packet loss and to avoid slowing down as quickly. But you have to know what these options are, and I'm pretty sure the OP's requirement of having `ssh foo.com` just work be complicated by TCP implementations defaulting to the "packet loss means congestion" behavior. Hmm... now that I think about it, I'm not even sure if the control plane options were integrated into the Linux kernel (or Mac or Wintel)
The guy in charge of Go's security decreed that SSL 1.3 (which he was a contributor to) was so secure that silly programmers should not be able to override what algorithms are allowed or not allowed, because why would they possibly need to do that, because he's such a genius, and even if someone DID find a security vulnerability, well....they can just wait for Google to publicly disclose it and release a patch, compile the new version, update their code to work with that version of Go, rebuild their containers, put stuff through testing, and then release it into production.
Versus...seeing there's a vulnerability, someone adding a one-line change to disable the vulnerable algorithm, compile, image update, test. And a lot less testing because you're not moving to a new version of the language / compiler.
The man has no practical experience in running a production network service, an ego the size of a small moon, and yet was a major contributor to a security protocol now in use by billions of people.
But hey, you can be a handbag designer and end up head of design at Apple soooooooo
Lots of the real world vulnerabilities out there exist exactly because of people choosing to support a range of crypto algorithms.
Sure, if it's an internal tool you can recompile both ends and force a universal update. But anything else and you need to stay compatible with clients and anytime you allow negotiation of the cryptosuit you open yourself up to quite a few subtle attacks. Not saying that choice about go is clearly a good one but i don't think it's obviously wrong.
Very interesting, I hadn't heard of this obfuscation before so it was well worth clicking.
Another good trick for debugging ssh's exact behavior is patching in "None" cipher support for your test environment. It's about the same work as trying to set up a proxy but lets you see the raw content of the packets like it was telnet.
For terminal games where security does not matter but performance and scale does, just offering telnet in the first place can also be worth consideration.
1980s: 1 packet per keystroke is too much, we must find a solution to bundle them together, for efficiency (see Nagle's algorithm, delayed ACK), also let's send everything in plaintext, including passwords
2020s: ha! with some advanced probabilistic models, we may be able to deduce something about what is being typed behind one of our layers of encryption, let's sent 100 packets per keystroke to mitigate that
Unfortunate result of the security theater.. "Someone who has access to run privileged application can run side channel attacks! Let's drop cpu performance 20 percent over the world"
As I understood it’s enough to have “access to run privileged application” anywhere where the packet goes through. So, not necessarily at client or server sides. Or did I misunderstand?
I don't see how Claude helped the debugging at all. It seemed like the author knew what to do and it was more telling Claude to think about that.
I've used Claude a bit and it never speaks to me like that either, "Holy Cow!" etc. It sounds more annoying than interacting with real people. Perhaps AIs are good at sensing personalities from input text and doesn't act this way with my terse prompts..
Even if the chatbot served only as a Rubber Ducky [1], that's already valuable.
I've used Claude for debugging system behavior, and I kind of agree with the author. While Claude isn't always directly helpful (hallucinations remain, or at least outdated information), it helps me 1) spell out my understanding of the system (see [1]) and 2) help me keep momentum by supplying tasks.
A rubber ducky demands that you think about your own questions, rather than taking a mental back seat as you get pummeled with information that may or may not be relevant.
I assure you that if you rubber duck at another engineer that doesn't understand what you're doing, you will also be pummeled with information that may or may not be relevant. ;)
I don't think that's right. When you explain a technical problem to someone who isn't intimately familiar with it you're forced to think through the individual steps in quite a bit of detail. Of course that itself is an acquired skill but never mind that.
The point or rubber duck debugging then is to realize the benefit of verbally describing the problem without needing to interrupt your colleague and waste his time in order to do so. It's born of the recognition that often, midway through wasting your colleague's time, you'll trail off with an "oh ..." and exit the conversation. You've ended up figuring out the problem before ever actually receiving any feedback.
To that end an LLM works perfectly well as long as you still need to walk through a full explanation of the problem (ie minimal relevant context). An added bonus being that the LLM offers at least some of the benefits of a live person who can point out errors or alert you to new information as you go.
Basically my quibble is that to me the entire point of rubber duck debugging is "doesn't waste a real person's time" but it comes with the noticeable drawback of "plastic duck is incapable of contributing any useful insights".
> When you explain a technical problem to someone who isn't intimately familiar with it you're forced to think through the individual steps in quite a bit of detail.
The point of Rubber Ducking (or talking/praying to the Wooden Indian, to use an older phrase that is steeped in somewhat racist undertones so no longer generally used) is that it is an inanimate object that doesn't talk back. You still talk to it as if you were explaining to another person, so are forcing yourself to get your thoughts in order in a way that would make that possible, but actually talking to another person who is actively listening and actually asking questions is the next level.
I guess I can see where others are coming from (the LLM is different than a literal rubber duck) but I feel like the "can't reply" part was never more than an incidental consequence. To me the "why" of it was always that I need to solve my problem and I don't want to disturb my colleagues (or am unable to contact anyone in the first place for some reason).
So where others see "rubber ducking" as explaining to an object that is incapable of response, I've always seen it as explaining something without turning to others who are steeped in the problem. For example I would consider explaining something to a nontechnical friend to qualify as rubber ducking. The "WTF" interjections definitely make it more effective (the rubber duck consistently fails to notify me if I leave out key details).
AIs are exceptional at sensing personalities from text. Claude nailed it here, the author felt so good about the "holy cow" comments that he even included them in the blog post. I'm not just poking this, but saying that the bots are fantastic sycophants.
Funny to see this fixed in 2023 and the side effects. Back in 2004, before I focused on performance, I did some security work including inter-keystroke latency analysis of captured SSH sessions to estimate the commands typed:
> When you say “LLMs did not fully solve this problem” some people tend to respond with “you’re holding it wrong!”
>
> I think they’re sometimes right! Interacting with LLMs is a new skill, and it feels pretty weird if you’re used to writing software like it’s 2020. A more talented user of LLMs may have trivially solved this problem.
So one thing I only recently figured out is that using ChatGPT via the web browser chat is massively different from using OpenAI's code-focused Codex model / interface. Once I switched to using Codex (via the VS Code extension + my own ChatGPT subscription) the quality of answers I got improved massively.
So if you're trying to use LLM to help with debug, make sure you're using the right model!! There are apparently massive differences between models of the same generation from the same company
3. If you uncork the socket, or if the buffer hits MSS, the kernel sends the packet
Basically, the kernel waits until it has a full packet worth of data, or until you say you don't have any more data to send, and then it sends. Sort of an extreme TCP_YESDELAY.
Oh wow - I've never heard of TCP_CORK before. Without disabling pings I'd still pay the cost of receiving way more packets, but maybe that'd be tolerable if I didn't have to send so many pongs. This is super handy; excited to play around with it.
I am aware of TCP_NODELAY (funny enough I recently posted about TCP_NODELAY to HN[1] when I was thinking about it for the same game that I wrote about here). But I think the latency hit from disabling it just doesn't work for me.
I missed that thread originally, the post and the comments where a good read, thank you for sharing.
I got a kick out of this comment [0]. "BenjiWiebe" made a comment about the SSH packets you stumbled across in that thread. Obviously making the connection between what you were seeing in your game and this random off-hand comment would be insane (if you had seen the comment at all), but I got a smile out of it.
Can you explain how TCP_CORK helps here? The chaff packets are spaced 20ms apart and sent per socket, so I don’t see how TCP_CORK could help unless it coalesced across 20ms intervals? But coalescing is clearly not an option for the intended obfuscation effect of the original feature.
It is unrelated to SSH, it is a generic TCP thing.
"hello world" fits in a single TCP packet, but the kernel might end up sending one packet containing "hello" and another packet containing " world". It is completely opaque to userspace.
TCP_CORK lets userspace decide when packets are dispatched. You get to control whether "hello world" is sent across 1 packet or 11 packets.
Wow, I did not realize that SSH did that. Good to know, and it makes sense as a default, because the people who need it need to have it on by default. But I think I'm going to be turning that off, because it's a security measure that doesn't make sense for my particular environment:
1) I'm pretty much never typing secrets into an SSH tunnel; these days if there's a secret I need to transmit over SSH I'm going to be copying and pasting it, which will not reveal info from keyboard timing. (Or rsync'ing a file, which ditto).
2) I'm not in a high-security environment where nation-states have an interest in sniffing my keystrokes.
3) I often open SSH connections to servers in other continents. Those underwater cables have massive bandwidth, but they're also in constant use by thousands upon thousands of people. So anything I can do to reduce my bandwidth by 100x is probably worth doing.
Any reason you can think of why I should not be setting ObscureKeystrokeTiming=no in my ~/.ssh/config?
I think those all have reasonable counterarguments:
(1) This sounds brittle. Are you really going to have a good mental model about what's secret when using ssh and reliably refrain from typing those things? Seems to kinda defeat the idea of securing the channel. Also, as a collection your activities might be more confidential to you than single inputs, or correlated with your other activities outside ssh, etc - it's hard to keep a mental model of this as well. Aka optimism is not a form of security.
(2) There isn't a reason to think this is a difficult attack that only a powerful adversary could mount. Seems like a college lab level thing to me. And very amenable to AI help as well. Also here optimism is not a form of security. It's a 25 year old attack[1] so there's a lot of existing research[2] around.
(3) Saving 100x bandwidth on single keystrokes on an internet dominated by video traffic just because it's 100x doesn't make sense. Also it's good to cultivate a mindset that steers away from trading off security in favour of trivial resource savings.
Bandwidth is not the problem when you are using mobile connections (4G, weaker 5G). Videos work just fine, but ssh can be painful already without keystroke obfuscation. The problem is latency. Especially when roaming abroad it can 100s of ms.
Not sure whether the obfuscation is fully synchronous, i.e waiting for the server response before continuing. That would really kill it. Working with LTS distros I don't think I have seen it in practice yet. Need to try something modern on my next trip abroad.
> Not sure whether the obfuscation is fully synchronous, i.e waiting for the server response before continuing.
The people who designed SSH aren't idiots, and also, you can answer this question by simple observation: When you connect to a server with ~200ms ping, which is somewhat common in the scenarios you describe and which I've done many times, it does not take 20 seconds to show a keystroke.
There's no way to know in advance if some leaked cleartext will provide enough information to an attacker to be useful. Attackers profit from making creative use of information they didn't have before.
That said, plenty of people disable the most useful security features of SSH, like verifying host key signatures, with no ill-effects (as far as they know). For the majority of users, using Telnet and unencrypted HTTP would make no difference, as nobody's trying to hack them, and who really cares about privacy anyway?
Did you know SSH has long-standing performance limitations due to its design that need patches to eliminate? It was never intended to be a high-performance tool. If you want really high performance, use Telnet. If you want real security, use SSH with all strong security options enabled plus a server using ContainerSSH with the OAuth2 plugin (SSH's keys are static, which can be captured and reused, which is bad). If you don't care either way, use SSH with the defaults.
> I'm pretty much never typing secrets into an SSH tunnel; these days if there's a secret I need to transmit over SSH I'm going to be copying and pasting it, which will not reveal info from keyboard timing
One common secret that goes through a tty ssh connection is a sudo password. You are probably typing sudo command so without obfuscation the attacker can find out the sudo keystrokes, the command keystrokes and then the encrypted bytes of the password. They don't have the timing data to decode them as easily as the previous parts but if they record enough traffic they might be able to decrypt the password. But maybe they won't, because the ssh session key is probably different each time. Furthermore I don't know how many times they should capture your encrypted password to be able to decrypt it. Maybe it's unfeasible.
Anyway, in case of the sudo password, if the attacker gets it what would happen? The attacker is hopefully not able to get a shell into the server. If they do they have different ways to get root privileges.
By the way, I also copy and paste secrets from either the password manager or the clipboard, because nobody remembers long random strings. The only exceptions are the passwords of a few accounts.
sudo passwords are one of the things I'm copying and pasting from the password manager, because my shell account password is different on every system. But yes, if you type your sudo password without thinking about it, the timing attack might be feasible. (Though if you're laboriously copying a random password from a different screen, as I've had to do once or twice in situations where copy-and-paste was infeasible, the timing data will be useless as it's about 500 ms between keystrokes no matter what the previous keystroke is. Which is an interesting way to accidentally defeat this attack.)
In 2023, ssh added keystroke timing obfuscation. The idea is that the speed at which you type different letters betrays some information about which letters you’re typing. So ssh sends lots of “chaff” packets along with your keystrokes to make it hard for an attacker to determine when you’re actually entering keys.
Now that's solving the problem the wrong way. If you really want that, send all typed characters at 50ms intervals, to bound the timing resolution.
Wouldn't this just change the packet interval from 20ms to 50ms? Or did you mean a constant stream of packets at 50ms intervals, nonstop?
I think the idea behind the current implementation is that the keystrokes are batched in 20ms intervals, with the optimization that a sufficiently long silence stops the chaff stream, so the keystroke timing is obfucated with an increased error bar of 20ms multiplied by number of chaff packets.
I assume the problem, such as it is, relates to the fact that a real human typing in 20-50ms would generate a few characters at most but a program could generate gobs of data. So automatically you know what packets to watch. Then you know if there were more the likely keys were in set X, while if there were fewer the likely keys were in set Y.
So a clock doesn't solve the problem. The amount of data sent on each clock pulse also tells you something about what was sent.
The Chaff packets already fire on a timer. They inject random extra fake keystrokes so you can't tell how many keystrokes were actually made. The only other way I can think of to solve that is by using a step function: Send one larger packet (fragmented or the same number of individual packets) on each clock pulse if the actual data is less than some N where N is the maximum keystrokes ever recorded with some margin. Effectively almost every clock pulse will be one packet (or set of packets) of identical size. Of course if you do that then you'll end up consuming more data over time than sending random amounts of packets.
The problem is not knowing whether someone is typing, as far as I understand. But that you may extract some information about what keys are being typed, based on the small differences in timings between them.
I seem to hit this logic often recently for some reason.
There are two issues with it:
- a primary is not a totality: if "security is the #1 consideration for SSH", that implies there's a #2, maybe even a #3 and so on consideration. So the question that follows becomes tautological: "but if the author doesn't need security, why use ssh?" -> surely for one or more of the #2, #3, etc. considerations, right?
- overabstraction (*): you ended up strawmanning the author. What they had issue with was keystroke timing obfuscation, which is a privacy feature. Timing attacks are (in part) a privacy concern, and privacy is a security concern, yes, but security is not just privacy concern, and privacy concerns are not just about timing attacks; these groups are not equal. For example, they might very well want the transmitted keypresses themselves to remain confidential, or they might very well want to retain cryptographic assurance of their integrity. These are security features they can continue to utilize by sticking with SSH.
All of this is to say, it's not even necessarily them using SSH for a hypothetical #2 or #3 (...etc...) reason, but likely because they still very much want to make use of large chunks of #1, which disabling keypress obfuscation does not actually rid SSH of, only at most weakens it in ways they clearly seem to be okay with.
(*) although if I zoom out enough, this is once again just "a primary is not a totality", just implicitly
Depends on what kind of security. They might care about connection integrity. If a faulty (or malicious) router in-between client and server starts malforming packets, `nc` will display those malformed packets. SSH will only show you what the server intended, or nothing.
> I am working on a high-performance game that runs over ssh.
Found your problem.
But it is an interesting world where you can casually burrow into a crypto library and disable important security features more easily than selecting the right network layer solution.
Yea UDP is technically more performant, but then you need a crypto layer + reliable message delivery layer + bespoke client. Using a plain old SSH client is cool.
its not really a question of 'udp performs better'. in tcp we have to live to head-of-line blocking on losses and congestion control. if you don't care about receiving every packet, but only the most recent, then udp is a good choice.
running without congestion control means that you avoid slowstart. but at a certain rate you run into poorly defined 'fairness' issues where you can easily negatively impact other flows. past that point, you can actually self-interfere and cause excessive losses for yourself.
quic uses congestion control, but uses latency estimates and variance as a signal to back off. it still imposes an ordering on a per-stream basis. so it might not be ideal either.
sctp has a mode which supports reliable and unordered, which might be something to consider
so really - if you care about latency and have a different reliability model, its worth unpacking all these considerations and using them to select your transport layer or even consider writing a minimal one yourself
The reliance on LLMs is unfortunate. I bet this mystery could gave been solved much quicker by simply looking at the packet capture in Wireshark. The Wireshark dissectors are quite mature, SSH is covered fairly well.
> I bet this mystery could gave been solved much quicker by simply looking at the packet capture in Wireshark.
For some people who are used to using Wireshark and who know what to look for, probably yes. For the vast majority of even technical people, probably not.
In my case, I did a packet capture of a single keystroke using tcpdump and imported it into Wireshark and I get just over 200 'Client: encrypted packet' and 'Server: encrypted packet' entries. Nothing useful there at all. If I tcpdump the entire SSH connection setup from scratch I get just as much useful information - nothing - but, oddly, fewer packets than my one keystroke triggered.
So yeah, I dislike LLMs entirely and dislike the reliance on LLMs that we see today, but in this case the author learned a lot of interesting stuff and shared it with us, whereas without LLMs he might have just shrugged and moved on.
And thats a huge downside when people howl about "Encryption everywhere! ".
Try debugging that shit. Thats right, debugging interfaces aren't safe, by some wellakshually security goon.
You want a real fun one to debug, is a SAML login to a webapp, with internal Oauth passthrough between multiple servers. Sure, I can decrypt client-server stuff with tools, but server-server is damn near impossible. The tools that work break SSL, and invalidate validation of the ssl.
I used to share that opinion but after decades in industrial automation I find myself coming down much more on the "yeah, encryption everywhere" because while many vendors do not provide good tools for debugging, that's really the problem, and we've been covering for them by being able to snoop the traffic.
Having to MITM a connection to snoop it is annoying, but the alternative appears to be still using unencrypted protocols from the 1970s within the limitations of a 6502 to operate life-safety equipment.
Problem is, security people don't want you to MITM connections, because it's insecure (mostly to business interests). Hence stuff like certificate pinning, HSTS, DoH...
If you're debugging your own equipment you should have the certificates or keys to make it work. I'm not saying that's easy in a lot of scenarios, in fact it's frequently tedious as hell. But for example there are debug tools for like DNP3 or RPC over TLS, etc that can watch the whole exchange if provided the keys and parse the SCADA traffic or JSON objects as if it was plaintext.
But this goes back to the vendors not providing better tools in the first place. We shouldn't NEED to be picking apart packet streams to prove to some jackass tech support ticket that their code is FUBAR. They're basically outsourcing support to their customer or userbase and we tolerated it because it was more expedient.
This really does not need to be that hard. For TLS, many tools support setting the SSLKEYLOGFILE environment variable to log the session keys used in connections. Wireshark can import those to decrypt everything. [1]
Unfortunately, nothing exists for SSH (yet?). [2]
I do agree that if you design a protocol that enforces encryption, you should include some debugging interface. It is much more straightforward to do this by logging the session secrets on the endpoints rather than trying to break it through a man-in-the-middle, the main thing the protocol is protecting you against.
Sounds like blaming a tool on a problem it did not cause. Either way, solvable and encryption is important. Badly designed systems and or lack of tooling isn't really an encryption problem.
Anyway, VMs should not have authentication, it makes access sooo much easier. Also drop your IPs while you're at it. Might be useful for debugging later.
Unfortunately with SSH specifically, the dissectors aren't very mature - you only get valid parsing up to the KeX completion messages (NEWKEYS), and after that, even if the encryption is set to `none` via custom patches, the rest of the message flow is not parsed.
Seems because dumping the session keys is not at all a common thing. It's just a matter of effort though - if someone put in the time to improve the SSH story for dissectors, most of the groundwork is there.
Interesting, I thought it was possible to decrypt SSH in Wireshark a la TLS, but it seems I'm mistaken. It still would have been my first goto, likely with encryption patched out as you stated. With well documented protocols, it's generally not too difficult deciphering the raw interior bits as needed with the orientation provided by the dissected pieces. So let me revise my statement: this probably would have been a fairly easy task with protocol analysis guided code review (or simply CR alone).
It all depends on the key exchange mechanism (KEM) used at the start of the TLS session. Some KEM have a property called “perfect forward secrecy” (PFS) which means it’s not possible to decrypt the TLS session after the fact unless one of the nodes logs out the session key(s). Diffie Helman and ECDH are two KEM that provide a PFS guarantee.
* Yes, I clearly know what tcpdump is / how to capture network traffic
* It has been several years since I have looked at a pcap
* I don't have wireshark installed on this computer
* I've done the thing where you decrypt TLS with wireshark exactly once, years ago, and I found it frustrating for reasons I can't remember[1]. Wasn't sure if I could do this with ssh
* When I started investigating this, I didn't remotely think that ssh was the root cause. I thought it was a quirk of my game
* I *did* make a client that printed out all the data it was receiving, but it was useless because it was operating at the wrong layer (e.g. it connected over SSH and logged the bytes SSH handed it)
* I'm experimenting with Claude Code a lot because it has a lot of hype and I would like to form an opinion
* Looking up flags is annoying
* Being able to tell an agent "look at this pcap and tell me what you see" is *cool*
So idk. I'm sure that you would have solved this much more quickly than I did! I'm not sure that (for me) opening up the packet in Wireshark would have solved this faster. Maybe reading the SSH spec would have, but debugging also just didn't take that long.
And the big leap here was realizing that this was my SSH client and not a quirk of my game. The time at which I would have read the SSH spec was after I captured traffic from a regular SSH session and observed the same pattern; before that I was thinking about the problem wrong.
I don't think that this is unfortunate. In fact, I think I got what I wanted here (a better sense of Claude Code's strengths and weaknesses). You're right that an alternative approach would have taught me different things, and that's a worthy goal too.
[1] I suspect this is because I was doing it for an old job and I had to figure out how to run some application with keys I controlled? It would have been easier here. I don't remember.
Thanks for taking the time to respond, and apologies for the contentiousness. I'm a jaded old man suffering from severe LLM fatigue, so I may have come off a bit harsh. Your write-up was a good read, and while I might be critical of your methodology, what you did clearly worked, and that's what matters in the end. Best of luck with your project, especially the go lib fork.
obviously OPs empirical and analytical rigor are top notch. He applied LLMs in the best way possible: fill gaps with clumsy command line flags or protocol implementations. Those aren't things one needs to keep in their head all the time.
Not even remotely accurate. While the dissector is not as mature as I thought and there's no built-in decryption as there is for TLS, that doesn't matter much. Hint: every component of the system is attacker controlled in this scenario.
>Is that because wireshark can't do that just from packet captures?
Well, not quite. I think it's more that nobody has taken the time to implement it. That's not to say such an implementation would automatically decrypt the traffic from a capture with no extra leg work, of course. Wireshark dissectors have user configurable preferences, and presumably this would be where captured secrets could be set for use. This is how it handles TLS decryption [1], which works beautifully.
My thoughts exactly. The OP used AI to get a starting point to their investigation, then used their skills to improve their game, with actual (I guess according to the article itself) proof of that, as opposed to just approving changes from the LLM.
This looks like an actual productivity boost with AI.
Well, I spent a good part of my career reverse engineering network protocols for the purpose of developing exploits against closed source software, so I'm pretty sure I could do this quickly. Not that it matters unless you're going to pay me.
What are you even trying to say? I suppose I'll clarify for you: Yes, I'm confident I could have identified the cause of the mysterious packets quickly. No, I'm not going to go through the motions because I have no particular inclination toward the work outside of banter on the internet. And what's more, it would be contrived since the answer has already shared.
I think the point they're making is that "I, a seasoned network security and red-team-type person, could have done this in Wireshark without AI assistance" is neither surprising nor interesting.
That'd be like saying "I, an emergency room doctor, do not need AI assistance to interpret an EKG"
Sure, but that is aside from my original point. If somebody:
a) Has the knowledge to run tcpdump or similar from the command line
b) Has the ambition to document and publish their effort on the internet
c) Has the ability identify and patch the target behaviors in code
I argue that, had they not run to an LLM, they likely would have solved this problem more efficiently, and would have learned more along the way. Forgive me for being so critical, but the LLM use here simply comes off as lazy. And not lazy in a good efficiency amplifying way, but lazy in a sloppy way. Ultimately this person achieved their goal, but this is a pattern I am seeing on a daily basis at this point, and I worry that heavy LLM users will see their skill sets stagnate and likely atrophy.
>I argue that, had they not run to an LLM, they likely would have solved this problem more efficiently
Hard disagree. Asking an LLM is 1000% more efficient than reading docs, lots of which are poorly written and thus dense and time-consuming to wade through.
The problem is hallucinations. It's incredibly frustrating to have an LLM describe an API or piece of functionality that fulfills all requirements perfectly, only to find it was a hallucination. They are impressive sometimes though. Recently I had an issue with a regression in some of our test capabilities after a pivot to Microsoft Orleans. After trying everything I could think of, I asked Sonnet 4.5, and it came up with a solution to a problem I could not even find described on the internet, let alone solved. That was quite impressive, but I almost gave up on it because it hallucinated wildly before and after the workable solution.
The same stuff happens when summarizing documentation. In that regard, I would say that, at best, modern LLMs are only good for finding an entrypoint into the docs.
While my reply was snarky I am prepared to take a reasonable bet with a reasonable test case. And pay out.
Why I think I’d win the bet is I’m proficient with tcpdump and wireshark and I’m reasonably confident that running to a frontier model and dealing with any hallucinations is more efficient and faster than recalling the incantantions and parsing the output myself.
Oh come on, the fact that the author was able to pull this off is surely indicative of some expertise. If the story started had started off with, "I asked the LLM how to capture network traffic," then yeah, what I said would not be applicable. But that's not how this was presented. tcpdump was used, profiling tools were mentioned, etc. It is not a stretch to expect somebody who develops networked applications knows a thing or two about protocol analysis.
The specific point I was trying to make was along the lines of, "I, a seasoned network security and red-team-type person, could have done this in Wireshark without AI assistance. And yet, I’d probably lose a bet on a race against someone like me using an LLM."
I'm still waiting for a systems engineering tool that can log every layer, and handle SSL the whole pipe wide.
Im covering everything from strafe and ltrace on the machine, file reads, IO profiling, bandwidth profiling. Like, the whole thing, from beginning to end.
Real talk though, how much would such a tool be worth to you? Would you pay, say, $3,000/license/year for it? Or, after someone puts in the work to develop it, would you wait for someone else to duct tape something together approximately similar enough using regexps that open source but 10% as good, and then not pay for the good proprietary tool because we're all a bunch of cheap bastards?
We have only ourselves to blame that there aren't better tools (publicly) available. If I hypothetically (really!) had such a tool, it would be an advantage over every other SRE out there that could use it. Trying to sell it directly comes with more headaches than money, selling it to corporations has different headaches, open-sourcing it don't pay the bills, nevermind the burnout (people don't donate for shit). So the way to do it is make a pitch deck, get VC funding so you're able to pay rent until it gets acquired by Oracle/RedHat/IBM (aka the greatest hits for Linux tool acquisition), or try and charge money for it when you run out of VC funding, leading to accusations of "rug pull" and development of alternatives (see also: docker) just to spite you.
In the base case you sell Hashimoto and your bank account has two (three!) commas, but worst case you don't make rent and go homeless when instead you could've gone to a FAANG and made $250k/yr instead of getting paid $50k/yr as the founder and burning VC cash and eating ramen that you have to make yourself.
I agree, that would be an awesome tool! Best case scenario, a company pays for that tool to be developed internally, the company goes under, it gets sold as an asset and whomever buys it forms a compnay and tries to sell it directly and then that company goes under but that whomever finally open sources it because they don't want it to slip into obscurity but if falls into obscurity anyway because it only works on Linux 5.x kernels and can't be ported to the 6.x series that we're on now easily.
> That 20ms is a smoking gun - it lines up perfectly with the mysterious pattern we saw earlier!
Speaking of smoking guns, anybody else reckon Claude overuses that term a lot? Seems anytime I give it some debugging question, it'll claim some random thing like a version number or whatever, is a "smoking gun"
Yes! While this post was written entirely by me, I wouldn't be surprised if I had "smoking gun" ready to go because I spent so much time debugging with Claude last night.
Serious question though, since AI seems to be so all capable and intelligent. Why wouldn't it be able to tell you the exact reason that I could tell you just by reading the title of this post on HN? It is failing even at the one thing it could probably do decently, is being a search engine.
I've had gemini tell me "We are debugging this problem here in İstanbul" and talking about an istanbul evening, trying to give uplifting or familiar vibes while being creepy.
I think there was a setting about time and location which finally got rid of that behavior.
It's not just a coincidence, it's the emergence of spurious statistical correlations when observations happen across sessions rather than within sessions.
Or the "Eureka! That's not just a smoking gun, it's a classic case of LLMspeak."
Grok, ChatGPT, and Claude all have these tics, and even the pro versions will use their signature phrases multiple times in an answer. I have to wonder if it's deliberate, to make detecting AI easier?
Without knowing how LLM's personality tuning works, I'd just hazard a guess that the excitability (tendency to use excided phrases) is turned up. "smoking gun" must be highly rated as a term of excitability. This should apply to other phrases like "outstanding!" or "good find!" "You're right!" etc.
smoking gun, you're absolutely right, good question, em dash, "it isn't just foo, it's also bar", real honest truth, brutal truth, underscores the issue, delves into, more em dashes, <20 different hr/corporate/cringe phrases>.
Maybe we need a real AI which creates new phrases and teaches the poor LLMs?
Looking back we already had similar problems, when we had to ask our colleagues, students, whomever "Did you get your proposed solution from the answers part or the questions part of a stackoverflow article?" :-0
That's the point though, it doesn't reflect human usage of the word. If
delve were so commonly used by humans too, we wouldn't be discussing
how it's overused by LLMs.
You might see certain phrases and mdashes ;-) rather often, because … these programs are trained on data written by people (or Microsoft's spelling correction) which overused them in the last n years? So what should these poor LLMs generate instead?
They love clichés, and hate repeating the same words for something (repetition penalty) so they'll say something like "cause" then it's a "smoking gun" then it's something else
What is the usecase for using ssh at all where you don't need to be resistant against timing analysis? Either it's not sensitive and you can use telnet (if necessary after using ssh to authenticate) or the game (or other stuff on the connection) might be sensitive and you need traffic analysis resistance.
If you get clever and write a client to ensure sensitive data like passwords or email are sent in a burst you could just use an encryption library just for that data instead.
Dont let this article be blinders to you. Ssh does much more than obfuscate keypress timings. Not needing the chaff means turn it off and keep all the other benefits. It doesn't mean "revert to telnet"
Or you could use anycasting to terminate SSH sessions on the moral equivalent of one of a number of geography based reverse proxies and then forward the packet over an internal network to the app server over a link tuned for low latency. The big guys already do something similar with HTTP over TLS for DDoS protection and to limit end to end latency on TLS.
Granted... it would increase the cost (since you're adding reverse proxies) but it would be a quick way to get acceptable latency, rudimentary DDoS protection, and you could try different connection options independent of the main app's logic.
It would be hard to estimate how much latency you're adding with a SSH2 reverse proxy in this case, but it's probably lower than one might think.
The idea of letting Claude loose on my crypto[graphy] implementation is about the most frightening thing I've heard of in a while [though libnss is so craptastic, I can't see how it would hurt in that case.] But I loved this write-up. It was readable and explained the problem the OP was encountering and proposed solutions well.
> Or you could use anycasting to terminate SSH sessions on the moral equivalent of one of a number of geography based reverse proxies and then forward the packet over an internal network to the app server over a link tuned for low latency.
I've been thinking about some stuff like this! Not being able to put my game behind Cloudflare[1] is a bummer. Substantial architectural overhead though.
> The idea of letting Claude loose on my crypto[graphy] implementation is about the most frightening thing I've heard of in a while [though libnss is so craptastic, I can't see how it would hurt in that case.]
I hear you, but FWIW the patch I was reverting was trivial (and it's also in the go crypto library, which is pretty easy to read). It's a couple-of-line change[2], and Claude did almost exactly what I would have done (I was tired and would have forgotten to shrink the handshake payload).
[1] This isn't strictly true, Cloudflare spectrum exists, but its pricing is an insane $1/GB last I checked.
Nice, but shouldn't the behaviour change be behind a config setting? And it's not clear what the intent of the change is. Implementing PING/PONG seems different from what you said you were trying to do. And it's section 1.8 of the OpenSSH [PROTOCOL] reference, not section 1.9.
But... before you think I'm trying to be negative... good on you. I wish you well. Getting crypto/security code into open source projects can be a slog as people frequently come out of the woodwork, so don't get discouraged.
And the more I think about this... there's plenty of examples out there about doing HTTP based reverse proxying, but essentially zero for SSH proxying, so if you do that, it would make a great blog post.
> I am working on a high-performance game that runs over ssh. The TUI for the game is created in bubbletea 1 and sent over ssh via wish.
> The game is played in an 80x60 window that I update 10 times a second. I’m targeting at least 2,000 concurrent players, which means updating ~100 million cells a second. I care about performance.
High performance with ssh and wish? For sure not. Rather use UDP over secure sockets. Or just normal sockets. Even Claude would come up with much faster code than the ssh/wish nonsense. Or mosh, but this also too complicated.
I didn't think about such throwback to the 80ies. Could be, yes. But then he cannot control the ssh option, and with 2000 users, maybe 10 would set it. I don't think so.
The really mysterious part is how ~10,000 packets per second costs ~20% of a core. That would mean SSH is bottlenecking in its code at ~50,000 packets per second per core which would be ~500 Mbps per core (assuming full packets) which is ludicrously slow. It is trivial to do 10x that packet per second rate. Is SSH really that poorly designed?
I do not know where people get the idea that encryption is that slow. Standard AES hardware acceleration instructions do ~25 Gbps per core (on a 2023 CPU) which is ~50x that rate [1]. I have heard modern cores can do ~40-50 Gbps, but I have not been able to find any independent benchmarks of that. Even the Intel i5-2500, a CPU from 2011, averages ~10 Gbps which is ~20x that rate. Even unaccelerated encryption can do ~2-5 Gbps in pure software which is 4-10x the SSH rate.
And in this situation, the amount of encrypted payload in each packet is 36 bytes which is ~40x less than a full packet of ~1500 bytes. You would almost surely hit packet per second limits before you hit payload throughput limits at these small sizes.
Encryption is slow when compared to data throughput you can get with a properly designed transport stack, but that is because it is in comparison to 100 Gbps per core even with no hardware offload. Anything less than ~10 Gbps/1 million packets per second (ignoring other bottlenecks, so only the software transport is the limit) is not merely unoptimized, it is pessimized.
> I am working on a high-performance game that runs over ssh.
Step one, run https://www.psc.edu/hpn-ssh-home/introduction/ instead
Step two, tune TCP/IP stack
Step... much later: write your own "crypto". (I'm using quotes because, before someone points out the obvious, packets-per-keystroke isn't, itself, a cryptographic algorithm, but because it's being done to protect connections from being decrypted/etc, mess with it at your own peril.)
Funny that this comes up today! I was just looking into adding a keyboard monitor to my website (I have a goal of making my 'contact me' page have oddly specific information). I wouldn't show the actual keys, just show a blinking light when there's activity, but I guess the timing really could expose quite a lot of information.
I did add a trackpad monitor though. It shows my raw MacBook trackpad data.
>In 2023, ssh added keystroke timing obfuscation. The idea is that the speed at which you type different letters betrays some information about which letters you’re typing. So ssh sends lots of “chaff” packets along with your keystrokes to make it hard for an attacker to determine when you’re actually entering keys.
Why not just add random "jitter" to the keystroke packets, but keeping just the 1 actual packet?
How? You can't average out the noise here because the attack involves discriminating the different types of events from one another based on the thing you'd be averaging.
One clue is that you cannot predict what key user is going to press next reliably, so the jitter would always be added to actual key press. You can minimise that by adding constant latency, so that you could simulate pulling events back in time, but still this is going to get complex quick and still could be filtered out. As for methods, it depends on the jitter. Think of things like noise removal in audio and adaptive filtering.
Adding extra packets is much easier and more secure.
Okay I think I see the issue (and slight misunderstanding). I believe the problem is actually latency. I was assuming the jitter interval would be noticeably larger than the gap between typical (say 95%) of key presses. Any smaller than that and you start to need cover traffic.
Such an interval would still face correlation issues due to the varying nature of the overlap between the jitter intervals, however it seems like that should be trivial to address. That said, just throwing in some cover traffic is bound to be simpler.
But a jitter interval long enough that keystroke packets can change order is going to be noticeable to a human typing quickly on what should be a solid connection - my WiFi is only at 3 to 6 ms RTT and I already notice that versus a wired connection. That doesn't sound so trivial to fix, and once again just throwing in some cover traffic completely solves the issue.
So just do what's simple.
My next question was going to be, why on the order of 100 extra packets instead of just 1 or 2? But of course an attacker could attempt to search some set of permutations for recognizable words. So either you drown everything out (simple) or you hook a multilingual dictionary up to a key stroke delay model for your cover traffic generator (complex).
But really shouldn't this feature be implemented as some constant (low) background level of cover traffic that scales up as your typing frequency increases but caps out at some (still fairly low) rate? That seems both less likely to suffer from inadvertent leaks as well as not running afoul of the issue in the article.
Not related to SSH, but does the eieio.games website make anyone else's monitor flicker? When the website is fullscreen it overwhelms something. I thought my monitor's backlight was going.
I wonder if this is the same reason why Microsoft's Remote SSH plugin on VS Code is so flaky even with a decent internet connection. Every couple of months I try to give it another go and give up due to the poor keyboard latency I inevitably experience. And the slow reconnects whenever I glance away from my computer monitor briefly. This is on a fiber connection with a 20ms ping to the remote machine.
You surely mean the latency in its embedded terminal and not the code editor, right? I use VSCode’s remote SSH specifically so that code editing doesn’t suck. It really does not.
>I am working on a high-performance game that runs over ssh.
By 'ssh', you mean 'ssh' (library/program + protocol + encryption + decryption) on top of TCP/IP, on top of the Internet, right?
OK, I'm not against it... but you do understand that there are all kinds of ways for that to slow things down, right?
Your issues may (or may not!) include such things as:
o Nagle's algorithm AKA buffering AKA packets not being sent until N bytes (where N > 1) ready to send, as other posters have suggested;
o Slower encryption/decryption on older hardware (if users with older hardware is a target market, and if the added loss in speed makes an impact in gameplay, depending on the game, this may or may not be the case...)
o The fact that TCP/IP (as opposed to UDP / Datagrams / "Raw" sockets) imposes a connection-oriented abstraction, requiring additional round trips of ACK ("I got the packet") RESEND ("I didn't get the packet") on top of the connectionless architecture that is the Internet (https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...), which adds additonal latency, so, for example if a rural user in Australia experiences a 350ms delay for a raw packet to get to a U.S. server (or vice versa), then TCP/IP might make this 700ms or more, depending on the quality of the connection!
o The speed of the game limited to both the bandwidth and latency of the slowest user (if a multi-player game, and if the game must not update until that user "moves"... again, game architecture will determine this, and it wouldn't be applicable to all games...)
Now, you could use UDP, as other posters have suggested, but then you must manually manage connections and encryption...
That may be the right choice for some types programmers, some types of games/applications -- but equal-and-oppositely it may be the wrong choice for others...
Anyway, wishing you well with your game development!
I haven't used SSH (at least, not in a debug capacity), so I'm not sure what SSH debugging options exist -- but it would be nice if SSH had a full logging debug mode, which would explain exactly WHY it chose to send any given packet that it did along with related helpful information, such as latency/time/other metrics, etc., if it doesn't have this/these feature(s) already...
And of course it totally doesn't work if the client doesn't have JavaScript at all. I read the HN front-page through an AI summary and it also got censored when it scraped the article.
>>> That makes a lot of sense for regular ssh sessions, where privacy is critical. But it’s a lot of overhead for an open-to-the-whole-internet game where latency is critical.
Switching to telnet instead of SSH might be an option.
Why not "amortise" the period of sending keystrokes -- buffer them in a queue, and process the queue for sending these on a regular (and short enough for the human at the client end feeling the interactivity) interval, so there's no latency difference between sending an 'a' vs a 'q' and so on. If we assume some average typing speed on the bell curve, say, around 250 keystrokes per minute, the queue can be picked for sending every 250 milliseconds or so. That solution wouldn't require injecting extra packets on the network. What am I missing?
Am I missing something? This isn't what ssh's purpose is. Why should anyone care? We're talking about a game built to run over an encryption protocol? What are we even doing anymore? Also, correct me if I'm wrong, but client-side option existing is secure design, really feels like it shouldn't be circumvented server-side without giving the client the choice to do so or not by default. Don't lobby for watering down security for convenience, especially for trivially important objectives, please?
Telnet nowadays typically isn’t available by default for security reasons, and OP wants people to be able to play the game just by typing “ssh thegamehost”.
> Keystroke obfuscation can be disabled client-side.
please never do that (in production)
if anyone half way serious tries they _will_ be able to break you encryption end find what you typed
this isn't a hypothetical niche case obfuscation mechanism, it's a people broke SSH then a fix was found case. I don't even know why you can disable it tbh.
That doesn't sound right to me. This obfuscation isn't about a side-channel on a crypto implementation, this is about literally when your keystrokes happen. In the right circumstances, keystroke timing can reduce the search space for bruteforcing a password [1] but it's overstating to describe that as broken encryption.
I'm baffled about this "security feature". Besides from this only being relevant to timing keystrokes during the SSH session, not while typing the SSH password, I really don't understand how can someone eavesdrop on this? They'd have to have access to the client or server shell (root?) in order to be able to get the keystrokes typing speed. I've also never heard of keystroke typing speed hacking/guessing keystrokes. The odds are very low IMO to get that right.
I'd be much more scared of someone literally watching me type on my computer, where you can see/record the keys being pressed.
Anyone who can spy on the network between the client and server can see the timing. This includes basically anyone on the same LAN as you, anyone who sets up a WiFi access point with a SSID you auto-connect to, anyone at your ISP or VPN provider, the NSA and god knows who else.
And the timing is still sensitive. [1] does suggest that it can be used to significantly narrow the possible passwords you have, which could lead to a compromise. Not only that, but timing can be sensitive in other ways --- it can lead to de-anonymization by correlating with other events, it can lead to profiling of what kind of activity you are doing over ssh.
So this does solve a potentially sensitive issue, it's just nuanced and not a complete security break.
It is to prevent timing attacks but there are many ssh use cases where it is 100% computer to computer communications where there is no key based timing attack possible.
- you are listening to an SSH session between devices
- and you know what protocol is being talked over the connection (i.e. what they are talking about)
- and the protocol is reasonably predictable
then you gain enough information about the plaintext to start extracting information about the cipher and keys.
It's a non-trivial attack by all means but it's totally feasible. Especially if there's some amount of observable state about the participants being leaked by a third party source (i.e. other services hosted by the participants involved in the same protocol).
this only works for manually typed text, not computer to computer communication where you can't deduce much from what is being "typed" as it's not typed but produced by a program to which every letter is the same and there is no different delay in sending some letters (as people have when typing by hand)
I'd love to hear more about this kind of attack being exploited in the wild. I understand it's theoretically possible, but...good luck! :)
You're guessing a cipher key by guessing typed characters with the only information being number of packets sent and the time they were sent at. Good luck. :)
I agree it is more nuanced than a simple 'good for computer-to-computer' and 'bad for person-to-computer'. I'm sure there are cases where both are wrong but I don't think that necessarily changes that it makes a reasonable baseline heuristic.
I haven't given this more than 5 seconds of thought, but wouldn't it make sense to only enable the timing attack prevention for pseudo-terminal sessions (-t)?
The fix seems kind of crazy though, adding so much traffic overhead to every ssh session. I assume there's a reason they didn't go that route, but on a first pass seems weird they didn't just buffer password strokes to be sent in one packet, or just add some artificial timing jitter to each keystroke.
I'm just guessing but this chaff sounds like it wouldn't actually change the latency or delivery of your actual keystrokes while buffering or jitter would.
So the "real" keystrokes are 100% the same but the fake ones which are never seen except as network packets are what is randomized.
Hey, if ECHELON snuck a listener into my house, where six devices hang out on a local router... Good for them, they're welcome to my TODO lists and vast collection of public-domain 1950s informational videos.
(I wouldn't recommend switching the option off for anything that could transit the Internet or be on a LAN with untrusted devices. I am one of those old sods who doesn't believe in the max-paranoia setting for things like "my own house," especially since if I dial that knob all the way up the point is moot; they've already compromised every individual device at the max-knob setting, so a timing attack on my SSH packet speed is a waste of effort).
One thing you notice if you have ADSL is that some services are built as if slower connections matter and others are not. Like Google's voice and audio chat services work poorly but most of the others work well. Uploading images to Mastodon, Bluesky, Facebook, LinkedIn, Instagram and Nextdoor is reliable, but for Tumblr you have to try it twice. I don't what they are doing wrong but they are doing something wrong and not finding out what they're doing wrong because they're not testing and they're not listening to users.
Nobody consulted me about their decision not to run fiber by my house. If some committee decides to make ssh bloated they are, together with the others, conspiring to steal my livelihood and I think it would be fair for me to sue them for the $50k it would take to run that fiber myself.
It's OK if you work for Google where there is limitless dark fiber but what about people in African countries?
It's the typical corporate attitude where latency never matters: Adobe thinks it is totally normal that it takes 1-5s for a keystroke to appear when you are typing into Dreamweaver.
I agree with your general point that most companies/projects do a terrible job optimizing for slow computers/networks, but OpenSSH is from the OpenBSD people, who are well-known for supporting ancient hardware [0]. Picking a random architecture, they fully support a system with only 64MB of memory [1], and the base install includes SSH. So I suspect that OpenSSH is fairly well tested on crappy computers/networks.
There's a good chance you have other options. Regardless of how you feel about the company's head, Starlink would probably be one of them, with likely better performance than you're dealing with on ADSL.
But you cannot just sue a company because their network connected software doesn't work well on slow networks. Let alone a project like OpenSSH. It would be like me suing a game studio because my PC doesn't meet their listed minimum requirements to play the game.
Hey, it is one thing to buy a new computer, it is another thing to ask people to move.
A better analogy is a bank redlining neighborhoods. The cost to run fiber to difficult rural locations pays itself easily if you look at a 25-year time span and is an order of magnitude less than building a new housing unit on the West Coast.
One half of my comment was actually about how you most likely have a better performing alternative option right where you already live. And even if you didn't, they're not asking you to move. You could argue they're not even asking you to use their software, you're electing to.
It's OSS with no warranty. You can compile it yourself with the option disabled. It's only ever on for pty connections (physical user with a keyboard), there's no added traffic for ttys.
You just opened a huge nostalgia portal, never thought that Dreamweaver would still be around, I used that somewhere around 2003 I believe. Good memories
Frankly I wish there was an HTML editor that delivers on what it promised. I mean, markdown is almost as rife with edge cases as YAML and somehow the link syntax still eludes me. If we could “just” template by merging at the DOM level and had decent HTML editors the world would be a different place. But yeah, Adobe probably thinks Dreamweaver isn’t worth maintaining just as they seem to think Photoshop is barely worth maintaining (they keep adding AI features that sorta work but the foundations seem to be much worse than Illustrator)
"The smoking gun!" got me laughing, i am not a native english speaker and only ever seen that expression from Claude, and who knew? Its gaining popularity!
SSH now deliberately sends many dummy (“chaff”) packets per keystroke to obscure typing patterns and defeat keystroke-timing attacks. This privacy feature makes a single keypress look like heavy network activity, explaining why it can generate around 100 packets.
The 2023 timing obfuscation is a nice case study in security defaults vs edge cases. Most SSH users won't notice 100 packets per keystroke - it's noise in the bandwidth budget. But for high-frequency terminal apps, it becomes the dominant cost. At 2000 concurrent players updating 80x60 chars at 10fps, a custom protocol might be the right answer regardless of obfuscation settings.
Each of our devices spents a lot of energy dedicated to encryption. By now, all disks you did not set up manually are most likely encrypted and hardly any unencrypted package will travel out of your network. That's not to mention the tons of load and dedicated hardware we have just to terminate https and scan traffic for suspicious activity or the hardware being replaced because it's internal security triggered/broke.
In a perfect world, we could send all traffic completely unencrypted and never scan for a malicious payload, saving all that energy and hardware. But we do not live in that world and drawing the line with this minor, mostly unintrusive security feature seems strange.
That's the judgement made with all consumption of energy. The benefits weighed against the costs.
Because of the harms of environmental change, there should be pressure placed to avoid damaging ways to generate that energy.
When people complain about the amount of energy being used, they are making the judgement on the benefits. This is subjective and people do not agree on the benefits. The argument you shouldn't do this because of the energy consumed is implicitly saying "My judgement on the worth of this supercedes yours"
Pretty soon it devolves into criticizing the energy use of things you just don't like.
A society has to accept that people have different opinions on things. That includes what it is worth using energy for.
Producing clean energy is something everyone should be able to get behind. There is a solid consensus that it would make a better world.
Producing cheap energy is something everyone should be able to get behind. There is a solid consensus that it would make a better world.
See what I did there? As long as you preach any ideology of yours without talking about its trade-offs you can claim everyone should get behind it. Obviously.
> In a perfect world, we could send all traffic completely unencrypted and never scan for a malicious payload, saving all that energy and hardware.
In a world with such social cohesion, we'd be defeated by an alien species being able to quickly interpret and exploit our technology like in the hit film Independence Day(note, we're the defeating alien species in this example). https://www.youtube.com/watch?v=9DIjBGierkA
Airline isn't about power consumption but rather reliability. You don't introduce failure modes to safety critical systems unless absolutely necessary.
Meanwhile the power consumption of a few extra packets or even AES on your block storage device is approximately nothing relative to the other things the device is doing. Unless the CPU or GPU is going full tilt the screen on a mobile device consumes more power than the rest of the system combined (not sure about a laptop but it's likely a similar story).
It's a bit like worrying about saving a single glass of drinking water, then turning around and hopping in the shower for an hour. Or not flushing the toilet then immediately drawing a bath.
> Obviously forking go’s crypto library is a little scary, and I’m gonna have to do some thinking about how to maintain my little patch in a safe way
This should really be upstreamed as an option on the ssh library. Its good to default to sending chaff in untrusted environments, but there are plenty of places where we might as well save the bandwidth
"where we might as well save the bandwidth"
I come from a world (yesteryear) where a computer had 1KB of RAM (ZX80). I've used links with modems rocking 1200 bps (1200 bits per second). I recall US Robotics modems getting to speeds of 56K - well that was mostly a fib worse than MS doing QA these days. Ooh I could chat with some bloke from Novell on Compuserve.
In 1994ish I was asked to look into this fancy new world wide web thing on the internet. I was working at a UK military college as an IT bod, I was 24. I had a Windows 3.1 PC. I telnetted into a local VAX, then onto the X25 PAD. I used JANET to get to somewhere in the US (NIST) and from there to Switzerland to where this www thing started off. I was using telnet and WAIS and Gopher and then I was apparently using something called "www".
I described this www thing as "a bit wank", which shows what a visionary I am!
Fellow old here, I had several 56k baud modems but even my USR (the best of the bunch) never got more than half way to 56k throughput. Took forever to download shit over BBS...
The real analog copper lines were kind of limited to approx 28K - more or less the nyquist limit. However, the lines at the time were increasingly replaced with digital 64Kbit lines that sampled the analog tone. So, the 56k standard aligned itself to the actual sample times, and that allowed it to reach a 56k bps rate (some time/error tolerance still eats away at your bandwidth)
If you never got more than 24-28k, you likely still had an analog line.
56k was also unidirectional, you had to have special hardware on the other side to send at 56k downstream. The upstream was 33.6kbps I think, and that was in ideal conditions.
The special hardware was actually just a DSP at the ISP end. The big difference was before 56k modems, we had multiple analog lines coming into the ISP. We had to upgrade to digital service (DS1 or ISDN PRI) and break out the 64k digital channels to separate DSPs.
The economical way to do that was integrated RAS systems like the Livingston Portmaster, Cisco 5x00 seriers, or Ascend Max. Those would take the aggregated digital line, break out the channels, hold multiple DSPs on multiple boards, and have an Ethernet (or sometimes another DS1 or DS3 for more direct uplink) with all those parts communicating inside the same chassis. In theory, though, you could break out the line in one piece of hardware and then have a bunch of firmware modems.
The asymmetry of 56k standards was 2:1, so if you got a 56k6 link (the best you could get in theory IIRC) your upload rate would be ~28k3. In my expereience the best you would get in real world use was ~48k (so 48kbpd down, 24kbps up), and 42k (so 21k up) was the most I could guarantee would be stable (baring in mind “unstable” meant the link might completely drop randomly, not that there would be a blip here-or-there and all would be well again PDQ afterwards) for a significant length of time.
To get 33k6 up (or even just 28k8 - some ISPs had banks of modems that supported one the 56k6 standards but would not support more than 28k8 symmetric) you needed to force your modem to connect using the older symmetric standards.
Yeah 28k sounds more closer to what I got when things were going well. I also forget if they were tracking in lower case 'k' (x1000) or upper case 'K' (x1024) units/s which obviously has an effect as well.
The lower case "k" vs upper case "K" is an abomination. The official notation is lower case "k" for 1000 and lower case "ki" for 1024. It's an abomination too, but it's the correct abomination.
That's a newer representation, mostly because storage companies always (mis)represented their storage... I don't think any ISPs really misrepresent k/K in kilo bits/bytes
Line speed is always base 10. I think everything except RAM (memory, caches etc.) is base 10 really.
* 56k baud modems but even my USR (the best of the bunch) never got more than half way to 56k throughput*
56k modem standards were asymmetric, the upload rate being half that of the download. In my experience (UK based, calling UK ISPs) 42kbps was usually what I saw, though 46 or even 48k was stable¹ for a while sometimes.
But 42k down was 21k up, so if I was planning to upload anything much I'd set my modem to pretend it as a 36k6 unit: that was more stable and up to that speed things were symmetric (so I got 36k6 up as well as down, better than 24k/23k/21k). I could reliably get a 36k6 link, and it would generally stay up as long as I needed it to.
--------
[1] sometimes a 48k link would last many minutes then die randomly, forcing my modem to hold back to 42k resulted in much more stable connections
Even then, it required specialized hardware on the ISP side to connect above 33.6kbps at all, and almost never reliably so. I remember telling most of my friends just to get/stick with the 33.6k options. Especially considering the overhead a lot of those higher modems took, most of which were "winmodems" that used a fair amount of CPU overhead insstead of an actual COM/Serial port. It was kind of wild.
Yep. Though I found 42k reliable and a useful boost over 36k6 (14%) if I was planning on downloading something big¹. If you had a 56k capable modem and had a less than ideal line, it was important to force it to 36k6 because failure to connect using the enhanced protocol would usually result in fallback all the way to 28k8 (assuming, of course, that your line wasn't too noisy for even 36k6 to be stable).
I always avoided WinModems, in part because I used Linux a lot, and recommended friends/family do the same. “but it was cheaper!” was a regular refrain when one didn't work well, and I pulled out the good ol' “I told you so”.
--------
[1] Big by the standards of the day, not today!
> several 56k baud modems
These were almost definitely 8k baud.
In case anyone else is curious, since this is something I was always confused about until I looked it up just now:
"Baud rate" refers to the symbol rate, that is the number of pulses of the analog signal per second. A signal that has two voltage states can convey two bits of information per symbol.
"Bit rate" refers to the amount of digital data conveyed. If there are two states per symbol, then the baud rate and bit rate are equivalent. 56K modems used 7 bits per symbol, so the bit rate was 7x the baud rate.
Not sure about your last point but in serial comms there are start and stop bits and sometimes parity. We generally used 8 data bits with no parity so in effect there are 10 bits per character including the stop and start bits. That pretty much matched up with file transfer speeds achieved using one of the good protocols that used sliding windows to remove latency. To calculate expected speed just divide baud by 10 to covert from bits per second to characters per second then there is a little efficiency loss due to protocol overhead. This is direct without modems once you introduce those the speed could be variable.
Yes, except that in modern infra i.e. WiFi 6 is 1024-QAM, which is to say there are 1024 states per symbol, so you can transfer up to 10bits per symbol.
Yes, because at that time, a modem didn't actually talk to a modem over a switched analog line. Instead, line cards digitized the analog phone signal, the digital stream was then routed through the telecom network, and the converted back to analog. So the analog path was actually two short segments. The line cards digitized at 8kHz (enough for 4kHz analog bandwidth), using a logarithmic mapping (u-law? a-law?), and they managed to get 7 bits reliably through the two conversions.
ISDN essentially moved that line card into the consumer's phone. So ISDN "modems" talked directly digital, and got to 64kbit/s.
An ISDN BRI (basic copper) actually had 2 64kbps b channels, for pots dialup as an ISP you typically had a PRI with 23 b, and 1 d channel.
56k only allowed one ad/da from provider to customer.
When I was troubleshooting clients, the problem was almost always on the customer side of the demarc with old two line or insane star junctions being the primary source.
You didn’t even get 33k on analog switches, but at least US West and GTE had isdn capable switches backed by at least DS# by the time the commercial internet took off. Lata tariffs in the US killed BRIs for the most part.
T1 CAS was still around but in channel CID etc… didn’t really work for their needs.
33.6k still depended on DS# backhaul, but you could be pots on both sides, 56k depended on only one analog conversion.
56k relied on the TX modem to be digitally wired to the DAC that fed the analog segment of the line.
Confusing baud and bit rates is consistent with actually being there, though.
As someone that started with 300/300 and went via 1200/75 to 9600 etc - I don't believe conflating signalling changes with bps is an indication of physical or temporal proximity.
I think it was a joke implying you'd be old enough to forget because of age, which in my case is definitely true...
Oh, I got the implication, but I think it was such a common mistake back then, that I don't think it's age-related now - it's a bit of a trope, to assume baud and bps mean the same thing, and people tend to prefer to use a more technical term even when it's not fully understood. Hence we are where we are with terms like decimate, myriad, nubile, detox etc, forcefully redefined by common (mis)usage. I need a cup of tea, clearly.
Anyway, I didn't think my throw-away comment would engender such a large response. I guess we're not the only olds around here!
No, just that confusing the two was ubiquitous at the time 14.4k, 28k, and 56k modems were the standard.
Like it was more common than confusing Kbps and KBps.
I mean, the 3.5" floppy disk could store 1.44 MB... and by that people meant the capacity was 1,474,560 bytes = 1.44 * 1024 * 1000. Accuracy and consistency in terminology has never been particularly important to marketing and advertising, except marketing and advertising is exactly where most laypersons first learn technical terms.
I started out with a 2400 baud US Robotics modem with my "ISP" being my local university to surf gopher and BBS. When both baud rates and bits per second were being marketed side by side I kinda lost the thread tbh. Using different bases for storage vs transmission rates didn't help.
Yeah I got baud and bit rates confused. I also don't recall any hayes commands anymore either...
> I've used links with modems rocking 1200 bps
Yo, 300 baud, checking in.
Do I hear 110?
+++ATH0
Ah, the good old days. I remember dialing up local BBSes with QMODEM.
AT&C1&D2S36=7DT*70,,,5551212
PoiZoN BBS Sysop chiming in. I ran the BBS on a free phone line I found in my childhood bedroom. I alerted the phone company and a tech spent a day trying to untangle it, but gave up at the end of his shift. He even stopped by to tell me it wouldn’t be fixed.
I didn’t know the phone number, so I bought a Caller ID box, hooked it to my home line, and phoned home. It wasn’t long before every BBS in town had a listing for it.
That's awesome.
I had to wait til I was old enough to get a phone line in my own name before running a BBS. And also til I had a modem that would auto-answer, which was not a given back then!
But I confess my first question for a working but unassigned phone line would be: who gets the bill for long distance calls?
I had access to no-cost long distance calling through other administrative oversights, but they were a bit more effort to maintain! :)
Man that tech was cool and did you a solid.
Many techs went to work for the phone companies for a reason.
My first modem (from 1987) was 300 baud, but it could be used in a split mode called 75/1200.
Before that I used 50 baud systems in the military as well as civil telex systems.
Mine was 300 baud, probably 1982?
And I felt privileged because the configuration for my TI-99/4A Terminal Emulator (which I believe was called Terminal Emulator) had options for 110 or 300 baud, and I felt lucky to be able to use the "fast" one. :)
My first modem (you always remember your first) had no carrier detection (and no Hayes commands, and no speaker...), so I would dial the number manually, then flip a switch when I heard the remote end pick up and send carrier to get the synchronization started.
It was incredibly exciting at the time.
Ha, same! On a TRS-80 Color, nonetheless. But I think I used four times, because no one else in the country had a BBS at the time (small city in Latin America).
It took a couple of years until it would catch on, and by then 1200 and 2400 bps were already the norm - thankfully!
Same year, I tried this cool new "Mosaic" software and thought it was a cool proof of concept, but there was no way this web thing could ever displace gopher
Which was right, today gopher has more users than ever! :)
It sort of already is. This behavior is only applied to sessions with a TTY and then the client can disable it, which is a sensible default. This specific use case is tripping it up obviously since the server knows ahead of time that the connection is not important enough to obfuscate and this isn't a typical terminal session, but in almost any other scenario there is no way to make that determination and the client expects its ObscureKeystrokeTiming to be honored.
What's a concrete threat model here? If you're sending data to an ssh server, you already need to trust that it's handling your input responsibly. What's the scenario where it's fine that the client doesn't know if the server is using pastebin for backing up session dumps, but it's problematic that the server tells the client that it's not accepting a certain timing obfuscation technique?
The behavior exists to prevent a 3rd party from inferring keystrokes from active terminal sessions, which is surprisingly easy, particularly with knowledge about the user's typing speed, keyboard type, etc. The old CIA TEMPEST stuff used to make good guesses at keystrokes from the timing of AC power circuit draws for typewriters and real terminals. Someone with a laser and a nearby window can measure the vibrations in the glass from the sound of a keyboard. The problem is real and has been an OPSEC sort of consideration for a long time.
The client and server themselves obviously know the contents of the communications anyway, but the client option (and default behavior) expects this protection against someone that can capture network traffic in between. If there was some server side option they'd probably also want to include some sort of warning message that the option was requested but not honored, etc.
To clarify the point in the other reply -- imagine it sent one packet per keystroke. Now anyone sitting on the network gets a rough measurement of the delay between your keystrokes. If you are entering a password for something (perhaps not the initial auth) it can guess how many characters it is and turns out there are some systemic patterns in how that relates to the keys pressed -- eg letters typed with the same finger have longer delays between them. Given the redundancy in most text and especially structured input that's a serious security threat.
It's not just the pointless chaff, the SSH protocol is inherently very chatty, and SFTP even more so. The solution, for a high-performance game, is don't use SSH. Either run it over Wireguard or grab some standard crypto library and encrypt the packets yourself. You'll probably make a few minor mistakes but unless the other player is the NSA it'll be good enough.
For that matter, why does it need to be encrypted at all? What's the threat model?
If there really is a genuine need to encrypt and low latency is critical, consider using a stream cipher mode like AES-CTR to pregenerate keystream at times when the CPU is lightly loaded. Then when you need to encrypt (say) 128 bytes you peel off that many bytes of keystream and encrypt at close to zero cost. Just remember to also MAC the encrypted data, since AES-CTR provides zero integrity protection.
Serious question, why not just use websockets? AFAIK, it's effectively a TLS socket with a little bit of handshake overhead when starting.
I'm literally working on a web interface I want to use for classic BBS door play... currently working on a DOS era EGA interface, and intend to do similar for PETSCII/Comodore64/128 play as well. I've got a couple rendering bugs to explore for ansis submitted that messed up in the viewer test mode.
https://github.com/bbs-land/webterm-dos-ansi
It's been an opportunity to play with AI dev as well... spent as much time getting the scrollback working how I want as it took on the general rendering.
Websockets is just another layer on top of TLS, so you've got the size explosion and complexity/latency of TLS and then another layer on top of that. The OP hasn't provided additional info on what the requirements are but if it's a realtime game then they'll probably be "as close to zero latency and size increase as possible (compared to unencrypted messaging)", which websockets over TLS isn't.
Yes, but I wouldn't be surprised if the change is rejected. The crypto library is very opinionated, you're also not allowed to configure the order of TLS cipher suites, for example.
[flagged]
that's the point of opinionated crypto libraries, yes
Personally I like that it's secure by default.
Those same security guys also think that "just hope that no bad guy ever gets root access, lol" is a valid threat model analysis, so whatever.
That is a completely valid threat model analysis, though? "Just hope no bad guy ever gets into the safe" is rather the entire point of a safe. If you have a safe, in which you use the contents of the safe daily, does it make sense to lock everything inside the safe in 100 smaller safes in some kind of nesting doll scheme? Whatever marginal increase in security you might get by doing so is invalidated by the fact that you lose all utility of being able to use the things in the safe, and we already know that overburdensome security is counterproductive because if something is so secure that it becomes impossible to use, those security measures just get bypassed completely in the name of using the thing. At some level of security you have to have the freedom to use the thing you're securing. Anything that could keep a bad guy from doing anything ever would also keep the good guy, ie. you, from doing anything ever.
> That is a completely valid threat model analysis, though?
No it isn't. Here in 2026 timesharing accounts aren't a thing anymore and literally everyone who ever logs into your server has root access.
"Just make sure all those outsourced sysadmins working for a contractor you've never met are never bad guys" is not a valid security threat model.
> literally everyone
Perhaps figuratively? I manage several servers where the majority of (LDAP) accounts have no special privileges at all. They get their data in the directories and can launch processes as their user, that's...pretty much it.
Though the upstream comment is gone and I am perhaps missing some important context here.
When the question is "how do I communicate securely with a third party," there's nothing you can do if the third party in question gets possessed by a demon and turns evil. (Which is what happens if an attacker has root.)
Incorrect.
Random sysadmins who have access to your server have the permissions to steal whatever is communicated between third parties unrelated to this sysadmin.
Just because some random outsourced nightshift dude has the permissions to do "sudo systemctl restart" shouldn't mean he gets to read all the secret credentials the service uses.
As it is now, the dude has full unfettered access to all credentials of all services on that machine.
I guess if your org usually gives the keys to the castle to random idiots, then yeah, I can see why you'd wish the master key didn't exist.
Threats exist in both trusted and untrusted environments though.
This feels like a really niche use case for SSH. Exposing this more broadly could lead to set-it-and-forget-it scenarios and ultimately make someone less secure.
Resource-constrained environments might be niche to you, but they are not niche to the world.
+1... Given how much SSH is used for computer-to-computer communication it seems like there really should be a way to disable this when it isn't necessary.
It looks like it is only applied for PTY sessions, which most computer-computer connections wouldn't be using.
https://github.com/openssh/openssh-portable/blob/d7950aca8ea...
In practice I've never felt this was an issue. But I can see how with extremely low bandwidth devices it might be, for instance LoRa over a 40 km link into some embedded device.
Hah no.
Nobody is running TCP on that link, let alone SSH.
Once upon a time I worked on a project where we SSH'd into a satellite for debugging and updates via your standard electronics hobbiest-tier 915mhz radio. Performance was not great but it worked and was cheap.
This is still done today in the Arducopter community over similar radio links.
I haven't heard much about the ArduCopter (and ArduPilot) projects for a decade, are those projects still at it? I used to run a quadroter I made myself a while back until I crashed it in a tree and decided to find cheaper hobbies...
Well at least crashing drones into trees has never been cheaper hahaha. So it's super easy to get into nowadays, especially if it's just to play around with flight systems instead of going for pure performance.
They're alive and well and producing some pretty impressive software.
Crashing your drone is a learning experience ;)
Remote NSH over Mavlink is interesting, your drone is flying and you are talking to the controller in real time. Just don't type 'reboot'!
ELRS?
Nope this predated ELRS by a bit. I wasn't super involved with the RF stuff so not sure if we rolled our own or used an existing framework
You can run ELRS on 900 MHz but the bitrate is atrocious.
https://github.com/markqvist/Reticulum
and RNode would be a better match.
In aerial robotics, 900MHz telemetry links (like Microhard) are standard. And running SSH over them is common practice I guess.
Why do you guess? I wouldn't expect SSH to be used on a telemetry link. Nor TCP, and probably not IP either.
what's wrong with tcp, on a crappy link, when guaranteed delivery is required? wasn't it invented when slow crappy links were the norm?
Because TCP interprets packet loss as congestion and slows down. If you're already on a slow, lossy wireless link, bandwidth can rapidly fall below the usability threshold. After decades of DARPA attending IETF meetings to find solutions for this exact problem [turns out there were a lot of V4 connections over microwave links in Iraq] there are somewhat standard ways of setting options on sockets to tell the OS to consider packet loss as packet loss and to avoid slowing down as quickly. But you have to know what these options are, and I'm pretty sure the OP's requirement of having `ssh foo.com` just work be complicated by TCP implementations defaulting to the "packet loss means congestion" behavior. Hmm... now that I think about it, I'm not even sure if the control plane options were integrated into the Linux kernel (or Mac or Wintel)
Life is difficult sometimes.
It will time out before your packet gets through, or it will retransmit faster than the link can send packets.
The guy in charge of Go's security decreed that SSL 1.3 (which he was a contributor to) was so secure that silly programmers should not be able to override what algorithms are allowed or not allowed, because why would they possibly need to do that, because he's such a genius, and even if someone DID find a security vulnerability, well....they can just wait for Google to publicly disclose it and release a patch, compile the new version, update their code to work with that version of Go, rebuild their containers, put stuff through testing, and then release it into production.
Versus...seeing there's a vulnerability, someone adding a one-line change to disable the vulnerable algorithm, compile, image update, test. And a lot less testing because you're not moving to a new version of the language / compiler.
The man has no practical experience in running a production network service, an ego the size of a small moon, and yet was a major contributor to a security protocol now in use by billions of people.
But hey, you can be a handbag designer and end up head of design at Apple soooooooo
Lots of the real world vulnerabilities out there exist exactly because of people choosing to support a range of crypto algorithms.
Sure, if it's an internal tool you can recompile both ends and force a universal update. But anything else and you need to stay compatible with clients and anytime you allow negotiation of the cryptosuit you open yourself up to quite a few subtle attacks. Not saying that choice about go is clearly a good one but i don't think it's obviously wrong.
Relying on not advertising some feature for it is very janky way to do it.
The proper fix would be adding option server-side to signal client it's not needed and have client side have option to accept or warn about that
Very interesting, I hadn't heard of this obfuscation before so it was well worth clicking.
Another good trick for debugging ssh's exact behavior is patching in "None" cipher support for your test environment. It's about the same work as trying to set up a proxy but lets you see the raw content of the packets like it was telnet.
For terminal games where security does not matter but performance and scale does, just offering telnet in the first place can also be worth consideration.
It made the front page when it was added.
https://news.ycombinator.com/item?id=37307708
Not everyone sees the HN frontpage every day, and sometimes especially-esoteric things spend a fairly short timespan on there.
Sadly, much fewer computer systems have telnet nowadays.
Also, port 21 is often blocked.
23 (21 is ftp)
Yup
Telnet, I want her to know it was me.
1980s: 1 packet per keystroke is too much, we must find a solution to bundle them together, for efficiency (see Nagle's algorithm, delayed ACK), also let's send everything in plaintext, including passwords
2020s: ha! with some advanced probabilistic models, we may be able to deduce something about what is being typed behind one of our layers of encryption, let's sent 100 packets per keystroke to mitigate that
Unfortunate result of the security theater.. "Someone who has access to run privileged application can run side channel attacks! Let's drop cpu performance 20 percent over the world"
As I understood it’s enough to have “access to run privileged application” anywhere where the packet goes through. So, not necessarily at client or server sides. Or did I misunderstand?
I think he's referring to CPU mitigations: https://en.wikipedia.org/wiki/Transient_execution_CPU_vulner...
I don't see how Claude helped the debugging at all. It seemed like the author knew what to do and it was more telling Claude to think about that.
I've used Claude a bit and it never speaks to me like that either, "Holy Cow!" etc. It sounds more annoying than interacting with real people. Perhaps AIs are good at sensing personalities from input text and doesn't act this way with my terse prompts..
Even if the chatbot served only as a Rubber Ducky [1], that's already valuable.
I've used Claude for debugging system behavior, and I kind of agree with the author. While Claude isn't always directly helpful (hallucinations remain, or at least outdated information), it helps me 1) spell out my understanding of the system (see [1]) and 2) help me keep momentum by supplying tasks.
[1] https://en.wikipedia.org/wiki/Rubber_duck_debugging
A rubber ducky demands that you think about your own questions, rather than taking a mental back seat as you get pummeled with information that may or may not be relevant.
I assure you that if you rubber duck at another engineer that doesn't understand what you're doing, you will also be pummeled with information that may or may not be relevant. ;)
That isn't rubber duck debugging. It's just talking to someone about the problem.
The entire point of rubber duck debugging is that the other side literally cannot respond - it's an inanimate object, or even a literal duck/animal.
I don't think that's right. When you explain a technical problem to someone who isn't intimately familiar with it you're forced to think through the individual steps in quite a bit of detail. Of course that itself is an acquired skill but never mind that.
The point or rubber duck debugging then is to realize the benefit of verbally describing the problem without needing to interrupt your colleague and waste his time in order to do so. It's born of the recognition that often, midway through wasting your colleague's time, you'll trail off with an "oh ..." and exit the conversation. You've ended up figuring out the problem before ever actually receiving any feedback.
To that end an LLM works perfectly well as long as you still need to walk through a full explanation of the problem (ie minimal relevant context). An added bonus being that the LLM offers at least some of the benefits of a live person who can point out errors or alert you to new information as you go.
Basically my quibble is that to me the entire point of rubber duck debugging is "doesn't waste a real person's time" but it comes with the noticeable drawback of "plastic duck is incapable of contributing any useful insights".
> When you explain a technical problem to someone who isn't intimately familiar with it you're forced to think through the individual steps in quite a bit of detail.
The point of Rubber Ducking (or talking/praying to the Wooden Indian, to use an older phrase that is steeped in somewhat racist undertones so no longer generally used) is that it is an inanimate object that doesn't talk back. You still talk to it as if you were explaining to another person, so are forcing yourself to get your thoughts in order in a way that would make that possible, but actually talking to another person who is actively listening and actually asking questions is the next level.
I guess I can see where others are coming from (the LLM is different than a literal rubber duck) but I feel like the "can't reply" part was never more than an incidental consequence. To me the "why" of it was always that I need to solve my problem and I don't want to disturb my colleagues (or am unable to contact anyone in the first place for some reason).
So where others see "rubber ducking" as explaining to an object that is incapable of response, I've always seen it as explaining something without turning to others who are steeped in the problem. For example I would consider explaining something to a nontechnical friend to qualify as rubber ducking. The "WTF" interjections definitely make it more effective (the rubber duck consistently fails to notify me if I leave out key details).
To that end a notepad works just as well.
In reality vim is my usual approach. But I think LLMs are better in a lot of regards.
Oh it can definitely be a person. I've worked with a few!
Cue obligatory Ralph Fiennes "You're an inanimate fucking object".
Before you expand the definition to every object in the universe, maybe we could call it parrot debugging.
I'm not saying you should do this, but you can do this:
https://gist.github.com/shmup/100a7529724cedfcda1276a65664dc...
Amusingly that looks less like "rubber duck debugging" and more like "socratic questions". Which certainly isn't a bad thing.
That is so true I wanted to "fix it", granted, I'm not even using these at the moment, but I appreciated the idea
https://github.com/shmup/metacog-skills/
Lol not bad
They also don’t waste electricity, water, drive up the prices of critical computer components, or DDOS websites to steal their content.
Not to defend the extravagant power use of the AI datacenters, but I invite you to look up the ecological footprint of a human being.
The human being in this scenario exists either way.
The AI does not.
> Even if the chatbot served only as a Rubber Ducky [1], that's already valuable.
I use the Other Voices for that. I can't entirely turn them off, I might as well make use of them!
Rubber Ducky is a terrific name for a GPT.
Also, always reminds me of Kermit singing "...you make bath time so much fun!..."
Maybe Kermit has sung it at some point, but that's Ernie's song usually
Ha. Well spotted. I totally hallucinated that.
Maybe it was this one
https://muppet.fandom.com/wiki/Do_De_Rubber_Duck
You’re absolutely right!
Claude is much faster at extracting fields from a pcap and processing them with awk than I am!
Have you tried wireshark?
AIs are exceptional at sensing personalities from text. Claude nailed it here, the author felt so good about the "holy cow" comments that he even included them in the blog post. I'm not just poking this, but saying that the bots are fantastic sycophants.
No they aren't. Current LLMs always have that annoying over-eager tone.
The comment about Claude being pumped was a joke.
It depends how much the LLM has been beaten into submission by the system prompt.
ChatGPT set to "terse and professional" personality mode is refreshingly sparse on the "you're absolutely right" bullshit
It's like I keep saying, it probably wasn't a good idea to give our development tools Genuine People Personalities...
I think the blog post itself was partially written with LLMs, which explains some of the odd narrative style.
Funny to see this fixed in 2023 and the side effects. Back in 2004, before I focused on performance, I did some security work including inter-keystroke latency analysis of captured SSH sessions to estimate the commands typed:
https://www.brendangregg.com/sshanalysis.html
The 2023 patch should finally fix that 2004 issue.
Memories! I was at the hacking conference HAL2001 and listening to Dug Song and Solar Designer, who were talking about their SSH timing analysis: https://download.openwall.net/pub/advisories/OW-003-ssh-traf...
Time flies
> When you say “LLMs did not fully solve this problem” some people tend to respond with “you’re holding it wrong!” > > I think they’re sometimes right! Interacting with LLMs is a new skill, and it feels pretty weird if you’re used to writing software like it’s 2020. A more talented user of LLMs may have trivially solved this problem.
So one thing I only recently figured out is that using ChatGPT via the web browser chat is massively different from using OpenAI's code-focused Codex model / interface. Once I switched to using Codex (via the VS Code extension + my own ChatGPT subscription) the quality of answers I got improved massively.
So if you're trying to use LLM to help with debug, make sure you're using the right model!! There are apparently massive differences between models of the same generation from the same company
You can also use TCP_CORK to reduce the number of packets without any increased latency.
Disabling TCP_NODELAY would also reduce number of packets + be portable & simpler to implement - but would incur a latency penalty.
Haven't heard of TCP_CORK, very interesting.
For people who don't feel like googling it:
1. You TCP_CORK a socket
2. You put data into it and the kernel buffers it
3. If you uncork the socket, or if the buffer hits MSS, the kernel sends the packet
Basically, the kernel waits until it has a full packet worth of data, or until you say you don't have any more data to send, and then it sends. Sort of an extreme TCP_YESDELAY.
See https://catonmat.net/tcp-cork for where I learned it all from.
Oh wow - I've never heard of TCP_CORK before. Without disabling pings I'd still pay the cost of receiving way more packets, but maybe that'd be tolerable if I didn't have to send so many pongs. This is super handy; excited to play around with it.
I am aware of TCP_NODELAY (funny enough I recently posted about TCP_NODELAY to HN[1] when I was thinking about it for the same game that I wrote about here). But I think the latency hit from disabling it just doesn't work for me.
[1] https://news.ycombinator.com/item?id=46359120
I missed that thread originally, the post and the comments where a good read, thank you for sharing.
I got a kick out of this comment [0]. "BenjiWiebe" made a comment about the SSH packets you stumbled across in that thread. Obviously making the connection between what you were seeing in your game and this random off-hand comment would be insane (if you had seen the comment at all), but I got a smile out of it.
[0] https://news.ycombinator.com/item?id=46366291
wow, I missed that comment, that's an incredible connection. Thank you!
First time I've been reading on HN and come across my name randomly.
Can you explain how TCP_CORK helps here? The chaff packets are spaced 20ms apart and sent per socket, so I don’t see how TCP_CORK could help unless it coalesced across 20ms intervals? But coalescing is clearly not an option for the intended obfuscation effect of the original feature.
It is unrelated to SSH, it is a generic TCP thing.
"hello world" fits in a single TCP packet, but the kernel might end up sending one packet containing "hello" and another packet containing " world". It is completely opaque to userspace.
TCP_CORK lets userspace decide when packets are dispatched. You get to control whether "hello world" is sent across 1 packet or 11 packets.
I’m aware what TCP_CORK does. I’m not seeing how it helps the situation in the post.
Ah, maybe you are saying it doesn’t help the situation in the post. That’s what I misunderstood.
Wow, I did not realize that SSH did that. Good to know, and it makes sense as a default, because the people who need it need to have it on by default. But I think I'm going to be turning that off, because it's a security measure that doesn't make sense for my particular environment:
1) I'm pretty much never typing secrets into an SSH tunnel; these days if there's a secret I need to transmit over SSH I'm going to be copying and pasting it, which will not reveal info from keyboard timing. (Or rsync'ing a file, which ditto).
2) I'm not in a high-security environment where nation-states have an interest in sniffing my keystrokes.
3) I often open SSH connections to servers in other continents. Those underwater cables have massive bandwidth, but they're also in constant use by thousands upon thousands of people. So anything I can do to reduce my bandwidth by 100x is probably worth doing.
Any reason you can think of why I should not be setting ObscureKeystrokeTiming=no in my ~/.ssh/config?
I think those all have reasonable counterarguments:
(1) This sounds brittle. Are you really going to have a good mental model about what's secret when using ssh and reliably refrain from typing those things? Seems to kinda defeat the idea of securing the channel. Also, as a collection your activities might be more confidential to you than single inputs, or correlated with your other activities outside ssh, etc - it's hard to keep a mental model of this as well. Aka optimism is not a form of security.
(2) There isn't a reason to think this is a difficult attack that only a powerful adversary could mount. Seems like a college lab level thing to me. And very amenable to AI help as well. Also here optimism is not a form of security. It's a 25 year old attack[1] so there's a lot of existing research[2] around.
(3) Saving 100x bandwidth on single keystrokes on an internet dominated by video traffic just because it's 100x doesn't make sense. Also it's good to cultivate a mindset that steers away from trading off security in favour of trivial resource savings.
[1] https://www.usenix.org/conference/10th-usenix-security-sympo... (probably older stuff exists outside open literature)
[2] eg https://crzphil.github.io/posts/ssh-obfuscation-bypass/
Bandwidth is not the problem when you are using mobile connections (4G, weaker 5G). Videos work just fine, but ssh can be painful already without keystroke obfuscation. The problem is latency. Especially when roaming abroad it can 100s of ms.
Not sure whether the obfuscation is fully synchronous, i.e waiting for the server response before continuing. That would really kill it. Working with LTS distros I don't think I have seen it in practice yet. Need to try something modern on my next trip abroad.
> Not sure whether the obfuscation is fully synchronous, i.e waiting for the server response before continuing.
The people who designed SSH aren't idiots, and also, you can answer this question by simple observation: When you connect to a server with ~200ms ping, which is somewhat common in the scenarios you describe and which I've done many times, it does not take 20 seconds to show a keystroke.
There's no way to know in advance if some leaked cleartext will provide enough information to an attacker to be useful. Attackers profit from making creative use of information they didn't have before.
That said, plenty of people disable the most useful security features of SSH, like verifying host key signatures, with no ill-effects (as far as they know). For the majority of users, using Telnet and unencrypted HTTP would make no difference, as nobody's trying to hack them, and who really cares about privacy anyway?
Did you know SSH has long-standing performance limitations due to its design that need patches to eliminate? It was never intended to be a high-performance tool. If you want really high performance, use Telnet. If you want real security, use SSH with all strong security options enabled plus a server using ContainerSSH with the OAuth2 plugin (SSH's keys are static, which can be captured and reused, which is bad). If you don't care either way, use SSH with the defaults.
> I'm pretty much never typing secrets into an SSH tunnel; these days if there's a secret I need to transmit over SSH I'm going to be copying and pasting it, which will not reveal info from keyboard timing
One common secret that goes through a tty ssh connection is a sudo password. You are probably typing sudo command so without obfuscation the attacker can find out the sudo keystrokes, the command keystrokes and then the encrypted bytes of the password. They don't have the timing data to decode them as easily as the previous parts but if they record enough traffic they might be able to decrypt the password. But maybe they won't, because the ssh session key is probably different each time. Furthermore I don't know how many times they should capture your encrypted password to be able to decrypt it. Maybe it's unfeasible.
Anyway, in case of the sudo password, if the attacker gets it what would happen? The attacker is hopefully not able to get a shell into the server. If they do they have different ways to get root privileges.
By the way, I also copy and paste secrets from either the password manager or the clipboard, because nobody remembers long random strings. The only exceptions are the passwords of a few accounts.
sudo passwords are one of the things I'm copying and pasting from the password manager, because my shell account password is different on every system. But yes, if you type your sudo password without thinking about it, the timing attack might be feasible. (Though if you're laboriously copying a random password from a different screen, as I've had to do once or twice in situations where copy-and-paste was infeasible, the timing data will be useless as it's about 500 ms between keystrokes no matter what the previous keystroke is. Which is an interesting way to accidentally defeat this attack.)
In 2023, ssh added keystroke timing obfuscation. The idea is that the speed at which you type different letters betrays some information about which letters you’re typing. So ssh sends lots of “chaff” packets along with your keystrokes to make it hard for an attacker to determine when you’re actually entering keys.
Now that's solving the problem the wrong way. If you really want that, send all typed characters at 50ms intervals, to bound the timing resolution.
Typing with an extra 50ms latency will be fairly unpleasant.
Average is 25ms. Just put sending on a clock.
Also considering ssh tunnels.
> send all typed characters at 50ms intervals
Wouldn't this just change the packet interval from 20ms to 50ms? Or did you mean a constant stream of packets at 50ms intervals, nonstop?
I think the idea behind the current implementation is that the keystrokes are batched in 20ms intervals, with the optimization that a sufficiently long silence stops the chaff stream, so the keystroke timing is obfucated with an increased error bar of 20ms multiplied by number of chaff packets.
I assume the problem, such as it is, relates to the fact that a real human typing in 20-50ms would generate a few characters at most but a program could generate gobs of data. So automatically you know what packets to watch. Then you know if there were more the likely keys were in set X, while if there were fewer the likely keys were in set Y.
So a clock doesn't solve the problem. The amount of data sent on each clock pulse also tells you something about what was sent.
The Chaff packets already fire on a timer. They inject random extra fake keystrokes so you can't tell how many keystrokes were actually made. The only other way I can think of to solve that is by using a step function: Send one larger packet (fragmented or the same number of individual packets) on each clock pulse if the actual data is less than some N where N is the maximum keystrokes ever recorded with some margin. Effectively almost every clock pulse will be one packet (or set of packets) of identical size. Of course if you do that then you'll end up consuming more data over time than sending random amounts of packets.
[flagged]
The problem is not knowing whether someone is typing, as far as I understand. But that you may extract some information about what keys are being typed, based on the small differences in timings between them.
Well, security is the #1 consideration for SSH, but if the author doesn't need security, why use ssh?
For example, "nc" (netcat) is pre-installed on all platforms where ssh is.
I seem to hit this logic often recently for some reason.
There are two issues with it:
- a primary is not a totality: if "security is the #1 consideration for SSH", that implies there's a #2, maybe even a #3 and so on consideration. So the question that follows becomes tautological: "but if the author doesn't need security, why use ssh?" -> surely for one or more of the #2, #3, etc. considerations, right?
- overabstraction (*): you ended up strawmanning the author. What they had issue with was keystroke timing obfuscation, which is a privacy feature. Timing attacks are (in part) a privacy concern, and privacy is a security concern, yes, but security is not just privacy concern, and privacy concerns are not just about timing attacks; these groups are not equal. For example, they might very well want the transmitted keypresses themselves to remain confidential, or they might very well want to retain cryptographic assurance of their integrity. These are security features they can continue to utilize by sticking with SSH.
All of this is to say, it's not even necessarily them using SSH for a hypothetical #2 or #3 (...etc...) reason, but likely because they still very much want to make use of large chunks of #1, which disabling keypress obfuscation does not actually rid SSH of, only at most weakens it in ways they clearly seem to be okay with.
(*) although if I zoom out enough, this is once again just "a primary is not a totality", just implicitly
> For example, "nc" (netcat) is pre-installed on all platforms where ssh is.
This is technically incorrect, because Windows now includes SSH too!
Depends on what kind of security. They might care about connection integrity. If a faulty (or malicious) router in-between client and server starts malforming packets, `nc` will display those malformed packets. SSH will only show you what the server intended, or nothing.
> I am working on a high-performance game that runs over ssh.
Found your problem.
But it is an interesting world where you can casually burrow into a crypto library and disable important security features more easily than selecting the right network layer solution.
the obtuseness is the point! This is true of a lot of my work[1][2][3].
The problems you run into when doing things you shouldn't do are often really fun.
[1] https://news.ycombinator.com/item?id=42342382
[2] https://news.ycombinator.com/item?id=37810144
[3] https://news.ycombinator.com/item?id=42674116
Ah I see now. The characteristic irony I associate with Brooklyn making its way online. Well done.
These were great reads, thanks for linking. The writeup around the UUID page was super interesting!
This is hackernews not consumer news
You should feel free to explore / abuse all options :)
Yea UDP is technically more performant, but then you need a crypto layer + reliable message delivery layer + bespoke client. Using a plain old SSH client is cool.
However, there are existing libraries for exactly this use case - see https://github.com/ValveSoftware/GameNetworkingSockets
I guess QUIC libraries would also work.
its not really a question of 'udp performs better'. in tcp we have to live to head-of-line blocking on losses and congestion control. if you don't care about receiving every packet, but only the most recent, then udp is a good choice.
running without congestion control means that you avoid slowstart. but at a certain rate you run into poorly defined 'fairness' issues where you can easily negatively impact other flows. past that point, you can actually self-interfere and cause excessive losses for yourself.
quic uses congestion control, but uses latency estimates and variance as a signal to back off. it still imposes an ordering on a per-stream basis. so it might not be ideal either.
sctp has a mode which supports reliable and unordered, which might be something to consider
so really - if you care about latency and have a different reliability model, its worth unpacking all these considerations and using them to select your transport layer or even consider writing a minimal one yourself
>in tcp we have to live to head-of-line blocking on losses and congestion control.
Is this not a performance consideration?
Either way, using plain old SSH means a metric bajillion computers have a client for your game built in.
The reliance on LLMs is unfortunate. I bet this mystery could gave been solved much quicker by simply looking at the packet capture in Wireshark. The Wireshark dissectors are quite mature, SSH is covered fairly well.
I'm anti-LLM in most cases, but:
> I bet this mystery could gave been solved much quicker by simply looking at the packet capture in Wireshark.
For some people who are used to using Wireshark and who know what to look for, probably yes. For the vast majority of even technical people, probably not.
In my case, I did a packet capture of a single keystroke using tcpdump and imported it into Wireshark and I get just over 200 'Client: encrypted packet' and 'Server: encrypted packet' entries. Nothing useful there at all. If I tcpdump the entire SSH connection setup from scratch I get just as much useful information - nothing - but, oddly, fewer packets than my one keystroke triggered.
So yeah, I dislike LLMs entirely and dislike the reliance on LLMs that we see today, but in this case the author learned a lot of interesting stuff and shared it with us, whereas without LLMs he might have just shrugged and moved on.
And thats a huge downside when people howl about "Encryption everywhere! ".
Try debugging that shit. Thats right, debugging interfaces aren't safe, by some wellakshually security goon.
You want a real fun one to debug, is a SAML login to a webapp, with internal Oauth passthrough between multiple servers. Sure, I can decrypt client-server stuff with tools, but server-server is damn near impossible. The tools that work break SSL, and invalidate validation of the ssl.
Yes, Esri products suck. Bad.
I used to share that opinion but after decades in industrial automation I find myself coming down much more on the "yeah, encryption everywhere" because while many vendors do not provide good tools for debugging, that's really the problem, and we've been covering for them by being able to snoop the traffic.
Having to MITM a connection to snoop it is annoying, but the alternative appears to be still using unencrypted protocols from the 1970s within the limitations of a 6502 to operate life-safety equipment.
Problem is, security people don't want you to MITM connections, because it's insecure (mostly to business interests). Hence stuff like certificate pinning, HSTS, DoH...
If you're debugging your own equipment you should have the certificates or keys to make it work. I'm not saying that's easy in a lot of scenarios, in fact it's frequently tedious as hell. But for example there are debug tools for like DNP3 or RPC over TLS, etc that can watch the whole exchange if provided the keys and parse the SCADA traffic or JSON objects as if it was plaintext.
But this goes back to the vendors not providing better tools in the first place. We shouldn't NEED to be picking apart packet streams to prove to some jackass tech support ticket that their code is FUBAR. They're basically outsourcing support to their customer or userbase and we tolerated it because it was more expedient.
This really does not need to be that hard. For TLS, many tools support setting the SSLKEYLOGFILE environment variable to log the session keys used in connections. Wireshark can import those to decrypt everything. [1]
Unfortunately, nothing exists for SSH (yet?). [2]
I do agree that if you design a protocol that enforces encryption, you should include some debugging interface. It is much more straightforward to do this by logging the session secrets on the endpoints rather than trying to break it through a man-in-the-middle, the main thing the protocol is protecting you against.
[1]: https://wiki.wireshark.org/TLS
[2]: https://gitlab.com/wireshark/wireshark/-/issues/16054
It seems like a leap to suggest we shouldn't have widely deployed encryption...rather than just fix the debugging tools.
Particularly in today's political climate, encryption has only become more necessary.
Sounds like blaming a tool on a problem it did not cause. Either way, solvable and encryption is important. Badly designed systems and or lack of tooling isn't really an encryption problem.
Anyway, VMs should not have authentication, it makes access sooo much easier. Also drop your IPs while you're at it. Might be useful for debugging later.
Unfortunately with SSH specifically, the dissectors aren't very mature - you only get valid parsing up to the KeX completion messages (NEWKEYS), and after that, even if the encryption is set to `none` via custom patches, the rest of the message flow is not parsed.
Seems because dumping the session keys is not at all a common thing. It's just a matter of effort though - if someone put in the time to improve the SSH story for dissectors, most of the groundwork is there.
Interesting, I thought it was possible to decrypt SSH in Wireshark a la TLS, but it seems I'm mistaken. It still would have been my first goto, likely with encryption patched out as you stated. With well documented protocols, it's generally not too difficult deciphering the raw interior bits as needed with the orientation provided by the dissected pieces. So let me revise my statement: this probably would have been a fairly easy task with protocol analysis guided code review (or simply CR alone).
It all depends on the key exchange mechanism (KEM) used at the start of the TLS session. Some KEM have a property called “perfect forward secrecy” (PFS) which means it’s not possible to decrypt the TLS session after the fact unless one of the nodes logs out the session key(s). Diffie Helman and ECDH are two KEM that provide a PFS guarantee.
Sure it could have been, if you knew about SSH packet inspectors in Wireshark...
The author didn't, and used a general tool to their aid - why is that unfortunate?
Hey! I'm the author.
My thinking was:
So idk. I'm sure that you would have solved this much more quickly than I did! I'm not sure that (for me) opening up the packet in Wireshark would have solved this faster. Maybe reading the SSH spec would have, but debugging also just didn't take that long.And the big leap here was realizing that this was my SSH client and not a quirk of my game. The time at which I would have read the SSH spec was after I captured traffic from a regular SSH session and observed the same pattern; before that I was thinking about the problem wrong.
I don't think that this is unfortunate. In fact, I think I got what I wanted here (a better sense of Claude Code's strengths and weaknesses). You're right that an alternative approach would have taught me different things, and that's a worthy goal too.
[1] I suspect this is because I was doing it for an old job and I had to figure out how to run some application with keys I controlled? It would have been easier here. I don't remember.
Thanks for taking the time to respond, and apologies for the contentiousness. I'm a jaded old man suffering from severe LLM fatigue, so I may have come off a bit harsh. Your write-up was a good read, and while I might be critical of your methodology, what you did clearly worked, and that's what matters in the end. Best of luck with your project, especially the go lib fork.
Eh, I was a little annoyed at the comment last night but read through the thread again today and you were clearly engaging in good faith.
I totally get being exhausted at LLMs. And I don't mind the nudge to be a little less lazy and install wireshark for next time.
hope I get you to play the game when it's out!
For sure. When it's out I'll give it a go.
obviously OPs empirical and analytical rigor are top notch. He applied LLMs in the best way possible: fill gaps with clumsy command line flags or protocol implementations. Those aren't things one needs to keep in their head all the time.
Asking an LLM about SSH (hint: the two S-es stand for security) would tell you why only having packet capture in Wireshark isn't going to reveal shit.
Not even remotely accurate. While the dissector is not as mature as I thought and there's no built-in decryption as there is for TLS, that doesn't matter much. Hint: every component of the system is attacker controlled in this scenario.
> Not even remotely accurate.
> there's no built-in decryption
Is that because wireshark can't do that just from packet captures?
>Is that because wireshark can't do that just from packet captures?
Well, not quite. I think it's more that nobody has taken the time to implement it. That's not to say such an implementation would automatically decrypt the traffic from a capture with no extra leg work, of course. Wireshark dissectors have user configurable preferences, and presumably this would be where captured secrets could be set for use. This is how it handles TLS decryption [1], which works beautifully.
[1] https://wiki.wireshark.org/TLS#tls-decryption
Wireshark can decrypt it, so I don't understand what you mean?
Not from packet captures, it can't.
Way to gatekeep. God forbid people use tools to help them investigate instead of knowing the exact approach to take.
My thoughts exactly. The OP used AI to get a starting point to their investigation, then used their skills to improve their game, with actual (I guess according to the article itself) proof of that, as opposed to just approving changes from the LLM.
This looks like an actual productivity boost with AI.
What I suggested (mistakenly so, see my revised suggested approach in response to one of your siblings) is the exact opposite of gate keeping.
ChatGPT gaslit the OP telling it there was no such thing as keystroke chafing. So yes, in this case it would have been better to do the work oneself.
How much are you staking on that bet?
Well, I spent a good part of my career reverse engineering network protocols for the purpose of developing exploits against closed source software, so I'm pretty sure I could do this quickly. Not that it matters unless you're going to pay me.
So you are basically overqualified to tell other people how to do it, especially with the payment part.
What are you even trying to say? I suppose I'll clarify for you: Yes, I'm confident I could have identified the cause of the mysterious packets quickly. No, I'm not going to go through the motions because I have no particular inclination toward the work outside of banter on the internet. And what's more, it would be contrived since the answer has already shared.
I think the point they're making is that "I, a seasoned network security and red-team-type person, could have done this in Wireshark without AI assistance" is neither surprising nor interesting.
That'd be like saying "I, an emergency room doctor, do not need AI assistance to interpret an EKG"
Consider that your expertise is atypical.
Sure, but that is aside from my original point. If somebody:
a) Has the knowledge to run tcpdump or similar from the command line
b) Has the ambition to document and publish their effort on the internet
c) Has the ability identify and patch the target behaviors in code
I argue that, had they not run to an LLM, they likely would have solved this problem more efficiently, and would have learned more along the way. Forgive me for being so critical, but the LLM use here simply comes off as lazy. And not lazy in a good efficiency amplifying way, but lazy in a sloppy way. Ultimately this person achieved their goal, but this is a pattern I am seeing on a daily basis at this point, and I worry that heavy LLM users will see their skill sets stagnate and likely atrophy.
>I argue that, had they not run to an LLM, they likely would have solved this problem more efficiently
Hard disagree. Asking an LLM is 1000% more efficient than reading docs, lots of which are poorly written and thus dense and time-consuming to wade through.
The problem is hallucinations. It's incredibly frustrating to have an LLM describe an API or piece of functionality that fulfills all requirements perfectly, only to find it was a hallucination. They are impressive sometimes though. Recently I had an issue with a regression in some of our test capabilities after a pivot to Microsoft Orleans. After trying everything I could think of, I asked Sonnet 4.5, and it came up with a solution to a problem I could not even find described on the internet, let alone solved. That was quite impressive, but I almost gave up on it because it hallucinated wildly before and after the workable solution.
The same stuff happens when summarizing documentation. In that regard, I would say that, at best, modern LLMs are only good for finding an entrypoint into the docs.
While my reply was snarky I am prepared to take a reasonable bet with a reasonable test case. And pay out.
Why I think I’d win the bet is I’m proficient with tcpdump and wireshark and I’m reasonably confident that running to a frontier model and dealing with any hallucinations is more efficient and faster than recalling the incantantions and parsing the output myself.
> I argue that, had they not run to an LLM, they likely would have solved this problem more efficiently
This is just expert blindness, and objectively, measurably wrong.
Oh come on, the fact that the author was able to pull this off is surely indicative of some expertise. If the story started had started off with, "I asked the LLM how to capture network traffic," then yeah, what I said would not be applicable. But that's not how this was presented. tcpdump was used, profiling tools were mentioned, etc. It is not a stretch to expect somebody who develops networked applications knows a thing or two about protocol analysis.
The specific point I was trying to make was along the lines of, "I, a seasoned network security and red-team-type person, could have done this in Wireshark without AI assistance. And yet, I’d probably lose a bet on a race against someone like me using an LLM."
Sigh.
I'm still waiting for a systems engineering tool that can log every layer, and handle SSL the whole pipe wide.
Im covering everything from strafe and ltrace on the machine, file reads, IO profiling, bandwidth profiling. Like, the whole thing, from beginning to end.
Theres no tool that does that.
Hell, I can't even see good network traces within a single Linux app. The closest you'll find is https://github.com/mozillazg/ptcpdump
But especially with Firefox, good luck.
Real talk though, how much would such a tool be worth to you? Would you pay, say, $3,000/license/year for it? Or, after someone puts in the work to develop it, would you wait for someone else to duct tape something together approximately similar enough using regexps that open source but 10% as good, and then not pay for the good proprietary tool because we're all a bunch of cheap bastards?
We have only ourselves to blame that there aren't better tools (publicly) available. If I hypothetically (really!) had such a tool, it would be an advantage over every other SRE out there that could use it. Trying to sell it directly comes with more headaches than money, selling it to corporations has different headaches, open-sourcing it don't pay the bills, nevermind the burnout (people don't donate for shit). So the way to do it is make a pitch deck, get VC funding so you're able to pay rent until it gets acquired by Oracle/RedHat/IBM (aka the greatest hits for Linux tool acquisition), or try and charge money for it when you run out of VC funding, leading to accusations of "rug pull" and development of alternatives (see also: docker) just to spite you.
In the base case you sell Hashimoto and your bank account has two (three!) commas, but worst case you don't make rent and go homeless when instead you could've gone to a FAANG and made $250k/yr instead of getting paid $50k/yr as the founder and burning VC cash and eating ramen that you have to make yourself.
I agree, that would be an awesome tool! Best case scenario, a company pays for that tool to be developed internally, the company goes under, it gets sold as an asset and whomever buys it forms a compnay and tries to sell it directly and then that company goes under but that whomever finally open sources it because they don't want it to slip into obscurity but if falls into obscurity anyway because it only works on Linux 5.x kernels and can't be ported to the 6.x series that we're on now easily.
> That 20ms is a smoking gun - it lines up perfectly with the mysterious pattern we saw earlier!
Speaking of smoking guns, anybody else reckon Claude overuses that term a lot? Seems anytime I give it some debugging question, it'll claim some random thing like a version number or whatever, is a "smoking gun"
Yes! While this post was written entirely by me, I wouldn't be surprised if I had "smoking gun" ready to go because I spent so much time debugging with Claude last night.
It's interesting how LLMs influence us, right? The opposite happened to me: I loved using em dashes, but AI ruined it for me.
I still love using emdashes, and people already thought I was a robot!
https://xkcd.com/3126/
Soon the Andy 3000 will finally be a reality...
That's a sweet ass—reference
I used to love using em dashes.
I still do - but I used to, too.
Hey wait, - isn't one! Did a human write this?
Serious question though, since AI seems to be so all capable and intelligent. Why wouldn't it be able to tell you the exact reason that I could tell you just by reading the title of this post on HN? It is failing even at the one thing it could probably do decently, is being a search engine.
Direct answers are often useless without building up context for them.
Reminds me of ethimology nerd's videos. He has some content about how LLMs will influence human language.
Some day in the future we will complain about AIs with a 2015 accent because that’s the last training data that wasn’t recursive.
The "maybe" of yesterday is the "you're absolutely right!" of tomorrow.
shouldn't it be "human language influences human language"?
ChatGPT too. And "lines up perfectly" when it doesnt actually line up with anything
Same with Gemini.
You can absolutely see this pattern in Gemini in 2026.
Btw, is the injection of "absolutely" and "in $YEAR" prevalent in other LLMs as well, or is it just in Gemini's dialect?
I've had gemini tell me "We are debugging this problem here in İstanbul" and talking about an istanbul evening, trying to give uplifting or familiar vibes while being creepy.
I think there was a setting about time and location which finally got rid of that behavior.
It's just Gemini. I'm guessing they changes the system prompt for the new year or something, but it's pretty annoying.
I chuckled out loud. It's funny cause it's true.
"You're so right, that nice catch lines up perfectly!"
It's not just a coincidence, it's the emergence of spurious statistical correlations when observations happen across sessions rather than within sessions.
You can add an M-dash, and we completed the bs-bingo. :)
Or the "Eureka! That's not just a smoking gun, it's a classic case of LLMspeak."
Grok, ChatGPT, and Claude all have these tics, and even the pro versions will use their signature phrases multiple times in an answer. I have to wonder if it's deliberate, to make detecting AI easier?
A computational necromancer has likely figured out a way to power a data center by making Archimedes spin in his grave very fast.
I've love to delve into that.
https://pshapira.net/2024/03/31/delving-into-delve/
Without knowing how LLM's personality tuning works, I'd just hazard a guess that the excitability (tendency to use excided phrases) is turned up. "smoking gun" must be highly rated as a term of excitability. This should apply to other phrases like "outstanding!" or "good find!" "You're right!" etc.
I'm working on a little SRE agent to pre-load tickets with information to help our on-call and I'm already tired of Claude finding 'smoking guns'.
smoking gun, you're absolutely right, good question, em dash, "it isn't just foo, it's also bar", real honest truth, brutal truth, underscores the issue, delves into, more em dashes, <20 different hr/corporate/cringe phrases>.
It's nauseating.
You might find this a fun read: https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing
It's what they read on The Internets when training, so don't expect them to generate new phrases, other than what they learned from it?
### The answer that fits everything (and what to do about it)
Maybe we need a real AI which creates new phrases and teaches the poor LLMs?
Looking back we already had similar problems, when we had to ask our colleagues, students, whomever "Did you get your proposed solution from the answers part or the questions part of a stackoverflow article?" :-0
cant wait for chatgpt to make me read about grandmas secret recipe and scroll through 6 ads to see the ingredients for my chicken teriyaki dinner
That's the point though, it doesn't reflect human usage of the word. If delve were so commonly used by humans too, we wouldn't be discussing how it's overused by LLMs.
Come on...haven't we all had to deal with the crazy smart lead who was loaded with those same types of annoying tics?
Considering what these LLMs bring to the table, I think a little tolerance for their cringe phrases is in order.
You might see certain phrases and mdashes ;-) rather often, because … these programs are trained on data written by people (or Microsoft's spelling correction) which overused them in the last n years? So what should these poor LLMs generate instead?
At this I'm just so glad that "you're absolutely right!" phase is over.
They love clichés, and hate repeating the same words for something (repetition penalty) so they'll say something like "cause" then it's a "smoking gun" then it's something else
I don't think claude has even once used this in my conversations (Claude Desktop, Claude Code, Voice conversations...) Sycophancy, yes absolutely!
Maybe it has something to do with your profile/memories?
It's a smoking gun of Claude usage.
Yes, it’s kind of a corpus delicti. ;)
I see it from GPT5 too a lot
> Speaking of smoking guns
Oh shoot! A shooting.
So the TL;DR of this post is: don't change this setting unless you know what you're doing.
Chastise it with a reminder that you're using smokeless powder.
What is the usecase for using ssh at all where you don't need to be resistant against timing analysis? Either it's not sensitive and you can use telnet (if necessary after using ssh to authenticate) or the game (or other stuff on the connection) might be sensitive and you need traffic analysis resistance.
If you get clever and write a client to ensure sensitive data like passwords or email are sent in a burst you could just use an encryption library just for that data instead.
Dont let this article be blinders to you. Ssh does much more than obfuscate keypress timings. Not needing the chaff means turn it off and keep all the other benefits. It doesn't mean "revert to telnet"
Or you could use anycasting to terminate SSH sessions on the moral equivalent of one of a number of geography based reverse proxies and then forward the packet over an internal network to the app server over a link tuned for low latency. The big guys already do something similar with HTTP over TLS for DDoS protection and to limit end to end latency on TLS.
Granted... it would increase the cost (since you're adding reverse proxies) but it would be a quick way to get acceptable latency, rudimentary DDoS protection, and you could try different connection options independent of the main app's logic.
It would be hard to estimate how much latency you're adding with a SSH2 reverse proxy in this case, but it's probably lower than one might think.
The idea of letting Claude loose on my crypto[graphy] implementation is about the most frightening thing I've heard of in a while [though libnss is so craptastic, I can't see how it would hurt in that case.] But I loved this write-up. It was readable and explained the problem the OP was encountering and proposed solutions well.
> Or you could use anycasting to terminate SSH sessions on the moral equivalent of one of a number of geography based reverse proxies and then forward the packet over an internal network to the app server over a link tuned for low latency.
I've been thinking about some stuff like this! Not being able to put my game behind Cloudflare[1] is a bummer. Substantial architectural overhead though.
> The idea of letting Claude loose on my crypto[graphy] implementation is about the most frightening thing I've heard of in a while [though libnss is so craptastic, I can't see how it would hurt in that case.]
I hear you, but FWIW the patch I was reverting was trivial (and it's also in the go crypto library, which is pretty easy to read). It's a couple-of-line change[2], and Claude did almost exactly what I would have done (I was tired and would have forgotten to shrink the handshake payload).
[1] This isn't strictly true, Cloudflare spectrum exists, but its pricing is an insane $1/GB last I checked.
[2] https://cs.opensource.google/go/x/crypto/+/833695f0a57b30373...
Nice, but shouldn't the behaviour change be behind a config setting? And it's not clear what the intent of the change is. Implementing PING/PONG seems different from what you said you were trying to do. And it's section 1.8 of the OpenSSH [PROTOCOL] reference, not section 1.9.
But... before you think I'm trying to be negative... good on you. I wish you well. Getting crypto/security code into open source projects can be a slog as people frequently come out of the woodwork, so don't get discouraged.
And the more I think about this... there's plenty of examples out there about doing HTTP based reverse proxying, but essentially zero for SSH proxying, so if you do that, it would make a great blog post.
Wait, go back to the first sentence:
> I am working on a high-performance game that runs over ssh. The TUI for the game is created in bubbletea 1 and sent over ssh via wish.
> The game is played in an 80x60 window that I update 10 times a second. I’m targeting at least 2,000 concurrent players, which means updating ~100 million cells a second. I care about performance.
High performance with ssh and wish? For sure not. Rather use UDP over secure sockets. Or just normal sockets. Even Claude would come up with much faster code than the ssh/wish nonsense. Or mosh, but this also too complicated.
The author wanted people to be able to just "ssh mygame", no? In that sense, ssh was a design requirement.
I didn't think about such throwback to the 80ies. Could be, yes. But then he cannot control the ssh option, and with 2000 users, maybe 10 would set it. I don't think so.
I enjoyed this write up as it touched on several topics I enjoy reading about.
Also I was unfamiliar with SSH being vulnerable in the past to keystroke timing!
https://news.ycombinator.com/item?id=37307708
2023 discussion about it here.
The really mysterious part is how ~10,000 packets per second costs ~20% of a core. That would mean SSH is bottlenecking in its code at ~50,000 packets per second per core which would be ~500 Mbps per core (assuming full packets) which is ludicrously slow. It is trivial to do 10x that packet per second rate. Is SSH really that poorly designed?
> It is trivial to do 10x that packet per second rate.
When making this statement, are you taking into account that SSH encrypts the traffic by default?
I do not know where people get the idea that encryption is that slow. Standard AES hardware acceleration instructions do ~25 Gbps per core (on a 2023 CPU) which is ~50x that rate [1]. I have heard modern cores can do ~40-50 Gbps, but I have not been able to find any independent benchmarks of that. Even the Intel i5-2500, a CPU from 2011, averages ~10 Gbps which is ~20x that rate. Even unaccelerated encryption can do ~2-5 Gbps in pure software which is 4-10x the SSH rate.
And in this situation, the amount of encrypted payload in each packet is 36 bytes which is ~40x less than a full packet of ~1500 bytes. You would almost surely hit packet per second limits before you hit payload throughput limits at these small sizes.
Encryption is slow when compared to data throughput you can get with a properly designed transport stack, but that is because it is in comparison to 100 Gbps per core even with no hardware offload. Anything less than ~10 Gbps/1 million packets per second (ignoring other bottlenecks, so only the software transport is the limit) is not merely unoptimized, it is pessimized.
[1] https://calomel.org/aesni_ssl_performance.html
doing a gigabit takes ~35% of single core to saturate my 1Gbit ethernet. On i3-3250 which is 12 years old CPU
Your assumptions are way off
> I am working on a high-performance game that runs over ssh.
Step one, run https://www.psc.edu/hpn-ssh-home/introduction/ instead Step two, tune TCP/IP stack Step... much later: write your own "crypto". (I'm using quotes because, before someone points out the obvious, packets-per-keystroke isn't, itself, a cryptographic algorithm, but because it's being done to protect connections from being decrypted/etc, mess with it at your own peril.)
Funny that this comes up today! I was just looking into adding a keyboard monitor to my website (I have a goal of making my 'contact me' page have oddly specific information). I wouldn't show the actual keys, just show a blinking light when there's activity, but I guess the timing really could expose quite a lot of information.
I did add a trackpad monitor though. It shows my raw MacBook trackpad data.
https://varun.ch/contact/
>In 2023, ssh added keystroke timing obfuscation. The idea is that the speed at which you type different letters betrays some information about which letters you’re typing. So ssh sends lots of “chaff” packets along with your keystrokes to make it hard for an attacker to determine when you’re actually entering keys.
Why not just add random "jitter" to the keystroke packets, but keeping just the 1 actual packet?
Jitter could be filtered out, I presume.
How? You can't average out the noise here because the attack involves discriminating the different types of events from one another based on the thing you'd be averaging.
One clue is that you cannot predict what key user is going to press next reliably, so the jitter would always be added to actual key press. You can minimise that by adding constant latency, so that you could simulate pulling events back in time, but still this is going to get complex quick and still could be filtered out. As for methods, it depends on the jitter. Think of things like noise removal in audio and adaptive filtering. Adding extra packets is much easier and more secure.
Okay I think I see the issue (and slight misunderstanding). I believe the problem is actually latency. I was assuming the jitter interval would be noticeably larger than the gap between typical (say 95%) of key presses. Any smaller than that and you start to need cover traffic.
Such an interval would still face correlation issues due to the varying nature of the overlap between the jitter intervals, however it seems like that should be trivial to address. That said, just throwing in some cover traffic is bound to be simpler.
But a jitter interval long enough that keystroke packets can change order is going to be noticeable to a human typing quickly on what should be a solid connection - my WiFi is only at 3 to 6 ms RTT and I already notice that versus a wired connection. That doesn't sound so trivial to fix, and once again just throwing in some cover traffic completely solves the issue.
So just do what's simple.
My next question was going to be, why on the order of 100 extra packets instead of just 1 or 2? But of course an attacker could attempt to search some set of permutations for recognizable words. So either you drown everything out (simple) or you hook a multilingual dictionary up to a key stroke delay model for your cover traffic generator (complex).
But really shouldn't this feature be implemented as some constant (low) background level of cover traffic that scales up as your typing frequency increases but caps out at some (still fairly low) rate? That seems both less likely to suffer from inadvertent leaks as well as not running afoul of the issue in the article.
Not related to SSH, but does the eieio.games website make anyone else's monitor flicker? When the website is fullscreen it overwhelms something. I thought my monitor's backlight was going.
I wonder if this is the same reason why Microsoft's Remote SSH plugin on VS Code is so flaky even with a decent internet connection. Every couple of months I try to give it another go and give up due to the poor keyboard latency I inevitably experience. And the slow reconnects whenever I glance away from my computer monitor briefly. This is on a fiber connection with a 20ms ping to the remote machine.
You surely mean the latency in its embedded terminal and not the code editor, right? I use VSCode’s remote SSH specifically so that code editing doesn’t suck. It really does not.
You're right, the latency is in the embedded terminal. Perhaps it is trying to run SSH inside SSH. Still, the disconnects are a pain too.
>I am working on a high-performance game that runs over ssh.
By 'ssh', you mean 'ssh' (library/program + protocol + encryption + decryption) on top of TCP/IP, on top of the Internet, right?
OK, I'm not against it... but you do understand that there are all kinds of ways for that to slow things down, right?
Your issues may (or may not!) include such things as:
o Nagle's algorithm AKA buffering AKA packets not being sent until N bytes (where N > 1) ready to send, as other posters have suggested;
o Slower encryption/decryption on older hardware (if users with older hardware is a target market, and if the added loss in speed makes an impact in gameplay, depending on the game, this may or may not be the case...)
o The fact that TCP/IP (as opposed to UDP / Datagrams / "Raw" sockets) imposes a connection-oriented abstraction, requiring additional round trips of ACK ("I got the packet") RESEND ("I didn't get the packet") on top of the connectionless architecture that is the Internet (https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...), which adds additonal latency, so, for example if a rural user in Australia experiences a 350ms delay for a raw packet to get to a U.S. server (or vice versa), then TCP/IP might make this 700ms or more, depending on the quality of the connection!
o The speed of the game limited to both the bandwidth and latency of the slowest user (if a multi-player game, and if the game must not update until that user "moves"... again, game architecture will determine this, and it wouldn't be applicable to all games...)
Now, you could use UDP, as other posters have suggested, but then you must manually manage connections and encryption...
That may be the right choice for some types programmers, some types of games/applications -- but equal-and-oppositely it may be the wrong choice for others...
Anyway, wishing you well with your game development!
I haven't used SSH (at least, not in a debug capacity), so I'm not sure what SSH debugging options exist -- but it would be nice if SSH had a full logging debug mode, which would explain exactly WHY it chose to send any given packet that it did along with related helpful information, such as latency/time/other metrics, etc., if it doesn't have this/these feature(s) already...
@eieio: whatever email protection you're running is triggering on the extension info. For example I see:
> And they’re sent to servers that advertise the availability of the [email protected] extension. What if we just…don’t advertise [email protected]?
Is it possible that this is on your end?
The extension is "ping@openssh.com." It shows up in the blog reliably for me across several browsers and devices.
No, it's Cloudflare munging the HTML. Cloudflare then provides JavaScript to un-munge it, but that's not reliable.
And of course it totally doesn't work if the client doesn't have JavaScript at all. I read the HN front-page through an AI summary and it also got censored when it scraped the article.
TIL! I'll see if I can change that.
>>> That makes a lot of sense for regular ssh sessions, where privacy is critical. But it’s a lot of overhead for an open-to-the-whole-internet game where latency is critical.
Switching to telnet instead of SSH might be an option.
Why not "amortise" the period of sending keystrokes -- buffer them in a queue, and process the queue for sending these on a regular (and short enough for the human at the client end feeling the interactivity) interval, so there's no latency difference between sending an 'a' vs a 'q' and so on. If we assume some average typing speed on the bell curve, say, around 250 keystrokes per minute, the queue can be picked for sending every 250 milliseconds or so. That solution wouldn't require injecting extra packets on the network. What am I missing?
It's wonderful to see LLM's being used to increase the programming community's general quality level of work, as more things become worth doing.
Am I missing something? This isn't what ssh's purpose is. Why should anyone care? We're talking about a game built to run over an encryption protocol? What are we even doing anymore? Also, correct me if I'm wrong, but client-side option existing is secure design, really feels like it shouldn't be circumvented server-side without giving the client the choice to do so or not by default. Don't lobby for watering down security for convenience, especially for trivially important objectives, please?
Your security is safe. This game isn’t going to cause SSH to degrade.
I assume this is done for novelty. There is also Terminal coffee which is a coffee company that takes orders for delivery over SSH.
Product behavior should be explainable without sleuthing. The well-named variables in the logs are serving this purpose.
Hmm, if the author is doing something high performance, they should probably use whatever mosh is doing to update the screen, not ssh.
That would require end users to install additional software though, which they do not want
Oh, true, ssh is not just the protocol, but also the name of the client software.
Though I would suggest to make mosh available, too. Many nethack servers are available via mosh and ssh. (And in an earlier age, telnet.)
Genuinely asking: Wasn't the reason this happens kind of obvious in the first place?
If security doesn’t matter then why not use telnet or something else besides ssh instead of forking a security library?
Telnet nowadays typically isn’t available by default for security reasons, and OP wants people to be able to play the game just by typing “ssh thegamehost”.
> Telnet nowadays typically isn’t available by default for security reasons
And with good reason. This CVE is from yesterday:
https://nvd.nist.gov/vuln/detail/CVE-2026-24061
> telnetd in GNU Inetutils through 2.7 allows remote authentication bypass via a "-f root" value for the USER environment variable.
Telnetd is the server though, and OP wouldn’t be using that.
ah, good point.
fwiw, I tcdumped between two systems running fedora43 and saw no chaff. (one packet out, one reply, one tcp ack.)
On Debian 13 I get a bunch when just typing interactively on shell instance
> Keystroke obfuscation can be disabled client-side.
please never do that (in production)
if anyone half way serious tries they _will_ be able to break you encryption end find what you typed
this isn't a hypothetical niche case obfuscation mechanism, it's a people broke SSH then a fix was found case. I don't even know why you can disable it tbh.
That doesn't sound right to me. This obfuscation isn't about a side-channel on a crypto implementation, this is about literally when your keystrokes happen. In the right circumstances, keystroke timing can reduce the search space for bruteforcing a password [1] but it's overstating to describe that as broken encryption.
[1] https://people.eecs.berkeley.edu/~daw/papers/ssh-use01.pdf
THANK YOU!
I'm baffled about this "security feature". Besides from this only being relevant to timing keystrokes during the SSH session, not while typing the SSH password, I really don't understand how can someone eavesdrop on this? They'd have to have access to the client or server shell (root?) in order to be able to get the keystrokes typing speed. I've also never heard of keystroke typing speed hacking/guessing keystrokes. The odds are very low IMO to get that right.
I'd be much more scared of someone literally watching me type on my computer, where you can see/record the keys being pressed.
Anyone who can spy on the network between the client and server can see the timing. This includes basically anyone on the same LAN as you, anyone who sets up a WiFi access point with a SSID you auto-connect to, anyone at your ISP or VPN provider, the NSA and god knows who else.
And the timing is still sensitive. [1] does suggest that it can be used to significantly narrow the possible passwords you have, which could lead to a compromise. Not only that, but timing can be sensitive in other ways --- it can lead to de-anonymization by correlating with other events, it can lead to profiling of what kind of activity you are doing over ssh.
So this does solve a potentially sensitive issue, it's just nuanced and not a complete security break.
[1] https://people.eecs.berkeley.edu/~daw/papers/ssh-use01.pdf
They literally explain the mechanism in the post and then explain why the security tradeoff made sense for their ssh game………
It is to prevent timing attacks but there are many ssh use cases where it is 100% computer to computer communications where there is no key based timing attack possible.
There is an argument that if:
- you are listening to an SSH session between devices
- and you know what protocol is being talked over the connection (i.e. what they are talking about)
- and the protocol is reasonably predictable
then you gain enough information about the plaintext to start extracting information about the cipher and keys.
It's a non-trivial attack by all means but it's totally feasible. Especially if there's some amount of observable state about the participants being leaked by a third party source (i.e. other services hosted by the participants involved in the same protocol).
this only works for manually typed text, not computer to computer communication where you can't deduce much from what is being "typed" as it's not typed but produced by a program to which every letter is the same and there is no different delay in sending some letters (as people have when typing by hand)
I'd love to hear more about this kind of attack being exploited in the wild. I understand it's theoretically possible, but...good luck! :)
You're guessing a cipher key by guessing typed characters with the only information being number of packets sent and the time they were sent at. Good luck. :)
I agree it is more nuanced than a simple 'good for computer-to-computer' and 'bad for person-to-computer'. I'm sure there are cases where both are wrong but I don't think that necessarily changes that it makes a reasonable baseline heuristic.
I haven't given this more than 5 seconds of thought, but wouldn't it make sense to only enable the timing attack prevention for pseudo-terminal sessions (-t)?
The fix seems kind of crazy though, adding so much traffic overhead to every ssh session. I assume there's a reason they didn't go that route, but on a first pass seems weird they didn't just buffer password strokes to be sent in one packet, or just add some artificial timing jitter to each keystroke.
I'm just guessing but this chaff sounds like it wouldn't actually change the latency or delivery of your actual keystrokes while buffering or jitter would.
So the "real" keystrokes are 100% the same but the fake ones which are never seen except as network packets are what is randomized.
It's actually really clever.
SSH has no way of knowing when a password is being typed. It can happen any time within the session after SSH auth.
But they'd have to be on the same network as me to do that attack, right?
Yep, like ECHELON and friends are. The metadata recorded about your (all of our) traffic is probably enough to perform the timing attack.
Hey, if ECHELON snuck a listener into my house, where six devices hang out on a local router... Good for them, they're welcome to my TODO lists and vast collection of public-domain 1950s informational videos.
(I wouldn't recommend switching the option off for anything that could transit the Internet or be on a LAN with untrusted devices. I am one of those old sods who doesn't believe in the max-paranoia setting for things like "my own house," especially since if I dial that knob all the way up the point is moot; they've already compromised every individual device at the max-knob setting, so a timing attack on my SSH packet speed is a waste of effort).
I find it disturbing.
One thing you notice if you have ADSL is that some services are built as if slower connections matter and others are not. Like Google's voice and audio chat services work poorly but most of the others work well. Uploading images to Mastodon, Bluesky, Facebook, LinkedIn, Instagram and Nextdoor is reliable, but for Tumblr you have to try it twice. I don't what they are doing wrong but they are doing something wrong and not finding out what they're doing wrong because they're not testing and they're not listening to users.
Nobody consulted me about their decision not to run fiber by my house. If some committee decides to make ssh bloated they are, together with the others, conspiring to steal my livelihood and I think it would be fair for me to sue them for the $50k it would take to run that fiber myself.
It's OK if you work for Google where there is limitless dark fiber but what about people in African countries?
It's the typical corporate attitude where latency never matters: Adobe thinks it is totally normal that it takes 1-5s for a keystroke to appear when you are typing into Dreamweaver.
I agree with your general point that most companies/projects do a terrible job optimizing for slow computers/networks, but OpenSSH is from the OpenBSD people, who are well-known for supporting ancient hardware [0]. Picking a random architecture, they fully support a system with only 64MB of memory [1], and the base install includes SSH. So I suspect that OpenSSH is fairly well tested on crappy computers/networks.
[0]: https://www.openbsd.org/plat.html
[1]: https://www.openbsd.org/landisk.html#hardware
There's a good chance you have other options. Regardless of how you feel about the company's head, Starlink would probably be one of them, with likely better performance than you're dealing with on ADSL.
But you cannot just sue a company because their network connected software doesn't work well on slow networks. Let alone a project like OpenSSH. It would be like me suing a game studio because my PC doesn't meet their listed minimum requirements to play the game.
Hey, it is one thing to buy a new computer, it is another thing to ask people to move.
A better analogy is a bank redlining neighborhoods. The cost to run fiber to difficult rural locations pays itself easily if you look at a 25-year time span and is an order of magnitude less than building a new housing unit on the West Coast.
> another thing to ask people to move.
One half of my comment was actually about how you most likely have a better performing alternative option right where you already live. And even if you didn't, they're not asking you to move. You could argue they're not even asking you to use their software, you're electing to.
It's OSS with no warranty. You can compile it yourself with the option disabled. It's only ever on for pty connections (physical user with a keyboard), there's no added traffic for ttys.
You're not ok with a security/privacy tool using defensive techniques because of ... the lack of fiber in Africa?
My backyard but people will take Africa more seriously than anywhere in the US 2 miles from the end of cable.
The openssh team does not owe you anything.
If you want a “1990s” mode, add it yourself or pay some to do it for you.
> One thing you notice if you have ADSL
This is funny to me, because ADSL used to be the fast thing, as opposed to dialup modems.
You just opened a huge nostalgia portal, never thought that Dreamweaver would still be around, I used that somewhere around 2003 I believe. Good memories
Frankly I wish there was an HTML editor that delivers on what it promised. I mean, markdown is almost as rife with edge cases as YAML and somehow the link syntax still eludes me. If we could “just” template by merging at the DOM level and had decent HTML editors the world would be a different place. But yeah, Adobe probably thinks Dreamweaver isn’t worth maintaining just as they seem to think Photoshop is barely worth maintaining (they keep adding AI features that sorta work but the foundations seem to be much worse than Illustrator)
Nice job! I need to learn how to use tcpdump apparently
Because we stopped coding for performance years ago.
"The smoking gun!" got me laughing, i am not a native english speaker and only ever seen that expression from Claude, and who knew? Its gaining popularity!
I got 99 packets but an SSH aint one
How do the HPN patches compare?
>very confidently told me that my tcpdump output was normal ssh behavior:
I mean, for modern version of Openssh it's not exactly wrong. The failure was to tell you why that is the normal behavior.
I loled and closed the article after 'I am working on a high-performance game that runs over ssh.'
Vibe coders man...
Amen brother
Tell me more about this game!
It's vibe coded
Just replace it with zeromq with curvemq.
I could vibecode an SSH zmq daemon in an afternoon.
SSH now deliberately sends many dummy (“chaff”) packets per keystroke to obscure typing patterns and defeat keystroke-timing attacks. This privacy feature makes a single keypress look like heavy network activity, explaining why it can generate around 100 packets.
Just wait until they discover head of line blocking
No TLDR. -.-
The 2023 timing obfuscation is a nice case study in security defaults vs edge cases. Most SSH users won't notice 100 packets per keystroke - it's noise in the bandwidth budget. But for high-frequency terminal apps, it becomes the dominant cost. At 2000 concurrent players updating 80x60 chars at 10fps, a custom protocol might be the right answer regardless of obfuscation settings.
You'd think the cover traffic would automatically cut out once the connection reached a certain rate though.
Just think of the trees burnt in the name of security!
Each of our devices spents a lot of energy dedicated to encryption. By now, all disks you did not set up manually are most likely encrypted and hardly any unencrypted package will travel out of your network. That's not to mention the tons of load and dedicated hardware we have just to terminate https and scan traffic for suspicious activity or the hardware being replaced because it's internal security triggered/broke.
In a perfect world, we could send all traffic completely unencrypted and never scan for a malicious payload, saving all that energy and hardware. But we do not live in that world and drawing the line with this minor, mostly unintrusive security feature seems strange.
That's the judgement made with all consumption of energy. The benefits weighed against the costs.
Because of the harms of environmental change, there should be pressure placed to avoid damaging ways to generate that energy.
When people complain about the amount of energy being used, they are making the judgement on the benefits. This is subjective and people do not agree on the benefits. The argument you shouldn't do this because of the energy consumed is implicitly saying "My judgement on the worth of this supercedes yours"
Pretty soon it devolves into criticizing the energy use of things you just don't like.
A society has to accept that people have different opinions on things. That includes what it is worth using energy for.
Producing clean energy is something everyone should be able to get behind. There is a solid consensus that it would make a better world.
Producing cheap energy is something everyone should be able to get behind. There is a solid consensus that it would make a better world.
See what I did there? As long as you preach any ideology of yours without talking about its trade-offs you can claim everyone should get behind it. Obviously.
I don't see a problem with that statement.
If you get the situation where the two are in conflict then you have to bring in a judgement, but it makes it explicit what you are prioritising.
Luckily for the case of energy, solar meets both goals.
> In a perfect world, we could send all traffic completely unencrypted and never scan for a malicious payload, saving all that energy and hardware.
In a world with such social cohesion, we'd be defeated by an alien species being able to quickly interpret and exploit our technology like in the hit film Independence Day(note, we're the defeating alien species in this example). https://www.youtube.com/watch?v=9DIjBGierkA
Shouldn't we sacrifice some security for convenience? And shouldn't we at least have a public discussion where to draw the line?
I already don't encrypt my Pinebook storage, because the device is low-powered.
I now disabled ObscureKeystrokeTiming on the ssh clients where it does not matter. And it should not matter in 99.9999% of cases.
P.S. There's a good reason airline frequencies are unencrypted AM and I hope IT "security" mindset does not reach its dirty hands up the air.
Airline isn't about power consumption but rather reliability. You don't introduce failure modes to safety critical systems unless absolutely necessary.
Meanwhile the power consumption of a few extra packets or even AES on your block storage device is approximately nothing relative to the other things the device is doing. Unless the CPU or GPU is going full tilt the screen on a mobile device consumes more power than the rest of the system combined (not sure about a laptop but it's likely a similar story).
It's a bit like worrying about saving a single glass of drinking water, then turning around and hopping in the shower for an hour. Or not flushing the toilet then immediately drawing a bath.
I can think of a few things that burn more energy per second that I'll burn in my entire lifetime via the use of encryption.
Coincidentally, they're all, somehow, insanely useless.
According to my Macbook Pro's energy sensors the predominant user of energy is the screen backlight, most of the time.
Crypto functions are so optimized in hardware that their energy consumption is insignificant...
[dead]
> I am working on a high-performance game that runs over ssh.
WAT. Please no.
Why not? If it's high-performance, it's fine.
SSH suffers from tcp-in-tcp issues which means it’ll always take a performance hit over other protocols
If you spend entire CPU to process few megabits of SSH traffic, it isn't high performance
Performing with highly elevated privileges? (Joke)
ssh the protocol doesn't imply any privileges of any kind
Unless you leave your ssh agent on, then it very much does.