Bypass SSH Keystroke Obfuscation
OpenSSH version 9.5 added measures to prevent keystroke timing attacks through traffic analysis. The patch included the addition of keystroke timing obfuscation in the SSH client. According to the release notes, this feature “attempts to hide the timing between keystrokes by sending interaction traffic at fixed intervals (default 20 ms) when little data is being sent.” It also sends fake chaff packets after the last keystroke, making traffic analysis much more difficult by hiding real keystrokes in a flood of artificial packets. This feature can be configured using the option ObscureKeystrokeTiming in SSH configuration.
I was investigating the impact of using keystroke delay analysis to detect commands sent by a client within an SSH session as part of my bachelor's thesis. In the process, I discovered a workaround for this obfuscation method that works up until the current release. I notified the developers on April 24th and received a response from the Damien Miller (the developer who added this patch), but unfortunately further correspondence was ignored. Hence the publication of this article.
Existing problem
Previous implementations of the SSH protocol exposed a significant amount of metadata, especially when used interactively. Even though the metadata is fully encrypted, it can be used to compromise the confidentiality of the session. Put simply, every time you press a key in an interactive SSH session, that keystroke is packaged, padded with bytes, encrypted, and sent to the server. It is then echoed back by the server. This means that each keystroke can be clearly identified and time-stamped, opening the possibility of attacks based on analyzing the delays between keystrokes to determine what the client was typing. The analysis can also be enhanced by adding additional context, such as server responses and other metadata, which I analyzed in detail as part of a university paper that will be published after evaluation.
The above can be seen using Wireshark with a filter for SSH. But to simplify the process, I wrote a tool SSHniff to automate the extraction of metadata. I also added Jupyter notebookswhich shows how intercepted delays can be used to determine the UNIX commands being transmitted, using algorithms such as Dynamic Time Warp (DTW) and/or Time Series Forests. Details on this are collected in in a separate article.
Obfuscation in a Nutshell
Although this attack vector was previously ignored, preventative measures were first added last October (2023). The idea was to hide real keystrokes among fake packets that would look the same to an outside observer. These are the chaff packets. Also, all outgoing packets are transmitted at regular intervals, about 20 ms. These chaff packets are actually packets SSH2_MSG_PING And SSH2_MSG_PONGwhich are the same size as the keystroke packets. Each time you press a key, these packets are sent en masse, hiding subsequent keystrokes. They are also sent for a specified amount of time after the last keystroke.
Bypass detection
Part of my dissertation was to look at the effectiveness of OpenSSH's mitigations, and while I expected this attack vector to be completely closed, after loading the Wireshark output into SSHniff I found that certain packets still stood out significantly among hundreds of packets with a 20ms delay.
In this session I executed the command uptime. As you can see, for each letter in uptime there is a corresponding peak on the graph, but it is worth noting that the first real press has a delay of 0 ms, and the last peak corresponds to pressing Enter, giving us a total of 7 presses.
To confirm, I ran a few additional commands and observed similar behavior. netstat ‑tlpnfor example, contains 13 characters. Pressing Enter in this case was omitted.
I realized that these spikes in the graphs were due to SSHniff dropping 3 packets at a time (resulting in a ~60ms delay for each spike). This was caused by the way the tool was implemented — it only looks for packets of a certain size K, which corresponds to the size of keystroke packets. This meant that among the chaff packets, some packets were larger or smaller than the expected size, which led to them being dropped.
This prompted me to take a closer look at the Wireshark output. Indeed, for each keystroke after the chaff call, the actual keystrokes were producing larger packets (as well as two responses from the server), allowing them to be clearly identified. I used previously mentioned techniques such as DTW to see if these outlier packets could be used to identify the command being sent, and it worked. This confirmed that these packets contained real keystrokes.
For more information on latency analysis, see the Keystroke Latency Analysis section of this article.
I then notified the OpenSSH developers and continued my research.
Large packages
The packets that were emitted were approximately twice the size of “normal” clicks (and therefore twice the size of chaff packets). I say “approximately” because the exact size depends on the encryption used and other factors.
Chaff packets are 102 bytes long, which would normally be the same size as normal keystrokes when using this cipher suite (those SSHniff would also filter out). Since the first packet of the three sent by the client is larger (138 bytes), all three packets are lost and lead to the previously mentioned peaks on the graph.
Interestingly, the initial keystroke is sent “normally”, without being wrapped in a larger packet. Large packets only start appearing once the chaff flood begins. Similarly, waiting for the flood to finish before entering the next keystroke results in a normal pair of packets.
OpenSSH verbose output
I compiled it OpenSSH v9.7 with a flag DPACKET_DEBUG to get a more verbose output. In the created session I executed the command whoami. Next we will show how the client creates and packages keystrokes.
Starting with the initial push, which as mentioned earlier is sent as a normal sized packet, after which chaff packets are sent.
debug1: packet_start[94]
plain: buffer len = 15
0000: 00 00 00 00 00 5e 00 00 00 00 00 00 00 01 77 .....^........w
debug1: send: len 20 (includes padlen 5, aadlen 4)
encrypted: buffer len = 36
0000: c7 e4 05 07 10 e1 f3 4b 24 ba 61 e8 fe 6e 0b 01 .......K$.a..n..
0016: 79 50 4e af 6a 96 31 5e ff fa ec bf 2b 3b 91 42 yPN.j.1^....+;.B
0032: a7 14 64 a6 ..d.
debug3: obfuscate_keystroke_timing: starting: interval ~20ms
debug1: input: packet len 20
debug1: partial packet: block 8, need 16, maclen 0, authlen 16, aadlen 4
read_poll enc/full: buffer len = 36
0000: d7 47 be 72 2c e8 e7 e1 ae 38 fe a2 f4 e9 e0 04 .G.r,....8......
0016: 9c 00 fd 1d 41 f8 d9 6e 61 4b 90 4e a4 e6 2c 30 ....A..naK.N..,0
0032: 92 92 42 54 ..BT
debug1: input: padlen 5
debug1: input: len before de-compress 10fPackage Type 94 — SSH2_MSG_CHANNEL_DATAgiven in ssh2.hstores a single keystroke. We can also see the obfuscation start at ~20ms. This packet is 36 bytes long after encryption, which matches the Wireshark output when looking at the TCP payload length.
Next comes the echo from the server, read by the client:
read/plain[94]:
buffer len = 9
0000: 00 00 00 00 00 00 00 01 77 ........w
debug1: received packet type 94
wNow let's look at the chaff packets that follow the first click. This is what the client sends:
debug1: packet_start[192]
plain: buffer len = 15
0000: 00 00 00 00 00 c0 00 00 00 05 50 49 4e 47 21 ..........PING!
debug1: send: len 20 (includes padlen 5, aadlen 4)
encrypted: buffer len = 36
0000: d7 cf 6b 64 25 d6 40 89 68 eb 4d 6c a0 cb de e6 ..kd%.@.h.Ml....
0016: d0 b5 14 81 c4 57 6f c4 3a 82 eb 55 44 d2 b4 9d .....Wo.:..UD...
0032: 3b 58 12 ac ;X..
debug1: input: packet len 20
debug1: partial packet: block 8, need 16, maclen 0, authlen 16, aadlen 4
read_poll enc/full: buffer len = 36
0000: 88 40 00 52 0e 4b fc eb 89 f7 72 1f d6 a4 3f dd .@.R.K....r...?.
0016: 0b dd 27 19 0e a8 84 f7 74 6f 43 e7 8c eb 16 9e ..\'.....toC.....
0032: 37 4e 89 95 7N..
debug1: input: padlen 5
debug1: input: len before de-compress 10йWe see that this is SSH2_MSG_PINGand most importantly, it is also 36 bytes long, which perfectly matches the length of the actual keystroke. Several such PING packets are sent, after each of which the server sends a PONG (SSH2_MSG_PONG), the length of each of which is also 36 bytes.
read/plain[193]:
buffer len = 9
0000: 00 00 00 05 50 49 4e 47 21 ....PING!
debug1: received packet type 193
debug1: Received SSH2_MSG_PONG len 5So far, everything looks as intended. But after we get to the second real press, namely the letter hthings take a different turn.
First of all, a package is formed, as before:
debug1: packet_start[94]
plain: buffer len = 15
0000: 00 00 00 00 00 5e 00 00 00 00 00 00 00 01 68 .....^........h
debug1: send: len 20 (includes padlen 5, aadlen 4)
encrypted: buffer len = 36
0000: c3 22 ea f0 f5 47 15 db 95 c9 64 ec e6 66 40 a2 .\"...G....d..f@.
0016: d2 fc 71 e2 59 35 c3 a7 85 90 4c b9 7f 17 fd 65 ..q.Y5....L....e
0032: 97 54 c3 e6 The length is the same, but what's interesting is that the debug lines partial packet And read_poll are missing because the packet has not yet been sent. Then the PING packet is generated before the keystroke is sent:
debug1: packet_start[192]
plain: buffer len = 15
0000: 00 00 00 00 00 c0 00 00 00 05 50 49 4e 47 21 ..........PING!
debug1: send: len 20 (includes padlen 5, aadlen 4)
encrypted: buffer len = 72
0000: c3 22 ea f0 f5 47 15 db 95 c9 64 ec e6 66 40 a2 .\"...G....d..f@.
0016: d2 fc 71 e2 59 35 c3 a7 85 90 4c b9 7f 17 fd 65 ..q.Y5....L....e
0032: 97 54 c3 e6 c6 59 df 64 eb c8 ba d4 f7 ed 5a 88 .T...Y.d......Z.
0048: 53 13 da 7e 7f 1d 63 9d dd 23 40 b4 b9 67 6e f3 S..~..c..#@..gn.
0064: 76 12 66 1b 89 5b 5a 21 v.f..[Z!
debug1: input: packet len 20
debug1: partial packet: block 8, need 16, maclen 0, authlen 16, aadlen 4
read_poll enc/full: buffer len = 36
0000: 47 3e ca 40 05 b8 a8 5b 1d 1a 2b bd bd c6 d5 35 G>.@...[..+....5
0016: d1 dc 56 f2 28 8a c4 07 df cb 73 e1 fb cc 0a 9e ..V.(.....s.....
0032: 20 73 c7 97 s..
debug1: input: padlen 5
debug1: input: len before de-compress 10Here we see lines that were previously missing. partial packet And read_poll. Also, due to packet merging, the encrypted length is 72 bytes instead of the expected 36.
And finally, we get two responses from the server: first PONGand then the response to the keystroke h.
read/plain[193]:
buffer len = 9
0000: 00 00 00 05 50 49 4e 47 21 ....PING!
debug1: received packet type 193
debug1: Received SSH2_MSG_PONG len 5
debug1: input: packet len 20
debug1: partial packet: block 8, need 16, maclen 0, authlen 16, aadlen 4
read_poll enc/full: buffer len = 36
0000: ec 9f ef a2 55 7e c3 4c f8 75 08 a9 8d 45 7e 14 ....U~.L.u...E~.
0016: 1f 55 b1 44 6e ea c7 f9 c9 ef ed ef 33 42 a7 29 .U.Dn.......3B.)
0032: 67 84 fa 94 g...
debug1: input: padlen 5
debug1: input: len before de-compress 10
read/plain[94]:
buffer len = 9
0000: 00 00 00 00 00 00 00 01 68 ........h
debug1: received packet type 94
hThis is what the packet triplets peaks look like in verbose debug output. This also explains the larger packet size and the duplicate server responses, since the real hits are packed together with the PING packets, which doubles the packet size and causes a double server response.
SSHniff
In the spirit of the good old “PoC or GTFO” approach, I wrote a nasty but working patch for SSHniff that, if it detects an SSH version higher than 9.4, assumes obfuscation is in use and applies a workaround. Keep in mind that this is truly obscene code that plagues my text editor, but it should be enough to show that the current obfuscation implementation is completely transparent.
Here is an example of running SSHniff on an intercepted SSH session that used obfuscation: I ran the commands iptables -S, whoami, ls -algot a typo exiand finally exit. You can check it yourself, PCAP is attached here.
<SNIP>
┃╭─────────────────Client─────────────────╮ ╭─────────────────Server─────────────────╮
┃│ 192.168.0.19:55932 │ │ 192.168.0.16:22 │
┃│ e42184b06d45385a906f0803d04c83da │----->│ aae6b9604f6f3356543709a376d7f657 │
┃│ SSH-2.0-OpenSSH_9.7 │ │ SSH-2.0-OpenSSH_9.7 │
┃╰────────────────────────────────────────╯ ╰────────────────────────────────────────╯
<SNIP>
┣━ tcp.seq ─ Latency μs ─ Type
┣ [4450] ─ ( 0) ─ Keystroke
┣ [4774] ─ ( 177182) ─ Keystroke
┣ [5026] ─ ( 119630) ─ Keystroke
┣ [5170] ─ ( 60477) ─ Keystroke
┣ [5530] ─ ( 182991) ─ Keystroke
┣ [5638] ─ ( 36727) ─ Keystroke
┣ [5998] ─ ( 175786) ─ Keystroke
┣ [6142] ─ ( 59886) ─ Keystroke
┣ [6394] ─ ( 119464) ─ Keystroke
┣ [6646] ─ ( 117633) ─ Keystroke
┣ [7078] ─ ( 219396) ─ Keystroke
┣╮ [10858] ─ ( 3478329) ─ Enter
┃╰─╼[236]
┣━
┣ [10858] ─ ( 0) ─ Keystroke
┣ [11290] ─ ( 238980) ─ Keystroke
┣ [11470] ─ ( 80064) ─ Keystroke
┣ [11650] ─ ( 79103) ─ Keystroke
┣ [11902] ─ ( 122768) ─ Keystroke
┣ [12226] ─ ( 158690) ─ Keystroke
┣╮ [15034] ─ ( 3324090) ─ Enter
┃╰─╼[204]
┣━
┣ [15034] ─ ( 0) ─ Keystroke
┣ [15322] ─ ( 162362) ─ Keystroke
┣ [15502] ─ ( 81398) ─ Keystroke
┣ [15682] ─ ( 83084) ─ Keystroke
┣ [15862] ─ ( 79398) ─ Keystroke
┣ [16114] ─ ( 123489) ─ Keystroke
┣╮ [18598] ─ ( 1363393) ─ Enter
┃╰─╼[3116]
┣━
┣ [18598] ─ ( 0) ─ Keystroke
┣ [18922] ─ ( 164250) ─ Keystroke
┣ [19210] ─ ( 144942) ─ Keystroke
┣╮ [22522] ─ ( 1822534) ─ Enter
┃╰─╼[256]
┣━
┣ [22522] ─ ( 0) ─ Keystroke
┣ [22846] ─ ( 162024) ─ Keystroke
┣ [23134] ─ ( 149977) ─ Keystroke
┣ [23458] ─ ( 158038) ─ Keystroke
┣╮ [27350] ─ ( 204709) ─ Enter
┃╰─╼[272]
┣━
┃
┣━━━━As you can see, the keystrokes are easily extracted and ready for processing by analysis tools.
Keystroke Delay Analysis
This is not part of the original goal of the article to describe obfuscation bypass, but it should help understand the impact of SSH protocol metadata leakage and the need for such measures. It also adds to the picture of the attack process itself.
To show how SSH metadata can be used to violate privacy, I will demonstrate a PoC of using SSHniff to extract keystrokes and obtain commands.
The output of Wireshark can be passed to the tool, which will give us the following output:
Among other information, the tool shows any sequence of keystrokes performed within a session, as well as the relative latency in microseconds, the TCP sequence number, and the suspected keystroke type. Using the packet size, we can differentiate between keystrokes such as Backspace, Enter, and the horizontal arrow keys, another important point in traffic analysis. The session shown only printed exitafter which the Enter key is pressed.
The tool can also serialize the data for further plotting and processing. I prepared this proof of concept using a Jupyter notebook and a small dataset collected for my dissertation.
Rhythmic commands
For a more complete look at the study, please see Jupyter notebook in the SSHniff repository. If you're only interested in bypassing obfuscation, look for the “Patch Analysis” section at the very bottom.
In fact, commands can have certain “profiles” or rhythms when entered, which can be identified by their latency. The graph below shows an example of when I entered the command sudo apt upgrade 18 times:
A dataset collected by other contributors also confirms this behavior, although some commands are more easily identified than others:
Using algorithms such as Euclidean Distance or DTW, the captured keystroke sequence can be matched to commands in the dataset, calculating the “similarity” of the sequence.
This is what a sequence obtained by SSHniff might look like:
Some results are shown in the following table:
![[Business English] How to talk about deadlines and times in English](https://prog.world/wp-content/uploads/2021/12/022ee7456124ded3365bd55429b03cec.jpeg)
