Removing the password from the PC keyboard after 0-60 seconds by thermal trace

Modern thermal imagers connect to a computer and record the temperature of each pixel in a CSV. This data is great for training a neural network. A few months ago, scientists from the University of Glasgow conducted an experiment and developed a really effective model ThermoSecure for recognition of pressed keys by a thermal trace.

Scientists have come to the conclusion that it is possible to relatively reliably remove entered passwords from the keyboard in 0–60 seconds after entering characters. Let’s see how the machine vision system works step by step.


The experimental setup is shown in the photo above, each object is labeled. In particular, an infrared camera was used for the experiment. Optris PI 450 (A) €3,950 (764×480 resolution, 80Hz, 40mK NETD, −20°C to 100°C temperature range, in photo A) mounted on a tripod next to a Microsoft Wired ISO Keyboard 600 (QWERTY) with ABS keycaps (B). The height of the tripod and the distance to the object changed in each session in the ranges from 60–90 cm and 50–121 cm, respectively. In addition, the angle of rotation of the keyboard also changed.

On the screen of laptop D, the subject was shown the characters to enter, while another laptop (C) recorded thermal 16-bit images from the camera. For each thermal image, the Optris APIs recorded a CSV file with the pixel temperature.

ThermoSecure Architecture

1. Keyboard localization: data collection and annotation, model training for reliable keyboard recognition in the frame.
   
2. Keystroke logging: Recognition of the layout, pressed keys, including those repeatedly pressed (they are warmer than once pressed).
   
3. Determining the order of clicksrecognition of the authentication stage against the background of general interaction with the system, that is, against the background of normal work.

Keyboard contour marking during model training annotation

Before entering the machine learning model, all images are pre-processed to reduce noise. This processing includes four main steps:

1. 5×5 median filter overlay
   
2. Convert to grayscale
   
3. Reapplying the median filter
   
4. Contrast-limited adaptive histogram equalization (CLAHE) (doi: 10.1109/ICACCI.2014.6968381)

Four stages of image preprocessing

At the second stage, the contour of the Rotated Bounding Box (RBBox) is overlaid from the Mask R-CNN, a special neural network that performs image segmentation. From this contour, the coordinates of the four vertices of the rectangle are extracted. The model then uses a template keyboard configuration (4 regions, 6 rows) to calculate key coordinates and correlate temperature clusters in the frame with the location of specific keys that are used to enter the password.

Finally, in the last step, the model uses the key coordinates obtained earlier and the temperature data from each thermal image to obtain the average, minimum, and maximum temperature of each keycap. These three values ​​(mean, max, min) are then averaged for each key to calculate the correct order in which they were pressed.

This procedure also includes several steps. After separating the username input phase from the password input phase, a list of password characters is determined. Then the temperature data of each key is used as the transition probability between different states (keys). Using the transition probability and the state probability, the probability of each sequence is calculated, as shown in the following diagram:

In this case, the sequence is most likely 6fbx9palthough all options are calculated:

( ' Password : ', '6 fbx9p ', ' with a probability : ' , 0.40740188103110087)
( ' Password : ', '6 fxx9p ', ' with a probability : ' , 0.4071952200350172)
( ' Password : ', ' ffbx9p ', ' with a probability : ' , 0.40670664210825924)
( ' Password : ', '6 fb99p ', ' with a probability : ' , 0.40659163972726375)
( ' Password : ', '6 f9x9p ', ' with a probability : ' , 0.4065814008654436)
( ' Password : ', ' ffxx9p ', ' with a probability : ' , 0.40649998111217556)
( ' Password : ', '6 fx99p ', ' with a probability : ' , 0.40638497873118007)
( ' Password : ', '6 bbx9p ', ' with a probability : ' , 0.4059259564615335)

Based on the results of the experiment, the scientists calculated the probability of password recognition by the heat signature, depending on the length of the password and the time between entering the password and removing the heat signature. As you can see, short six-character passwords are recognized with 100% accuracy 20 seconds after input, and 83% accuracy 60 seconds after input. Other results are shown in the table:

The faster a person types, the lower the probability of recognizing the correct sequence of keystrokes. Password recognition accuracy improves for users who tap hard on the keyboard.

The researchers also concluded that ABS plastic keycaps heat up much better than PBT keycaps and are therefore more vulnerable to thermal fingerprinting.

Research Article published September 15, 2022 in the magazine ACM Transactions on Privacy and Security (doi: 10.1145/3563693).

As for the practical implications of the study, in public places, it is better to cover the keyboard not only while entering the password, but also for 60 seconds after that. Since thermal imagers have moved into the category of inexpensive mass-produced devices, attackers can use them for espionage.

Learn more about the benefits of a code signing certificate here: www.globalsign.com/ru-ru/code-signing-certificate