Using Machine Learning to Detect Keystrokes

Researchers have trained a ML model to detect keystrokes by sound with 95% accuracy.

“A Practical Deep Learning-Based Acoustic Side Channel Attack on Keyboards”

Abstract: With recent developments in deep learning, the ubiquity of microphones and the rise in online services via personal devices, acoustic side channel attacks present a greater threat to keyboards than ever. This paper presents a practical implementation of a state-of-the-art deep learning model in order to classify laptop keystrokes, using a smartphone integrated microphone. When trained on keystrokes recorded by a nearby phone, the classifier achieved an accuracy of 95%, the highest accuracy seen without the use of a language model. When trained on keystrokes recorded using the video-conferencing software Zoom, an accuracy of 93% was achieved, a new best for the medium. Our results prove the practicality of these side channel attacks via off-the-shelf equipment and algorithms. We discuss a series of mitigation methods to protect users against these series of attacks.

News article.

Tags: machine learning, side-channel attacks, smartphones

Posted on August 9, 2023 at 7:08 AM • 10 Comments

Comments

Ted • August 9, 2023 8:15 AM

The lead author of the paper, Joshua Harrison, acknowledges this is a very single-use model, in response to a Twitter/X comment that the attack was trained on a single laptop.

SG: “… Seems like a very limited attack, so far as it goes, requiring a lot of prep and physical access… “

JH: “… News articles using my work to claim “AI knows what you type” are generalising my very specific example… “

Still, as Bill Toulas noted, it’s not a bad reason for using biometric authentication and a password manager.

modem phonemes • August 9, 2023 9:20 AM

@ Ted

Joshua Harrison, acknowledges this is a very single-use model

…

Bill Toulas noted, it’s not a bad reason for using biometric authentication and a password manager.

Which is a very general use model 😉

Keith Douglas • August 9, 2023 10:21 AM

Even if it is not too practical now, this is presumably the beginning of such investigations, not some sort of “final state” – whatever that would be.

iAPX • August 9, 2023 12:38 PM

As stated by @Keith, this is just the beginning.

Some people were fooled about 2 things: Zoom and the specific laptop.

Zoom is not the problem, and in fact it had noise cancelling features that protect from it.
They also used a smartphone with slight success too.
Every microphone in your environment could pick your typing and with help of AI (and dictionary/statistics) recover part of what you are actually writing.

Secondly, this is not a matter of a specific laptop, even if this one was sold in millions with the same exact keyboard, it’s not a noisy keyboard and the AI could have been trained on other laptop models with similar results.
And you have to consider that in some companies, as mine, there’s one or two model of laptops, also they might have the same keyboard too!

They did an incredible work to obtain these 90%+ results, and it’s just the beginning.
That’s totally problematic if you think at all the microphones around us, the ability to exploit some remotely, including cloud “services”, and also in open spaces how many people could record your typing.

Mexaly • August 9, 2023 1:52 PM

This gives, “The walls have ears,” a whole new meaning.
We need only continuously update our countermeasures.

JonKnowsNothing • August 9, 2023 2:24 PM

@Mexaly, All

re: The walls have ears

WikiP

The Thing, also known as the Great Seal bug, was one of the first covert listening devices (or “bugs”) to use passive techniques to transmit an audio signal.

It was concealed inside a gift given by the Soviet Union to W. Averell Harriman, the United States Ambassador to the Soviet Union, on August 4, 1945.

Because it was passive, needing electromagnetic energy from an outside source to become energized and active, it is considered a predecessor of radio-frequency identification (RFID) technology.

The Thing consisted of a tiny capacitive membrane connected to a small quarter-wavelength antenna; it had no power supply or active electronic components.

===

ht tps://en.wikipedia.o r g/wiki/The_Thing_(listening_device)

(url fractured)

Clive Robinson • August 9, 2023 5:20 PM

@ Mexaly, ALL,

Re : Unknown-unknowns are not known-knowns.

“We need only continuously update our countermeasures.”

To countermeasure what?

If they know an attack method that you do not the chances are moderate that you can not come up with a countermeasure…

The history of cold-war state level espionage says that in a lot of cases they might as well have just left the file cabinates unlocked and the front door open, for all the good the counter surveillance security measures put in place did…

It’s a game where the thoughtful attacker has significant advantage against the “military way of thinking” defender.

@ JonKnowsNothing, all,

Re : “The Thing” Great Seal Bug.

“connected to a small quarter-wavelength antenna”

It was not a “quater-wavelength antenna”.

Have a read of Peter Wright’s “Spy Catcher” book it goes into some considerable technical depth about what was called “The Thing”.

It used a cavity resonator set up to do “slope modulation” to get an AM signal of sufficient depth.

Mike • August 9, 2023 10:09 PM

Yet another example of tv technology brought to real life.

I remember an example of this from Spooks(MI-5 in the US) where the spys gave bugged cufflinks to a target. They then got the target to type in a known piece of text, a resume. After that the spys could read everything the target typed.

I tried to google a clip but no luck.

RobertT • August 14, 2023 5:18 AM

Machine learning is a funny area
I remember, many years age wondering why our handwriting to text software couldn’t figure out that I was writing a T (lots of them in my handle)
I screamed at the screen ffs it’s “t” but neither the screen nor the software listened.

What was my problem???
well this is when I learned an important lesson, culturally in most prats of Asia the cross stroke on a T comes before the down stroke.

I’m like, no f’ing way, but turns out I’m wrong that’s just how it is. Now with wrt ML it turns out that the ML software developers had better access to Chinese writing citizens writing English than actual English or American citizens writing English, so the software strongly favors the Chinese writing style (cross before down) in all handwriting analysis software.
None of this has anything to do with keystroke patterns/timing but I’d be extremely surprised if similar Asian vs Western keystroke timing differences didn’t exist.
I’ve seen young girls walk down the streets of Shanghai testing with both thumbs at a phenomenal rate. whereas I’m picking with my pointer finger and getting half the characters wrong because the algorithm strongly favors thumbs….precision has little to do with the selected character.

Clive Robinson • August 14, 2023 7:06 AM

@ RobertT,

Re : All fingers and thumbs.

“whereas I’m picking with my pointer finger and getting half the characters wrong”

I’ve found it matters not if I use finger or thumb, it comes out wrong…

I got so annoyed by it –after all my fingers could not be that fat– I actually set up a high frame rate camera and took a video of my typing…

As I thought the right keys came up highlighted on the virtual keyboad as I tapped them, but the wrong letter appeared on the screen.
I found two basic faults,

1, The wrong letter displayed, was an adjacent letter usually to the left.

2, If a top row key was pressed it often pulled a word from the spell check line above it.

After thinking about it I realises there was probavly a bias in their that favoured right index finger users not left.

I’m guessing they map the shape of the contact shape and make an incorrect average or similar.

But it’s not just asian girls that wizz the virtual keyboard a friends husband can wizz the keys at near normal talking rate, and actually takes meetings transcripts…

So yes you and I belong in the Dinosaurs School for typing…

Schneier on Security

Using Machine Learning to Detect Keystrokes

Comments

Leave a comment Cancel reply