Recovering Smartphone Typing from Microphone Sounds

Yet another side-channel attack on smartphones: “Hearing your touch: A new acoustic side channel on smartphones,” by Ilia Shumailov, Laurent Simon, Jeff Yan, and Ross Anderson.

Abstract: We present the first acoustic side-channel attack that recovers what users type on the virtual keyboard of their touch-screen smartphone or tablet. When a user taps the screen with a finger, the tap generates a sound wave that propagates on the screen surface and in the air. We found the device’s microphone(s) can recover this wave and “hear” the finger’s touch, and the wave’s distortions are characteristic of the tap’s location on the screen. Hence, by recording audio through the built-in microphone(s), a malicious app can infer text as the user enters it on their device. We evaluate the effectiveness of the attack with 45 participants in a real-world environment on an Android tablet and an Android smartphone. For the tablet, we recover 61% of 200 4-digit PIN-codes within 20 attempts, even if the model is not trained with the victim’s data. For the smartphone, we recover 9 words of size 7-13 letters with 50 attempts in a common side-channel attack benchmark. Our results suggest that it not always sufficient to rely on isolation mechanisms such as TrustZone to protect user input. We propose and discuss hardware, operating-system and application-level mechanisms to block this attack more effectively. Mobile devices may need a richer capability model, a more user-friendly notification system for sensor usage and a more thorough evaluation of the information leaked by the underlying hardware.

Blog post.

Tags: academic papers, cell phones, phones, PINs, side-channel attacks

Posted on April 1, 2019 at 9:44 AM • 15 Comments

Comments

Mitch • April 1, 2019 3:36 PM

Wonder if/how this works with a “swipe” keyboard?

Jkli • April 1, 2019 3:39 PM

This must assume a standard keyboard layout but various third party programs make that easy to change.

Jesse Thompson • April 1, 2019 4:20 PM

Ah ha, but I use voice dictation so I never even touch the keyboard. CHECKMATE spooks! >:D

Clive Robinson • April 1, 2019 4:20 PM

@ jkli,

This must assume a standard keyboard layout but various third party programs make that easy to change.

Not at all.

Each key press is diferent to the other key presses, but importantly with out other measures it becomes the same noise each time for the same key. This makes it the equivalent of a “simple substitution cipher” like the school child “pig-pen code”. This acoustic substitution cipher and an equivalent gravatometer cipher are well know side channels in certain quaters. And can be broken in as little as 20 keypresses.

There are ways to take counter measures, but unless you get out the wire cutters / soldering iron to disconnect both the microphone and speaker (which can be used as a microphone) your only other options are various types of random noise generator or making the keyboard layout randomised very frequently like every key press…

@ All,

Yes I’m still alive but for various reasons I’m not going to be showing up very often. Thank you to all who have shown concern about me and the others who have dropped off this blog, it has made my ears glow more than a little pink.

Who? • April 1, 2019 4:53 PM

@ Mitch

No, I think it should not work on a swipe keyboard at least not as easily as an accelerometer-based approach. However no one types passwords on a swipe keyboard, up to my knowledge.

An easy workaround to this thread would be putting the keys on the pin pad on a random order each time a pin must be typed.

Who? • April 1, 2019 4:55 PM

I should have typed “threat” instead on my previous post.

0laf T. Hairy • April 2, 2019 4:51 AM

Surely there are easier ways to grab the data than this. More effort put into getting a standard keylogger or screen grab software onto the device would likely pay off better.

Ilia • April 2, 2019 6:18 AM

@Mitch, if interested in acoustic attacks against swipe keyboards read https://arxiv.org/pdf/1808.10250.pdf

Petre Peter • April 2, 2019 7:09 AM

I thought I was supposed to see a popup when an app is trying to access my mic.

1&1~=Umm • April 2, 2019 7:20 AM

@0laf T. Hairy:

“Surely there are easier ways to grab the data than this.”

But with every one ‘grabing’ at the low hanging fruit it quickly ceases to be available, so they then have to work that bit harder to get a ‘free lunch’.

If it’s your tree they are stealing your apples from, maybe knowing how apples get ‘grabbed’ at all levels is something you want to stop, that way they steal your neighbors apples instead, or even go pay for their lunch, likevthey are supposed to do.

1&1~=Umm • April 2, 2019 7:59 AM

@Ilia Shumailov:

Thanks for droping in, it is actually nice when paper authors take part in the thread. Much appreciated.

Ilia Shumailov • April 2, 2019 9:22 AM

@Petre Peter, you are right — it is asking for your permissions, however, as we point in the paper, a lot of users simply give the applications access to everything that applications ask for. And applications ask for everything justifying that at some point in the future they will have a use for it.
Furthermore, one can imagine scenarios with legitimate sensor usage — imagine you talking on the phone with a bank, who asks you to open their app and confirm something. You do so and enter the app PIN (as for example Barclays asks you). If the recording of you typing in your PIN, we can decode this PIN with the knowledge of the phone model.

Cassandra • April 3, 2019 3:23 AM

@Clive Robinson

Good to hear from you. I too have had to cut back for non-optional reasons.

Best wishes and regards (for once, this is not just a formulaic bit of fluff),

Cassandra.

Clive Robinson • April 5, 2019 1:48 PM

@ Cassie,

Many thanks for the kind words.

Sadly life has to move on for all of us and change good or otherwise is part of that.

Heres to hoping there is a good light at the end of the tunnel for all of us.

vas pup • April 5, 2019 4:44 PM

Question:
Could China (Huawey)finally made smart phone with capability of real shutting off microphone/speaker other features by user to eliminate a lot of security weak spots related.
At least for export.

Schneier on Security

Recovering Smartphone Typing from Microphone Sounds

Comments

Leave a comment Cancel reply