Eavesdropping on Phone Taps from Voice Assistants

The microphones on voice assistants are very sensitive, and can snoop on all sorts of data:

In Hey Alexa what did I just type? we show that when sitting up to half a meter away, a voice assistant can still hear the taps you make on your phone, even in presence of noise. Modern voice assistants have two to seven microphones, so they can do directional localisation, just as human ears do, but with greater sensitivity. We assess the risk and show that a lot more work is needed to understand the privacy implications of the always-on microphones that are increasingly infesting our work spaces and our homes.

From the paper:

Abstract: Voice assistants are now ubiquitous and listen in on our everyday lives. Ever since they became commercially available, privacy advocates worried that the data they collect can be abused: might private conversations be extracted by third parties? In this paper we show that privacy threats go beyond spoken conversations and include sensitive data typed on nearby smartphones. Using two different smartphones and a tablet we demonstrate that the attacker can extract PIN codes and text messages from recordings collected by a voice assistant located up to half a meter away. This shows that remote keyboard-inference attacks are not limited to physical keyboards but extend to virtual keyboards too. As our homes become full of always-on microphones, we need to work through the implications.

Tags: academic papers, eavesdropping, privacy, side-channel attacks, surveillance

Posted on December 22, 2020 at 10:21 AM • 22 Comments

Comments

Chuckb • December 22, 2020 11:57 AM

Based on reading the paper that generated this story, no Alexas were harmed – or used- in the research.

internet individual • December 22, 2020 12:15 PM

Pretty slick stuff. One potential countermeasure could be using the pattern method (android) rather than punching in a pin to unlock the phone. I am not sure what an attacker could do with a pin, without having access to the actual device. If the attacker has access to the device than its game over anyways right? Still it might be worth learning someones 4 digit pin anyways, as I would imagine its the same as most peoples ATM codes.

humdee • December 22, 2020 1:42 PM

Modern tanker ship have been outfitted for years with all these microphones as well as digital data recorders. You can now hear the crew scream as they drown when their ship sinks. Consider the privacy implications of that!

Chilling stuff, really.

Clive Robinson • December 22, 2020 2:35 PM

@ ALL,

Whilst this was done with a relatively small microphone array remember for certain types of measurment things improve with array size.

That large screen Smart TV you have that has a microphone in each corner thus can have better discrimination.

But the big problem the authors realy had was “key press recognition” that is pulling real key click data from the ambient noise.

Whilst they partially overcame the issue, the addition of a triggering or correlation signal would improve things quite a bit bassed on the figures they give.

There are several sensors that could be used. The easiest is to pick up bursts of CPU activity in the smart device, this is generally not that hard to do in the near-field of the device.

But there are other attacks possible. When a key is pressed there is often quite a bit of LCD screen change on a smart device. That is the spell check bar above the virtual keyboard makes quite a bit of difference to the light intensity rather rapidly in a fairly predictable pattern.

A look at various papers on “Optical TEMPEST” indicates that in a domestic environment in the evening, the lower background illumination would make this a fruitful area to investigate to get a trigger or correlation signal.

Faustus • December 22, 2020 3:29 PM

I frequently have a phone on me. My computer microphones are disabled except on my designated comm computer. I have no other microphones. The need seems small. I did have a voice tv remote in a hotel once. It seemed convenient. It was sort of fun. But I’d never get one myself.

Is there a really good reason to have these microphones in your home? I assume that the audio is.mined and the results put in a profile. Does anybody have links about how audio eavesdropping is used?

Does anyone know of systems designed to override one’s profile with garbage audio data?

I also wonder about the mining of camera feeds that are accessed by app over the internet. Does it happen? Is it documented somewhere? Can it be subverted?

I take these attacks on my privacy as an assault. It seems that the best responses are obfuscation attacks to.insert garbage data and adversarial attacks on the data mining software. These seem to me to be legal and proportionate.

People complain about invasion of privacy but I see few ideas for large scale.responses. I do like Ad Nauseum that blocks ads but then clicks on them to deplete ad budgets.

Are there any ideas or references for pollution attacks on surveillance of audio or video feeds?

Joe • December 22, 2020 4:40 PM

Why does the microphone have to be always on? Couldn’t Apple include another microphone that listens for the start command, and turns the device on when the command is spoken, and turns the device off after a bit of silence?

Clive Robinson • December 22, 2020 7:07 PM

@ ,

Is there a really good reason to have these microphones in your home?

If you understand technology and you are a rational person then no, there is no reason when you balance with the known and assumed risks.

But, not everyone understands technology or behaves rationally, some are in effectct status / vanity driven, thus well over estimate the slim benifits over the very real risks, or just do not give them a thought.

The “status” of having a slave awake for every on of the 24hours of the day, just to do your bidding upon command for some is a status above any other. Thus they not just desire but crave such status.

Which when you think about it is quite sad, as they are in effect selling theory their souls to the Devil, for effectively nothing…

lurker • December 22, 2020 7:45 PM

From the paper: distance from victim device to attack array: 15 ~ 20cm, or for some in the Western hemisphere, 6 ~ 8 inches, or was I reading the wrong table? Here, let me shove this swab down your throat…

Clive Robinson • December 22, 2020 10:38 PM

My above at 7:07 PM was ment for @Faustus.

Goat • December 22, 2020 10:58 PM

“The ‘status’ of having a slave awake for every on of the 24hours of the day”

Think about the number of slaves these tech giants have manufactured from humans to bots. Facebook has so many slaves clicking the like button at any given point of time, amazon has yet another army scanning store shelves(Lightning Deals) they can compel them to just get a device(toy) to ease their “busy” lives while they are milked further for cents.

@Clive, re:not everyone understands technology
While everyone should understand such things, even if they don’t… If they open their minds for a second they will realize what’s going on.

To Qoute Rudyard Kipling(hugely relevant here)

I keep six honest serving-men
(They taught me all I knew);
Their names are What and Why and When
And How and Where and Who.
I send them over land and sea,
I send them east and west;
But after they have worked for me,
I give them all a rest.

I let them rest from nine till five,
For I am busy then,
As well as breakfast, lunch, and tea,
For they are hungry men.
But different folk have different views;
I know a person small
She keeps ten million serving-men,
Who get no rest at all!
She sends em abroad on her own affairs,
From the second she opens her eyes
One million Hows, Two million Wheres,
And seven million Whys!

TAP ME HARDER !!!! • December 22, 2020 11:07 PM

TAP ME YEAH
YEAH YEAH YEAH

YEEEEEEEEEEEEAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHH

oooooooooooohhh..

i came.

Clive Robinson • December 23, 2020 3:32 AM

@ Goat,

Ahh “Good Queen Vic” and her industrious ministers and minions, falling for the corporate propaganda line.

Kippling saw one side of Empire, that is the ordinary woman and man being exploited. What he apparently did not see was the likes of the Cecil Rhodes view, that is of the corporate raider or merchant venture.

Cecil’s techniques was to engineer a situation that then got blown up in the press by what we would now call “fake news”. This created much “stuff and humbug” in the press and thus the public which inturn caused “questions in the house” so that Government was forced to act…

Thus Cecil Rhodes had the power to divide up Africa in a way that made trible warfare a certainty then armed the minority side and put then in power, which made them dependent on “the white man”.

If you read other of Kiplings works you will see one exhorting the Americas to take up the white mans burden[1]. With in which you will find that Kipling was aware that between the Lords Corporate and the distant natives, were men cursed from above and below.

As I’ve indicated before history teaches us lessons if we are but to look.

What Cecil Rhodes did more blatently more than a Century ago the Big Silicon Valley Corps are trying to do not just in the physical world but the new “new world” of the information world we call the Internet, or in more gentle times “the information super highway” with all those oh so slow “on ramps” with all that whiney noise we used to call “dial up”. Atleast with dial up you knew when and why you were online, and you could keep an eye on things. Unlike today where a firehose not just deluges but drowns people such that they can not see what nasties jet in and what privacy gets washed out like strip mining.

All in the name of free markets…

[1] https://en.wikipedia.org/wiki/The_White_Man%27s_Burden

Petre Peter • December 23, 2020 6:55 AM

Microphones are just smaller phones no need to worry about them being always on. Now tell Alexa what you want for Christmas.

Goat • December 23, 2020 10:10 AM

“Microphones are just smaller phones no need to worry about them being always on.”

@Peter Peter, Leaving aside the factual incorrectness and interpreting the comment in its spirit, This is the kind of thinking that spells doom. Also if you are using a phone ypu can use foss systems or atleast disable ok google/hey siri.

You own a phone because you have to, you own alexa because they sold you..

Spy Mouse • December 23, 2020 10:14 AM

Somehow phone tapping pushed me to a different kind of practice than typing on a touch screen…

Not Ur Serf • December 23, 2020 12:04 PM

@Faustus @Clive Robinson re: microphone “necessity” – two issues more on point for me would be: how to permanently disable microphones (and cameras) in owned devices that are otherwise desirable and useful, e.g., a specific model of TV; a device to detect the near presence of a microphone or camera that is transmitting via WIFI or BT. I strongly suspect that there could be a pattern to such transmissions that would allow detection (within certain bounds of probability) even if those transmissions were encrypted via the usual consumer protocols. Possibly detection could be enhanced by having a microphone in the detection device in order to compare the acoustic and digital signals. I don’t have the hardware/comm chops to design such a device, but I suspect there could be a fairly robust market for it. I’d certainly consider ponying up $100 – 200 for one, if it was reasonably reliable

MikeA • December 23, 2020 12:23 PM

@Not Ur Serf

I suspect that possession of such a bug detector would be grounds for entry on the list of “suspected terrorists” pretty soon (within a year or two of their commercial introduction).

I would not be surprised to outlive the legal possession of electronic parts without permission from the Powers that be.

Karl • December 23, 2020 1:24 PM

You own a phone because you have to

While true enough, there’s still no need to have a mobile phone turned on at all times (hard to tell, though, without removable batteries…) or to carry it everywhere. Some people don’t even have them. Unless hacked by government spooks, the microphones of landline-style phones are known to be inactive when on hook; they can even be connected to VOIP or mobile networks with adapters.

Mobile phones are the opposite: the microphone is enabled unless hacked, which Ed Snowden demonstrated how to do (“Going Black”). One would then plug a headset as needed. FYI, the Librem and PhinePhone do have hardware microphone kill switches, unlike most mobile phones.

Why should anyone use one of those voice assistants? It’s been pointed out as useful for disabled people, but only the most severely disabled would be unable to push a button instead of using a wake word. Many people might be satisfied with the presumably-poorer results of local speech recognition. Funny how none of these systems provides any such options, nor any option for Tor or similar privacy layers.

Clive Robinson • December 23, 2020 2:26 PM

@ Karl,

Unless hacked by government spooks, the microphones of landline-style phones are known to be inactive when on hook

Not exactly…

In the traditional Plain Old Telephone System(POTS) the microphone was of a particular type thay pulled current from the batteries at the exchange that were fed via the large inductance of a relay coil in the exchange that provided the equipment with the “off hook current loop” to the customer and an issolated “switch contact” in the exchange.

At the consumer instrument or phone there was an “in series” hook switch between the line pair and the microphone and headset speaker in the instrument hand set.

The thing is whilst a switch might be an open circuit to DC, it is infact the equivalent of a small value capacitor of between 15-150pF. This is sufficient that an AC current in the gap between Long Wave and Medium wave frequencies can be introduced to the phone line pair via an RF Bridge Circuit of high sensitivity and thus could pick up most of what was being said in a room even thpugh the phone was “on hook”.

Back fourty odd years ago I used to sell “detector boxes” that could pick the signal up without the opperators of the surveilance equipment being aware that they had been detected… Suffice it to say that those buying were not crooks/Criminals in the general meaning, but company executives and various Government departments and agencies all looking for what you might call “rivals”… Obviously “no honour amongst bureaucrats” etc.

Back then it needed only be a broadband RF amp, a common base circuit as a diode detector and a simple comparitor circuit and meter so a threshold could be set. About £3:50 of components that used to sell for £250:00 for a basic unit plus another £50:00 for each additional feature added.

I would not say I was “rolling in it” but it kind of payed more than the full time day job for only half a days work once a month.

Goat • December 23, 2020 6:57 PM

“Funny how none of these systems provides any such options, nor any option for Tor or similar privacy layers.”

@Karl,
You don’t need tor if it’s not sent over the network, Almond is a good option for those who actually need it.

https://almond.stanford.edu/

👺 • December 24, 2020 3:06 PM

I wish I had a pet frog 🐸.

quantry • December 24, 2020 9:42 PM

@ Karl,

there’s still no need to have a mobile phone turned on at all

this practice does seem to be a factor in the motive behind non-removeable batteries: The unit IS powered-up, AND responsive to all manner of stimulations, and calls “home”.

But too many disconnects from the mothership seems to ‘justify’ permanent denial of service: Unrecoverable from this side of the iron curtain of tech-support. “Only a criminal would insist on that level of privacy”, seems to be the assertion.

Driving a thumb tack thru the mic hole has merits, but the hardware gain contol may underperform on mismatched alternative inputs, it seems: Hard enuf to make an impression with tiny tek: Do it on your burner phones. kyocera duraxv also has a development mode that might have a software solution on the no camera/removeable battery model?

Schneier on Security

Eavesdropping on Phone Taps from Voice Assistants

Comments

Leave a comment Cancel reply