Smartphone Keystroke Logging Using the Motion Sensor

Clever:

“When the user types on the soft keyboard on her smartphone (especially when she holds her phone by hand rather than placing it on a fixed surface), the phone vibrates. We discover that keystroke vibration on touch screens are highly correlated to the keys being typed.”

Applications like TouchLogger could be significant because they bypass protections built into both Android and Apple’s competing iOS that prevent a program from reading keystrokes unless it’s active and receives focus from the screen. It was designed to work on an HTC Evo 4G smartphone. It had an accuracy rate of more than 70 percent of the input typed into the number-only soft keyboard of the device. The app worked by using the phone’s accelerometer to gauge the motion of the device each time a soft key was pressed.

Paper here. More articles.

Tags: , , , , , ,

Posted on August 23, 2011 at 2:09 PM • 27 Comments

Comments

Rick Auricchio • August 23, 2011 2:59 PM

This isn’t a new invention, but an interesting application of the technology nonetheless.

Apple’s GarageBand app uses accelerometer info to adjust note velocity as one plays the soft music keyboard.

Scott • August 23, 2011 3:22 PM

Clever, but seems like too low an accuracy to have any practical value.

If you are trying to figure out a repetitive value like the unlock code, it seems like you would have enough trials to zero in on two or three possibilities.

mn • August 23, 2011 3:35 PM

Too easy to defeat – now the phone just need to vibrate so slightly and radomly to defeat this

jb • August 23, 2011 3:57 PM

@mn – So make sure you always shake your phone while punching in your passwords? Isn’t that a little impractical?

Clive Robinson • August 23, 2011 4:14 PM

What I like about the story is the way it showes the law of “unintended consequences” in secure system design that an attacker uses to “do an end run” around a mandated security policy.

In smart phones a lot of the OS goes into ensuring that any entry on the keybaord input goes only to the application that has the windows system focus.

The designers of the system did not realise that the accelerometers used had a sufficient degree of sensitivity and frequency to enable such a thing to happen.

As I’m often heard to note “efficiency is the enemy of security” (not that the converse is true) in that as we make systems more efficient in this case expressed as a usability feature we open up more and more side channels of various forms.

Making even simple single purpose designs secure against side channels is very very difficult, trying to do it with an all singing and dancing system with a myriad of value added features is making the life of a security engineer dificult at best.

Now a thought for all the boys and girls out there, smart phones have started to become a significant target of crackers for various reasons.

Now Microsoft and Apple have designed smart phone OS’s based on their mainline OS’s, have you stopped to think what this means?

The chances are that a bug discovered in the mobile OS has a sibling bug in the mainline OS…

Now it appears that the mainline OS of the smartphone world is Apples offering, unlike in the mainline PC world where it is Microsoft.

What are the chances that as we see a rise in attacks against the Apple Smart Phone OS we also see a rise in the Apple mainline OS attacks?

That is will the iPhone/Pad popularity actually make Apple a sufficiently viable target that the spillover will be seen by MAC users who will be more or less totaly unprepaired for it…

Clive Robinson • August 23, 2011 4:31 PM

Oh and another thought for people.

If this works with the accelerometers on the phone can you think of any other sensor that might be just as sensitive if not even more so?

I can and the chances are there is already software in the phone that could be exploited to use it…

I’s the camera, think about just how much camera shake the typing would induce. All the software would need is a single point of fixed refrence in the camera field of view and it would work just like a “mirror galvonometer” in reverse, and mirror galvos where famed for their sensitivity…

David • August 23, 2011 6:03 PM

@Clive: Many (most? all?) phone cameras have some sort of indication (light, typically) when they are running, which may at least be more discoverable than something sampling the accelerometer in the background.

In either case, it’d run down the battery faster (noticeably if it were running all the time, less so if it ran only when something had the keyboard enabled but I’m not sure whether that’s doable without access in the device that would make this attack redundant anyway).

Richard Steven Hack • August 23, 2011 6:12 PM

Clive: Re iPhone OS vs OS X. I’m not sure the two are that comparable in function, are they?

Wikipedia says “iOS is derived from Mac OS X, with which it shares the Darwin foundation.”

Android is based on Linux, where the comparison might be more apt – or not.

But while presumably some low level functions are the same or similar enough, I’m not sure the phone-specific functions would be similar to enough of the desktop functions to allow for exploits to be easily ported.

For example, the use of accelerometers isn’t likely to be found in a desktop. I find it odd that smart phone design even includes that sort of “motion interface.” That’s just bizarre to me that anyone would want to shake their phone to use it. I don’t even like browser “gestures”. 🙂

I can see the possibility. But I would expect attacks directed at the mobile market to take precedence over porting such attacks to the desktop and vice versa. If it happens at all, it will probably be as an afterthought and not really increase the risk on the desktop much, which is already rising slowly.

I read an article recently that asked where all the Mac exploits that were predicted months ago are. Apple had some bad publicity over some malware a while back and everyone predicted an “explosion” of Mac malware. But it just hasn’t happened. Instead it’s apparently all gone toward the mobile phones, not the desktop.

Which makes sense since there are probably more Apple phones than there are Mac desktops, or will be if there aren’t already. And since the phones are less secure, apparently, because they are a newer platform, and also a new more active market than desktop software (as well as being easier to develop for since apps are smaller) it’s smarter to attack them.

In any event, I think it will be hard to tell if any rise in Mac malware is due to porting from iOS vs a general rise in Mac malware. It might be easy to tell if new malware appears on iOS and then moves to OS X later, but if it appears in both platforms at once, it might not.

It will also be interesting to see whether Android malware exceeds Apple malware (given that Android has the larger market share) and whether any porting between mobiles and desktops occurs more on Android or more on iOS.

There may be effects due to Android developers working on more variants of the OS vs iOS developers have to contend with.

Anonymous 1 • August 23, 2011 7:20 PM

Maybe something like the selective availability bug GPS once had might be able to prevent applications from doing things they shouldn’t be doing with the motion sensor information (with the user able to trust certain applications to get the fully accurate information).

Very few programs running on a phone really need the motion sensor input anyway (and even fewer need it when they don’t have input focus).

c • August 23, 2011 7:47 PM

@Anonymous1 “…something like the selective availability bug GPS once had…”

feature not bug. it was intentional

RobertT • August 23, 2011 8:13 PM

The good thing about these obvious side channels, with in build hardware support is that it tells you exactly where information security belongs on the list of care abouts, which is somewhere after a fairly useless wow feature.

The amazing thing is, when you sell parts to these guys, you will get grilled over obscure details about the “randomness” of an RNG and sensitivity to power noise locking. you’ll find belt and braces fixes for problems that do not really exist, while at the same time they build a directly accessible 3 axis accelerometer into the product and somehow think it wont get used as a hacking support tool, incredible!

Anonymous 1 • August 23, 2011 9:16 PM

c: I’m quite aware of that (as well as that it worked by deliberately changing time and ephemeris values sent by the satellite).

Roger Wolff • August 24, 2011 12:43 AM

@Richard Steven Hack: the accelerometer is there to allow the phone to sense which side is up. This is used to rotate the display when you hold it differently. And it is used to tag photos with the right side “up”.

Now, you might be right in that just a 3-bit output would suffice: left-is-up, top-is-up… 6 options. But in fact the whole accelerometer is accessible. This allows extra features that “come for free” if you need the “which side is up”. But apparently also some side-channels.

tommy • August 24, 2011 2:24 AM

What if she holds her vibrator in her hand with the phone on top of it, and runs the vibrator while using the keypad?

bob • August 24, 2011 3:55 AM

@ Tim

Clever, but seems like too low an accuracy to have any practical value.

If “she” enters her password 10 times you’d have the password even at 70% accuracy.

David • August 24, 2011 9:55 AM

Of course, the real solution is not to rely on passwords. What percentage of people actually type secure ones on their phone anyway?

TS • August 24, 2011 10:04 AM

“Of course, the real solution is not to rely on passwords. What percentage of people actually type secure ones on their phone anyway?”

A lot. Many don’t trust their computers thanks to malware, so do online banking over their smartphones (connected to secure WiFi or over 3G).

Webmail, facebook, twitter, people do quite a bit with their phones.

Bryan Feir • August 24, 2011 11:57 AM

@Richard Steven Hack:
“Android is based on Linux, where the comparison might be more apt – or not.”

Mostly not.

Android may be using the Linux kernel, but the userspace system is completely different, even if you don’t count the fact that the primary Android userspace is Java. Low-level things such as the application loader (the code that checks which libraries the application needs and fixes any jump calls from the app to the libraries) are quite different on Android. Part of that seems to be so that most of the necessary system libraries are pre-loaded into RAM in a block, allowing the system to optimize memory usage a bit better.

Richard Steven Hack • August 24, 2011 3:11 PM

Roger: “to allow the phone to sense which side is up. This is used to rotate the display when you hold it differently. And it is used to tag photos with the right side “up”.”

Like anyone really needs those “features”, which is my point.

Bryan: Thanks for the clarification. I know next to nothing about Android or iOS.

JimFive • August 24, 2011 4:02 PM

@RSH, re: “Like anyone really needs those “features”, which is my point.”

Of course they don’t need them, just like they don’t need a smart phone, or even a cell phone.

The fact is that allowing the device to determine which side is up is very convenient. Allowing the entire device to be used as e.g. a game controller is pretty cool. The mistake was in not extending the “no keyboard without focus” idea to “no user input without focus” or even “no input without focus” (although that might kill multitasking)

JimFive

Richard Steven Hack • August 24, 2011 4:20 PM

JimFive: I don’t see it as an “either-or” issue, i.e., whether adding features is on a par with creating the platform in the first place.

The problem is endemic to the technology industry: throw in “features” the consequences of which and the number of people who will actually use them being completely out of consideration.

It’s really done as a selling point. The notion is if you accumulate enough selling points, everyone will buy your device.

i.e., your comments: “is very convenient..is pretty cool.”

But again you end up with the consequences of: 1) bloat; 2) unreliability; 3) insecurity; 4) expense; 5) poor usability.

Is this really a good notion? I think not.

And THAT’s where the mistake lies – not in some minor detail about forgetting about “input vs focus”.

As some people like to say, just because you CAN do something doesn’t mean you SHOULD.

Nick P • August 24, 2011 11:07 PM

@ tommy

“What if she holds her vibrator in her hand with the phone on top of it, and runs the vibrator while using the keypad? ”

LOL. Yeah, that would probably work. And the reference to the Villiage People was uniquely appropriate. They’re probably experts on the use of such products.

tommy • August 25, 2011 2:28 AM

@ Nick P.:

Thanks – and for catching the V-P ref. (Vibrators were also referenced directly.)

Seriously, it seems like it should mess up the attack. I’m sure those who read it took it as totally facetious, but wouldn’t it actually block the attack? Can the vibration-measuring experts say yea/nay?

AFC • August 26, 2011 10:10 AM

Well, this certainly proves that access to the accelerometer must be secured (along with the keyboard and camera).

@Richard Steven Hack: the problem you describe is endemic in the entire consumer market, not just gadgets. Here’s an example from a completely different area: the gym I go to has no duplicate machines, because every new machine type is a selling point — so that the most popular machines are always busy, while many exotic “selling point” machines are always idle… (Sound familiar?)

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.