Smart Watch that Monitors Typing

Here’s a watch that monitors the movements of your hand and can guess what you’re typing.

Using the watch’s built-in motion sensors, more specifically data from the accelerometer and gyroscope, researchers were able to create a 3D map of the user’s hand movements while typing on a keyboard.

The researchers then created two algorithms, one for detecting what keys were being pressed, and one for guessing what word was typed.

The first algorithm recorded the places where the smartwatch’s sensors would detect a dip in movement, considering this spot as a keystroke, and then created a heatmap of common spots where the user would press down.

Based on known keyboard layouts, these spots were attributed to letters on the left side of the keyboard.

The second algorithm took this data, and analyzing the pauses between smartwatch (left hand) keystrokes, it was able to detect how many letters were pressed with the right hand, based on the user’s regular keystroke frequency.

Based on a simple dictionary lookup, the algorithm then managed to reliably reproduce what words were typed on the keyboard.

Tags: algorithms, concealment, eavesdropping, key logging, sensors

Posted on September 18, 2015 at 5:20 AM • 34 Comments

Comments

Greg Surplus • September 18, 2015 7:01 AM

Another reason to consider greater use of 1-time passwords, multi-factor authentication or using login alternatives such as SQRL.

Moo • September 18, 2015 7:14 AM

There’s a movie plot in this!

AJWM • September 18, 2015 8:31 AM

@Greg Surplus –
Or at least a password that isn’t in a dictionary, or uses only the right (non-watch hand) side of the keyboard.

Or just take the damned watch off.

Clive Robinson • September 18, 2015 8:34 AM

I wonder if mechanical watches, will become popular in the Kremlin along with those mechanical typewriters?

I wonder if it would still work with the “microwriter” and similar simultanious key press input devices?

Think of just five keys, one for each finger and the thumb of one hand and a simple binary code for letters like Baudot, or apictorial representation of morse.

Alan Kaminsky • September 18, 2015 8:40 AM

Or just don’t buy the damned smart watch in the first place, and donate the money to the EFF or your favorite charity.

Clive Robinson • September 18, 2015 9:14 AM

The problem with “not buying the watch” is that eventually it will be a case of “not buying any watch”.

I remember when phones did not have, cameras, gps receivers, gravitometers and other things that have been used to spy on us. Now you would be hard pushed to find a smart phone without any of those, and even harder pushed to not buy a smart phone…

Like those “fit strip” etc “health & excercis” gizmos accelerometers etc are going to move into “bodywear” like watches, glasses, jewelry and clothing etc. It’s something that will take maybe a decade at most before you are considered “odd” for not having them, such is the nature of progress in this part of the consumer industry.

Sara P. • September 18, 2015 9:16 AM

I like Alan’s plan. Why would anybody buy something like this?

Mace Moneta • September 18, 2015 9:19 AM

Sure they made fun of me because I type (mostly) one handed instead of touch-typing. Who’s laughing now? 😉

Alan Kaminsky • September 18, 2015 9:49 AM

@Clive, I disagree with your prognostication. I say non-smart watches will always be available, and non-smart phones will always be available. As technology gets cheaper, what happens is that devices get more powerful but stay the same price or even increase in price, not that devices stay the same but get cheaper. There will always be a market for low-end “dumb” devices for those who can’t afford or don’t want high-end “smart” devices.

I also disagree with your assertion that everyone is “harder pushed to not buy a smart phone.” I used to have a smart phone, but I got rid of it once I realized I never used its “smart” features, and once I realized it was just a Google spy device (Android). Now I have a dumb cellphone, I am perfectly happy with it, and I feel no pressure to go back to a spyware-phone.

Also, I don’t have a watch, either smart or dumb.

David Leppik • September 18, 2015 9:56 AM

If you have a watch that can tell what you’re typing, you don’t even need a keyboard. (In “Rainbows End” by Vernor Vinge, this is exactly how people typed into computers.)

But it’s nowhere near that good. The problem with this sort of Bayesian guessing is that, at the end of the day, it’s still an educated guess. They have no idea what you’re doing with your right hand, and they aren’t completely sure what you’re doing with your left. They’ve got your A’s and E’s, but none of the other vowels. If you think auto-correct and auto-complete are error prone, this is worse.

At the end of the day, it’s going to be easier to bug the keyboard or hide a camera in the room.

John Macdonald • September 18, 2015 9:57 AM

@Alan

While you will long be able to buy a “non-smart” phone, it won’t be too long before the cheap dumb phones are actually build from smart phone components that simply pretend that the extra features aren’t there and don’t present them to the user. The smart capabilities will be integrated onto a single chip that is cheaper to buy than a chip that is missing them (because of lack of demand).

Adrian • September 18, 2015 10:38 AM

I, too, switched from a smart phone back to a flip phone. I almost got the one without the camera, but the store was out of stock on those.

I wouldn’t have a cell phone at all if my wife would let me. Cell phones are easily the least reliable bit of tech I’ve ever used. I have no idea how people can grow so attached to a device that so rarely works. I routinely forget to take mine along, and some days I forget to turn it on.

I’m not that worried about the capabilities of wearables if they’re all going to depend on a cell phone hub.

Anura • September 18, 2015 10:45 AM

@Clive

I think mechanical watches will always be available, as they are more about jewelry than functionality. No one has really needed a watch since they have gotten cell phones. I used to always wear a watch, but haven’t done so in nearly a decade.

Randal • September 18, 2015 10:52 AM

Squid don’t wear watches.

albert • September 18, 2015 11:02 AM

@Clive,
I owned a cheap Molnija pocket watch years ago. Built like a Russian tank. The company made expensive models as well. They stopped production in aught-seven, and are becoming collectible. I first encountered the name (Molyna) in a spy novel. Tanks for the memories…Gold pocket watches are still made in Switzerland, pricing from, say, a good used car, to a good new car:)

I went to a sensor technology trade show a few years back. Cheaper-to-build and nano-miniaturization were, and still are, the watch words of the industry.
.
I’ve noticed fewer folks wearing watches (except as jewelry), as most everyone (as I) gets the time from their mobile.
.
‘Smart watches’ are a fad, just like Google Glass (you think they’d know better). Yet another way to separate us from our money.
.
@John MAC,
I think the IC would love to see that. I’d guess that anything ‘off chip’ is the main expense in smart phones, with display/keyboard, battery, and antenna being the limiting factors. I imagine a sandwich design, with display, sealed electronics module (no circuit board), battery, and antenna, all glued together like ham & cheese on rye, heavy mayo.
.
. .. . .. oh

Wael • September 18, 2015 11:43 AM

@Mace Moneta,

Who’s laughing now?

People who believe you wear the watch on the hand you type with 😉

Anura • September 18, 2015 12:02 PM

@Wael

Of course, if the watch is on your non-typing hand when you are “one-handed typing” then you get potentially more embarrassing information leaked.

Wael • September 18, 2015 12:36 PM

@Anura,

if the watch is on your non-typing hand

You mean wearing the watch on your non-browsing hand, I’m sure 🙂 Leaked information showing sinusoidal data can mean many things! After all, all things in this world are sinusoidal, so they say…

Clive Robinson • September 18, 2015 1:50 PM

@ Anura, Wael,

After all, all things in this world are sinusoidal, so they say..

Hmm do you mean “feeling the beat” to something like “Willy and the hand jive” from CCS?

CallMeLateForSupper • September 18, 2015 2:05 PM

Duly added to my rapidly growing list of Things whose very existance reveals that the new smart is the old dumb.

How long before the appearance of the first article claiming this Thing “will forever change your relationship with your keyboards”?

Wael • September 18, 2015 2:10 PM

@Clive Robinson,

Been a while since I laughed 🙂 You, sir, are as subtle as a freight train (sometimes.)

CallMeLateForSupper • September 18, 2015 2:17 PM

Come to think of it, my Dumb Wristwatch will be 48 years young next month. Off the top of my head, it’s the oldest thing I own AND still use. Jungle boots (circa 1970) are second-oldest.

r • September 18, 2015 3:50 PM

@clive, i wonder if this can be applied to handwriting and signatures the same way those document storing ink pens work?

this still doesn’t dissuade me from wanting a live bluetooth heartrate monitor though. 🙂

but again these bluetooth devices, even if iOS and android randomize their wifi mac’s the bluetooth mac’s should remain static and trackable?

@albert, google glass may have been a fad but has anyone tried to modify one to use them as night vision goggles yet?

from what i recently learned about webcam’s is that there’s a removable red-filter from the lens – it may only be a minor hack if google glass had some sort of overlay?

Robert in San Diego • September 18, 2015 3:53 PM

Thank you Adrian for the comment on the lack of robustness in cell phones. I work in a call center. I love my customers. I hate their cell phone service providers (and cell phone designers — hate it when the user accidentally presses buttons while talking to me). Oddly, even the least frequent of cell phone users in my family, the flip-phone packing grandparents, don’t have that button press problem. Just my customers….

Whocares • September 18, 2015 5:28 PM

I won’t buy any body-fit products. People use their time on watching smartphone too much.
Of course I know today is a IoT era, but I don’t use smartphone when I’m out.

Clive Robinson • September 18, 2015 7:06 PM

@ Wael,

Been a while since I laughed 🙂

You know that laughing is not just good excercise, it also improve mental function as well as helping release both mental and physical stress. A Dr who is sadly nolonger with us (got to 95 though) once advised me that the way to improve ones life was to “Halve the meat and halve the wheat and tripple up on the greens, drink plenty of water and laugh long and deeply to help it all on it’s way”.

And to help you, the words of British Comedian Ken Dodd who appears to have been around for ever,

“Oh aye, there’s a woman over there with the ready for bed look… She’s just put her teeth in a glass… Speaking of things that come from your face… a laugh ladies and gentleman, I think you will agree, is a beautiful noise… which comes from a hole in your face… from any where else, then you best see your doctor”.

albert • September 19, 2015 9:45 AM

@r,

IR-sensitive Google Glass would be a fascinating product. I can see lots of legal and ethical uses. Google took a lot of shit from ‘privacy advocates’, but really, aren’t the abusers the guilty ones? Google Glass is hardly a hidden device. The use of hidden cameras goes back a long way. Wiki enumerated productive and useful applications of the product. You’re damned if you do, and damned if you don’t. Folks love to hate big companies….

. .. . .. oh

ianf • September 19, 2015 6:15 PM

NEXT THREAT: a Fitbit-like pulse/ heartbeat/ blood pressure/ breathing profile/ skin temperature/ humidity/ electrical resistance/ conductivity sensor embedded in a luxury  iWatch that’s been awarded the target. Paired to a hostile hidden base station that’s capable of recognizing infinitesimal pattern changes in the wearer’s extremities, changes detected IN ANTICIPATION OF about-to-be-typed-in ZECRET passwords, or other critical data that the subject has made sure can not be captured otherwise. Just sayin’

Wael • September 19, 2015 10:31 PM

Poor @tyr won’t comment on this thread because he only has one left hand 😉

Wael • September 19, 2015 10:46 PM

@Clive Robinson,

Hmm do you mean “feeling the beat” to something like “Willy and the hand jive” from CCS?

I mean like the sign on an out of business brothel that says: Beat it, we’re out of business 🙂

Mr zero knowledge • September 20, 2015 12:20 AM

Its safe to say a great many folks on here learned typing on mechanical. It does give life a little more English. like johnny cage, the songs gets better sang with age. Tried for the first time the other day.

ianf • September 21, 2015 8:39 AM

“… would [that watch] still work with the “microwriter“[*] and similar chord-input devices?”

@ Clive, instead of dazzling us time and time again with vintage hardware and software insights, wouldn’t it be simpler if you started chronologically at the beginning, and disclosed—I’m quite sure by now long declassified—Antikythera hacks of yours?

[^*] in 1988 or thereabouts I helped a guy with one of these interface it to a 1200/75 baud modem. He was overjoyed, but a short time after that he up sticks and moved to Copenhagen.

Clive Robinson • September 21, 2015 10:36 AM

@ ianf,

I’m quite sure by now long declassified — Antikythera hacks of yours?

It’s funny you should mention the Antikythera Mechanism, I occasionaly went to visit a friend who operated GB3SM and also help out with some old SiS kit from WWII and onwards due to my involvment with 39(SC)Sigs which originated from those supporting SiS via what became the DWS (and still had spares in store…). Anyway I was pooching around there when I was introduced to Michael Wright, (who probably knows more about the mechanism than anybody else alive). Any way we got chatting and the subject came around to how the “wheels within wheels” worked and the similarity to sun and planet gear trains. Anyway I mentioned that modern astronomers use the addition of sinewaves to predict orbital motion, and how it was related to Fourier Analysis and had a gentle argument as to if Fourier had been beaten to the punch on it…

Any way all that aside I still do keenly await the full analysis that has been promised by various people arising from the scans done by the Uni of Cardiff, even though it was done a decade ago.

ianf • September 21, 2015 11:53 AM

Isn’t there a copy of this 2002 Science Museum-funded reconstruction of the Antikythera Mechanism on display in London? I don’t follow news of it that closely, but sort of wonder… what if the modern boffins discover some fundamental, or non-critical but cumulative, flaw in it, just like what happened during the computer-aided validation stage of the first completed buildup of Charles Babbage’s Difference Engine No. 2 in 2000? Were that to happen even now, we’d perhaps have a workable theory of why these (now-Antikythera) Celestial Navigation Boxes disappeared from the maritime record: they sailed off the edge of the horizon.

Smart Watch that Monitors Typing

Comments

Leave a comment Cancel reply