Auditory Eavesdropping

In the information age, surveillance isn’t just for the police. Marketers want to watch you, too: what you do, where you go, what you buy. Integrated Media Measurement, Inc. wants to know what you watch and what you listen to—wherever you are.

They do this by turning traditional ratings collection on its head. Instead of a Nielsen-like system, which monitors individual televisions in an effort to figure out who’s watching, IMMI measures individual people and tries to figure out what they’re watching (or listening to). They do this through specially designed cell phones that automatically eavesdrop on what’s going on in the room they’re in:

The IMMI phone randomly samples 10 seconds of room audio every 30 seconds. These samples are reduced to digital signatures, which are uploaded continuously to the IMMI servers.

IMMI also tracks all local media outlets actively broadcasting in any given designated media area (DMA). To identify media, IMMI compares the uploaded audio signatures computed by the phones with audio signatures computed on the IMMI servers monitoring TV and radio broadcasts. IMMI also maintains client-provided content files, such as commercials, promos, movies, and songs.

By matching the signatures, IMMI couples media broadcasts with the individuals who are exposed to them. The process takes just a few seconds.

Panel Members may sometimes delay watching or listening to a program by using satellite radio, DVRs, VCRs, or TiVo. IMMI captures these viewings with a “look-back” feature that recognizes when a Panel Member is exposed to a program outside of its normal broadcast hour, and then goes back in time (roughly two weeks) to identify it.

These cell phones are given away to test subjects, who get free service in exchange for giving up all their privacy.

I’m sure the company will claim not to actually eavesdrop on in-room conversations, or cell phone conversations. And just how different are these special phones, anyway? Can the software be installed on off-the-shelf phones? Can it be done without the owner’s knowledge or consent? The potential for abuse here is enormous.

Remember, the threats to privacy in the information age are not solely from government; they’re from private industry as well. And the real threat is the alliance between the two.

Tags: , ,

Posted on December 19, 2006 at 6:54 AM • 42 Comments

Comments

Wyle_E • December 19, 2006 7:26 AM

I wonder how IMMI informs the recipients of the ‘free’ phones that these phones are really room bugs. Does the company rep actually say it out loud, or is that fact buried, in Ultra-Eyestrain type, at the bottom of several pages of soporific lawyerspeak in the service agreement? I’m drawing on my Navy experience to compose a suitable reply to any salesdroid who pushes one of these items at me.

C Gomez • December 19, 2006 8:06 AM

Look, I’m all for privacy and its protections, but people are signing up for this stuff. Yes, marketers increasingly ask us to give up privacy in exchange for a goodie, but I don’t recall ever being forced. I could always choose not to participate.

Anon 31 • December 19, 2006 8:17 AM

@C Gomez

But that’s just the point with this setup….even if you choose not to participate I (or someone else in the room) might have. Then YOUR privacy is gone as well. What if your child signed up for this (or wife, or co-worker) without telling you they now had a bug on them at all times?

paul • December 19, 2006 8:23 AM

An important point, I think, is that it’s not just the ratings “volunteers” who are giving up their privacy, it’s everyone they may associate with. Family, friends, business contacts, everyone in the same elevator, subway car or airline waiting room…

Ideally you’d want each of these people to wear, say, a yellow badge warning others that what they’re saying might be bugged.

Incidentally, it seems to me that the data gathered by these phones isn’t going to be terribly useful for its stated purpose — you’re going to get a particularly skewed sample of the country, which even the most aggressive adjustments won’t entirely fix. And between portable radio players at low volume and DVRs left on during slack periods the system should be ripe for gaming.

Of course, you’ll get a much better sample once a couple of major cell carriers sign contracts to deliver this service to all of their customers.

wade • December 19, 2006 8:30 AM

This bothers me:

“The IMMI phone randomly samples 10 seconds of room audio every 30 seconds. These samples are reduced to digital signatures, which are uploaded continuously to the IMMI servers.”

What kind of “digital signature” algorithm could reduce 10 seconds of audio to a key that would aid in distinguishing a music program from, say, the sound of a toilet flushing?

Limboduck • December 19, 2006 8:35 AM

1) Get one (or several!)
2) record an advert
3) put your MP3 player onto repeat
4) Leave them both turned on in your garage.

Confused • December 19, 2006 8:36 AM

Not sure I understand how effective this could be. I can’t imagine that a digital signature is going to be able to generate effective points of comparison between sound samples. There is too much ambient noise to generate a reliable signature (e.g. refrigerators, ringing phones). In addition, each room is going to have unique acoustic characteristics based on room size and clutter.

Ed T. • December 19, 2006 8:43 AM

“The potential for abuse here is enormous.”

And all it would take is someone couching it in terms of “protecting children from sexual predators”. We could than have various sound patterns associated with sexual abuse of children digitized (who would want to take part in such an exercise? ewwwww.) and the signatures stored away in a database, and then everyone would be required to wear one of these devices at all times – with failure to wear the device being a Class 3 felony, and automated presumption of guilt for either (a) sexual abuse of a child, or (b) high treason.

~EdT.

kashmarek • December 19, 2006 8:49 AM

What makes anyone think that the individual carrying such a device is giving up their rights? They are being paid (by free services) not necessarily to gather data about themselves but to gather data about everyone else (whether they realize it or not). Recording of conversations (the 10 second sample) without the permission of others involved, would be construed an invasion of privacy and possibly illegal (at least in some circumstances). Even at my workplace, and others I presume, you cannot use a cell phone camera (or any camera) to take photographs. Stealth recording of conversations is a problem.

Vicki • December 19, 2006 8:49 AM

C. Gomez:

Yes, people are choosing to sign up for this. Some of them presumably are doing so because they haven’t thought through what it means, in which case it’s more than reasonable for Bruce to be pointing that out.

Also, as Anon 31 points out, if I walk into a room, I don’t know whether someone has one of those. Most of us are calmer about the possibility of one or a few random strangers overhearing bits of personal or business conversation while thinking about their own affairs than about it possibly being recorded and stored for use by dozens or thousands of other strangers.

paul • December 19, 2006 9:12 AM

And remember, even if IMMI’s heart is pure as the driven snow, their database is going to be a delightful target for hackers. And even stored in compressed form (I’m assuming the ‘digital signature is some kind of hash/compression of the 10 seconds of captured audio, likely rather longer than 128 or 256 bits) the eavesdropping data would be subject at the very least to known-plaintext attacks.

Andy Dingley • December 19, 2006 9:15 AM

How do I get one of these things?
Some friends have a band that could use a deliberate plugging….

Isn’t any sort of snoop-o-matic like this just asking for an exploit of deliberately feeding it?

Z • December 19, 2006 9:18 AM

Sounds like an excellent alpha test for how well this might work in a covert mass monitoring situation. I had a cell phone for years, but I got rid of it because the cost outweighed the benefit. It is nice now that I am used to it and I don’t even miss it. (Then again my “commute” door to door is a whole 12min drive or a 25min walk.) Now all I have to worry about is tracking of my web surfing habits which consist of, among other things, requests per day to this site <;)

Mike Sherwood • December 19, 2006 9:43 AM

Teenagers seem like the natural target for this. I wonder if that’s their primary target demographic.

I think most, if not all companies would be opposed to surreptitious monitoring of the work place being provided to a third party. IMMI says they’re only sending signatures, but without a lot more detail about what they’re doing, it’s hard for someone to evaluate it. It seems like this would be banned just based on a quick summary like “leaves microphone on and sends data to third party”. Just like many places don’t allow camera phones because of restrictions against photographs, even though it’s hard to get a cell phone without a camera these days.

Privacy is a tradeoff that individuals can make, informed or not, but the people surrounding them do not get the same choice. For those who believe that no one is coercing them into giving up marketing information, have you ever sent in a mail in rebate or a warranty registration form? They never state it explicitly, but that information is being collected and sold for marketing purposes.

jammit • December 19, 2006 10:05 AM

Please give me one. I’ll just drop it near one of the speakers at a nearby porno theater. I wonder if in the agreement you sign if there is a rider protecting your speech from improper use? Is there anything guaranteeing that the info won’t be stolen or fall into another parties hands? I’m pretty sure there isn’t, although I can’t blame the company because unlimited eavesdropping can’t be secured any way.

Anon 31 • December 19, 2006 11:48 AM

One other thing, does this violate any wiretapping laws because it records without the explicit permission of those being recorded? Certainly sounds like it can (and does) record snippets of your cell phone conversations.

Davi Ottenheimer • December 19, 2006 11:54 AM

This system seems so unbelievably primitive it’s hard to imagine it’s real. If you’re worried about collusion, the fact that the media moguls, phone companies, and advertisers have to resort to interpreting sound from the mic in a mobile device in order to figure out what is going on…

My experience is that these companies work very closely together and it will be (already is, in some cases) trivial to correlate what is on your TV and and where your mobile is actually located. I mean there are several services where we can already program our TV shows from our mobile phone, and we can interact and be tracked on both. The audio would thus be useful only to pick up actual spoken conversations or audible reactions.

Geoff Lane • December 19, 2006 12:09 PM

Isn’t the battery life going to be terrible? Active for 30% of the time is way more than a normal cellphone.

Davi Ottenheimer • December 19, 2006 12:21 PM

Ah, I should have looked more closely at who’s running IMMI…

http://www.immi.com/aboutUs.html

CTO Al Alcorn – Atari (cofounder)
SVP Research Amanda Welsh – Netscape
VP Engineering David Brudnicki – AT&T

…not to mention York-Fess.

That’s a big clue. Given that they are so experienced in telco/media convergence the first question I would have for these folks is who “gets” to convert the live data to aggregate (de-weaponizing it) and is it a one-way? Even as we are forced to think of location and behavior as a keys to our identity (since they now are being meticulously recorded and correlated), we still should be able to call for basic (privacy) rights for our own protection.

In other words, in demonstrating any kind of accuracy of the system, companies like IMMI should be asked to also show how they have comparably raised the bar for protecting individuals from harm due to their increased accuracy.

another_bruce • December 19, 2006 12:28 PM

great caution is indicated. somebody brings one of these things into my presence without my knowledge, i have the means to break both the spy and the company in court.
i’ve said it before in this space: no sex in the same room with a telephone!

Woody • December 19, 2006 12:47 PM

The sampling technology is probably real, I’m aware of a cell-phone app/service that allows you to record music playing, and then it sends the sample to the server, which responds with the ID of the music. Great for clubs/radio. Works fairly well, too.

If the hashing was done in DSP instead of on-processor, this might be very easy to handle on-phone, instead of sending the raw samples up to the server. Which would also help assuage the privacy concerns. And most cell phones tend to have a couple DSPs lying around for audio compression and such…

Also, the phone can store the audio locally, and send it up in bulk to the server every hour or so. That wouldn’t be much different to the battery than a teen that’s sending an SMS every couple minutes.

OTOH, just because it’s possible to do this without totally violating everyones privacy, doesn’t mean that they have.

It’s much like OnStar, where they do in fact have the capability to listen in on the audio feed from the interior of the car, but procedures disallow it, and you need to put it into test-mode to do it remotely, anyway.

M.Research • December 19, 2006 12:50 PM

That kind of behavioral studies are very important for equipment and service providers: they want how you use their stuff. For example a mobile phone with bt and wlan can tell lot of what is happening around.

Those can be realy invasive for privacy, but they do not need to be. One can respect user privacy and whithen data before it is permanently stored thus protecting user. Even then the data is available only to limited number of people. Some corporations I know act responsible way.

Of course, average U.S. corporation does not have any respectn privacy. On here other side of Atlantic, it is quite different because of privacy laws.

Elliott • December 19, 2006 1:12 PM

@C Gomez:
Those who do not agree to give away their data must pay the rebates that the others get.

TimH • December 19, 2006 1:41 PM

@ M.Research
Re “Of course, average U.S. corporation does not have any respectn privacy. On here other side of Atlantic, it is quite different because of privacy laws.”

Actually, so much of the EU protected stuff finds its way to USA to help find terrorists, save the children from porn, whatever the current excuse. Recent examples are passenger travel data, the SWIFT transaction records – both heavily protected by EU law.

Remember that the USA gov. uses contractors to process the data because gov. is constitutionally prevented from monitoring without cause. Once in contractors’ hands, realistically data is unprotected because there is not privacy law in USA apart from select items (wiretapping is one example), so release of data inappropriately would be civil breach of contract (between gov. and contractor) only.

Basically, USA is screwing over privacy in EU for anyone who travels or has international money connections.

Jungsonn • December 19, 2006 2:23 PM

“These cell phones are given away to test subjects, who get free service in exchange for giving up all their privacy.”

Yay! for the freebees.

I have mixed feelings about this:

a) I never thought this would happen.
b) I never thought it would happen SOON
c) I’m still ignorant by thinking this isn’t already done by now.

bob • December 19, 2006 3:51 PM

I suspect the “digital signature” may be spectral domain, not time-domain. That would be where I’d start if I had a signal with lots of unwanted noise elements, because the time-varying aspects of the media should be discernible even with refrigerators, toilets, people talking, etc. It’d also tend to obscure dialog, although not entirely. I’ll have to see if I can find out what tech is used.

john • December 19, 2006 4:18 PM

bob, “An Industrial Strength Audio Search Algorithm,” Wang, 2003, is a good place to start reading.

X the Unknown • December 19, 2006 4:32 PM

@paul: “And remember, even if IMMI’s heart is pure as the driven snow, their database is going to be a delightful target for hackers…”

Not to mention being a prime target for Law-Enforcement search-warrants…

MikeB • December 19, 2006 6:41 PM

Their description and info talks about ‘open architecture’ phones. A search revealed the URL below, which provides an example of a Cingular 3125 windows 5.0 smartphone.

Contrary to the original article text, I don’t think these are “..specially designed cell phones..”. These are standard cellphones, running a native .net smartphone app which would always run when the phone is on (for those that use Goodlink email client – I guess it would be like this).

I would expect the App would use the GPRS/EDGE/3G data channel to convey compressed audio data / ‘digital signature’ info up to the IMMI server. 30sec of audio at 8kbit/sec (e.g. AMR normal) would give a 30kbyte upload. This app is going to need an unlimited data plan though 🙂

As for gaining access / building an app like this:
– smartphones are relatively open from a file system perspective (just connect one to a pc with a USB cable). Anyone accessing a phone with this app installed could grab the IMMI binary.
– now whether the binary could be hacked to redirect the audio to another destination – depends on the design of the app & binary. I wouldn’t be surprised if it could – unless it was designed to prevent this. Biggest issues would be those following
– I’m fairly sure you need a ‘signed’ app for the app not to prompt you whenever you access various APIs including data transfer
– other than the ‘signing’ issue, I don’t think it would be a hard app to build.
– main issue would be getting access to the phone to install (OTA may be possible but again would need user confirmation). ALternatively, could also store audio to a storage card which most users would not know they had. If you had access to the phone, this would be easy. 2GB micro SD = 500 hours of audio logging? (is my math right?).
– I think the battery life would be reduced

For more info see:
http://www.immi.com/teen/

GregS • December 19, 2006 8:14 PM

One of the key points is not being mentioned. Maybe it’s obvious to everyone but me. The encoding into a signature is done in the phone and only the signature is transmitted home to IMMI. The actual sound sample presumably can’t be reconstructed from the signature, period, no matter how many GB of data might be hacked from IMMI servers. So it’s not an eavesdropper in ths sense being disucssed.

Tarkeel • December 20, 2006 5:38 AM

This reads like right out of a fiction story – namely “Interface” by Neal Stephenson. A subset of voters were given a special “watch” that would automatically broadcast all political debates etc, and monitor their pulse and other biometric signs and send back for evaluating the response to the broadcast.

Ofcourse, that story did take it quite a bit further then this, but..

Voyeur • December 20, 2006 7:42 AM

“Can it be done without the owner’s knowledge or consent? The potential for abuse here is enormous.”

Yes, and perhaps quite amusing too. You discuss the possibility of eavesdropping by phone but what about getting at the pictures on phones with cameras? I know some people who keep some ‘dodgy’ pictures on their mobile. If you can hack the phone to eavesdrop, surely it’s not too hard to browse through the pics on the phone as well?

paul • December 20, 2006 10:10 AM

@GregS:

“The actual sound sample presumably can’t be reconstructed from the signature, period”

Why do you think that? There are obviously hashes that could be used to make that mostly true (although having bunches of known plaintext in the form of all the broadcast audio being tracked makes analysis a bit easier). But all of the simpler encodings that a DSP would apply to create a “signature” file tend to be reversible. Furthermore, the claimed robustness in the face of background noise and conversation suggests encodings that map fairly closely to the original signal (because otherwise even one or two off bits could kill your recognition).

stf • December 20, 2006 10:47 AM

In Switzerland, this monitoring (“radiocontrol”) is done offline with a watch, that encodes “parts of high, medium and low frequencies into number sequences, 4sec/min” (my translation from the German version [1]). The watch stores the numbers and gets sent back after a week.
Apparently the system is based on comparison with numbers generated by the Swiss radio stations. I suppose this is a better way of doing it than with the cellphone: no online connection needed, therefore hacking much more difficult; dedicated system, again more difficult to hack; time delayed, so less interesting for eavesdropping.
However, I don’t know about the encoding algorithm; they claim on [1] that only “one thousandth of the sounds are detected” by the watch.

[1] http://www.drs.ch/index.cfm?gbAction=3CA2B391-49F8-45A1-9AC1A217A5973462&CategoryNodeID=4F1908F4-7F7E-4EB1-AE376BFC710A3258&prg=srdrs&szm=about%2Fsrdrs&szmcha=drs

OpenMike • December 23, 2006 12:16 AM

Reminded me of this story from “Spyware Weekly Newsletter” 6Sep06 http://www.spywareinfo.com:

Google Listens… Literally

I found a story at Slashdot http://yro.slashdot.org/article.pl?sid=06/09/03/2148216 the other night that knocked me right out of my chair. Google wants to use your computer’s microphone to listen to your TV, then deliver targeted advertisements at you based on what you are watching….

He gives a link with more information: http://www.mangolassi.org/covell/pubs/euroITV-2006.pdf

The newsletter’s author reckoned the whole thing is innocuous, since, after all, it’s good ol’ Google and it’s an “opt-in” program, but I seem to remember that Google reserves the right to make modifications (er, “updates”) to any of their code running on your machine at any time and without notification… And I did notice that the scheme originates in Israel. Guess it might not be too popular on Google Arabic. But maybe it won’t be offered on an “opt-in” basis there…

Elena V • February 6, 2007 10:05 AM

This sounds like exactly the sort of thing I’ve been looking for to …..well, hell… stalk my ex-semi boyfriend in California.

Can I use this to listen in on his “phone sex” phone calls to other girls.

I hate him. But I must know everthing he does.

Elena ExinAustin • February 6, 2007 10:08 AM

I’m just a little obsessed with this guy I dated in California. Will this technology help me find out what he’s doing while I’m sitting here thinking about him?

T • July 21, 2007 12:28 AM

I’m going to be one of the test subjects. They contacted me via mail and gave me a test over the phone. It’s a Cingular smartphone. They give you 1000 anytime minutes, unlimited text and data plan. You can do it for 2 years as long as you keep the phone on and charged. I was skeptical at first. When I called to ask about it, a kid at Washington State answered and told me it was a combined deal with the IMMI company and a marketing survey with the college. I agreed to do it as my current provider has the worst coverage ever and my job requires me to be able to be contacted 24-7 and it just happened to be conveniently timed. The kid told me that voice converstions don’t get encrypted and sent to the server, thats how they don’t violate privacy. I don’t know how I feel about it quite yet. The good thing is there is NO agreement to sign and you can stop at any time.

disainrite • September 22, 2007 2:19 PM

well dis thing is kinda dumb it makes me mad that im being bugged because how could it tell the difference between a sson and an argument????

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.