Eavesdropping on Computer Screens through the Webcam Mic

Yet another way of eavesdropping on someone's computer activity: using the webcam microphone to "listen" to the computer's screen.

Posted on August 31, 2018 at 6:29 AM • 28 Comments

Comments

echoAugust 31, 2018 8:17 AM

This is clever. I have noticed screen whine before and/or some kind of feedback loop with speakers. It never crossed my mind it had a useful resolution when combined with smart pattern recognition.

This is pretty good domain expertise. I'm impressed!

hermanAugust 31, 2018 9:12 AM

Hmm, small fonts will defeat it, but instead of trying to decode the video with machine learning, it may be better to simply render what was received in blurry monochrome and let a human being interpret it.

flaskerAugust 31, 2018 11:16 AM

@herman

It may be simpler to have a human decode blurry monochrome, but that doesn't scale. If this attack is not being used on a single target, but focused on hundreds of thousands, it will be much easier to have the machines do the busy work and have humans review the interesting bits.

vas pupAugust 31, 2018 11:23 AM

"And mitigating the risk would require re-engineering of current screen technology."
Is it better to re-engineering current microphone wiring by providing kill switch controlled by user?

AlejandroAugust 31, 2018 11:55 AM

To mechanically turn off a microphone on a laptop or other device with a mic. jack, find that old cheap external microphone you never use anymore, cut off everything except the male jack portion and then plug that in to the computer, and leave it in. That will turn off the mic. regardless of any electronic wizardry.

But, if an attacker is remotely recording your audio whine, aren't you pretty well screwed, regardless. I mean, that seems like a bit much. No?

Clive RobinsonAugust 31, 2018 12:07 PM

So yet another side channel leaks information by acoustic means... Can there be any more to exploit ;-)

It kind of points out two things,

1, Just about any source of energy can be an information leaking side channel.

2, Nobody can afford a system designed to stop all the side channels.

Thus nobody realy designs and builds "secure systems" of any upto date use any more...

Thus to be secure you have to use the fact that the energy density (signal strength) of side channels drops of at either 1/d for "conduction" in a transmission line or 1/(d^2) for normal surface "radiation" or very occassionaly at 1/(d^3) for volumetric energy density systems[1]. Or somehow reduce the bandwidth of all side channels below a usefull rate[2].

That's it, whilst distance is fairly easy to achive upto certain limits, locking down the bandwidth of all energy channels is nowhere as easy to do. For instance you could use the load on a CPU to get it to heat up and cool down. You can buy FLIRs that will "see the heat" and it will reflect of some surfaces. But that heat also changes the resonant frequency of the CPU XTAL, this can be seen via "timestamps" just about anywhere on the Internet...

It's why several people hear talk about not just "air gaps" that might stop EM side channels but probably not acoustic, and "energy gaps" that should stop all forms of conduction, radiation, and even convection of energy.

But what used not to be a problem now is and that is "store and forward" "opportunistic time shifting". That is malware once on a device with storage can use it to record signals that it later transmitts to a "cut out" "sacrificial goat" when it gets an Internet or other usable communications path. Thus you can nolonger use a "single device" but at a minimum two. The first is used for general connectivity to the rest of the world, but nothing "sensitive or private". The second is for "sensitive or private" work an is never ever connected to communications devices or even unfiltered mains conbections. It's this second device that is the one that is "energy gapped". It might sound hard, but believe me when I say the "OpSec" involved is way harder than you would think. As I've mentioned before I keep some devices in safes these devices have no hard drives or flash memory etc which makes the use of malware more than a tads difficult for an attacker, and "store and forward" just about impossible.

There are "mittigations" but as Meltdown and Spector show, you just can not trust the underlying hardware in any way shape or form... you have to mitigate all of it... Which is a point that's been raised over and over on this blog going back for a very long time. A situation that is only going to get worse not better given time :-(

[1] Think sound traveling along a rail or rod for 1/d, a radio wave or sound wave through "free space" for 1/(d^2), or energy such as heat building up in a volume like a kettle put on to boil for 1/(d^3).

[2] The problem of "below a usefull rate" is it's dificult to determine what is usefull. I've sent information hundreds of Km using bandwidths below 1/10hz on synchronized signals (DSSS) that most receivers would only see as maybe a little noise in the noise floor even just a few meters away.

Look at it this way we know that students have extracted key information at a rate of one bit every 15-20 seconds which sounds realy slow when normal typing is around 1200 bits per second. But if they are also recording your AES data transmission it will only take half a minute to get the key for it...

Clive RobinsonAugust 31, 2018 12:14 PM

@ Alejandro,

find that old cheap external microphone you never use anymore, cut off everything except the male jack portion and then plug that in to the computer, and leave it in. That will turn off the mic. regardless of any electronic wizardry.

Sorry no not these days, many computers nolonger use mechanical switches in the jack pair connector as it cuts down noise and considerably ups the reliability.

Thus the internal mic is hardwired into the chip, and software is what turns it off.

vas pupAugust 31, 2018 2:46 PM

@Clive Robinson • August 31, 2018 12:14 PM
Is the same true for smart phones?

echoAugust 31, 2018 3:09 PM

This isn't the clearest explanation of "ghost images". By this I think they mean the experiment where a single photon can be used to reconstruct an image of an object around a corner of all things. Re-reading it I think it's a completely different experiment but similarly using a single photon. I'm a bit lost as to what they are saying here. It's something about normally out of range wavelengths and what refresh rate is optimal for the eye reconstructing partial images into a coherent whole. They also mention wanting to discover what additional computation is possible by the eye. Hacking eyeballs? Whatever next!

https://www.technologyreview.com/s/611936/physicists-hack-the-human-visual-system-to-create-ghost-images/
Physicists hack the human visual system to create “ghost images”.
A pioneering experiment will extend human vision to invisible wavelengths, say researchers.

Jesse ThompsonAugust 31, 2018 3:17 PM

@Clive Robinson

> when normal typing is around 1200 bits per second.

Hmm.. did you forget to shift unit from minute to second here?

typingtest.com just told me I clock in at 415 characters per minute, which is an average of 6.9 characters per second and in 7-bit ASCII that's 48.4 bits per second. Uncompressed, of course.

If the average typer is really 25 times faster than me, perhaps there really is something to this home row fad! xD

AlejandroAugust 31, 2018 5:58 PM

@Clive Robinson

Re: Mic jack plug disables mic.

I will certainly defer to your expertise. I would suppose whether the jack-hack will disable the microphone can be tested easily. It does on my beat up old laptop.

With software switches, I wonder if that would be good enough, too: Simply turn off the mic with the software command?

The question might be: could the remote exploit described ALSO switch the mic. to on?

And, I would have to surmise maybe it doesn't matter. The exploit might not need access to mic. input or output.

There's always something to worry about, isn't there?

Clive RobinsonAugust 31, 2018 6:43 PM

@ vas pup,

Is the same true for smart phones?

I don't know I'm not in the habit of pulling more than one or two Smart Phones apart these days (they make them increasingly "not serviceable" with glue and other one way mechanics).

But the ones I've looked at don't appear to have them. Also the spec for the "Tip two rings and body" 3.5mm jack as given in the Android Spec certainly does not need them as it senses via loading resistors.

@ Alejandro,

There's always something to worry about, isn't there?

Yup it's like the old jokes about paranoia...

@ @Jesse Thompson,

Hmm.. did you forget to shift unit from minute to second here?

Let me check, 60wpm is sixty lots of five printing chars plus space, so 360cpm or 6c/s. In ASCII it's seven data bits, a parity bit and a couple of framing bits or 10b/c or 60baud which is 60b/sec which would be 3600b/min...

So yup guilty as charged...

Thinking back yes I was thinking in terms of "4 key bits" per minute or 15secs/bit. And I must have got secs/bit and bits/sec muddled up somewhere and once the muddle starts...

Clive RobinsonSeptember 1, 2018 8:05 AM

@ echo,

This isn't the clearest explanation of "ghost images".

The diagram in the article you link to reminds me of those you used to see explaining holograms.

They two use seperate beams (although split from a single source). Aproximately they worked by taking a refrence beam and a reflected beam then producing an "interferance refrence" at the photo gel. You then played it back using another refrence beam that did not have to be the same frequency or in some way cohearent with the original refrence beam.

In effect the photo plate when developed had a complex transform embedded on it, that to most views without the refrence beam was vague or random mess.

I think it's time for a little more digging and thinking ;-)

Alyer Babtu September 1, 2018 4:07 PM

"listen" to the computer's screen.

As Victor Borge used to say, while contemplating a piece of music he was about to play, turning the score this way and that, “let’s see what this sounds like; or, let’s hear what this looks like”.

echoSeptember 1, 2018 5:05 PM

@Clive

Since you mention it the connection seems fair. This stuff gets complicated... I only just heard of quantum information theory yesterday. At a classical level in some senses all systems are information processors too.

Myself, I'm intrigued by neuro-psycho-social processing and perception. I have no idea what you are up to but it sounds intriguing.

OtterSeptember 1, 2018 6:28 PM

Clive : "Nobody can afford a system designed to stop all the side channels."

Yet, everybody can afford systems with evermore additional sidechannels.

"Thus nobody realy designs and builds "secure systems" of any upto date use any more..."

No, they really design and build insecure systems.

Clive RobinsonSeptember 1, 2018 8:03 PM

@ echo,

I only just heard of quantum information theory yesterday.

So you've not got the "Bloch sphere" under your belt then ;-)

If you want something to realy drop your jaw, have a look at Seth Lloyd over at MIT, he has a fair argument the whole universe is a quantum computer, but nobody knows --or could no-- for what purpose.

An interesting little thought for you. Information is not of the physical universe quantum teleportation tells us this. We simply impress/modulate information onto physical universe objects such as matter and energy, to store, communicate, and process it. The implication is that our physical universe is a subset of an information space of some kind, where the laws of physics do not of necessity apply. But also our physical universe is finite and thus only a very limited subset of what is knowable will be known at any one time.

It's thoughts along these lines a hundred or so years ago that drove mathmeticians not just mad but as one eminent physicist only half jokingly said "bat shit insane"...

echoSeptember 2, 2018 2:45 AM

@Clive

I'm leaving all the hard work to the eggheads. Apart from this it's all very entertaining.

The problem with concepts is as much as they help us understand they can also placelimits on our understanding. So much is dependent on ignorance and society. Scientistsare beginning to attempt to question their boundaries of udnerstanding and work their way through the muddle. Another intriguing possibility is thata deeper understanding may be translatable to everyday language. From a cognitive and artistic point of view this is interesting especially given the crossover of ideas between the three different forms of knowledge: religion, science, and drama. Things get even weirder as at the extreme laws may disappear entirely, or be a wholly subjective manifestation of observation. Nontheless the discussion about the whole subject area seems to be acknowledging the limits of understanding which is creating a good base for more discussion and scientific discovery.

https://www.scientificamerican.com/article/beyond-the-quantum-horizon-extreme-physics-special/

https://www.wired.com/story/physicists-want-to-rebuild-quantum-theory-from-scratch/

Clive RobinsonSeptember 2, 2018 6:16 AM

@ echo,

Sorry a nights sleep intruded in my reply (they tell me it happens to everyone ;-)

With regards,

I have no idea what you are up to but it sounds intriguing.

I'm upto many things, some dull some interesting, some I can't talk about or talk about currently whilst others I can. But at all times I'm curious about things that are new. That is part of what I do all the time is research into the viability of the new in fields of endevor the originators might not have thought of.

Mostly these days the research I do is in my own time[1] and is about using or mittigating the very new for security purposes. Usually from a bottom up approach using external communications technology or other energy source to do Active EmSec Attacks.

What most forget is the usuall solution to security issues is much like that of Victorian artifisers. Which is the "We will make it stronger by design" PR sound bite of the reality of

    Bolt another bit on till the bl@@dy thing stops breaking and just get a bigger horse to drag it to market.

That is don't find out why it's breaking just glue/weld stuff over the cracks and cross your fingers.

Such an aproach is almost always "top down" thus does not fix "bottom up" attacks.

Also what most forget is that almost always at low level the movment of information is not one way due to the likes of handshaking and error correction. Software writers are notorious for how they handle "errors and exceptions" from the right... That is down stream functions, processes and devices. Likewise unexpected input from the left especially when it is assumed to be another process not a user...

Which means injecting faults is a happy hunting ground for attackers. Whilst the industry is slowely waking up to input side faults, they are still sleepwalking on the output side faults... So any "bubbling up" or "bottom up" attack is going to be quite devastating. As @Thoth indicated "the usual suspects" here have been talking about the problem for quite a long time and it is only now with Meltdown and Spector being reported even in the MSM that people are waking up.

As @Thoth has pointed out Security Enclaves are a bit of a problem.

The real reason for this is that they use the main memory just as every other process does. At the memory level there is generally no protection of process space as at that level it has no meaning what so ever. It's just address bus and data bus and a control bus designed to have as few gates as possible to get maximum speed of response.

It's been known for multiple decades that anything with that bus level access can do as it pleases, thus Direct Memmory Access (DMA) is a major system integrity risk. Unfortunately trying to get high communications transfer speeds and bandwidth with low latency response more or less predicates the use of DMA...

If you look back at Apples Firewire interface and Intels later lightPeak designed to replace it which later still became ThunderBolt they all provide DMA intetfaces to the outside world on conbectors. So pluging in a malicious device to give a simple "bubbling up" attack whilst not childs play to develop realy is childs play to deploy...

The same is true of other high bandwidth low latence comms. Even USB suffers from similar DMA issues but with it's own interface CPU.

There is no top down soloution to these types of "bubling up" attacks because software works much further up the computing stack. Thus it does not matter a tinkers cuss what you do in the way of formal methods, provably correct by design etc, they can not stop DMA changing the process memory. Only correctly configured hardware can do that IF AND ONLY IF it has the capability built in from day zero.

But it gets better, once you know the basic security fault behind DMA and you know how the extra hardware protection is configured by the OS software, it's simple to realise if you can change the configutation you change the security. Thus you can turn security off to gain access to a protected processes memory or you can turn security on to a process such that it can not access the memory resources it needs to function...

Well guess what you can... You can actually perform an attack at a lower level in the computing stack than the memory interface level... Due to issues in the design of high density memory chips it's possible to attack them not at the gate level but even lower still at the basic storage component (capacitor) level... The fact you can do this from software is more than a bit devistating and you can look it up as the RowHammer attack. It requires fairly major hardware changes to fix but there are some OS mitigations that limit it's potential, but there are Systems Performance Issues...

But RowHammer is just one instance of a whole class of "Reach Around" attacks in this case a "reach down" subclass. Meltdown and Spector are other reach around attacks but there are more, many more and the vague fiddling Intel, AMD, ARM, and others are doing will not solve the issue of this class of attack, at best they will mitigate an instance or two with a significant system performance hit which will accumulate for each mitigation...

So you can see the potential of all such bottom up attacks be they "reach around" from malware or "bubbling up" from malicious hardware.

But... My favourit areas of research are ones you don't hear very much about in "The Open or Academic Communities" in fact next to nothing[3]. They are mainly Active EmSec attacks not passive, where by you get a signal in at the basic electronic component level that takes advantage of fundemental analog problems such as metastability to cause execution to diverge from the designers intention. The routes in are many and I've described a few in the past. But two simple to exploit ones are due to "System Transparancy" issues,

The first is time delay or jitter, whereby you hide a time based signal on top of a valid input signal. Even though the system may change the input the time signal is still seen on the output. The classic example is keyboard data goes into the system as valid looking key presses which then gets wrapped up in network packets for originally Telnet but later SSH etc remote terminal access. Thus the timing data ends up being transfered onto the network packets irrespective of any encryption etc. But... because the system will alow other processes to connect to "stdin" an application can be inserted into the input stream of any existing service alowed to run. Thus the app could be a generator or an input Tee acting as a modulator to an existing stream, sending out confidential information in the timing...

The second transparency issue goes in the opposite direction, that is from the output back through the system to the target of interest. You will not find this talked about, I guess because few even think it's possible. Unfortunately for them and many others it is and it quite happily works bakwards through data diodes and other supposed one way protection devices. At a low level systems are generally designed by people who have trained as electronic or communications engineers thus are, unlike most software coders, aware that there are realities in life such as line noise and disrupted communications. In short they know things are inherently unreliable and that you need to employ multiple levels of error correction in the right ways. The most obvious to look up is "flow control" that is a transmitting source also has a receiving flow control port you can see an early example in the old RS232 CTS/RTS signals [4]. A terminal (DTE) sends data to a modem (DCE) but for various reasons the modem may not be able to process data at the rate the terminal sends it at. Thus the modem uses a control line output (DCE-CTS out) to a control line input on the terminal (DTE-CTS in) to tell it to pause in transmission. Thus data will wait at the DTE filling it's serial port device driver buffer such that it sends an error signal back to the process connected to it and so on backwards up the transmission chain. Eventually when buffers are full the generating process gets the error signal as an exception and has to deal with it. Due to the way many coders think and write software it might send an error message to the error log before crashing out. Due to the way modern computer caches work other processes can find out the error status thus have a covert channel back to them most would never have thought to check for... Likewise due to the way network switches and ports work you can do similar, which is why even Data Diodes are vulnerable to this reverse error/exception timing covert communications channel. Oddly though the software coders should be aware of not just system transparancy but reverse communications are, if they know what "Port Knocking" is[5]. It makes a good question when interviewing.

Anyway I hope that answers your question without me having to go into specifics.

[1] One of the major issues these days I'm guessing you are probavly aware of is "employer over reach" in some form or another. One aspect of is if an employee comes up with an original idea and even if entirely unrelated to the emoloyers activities and done entirely in the employees own unpaid time with entirely the employees own resources the employer will lay claim to it[2]. Even "zero hour" supposadly "self employed gigs" come with a "we grab it all clause".

[2] There have been many reasons for employers attacking employees over their creativity or IP. Mostly it's not about going on and using the IP but stoping it being used to protect the often exemployer's rice bowl or other employers rice bowl. In the same way patents are frequently used. But employees can be unaware it is also directed at them. The Silicon Valley employers deliberately not employing from other Silicon Valley employeers was not about the noise they made of potential legal issues, but an illegal cartel designed to drive and keep wages down.

[3] One of the reasons you do not hear very much about it goes back to the end of the last centure and research into attacks on Smart Cards. The idea of "Power Supply Noise Analysis" came up again after a nearly a couple of decades, as a way to detect a CPU's internal function thus how it behaved in things like crypto algorithms that could and did leak KeyMat information easily to the outside world... Well the smart card industry did the usual top down approach to in effect mask the noise in random data. But as any communications engineer will tell you that approach is doomed to failure if the algorithm is run repratedly and the signal averaged. What happens is the wanted signal adds up and the random noise averages out towards zero, so you get back to where you were before the madking technology was bolted on. Well a person who decided to publish what was openly known in the communication engineering community gave it the name "Differential Power Analysis" and grabbed a bit of fame with it, which is fair enough. But what he also did was seek and gain overly broad patents in the US (you can only do it in the US). But the result was a "major chilling effect" on the various communities on what is or should be basic research and thus exempt from patent issues. You can be sure that research is on going in commercial entities but as they will want to access the US market they simply say nothing about it outside of select internal groups. Which is just one reason I have no intention of visiting the US again and thus am happy to talk about it in my personal capacity to make the issues more widely known.

[4] Although most call it "RS232 hardware signalling" it's actually the TIA-232-F 1997 Rev 2012 interface specification that is current. What is worse, is it's probably the most abused data communications specification for single user compiters ever. Thus it involves one heck of a lot of "magic thinking" to get it to work, and yes books have been writen about it (I wrote one back at the birth of the UK 8bit Home Computer revelotion, and although I got the miniscule advance the publisher joined that great big publishing house in the sky and closed it's doors befor it got around to getting the draft let alone publishing it).

[5] our host @Bruce wrote briefly on port knocking getting on for a decade and a half ago,

https://www.schneier.com/crypto-gram/archives/2004/0315.html#5

I'm thinking it's time he wrote an update about it, but this time including the risks as a covert communications channel for attackers ;-)

echoSeptember 2, 2018 4:04 PM

@Clive

Thanks for the explanation. I'm aware of most of what you say although get fuzzy with some of longer detailed explanations and how everything fits together in practice.

I wish I could discuss what I want to progress more. The problem is keeping it on topic and not making things worse. It's very difficult discussing this with people who lack an appropriate level of expertise plus rice bowls and IP theft are issues too. What happened to me isn't unlike what happens to a whistleblower. Nobody will say anything but I have effectively been given a death sentence.

I was warned a few years ago by a high level executive in a comemrcial company I would be "used or abused". I was also told by a mid level state sector manager who knew what was happening to give up on the UK because it was so backward and rebuild my life abroad. They choked me off so badly I'm struggling to save up to buy the passport and airfare. I know "they" have treated people like this before. I have plenty of media in my database. I even have audio recordings of when it happened to me not that any authority wants to know because they are implicated too. I'm not the one who did anything wrong and it's infuriating to have the reports which prove it. It's a horrible situation.

samantha September 2, 2018 4:23 PM

I only recently discoved this blog upon stumbling across a 2016 article regarding who is implementing a bunch of test bed attacks on core infrastructure. it was for me in many ways a relief to read. I just wanted to pass comment on the audio channels its unfortunatley not just limited to all those mentioned above. Have you come across the various methods of "DIY" converting other appliances with simular electronics base and a few virtual connections and even some emulated hardware can be thrown in for good measure

meSeptember 4, 2018 6:33 AM

@Alejandro

Re: Re: Re: Mic jack plug disables mic.
> I would suppose whether the jack-hack will disable the microphone can be tested easily. It does on my beat up old laptop.

Just because inserting a jack disable mic doesn't mean that it is phisically disconnected.
as Clive said, it's software that sees the jack and suppose that you want to use it instead of the builtin one.

not only, inputs and outputs can be remapped and you can use headphones (connected to the headphones jack) as microphone by remapping the output as input.
you just need to tell the chip "that is a mic input, no more headphones output"
since the working principle is the same you will have (lower quality) mic but it works.

> The question might be: could the remote exploit described ALSO switch the mic. to on?

yes, an exploit is a program and as any program that you have on your pc can turn on the mic & webcam without any kind of permission, windows doesn't have permissions unlike android.

MSeptember 4, 2018 7:30 AM

@me the most recent versions of windows actually do have microphone permissions (per app for store apps and global for desktop apps). But I suspect if a program is running at a privilege level where it can remap inputs and outputs, any permissions are also irrelevant since they can be changed

vas pupSeptember 4, 2018 10:33 AM

@Clive, Alejandro, me:
Should small audio jammer placed next to PC/smart phone do the job, i.e. mic is still working as you suggested, but can't extract useful information?
If answer is yes - that is million $ market - I guess.

Clive RobinsonSeptember 5, 2018 6:29 AM

@ vas pup, Alejandro, me,

Should small audio jammer placed next to PC/smart phone do the job, i.e. mic is still working as you suggested, but can't extract useful information?

The answer is the mic will always extract the information around it. All you are effectively doing is adding a lot of noise to the desired signal.

This is the mistake they made with smart cards and power analysis, which became obvious with Differential Power Analysis (DPA) back last century.

With DPA you simply keep adding aligned trace after aligned trace thus the wanted signal goes up by n but as the noise is random it goes down by sqrt(n). With enough aligned samples the noise is effectively striped of and other signal processing meyhods then applied to recover the wanted signal and determin the wanted key bits.

With an external source you need to get a second signal with a different aplitude differential between the desired signal and the jamma signal. By making the desired phase and amplitude correction it enables you to add the two together to eliminate the jamming signal and recover the underlying audio.

It's why proffessional surveillance people always try and get two fairly widely spaced audio signals when ever possible.

The same technique can be used effectively in reverse to detect GPS jamming with a synthetic or translocated signal.

vas pupSeptember 5, 2018 9:55 AM

@Clive:
Thank you!
As my humble understanding is to manufacture and market as privacy efficient phone, tablet, laptop with microphone, speaker, GPS unit ALL wired through kill switch eliminating their power supply and possibility to activate those functions remotely by software without user's awareness.
I am curious why Samsung, LG, other manufacturers did not do this for now. Does Big Brother/FCC object openly or secretly for that?
I guess China manufacturing for their officials abroad electronics with such functions.

Leave a comment

Allowed HTML: <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre>

Photo of Bruce Schneier by Per Ervland.

Schneier on Security is a personal website. Opinions expressed are not necessarily those of IBM Resilient.