Fawkes: Digital Image Cloaking

Fawkes is a system for manipulating digital images so that they aren’t recognized by facial recognition systems.

At a high level, Fawkes takes your personal images, and makes tiny, pixel-level changes to them that are invisible to the human eye, in a process we call image cloaking. You can then use these “cloaked” photos as you normally would, sharing them on social media, sending them to friends, printing them or displaying them on digital devices, the same way you would any other photo. The difference, however, is that if and when someone tries to use these photos to build a facial recognition model, “cloaked” images will teach the model an highly distorted version of what makes you look like you. The cloak effect is not easily detectable, and will not cause errors in model training. However, when someone tries to identify you using an unaltered image of you (e.g. a photo taken in public), and tries to identify you, they will fail.

Research paper.

EDITED TO ADD (8/3): Kashmir Hill checks it out, and it’s got problems.

Another article.

Tags: , , ,

Posted on July 22, 2020 at 9:12 AM • 48 Comments

Comments

Alan Kaminsky • July 22, 2020 10:04 AM

As I understand it after skimming the paper, Fawkes “cloaks” a photo of you by adding noise, imperceptible to the human eye, such that an image classification AI would classify your photo as that of someone else — and you can pick the target for the misclassification. For example, I could post a cloaked photo of me (an old, balding male) that the AI would think is a photo of Taylor Swift.

bcs • July 22, 2020 10:19 AM

I haven’t read anything other than the snippet but; while I’m willing to believe this works for existing ML process, I somehow suspect that this would be rather easy to defeat by applying the same process to parts of the training data. Or by lightly blurring the images. Or by training another system, using before and after pairs, to remove the modifications. Or by tuning the ML towards other kinds of features. Or any number of other things.

Ultimately, as long as the photo is useful to you, the information the other side wants is still there. The only thing you can do is make it more expensive to get.

g • July 22, 2020 10:53 AM

@czerno, nope this is a known problem with machine learning systems. It’s very easy to trick them into completely misidentifying input. Especially ones using images since there is so many creative ways to add noise.

For an easy to digest example using MNIST: https://ml.berkeley.edu/blog/posts/adversarial-examples/
Basically you can arbitrarily make the NN identify any number as any other number.

Yes there are countermeasures, but that becomes much more difficult with unsupervised learning.

Clive Robinson • July 22, 2020 10:58 AM

@ Alan Kaminsky,

For example, I could post a cloaked photo of me (an old, balding male) that the AI would think is a photo of Taylor Swift.

Hmm some might be “Enchanted”, but that takes Rick Rolling to a new level in fact for some –not sure who though– it might be a “Holy Ground” 😉

(Yes there are other TS titles I could use but those are supposed to be two of her more popular ones).

Clive Robinson • July 22, 2020 11:05 AM

@ ALL,

One the face of it the way it is described as working,

    “Fawkes takes your personal images, and makes tiny, pixel-level changes to them that are invisible to the human eye, in a process we call image cloaking.”

Sounds a lot like the faild DRM watermarking scheme of the late 1990’s early 2000’s.

It would be interesting to see what happens to a Fawked image after it’s been through the UK Cambridge University Computer Labs 2D image mangler that put the final nail in the coffin of DRM Watermarking.

That is what would the AI image software make of it, even though humans would still as easily recognise it.

uh, Mike • July 22, 2020 12:25 PM

This demonstrates a low-tech resistance technique: inject garbage into the adversary’s data stream. Even more effective if the adversary seeks the garbage.

Banksy • July 22, 2020 3:32 PM

How is this different than (seemingly) random graffiti on, for instance, a STOP sign, fooling the self driving AI into registering a 55 speed limit?

vas pup • July 22, 2020 3:39 PM

@ALL:
Could the same idea be applied to your voice? Just curious.
Making it (voice print) recognizable by your human counterpart, but not by Big Brother surveillance applications?

Weather • July 22, 2020 4:06 PM

@vas pup
The one I use for IRS can detect it even if drunk or with a cold, I think the encoding is just amplitude between different frequency but all so what patterns.
A computer program that changes it can match or fool a Ai probably want be recognized by someone that really knows the person, if they don’t it might pass.

Al • July 22, 2020 5:13 PM

Downloaded the windows binary, checked against virus total, one av flagged it.
Tried it out in a sandbox, it made one or more TCP outbound connections to some .edu domain, didn’t catch the full host.
Tried command line:
protection -d img2 -m low –format jpg
Interesting. -m is mode, low, medium, or high. It defaults to producing a png unless output format is specified. Haven’t tested it yet against an image recognition website. To get an idea of the command line switches see:
https://github.com/Shawn-Shan/fawkes/blob/master/fawkes/protection.py

Rachel • July 22, 2020 5:32 PM

Is this service offered in reverse? Tiny, pixel-level changes are made to the humans facial features before the photographi is taken

Metaschima • July 22, 2020 6:33 PM

Really cool technology. My only concern is that my photos are already out there and already tagged with my name, so I wish this had come out in like 2013 when I didn’t have a social media account. It’s my fault of course for being pressured into having one in the first place.

Ismar • July 22, 2020 7:02 PM

I am a bit confused? Is this software to be used on an existing ‘raw’ photos and the results are to be uploaded somewhere on the Internet for the AI engines to process? Is the aim of the whole exercise to have these manipulated photos then be recognisable by the people who may or may not know the way you look, while fooling only the machines?
Any more details on the intended scenario?

echo • July 22, 2020 8:01 PM

This is conceptually interesting and could be layered with other exploits. Some of the more inventive “B” movies and sci-fi television series have explored this. Ultimately stealth is a system and you don’t go foghorning your best tricks. It also helps not to be a “person of interest”.

It’s interesting to compare technical with social systems. They both go about the same problem by different means. Who stands out. Who displays behaviour which is off the mainline. Who is in the wrong place at the wrong time. So an authoritarian regime tracks me from leaving home to wandering around most of every transaction I do with varying degrees of accuracy. Big deal. I just went to the shops. Security by obscurity house of cards made of Swiss cheese wins again.

How much are authoritarian regimes about “maintaining appearances” or simply run by someone who is an overt psychopath or has some other form of mental illness? They are actually a minority.

I think we can get obsessed with technical solutions and forget they don’t exist in a vacuum.

Weather • July 22, 2020 8:54 PM

@Ai
I’m not going to look at it to see if it has a virus, the type of topic and lead removed that.

@rrd
You still there, what was the string you used?

Erdem Memisyazici • July 23, 2020 1:58 AM

You could always run all uploaded images through a filtration process and input that copy if you were running a social media platform so that you would sacrifice precision for noise removal. It shouldn’t make too much of a difference in quality if the noise is undetectable to the human eye.

I wouldn’t be surprised if social media platforms were doing this already as it seems like a security precation from that perspective.

Clive Robinson • July 23, 2020 2:24 AM

@ Rachel,

Tiny, pixel-level changes are made to the humans facial features before the photographi is taken

Oh no don’t wish for things that might happen 😉

Just think of the down side a veritable armarda of californian sail boats heading for you all captained by angry plastic surgeons :-S

Clive Robinson • July 23, 2020 3:04 AM

@ Metaschima,

It’s my fault of course for being pressured into having one in the first place.

It’s not just your peers, imagine being a parent of an averagely bright child, the schools all have web sites these days and they like to advertise…

Luckily back when my offspring were to young to be able to see further into the future and defend themselves schools did not make actually attending the school dependent on you signing up your children into accepting…

One school made noise about my polite decline and made the “if you have nothing to hide” noises. My reply was a week later presenting the administrator with copies of what I had found about them online and in other “Open Source” information. They were not happy but finally understood the point I was making…

I suspect a lot of people in a few years when AI’s will trawl the Internet in full and also “image match backgrounds” etc will have remorse over how they “over shared” information. Where over sharing was just a single photo or hasty comment.

And the people running such trawling AI will make money, because if there is on thing the “Kates and Kevins” of this world want is petty power over other people that they can spin up in their heads to something a lot worse than it could ever be in reality.

And before people say that others could not be that petty, have a look back at what history teaches us. Religion is a good place to start there is a reason why we have the old sayings of “holier than thou”.

Even the bible warns against people with attitudes of Self-righteous, sanctimonious, priggish, sniping and petty behaviour and gives examples. But you can also look up the methods of punishment for some felt to be agregiously so in public of the ducking stool, brides scould, and even the stocks (though women got a seat to protect their virtue). Sadly people with such “I’m better than you” thoughts are generally ineffectual and “not worth the feed”. But unfortunatly as with all vindictive people they are usefull to others who use them often with what we might call evil intent. By simply using a little undeserved flattery or praise of them they can use such people to be set against others.

They are also the people that “demand blood” as part of the “beast” that is drummed up by the media and will be found rousting and inciting in vigilante crowds.

David • July 23, 2020 3:36 AM

So the ML has been processing images in a way completely different to the way that a human recognizes a face. Probably too focused on details like the angles of the extreme ends of the eyebrows to the tip of the nose rather than our bigger picture pattern matching. This allows tiny manipulations to destroy the ML accuracy.

Name • July 23, 2020 4:24 AM

Correct me if I’m wrong, but all it takes is to retrain the model using both “Fawked” and “Unfawked”, correctly labelled images. So it’s only useful while relatively novel.

Anders • July 23, 2020 6:03 AM

@Czerno

You can consider this how our, human brain reacts to optical
illusions. We see for example moving circles while in fact they
are standing still.

Same thing here – you find a vulnerability in AI processing
and you exploit it.

Petre Peter • July 23, 2020 6:42 AM

cs assignment: fool the image classifier. This must have something to do with cs assignments.

c1ue • July 23, 2020 11:36 AM

Interesting and useful tool, but from a system perspective – not clear how beneficial.
Fawkes seems like PGP: yes, secure but use of PGP encrypted email also puts you in the <0.5% of traffic/users, which in turn has now filtered you out as an anomaly for further analysis.
Unless Fawkes can be pushed into major use… That’s why TOR is brilliant: it covers the natsec traffic with hordes of criminal traffic.

c1ue • July 23, 2020 11:38 AM

Sorry, should be previewed above. Meant to say use of PGP encrypted email puts a user in the under 0.5% of email traffic category as an anomaly to be further investigated.

echo • July 23, 2020 7:13 PM

@Clive @Metaschima

Schools have always applied pressure in various ways to comply with agendas. Apart from police officers visiting to give a chat and collect finegerprints under the guise of children playing with fingerprint kits there was the not actually mandatory school photograph. Oh, the peer pressure if you didn’t make a donation to the swimming pool repair fund or new sports van fund.

I put my foot down on this last one because I would never in my life use the sports van which was pretty much exclusively used by the school cricket and rugby teams plus my mum wasn’t rich and they were asking for a relatively hefty donation. I have no idea what my mum thought or whether she had the fist single clue what was going on but I wasn’t having any of this. The fact is I said “no” and wouldn’t bend to their demands or pressures and there wasn’t a lot they could do about it. A pity they didn’t learn no meant no because pretty much all the sports teachers were later prosecuted and did jail time after being caught up in a government led child abuse purge.

So the old state sector trick of trying to get people to sign up to preconditions a.k.a. unlawful T&Cs is doing the rounds with schools? God, they never stop trying it on do they? I always take these things as indicators of abuse of power or greed. If something is going wrong here there’s usually something else going on and someone somewhere is being hurt.

Gemalt-eration • July 24, 2020 12:31 AM

Has anyone had experience on the “subject side” with ring-flash-equipped Gemalto* cameras? I’m quite sure they’re not just the traditional high-res image device most people are used to.

(*) SIM card manufacturer so I would assume some significant undocumented “security features” in their cameras.

echo • July 24, 2020 1:58 AM

Don’t use ring flash on any camera if you can avoid it. I know lots of people use them for “creative” reasons or because they are cheap but seriously they are terrible for anything other than shooting small objects with macro.

I have no idea what Gemalto do or do not build into their cameras. I had no idea they made cameras.

Tatütata • July 24, 2020 7:47 AM

@AI

I wouldn’t worry too much. A glance at the source code shows that module “utils.py” pulls during its initial run some model files from “http://sandlab.cs.uchicago.edu/fawkes/files/” which are then stored locally in a cached directory “.fawkes”. I don’t see this as a big deal, even though https could have been used, and these data could have been distributed alongside the code at Github. If the files contained some nefarious elements, these could be distributed just as well through Github as through that UChicago site. At first glance, they appear to be some sort of neural model for extracting features from a picture.

The code isn’t very long, and I haven’t seen anything suggesting usage logging or uploading of information. If you’re really worried, in Linux you could always run a Python instance under firejail restricting network access. Since the models have been cached on the first run, no further access should be necessary. You could also patch utils.py to make the process more visible.

Since the images modifications are supposed to be transparent to the casual viewer, the countermeasure would therefore be to filter the images according to model of human vision (which would discard the invisible interference) before they are sent on to the classifier. I.e., an arms race should develop.

The Github page states that “We are currently exploring the filing of a provisional patent on the Fawkes algorithm.“. If they haven’t actually filed a (preferably non-provisional) application BEFORE their publication, they are most probably underwater. As others noted, this isn’t the first attempt in this area, so they need(ed) to give a long hard thought as to what makes their approach original and distinguishable from previous ones.

echo • July 24, 2020 8:53 AM

I had a read through the paper and Fawkes basically adds feature morphing just enough to throw off an image to disrupt relatively crude recognition software. It won’t protect against clean samples by law enforcment or other authorised parties nor protect against snoopers. If you can fly under the radar then great otherwise I can’t get too excited about it. Another win for security by obscurity house of cards made out of Swiss cheese.

Tatütata • July 24, 2020 9:55 AM

It won’t protect against clean samples by law enforcment or other authorised parties nor protect against snoopers.

If I were a passport office, I would try to see whether application pictures have been doctored in some fashion. If there is a history of consecutive documents issued to the same person, one could check whether the algorithm can match the pixes over time.

One could operate like the ID card issuers or DMVs in some countries, and take a picture of the subject on the spot with their own equipment.

I installed fawkes on a fresh Python3 setup. I was taken a bit aback when pip3 immediately fetched 50+ different additional packages, resulting in the download of the better part of a gigabyte. But there was nothing really surprising or untoward in that batch.

During the first run, several files were downloaded from the uchicago site, as expected. The ./.fawkes directory contains three files, totalling about 85MB in their compressed form:

high_extract.h5 contains an HDF data structure called “model_weights” with about 400 members. For example, a member called “model_weights.conv5_block8_2_conv.conv5_block8_2_conv.kernel_0” contains a 32x128x3x3 matrix. Phew.

high_extract_emb.p.gz and mtcnn.p.gz appear to be some sort of native NumPy data container, which I’m not familiar with.

I’ll try to make some before-and-after comparisons, when I feel like it.

echo • July 24, 2020 11:06 AM

@Tatütata

I’m sure something similar to what you suggest will happen in practice.

It will be interesting to here how before and after images compare if you ever post results. I’ll admit the thought of downloading and looking into it is a headache plus I’m feeling lazy. The idea is intriguing and one I first heard while watching an episode of “Burn Notice” but loses its usefulness when everyone knows about it. I somehow doubt any amount of magick trickery will make anyone confuse me with Gwynth Paltrow so I wouldn’t bother. It would be more effective going around with a bag over my head and save a few mirrors in the process.

I’m left wondering if this is another one of those solutions in search of a problem things.

JonKnowsNothing • July 24, 2020 11:37 AM

There are existing forensic protocols for evaluating documents and images.

The meta-data and image manipulation audit trails within a document are starting points.

Long time past, when UnDo and ReDo options were added to programs there was a concern because they show the editing and assembly of a document or image. Much of that exists in all pictures as default and piggybacks when you share the images/documents.

That concern is more than valid today but you can hardly find a program that doesn’t have something similar.

There are a few scrubbing options, as in Properties/Details /Remove Properties and Personal Information, but this is not likely to go deep enough to scrub out data inside the file. Mostly it removes your name and tags from the external parts. Some will remove geolocation tags but again, deeper inside the file structures it is still likely such removals can be still detected.

The massive document sets in use would be much harder to fudge because there are “Beellions and Beellions” of source images to validate against.

Then, there are your friends who share your picture on social media…

iirc(badly)
A government report with redactions was obtained by FOIA. There was a paragraph with a big black box in one sentence. It didn’t take long before simple comparative font size calculations determined the number of letters in the redaction. Given the topic, it wasn’t hard to determine there were 3 or 5 words (English) that would fit the redacted space. As only 1 word fit the full redacted space and matched the content of the document the redaction was moot.

Door Number 2: Afghanistan

Smith Mitchell • July 24, 2020 11:40 AM

I liked the answer in Frequently Asked Questions. “Protection level will vary depending on your willingness to tolerate small tweaks to your photos.”

Clive Robinson • July 24, 2020 4:55 PM

@ echo,

Another win for security by obscurity house of cards made out of Swiss cheese.

Well kind of but…

What it demonstrates is that ML currently iscto fragile to be relied upon. Which in some circumstances is actually quite important.

As you know in the UK the Met Police especially are prone to “fudging the figures” and trying all sorts of “technobable” to try to gain a conviction (Colin Stagg over Rachel Nickell being one of the more egregious examples).

The simple fact is the way the police work is to first build a list of suspects then go after the ones on the top. This frequently involves the build up of significant cognative bias. That is the further along in the case they are the more likely they are to loose or forget any evidence that indicates innocence, whilst trying to turn any piece of nonsense into circumstantial evidence… That is they quickly cease to be impartial and that is problematic.

Thus anything that highlights the technobable nonsence for what it is the more pleased I am to see it.

Whilst I respect science and the scientific process, it is all to frequently abused beyond belief by those seeking a prosecutorial advantage. Thus forensics frequently abuses both science and it’s process beyond belief. The biggest in terms of how frequently it is used in the wrong way in forensics is to argue from effect to cause. For good reason science goes from cause to effect primarily because it is reliable in testing a hypothesis, which forensics rarely if ever is.

echo • July 24, 2020 8:40 PM

@Clive

I’ve come across a lot of other police nonsense but they aren’t the only ones pulling stunts based on tricks pulled from the big bag of nonsense. Technobabble and biases seems to be part of the British civil service attitude which permeats the more medicore authoritarian aspects of the UK state. That science and process is abused isn’t a big surprise either.

Having read the wiki article on the case you mention it sounds like it’s one of those where none of theactors in this case come outlooking good. There’s a lot I would take issue with not just the before but during and after.

Commit an act of murder in an “official capacity” and look at all the blind eyes being turned. The police aren’t the only ones for the reasons you describe fudging figures.

Guy • July 25, 2020 1:34 AM

What would really make this soar is if it made all images fed into it look like variations of Guy Fawkes masks or random Juggalo clown faces.

1&1~=Umm • July 25, 2020 4:18 AM

@Guy:

“is if it made all images fed into it look like variations of Guy Fawkes masks”

Do you know how painfull it has been cultivating that image by more traditional means?

It’s not just getting the mustachio right 😉

Fun though it is to twirl between thumb and forefinger, you can not help but ponder why it is “Must Ach IO”… The eyebrow shaping alone is an endless and painful persuit, then there is ensuring the correct palour, living a life of “Vampire Hours” is not as much fun as people think.

People wonder what is done for the sake of art, but hobestly that’s easy compared to cultivating an anonymous life and look.

PattiMichelle • August 14, 2020 10:39 AM

Note that calculation-intensive programs, such as computational fluid dynamics, weather prediction, etc., (matrix-solving software) usually only profit from hardware CPU’s, not threads (virtual CPUs). Using CPUs plus more threads than cores will slow things down. So there should probably be an argument to set the number of CPUs used to the number of hardware CPUs available.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.