Split-Second Phantom Images Fool Autopilots

Researchers are tricking autopilots by inserting split-second images into roadside billboards.

Researchers at Israel’s Ben Gurion University of the Negev … previously revealed that they could use split-second light projections on roads to successfully trick Tesla’s driver-assistance systems into automatically stopping without warning when its camera sees spoofed images of road signs or pedestrians. In new research, they’ve found they can pull off the same trick with just a few frames of a road sign injected on a billboard’s video. And they warn that if hackers hijacked an internet-connected billboard to carry out the trick, it could be used to cause traffic jams or even road accidents while leaving little evidence behind.

[…]

In this latest set of experiments, the researchers injected frames of a phantom stop sign on digital billboards, simulating what they describe as a scenario in which someone hacked into a roadside billboard to alter its video. They also upgraded to Tesla’s most recent version of Autopilot known as HW3. They found that they could again trick a Tesla or cause the same Mobileye device to give the driver mistaken alerts with just a few frames of altered video.

The researchers found that an image that appeared for 0.42 seconds would reliably trick the Tesla, while one that appeared for just an eighth of a second would fool the Mobileye device. They also experimented with finding spots in a video frame that would attract the least notice from a human eye, going so far as to develop their own algorithm for identifying key blocks of pixels in an image so that a half-second phantom road sign could be slipped into the “uninteresting” portions.

The paper:

Abstract: In this paper, we investigate “split-second phantom attacks,” a scientific gap that causes two commercial advanced driver-assistance systems (ADASs), Telsa Model X (HW 2.5 and HW 3) and Mobileye 630, to treat a depthless object that appears for a few milliseconds as a real obstacle/object. We discuss the challenge that split-second phantom attacks create for ADASs. We demonstrate how attackers can apply split-second phantom attacks remotely by embedding phantom road signs into an advertisement presented on a digital billboard which causes Tesla’s autopilot to suddenly stop the car in the middle of a road and Mobileye 630 to issue false notifications. We also demonstrate how attackers can use a projector in order to cause Tesla’s autopilot to apply the brakes in response to a phantom of a pedestrian that was projected on the road and Mobileye 630 to issue false notifications in response to a projected road sign. To counter this threat, we propose a countermeasure which can determine whether a detected object is a phantom or real using just the camera sensor. The countermeasure (GhostBusters) uses a “committee of experts” approach and combines the results obtained from four lightweight deep convolutional neural networks that assess the authenticity of an object based on the object’s light, context, surface, and depth. We demonstrate our countermeasure’s effectiveness (it obtains a TPR of 0.994 with an FPR of zero) and test its robustness to adversarial machine learning attacks.

Tags: academic papers, cars, machine learning, spoofing

Posted on October 19, 2020 at 6:28 AM • 30 Comments

Comments

Winter • October 19, 2020 7:20 AM

Billboards should not be so close to a road that anything displayed on it would be of interest to an automatic pilot.

If they are that close, the moving image on the billboard will itself distract drivers and cause accidents.

A not so reliable source:
http://www.dailymail.co.uk/sciencetech/article-2280521/Keep-eyes-road-How-emotional-billboards-dramatically-affect-driving-speeds.html

A more reliable source:
dl.uswr.ac.ir/bitstream/Hannan/34171/1/2018%20AAP%20Volume%20117%20August%20%2838%29.pdf

Colin • October 19, 2020 7:55 AM

Sounds like a simple coding fix to me…

If image less than 2 frames or 5 mSecs = ignore

Comments?

Phaete • October 19, 2020 8:33 AM

Looks like they are looking to license out their own software.
If they can catch a big fish in this (unethical?) way it would mean millions.

The Ben Gurion researchers didn’t test their attacks against those other, more multi-sensor setups. But they did demonstrate ways to detect the phantoms they created even on a camera-based platform. They developed a system they call “Ghostbusters” that’s designed to take into account a collection of factors like depth, light, and the context around a perceived traffic sign, then weigh all those factors before deciding whether a road sign image is real.

Phaete • October 19, 2020 8:36 AM

Looks like they are looking to license out their own software.
If they can catch a big fish in this (unethical?) way it would mean millions.

This website gets fuzzier every day, posting without getting any conformation, error or a post. service unavailable, html gets mangled.

Try 2 hope this makes it.

Jerry • October 19, 2020 1:54 PM

They can just make their image recognizer recognize billboards and ignore them entirely.

Thunderbird • October 19, 2020 3:40 PM

For everyone commenting along the lines of “this is a simple fix, just <>,” remember that once you have thirty years of “simple fixes,” you end up with Windows.

Also, recall that security problems frequently lead to arms races. You spend six months on your code to recognize and ignore billboards, and I spend a week setting up projectors to flash stuff onto reflective road signs. Now your fix is either ineffective or it inexplicably starts ignoring road signs.

I do agree about the distractions of billboards, especially the ever-changing ones that seem to be so common today. Hopefully someone will wipe out a few billboard companies with gigantic damage judgements, but that will probably just mean that the new owners of the billboard companies will lobby for legal immunity from suit.

tfb • October 19, 2020 3:41 PM

The obvious weaponization of this is to have the ‘billboard’ be the back of a van / lorry.

metaschima • October 19, 2020 4:53 PM

Great stuff. Definitely highlights the real world implications of autopilot on vehicles. It’s not at all easy to develop an autopilot that handles all instances appropriately, in fact I argue that it’s next to impossible without some type of machine learning embedded in it. Heck, most if not all drivers cannot handle everything that comes at them on the road, me included. Where do you draw the line? You’d have to make sure the driverless car software performs at least as well as your average driver, which I’m thinking is getting pretty close besides these bugs being worked out.

Anon Y. Mouse • October 19, 2020 6:02 PM

Still waiting for an answer from autonomous vehicle advocates
for a simple question: by what mechanism will the police pull over
an autonomous vehicle? There are valid reasons to have this ability
besides traffic infractions, and it will be demanded by law enforcement
agencies and politicians. How will unauthorized use of this mechanism
be prevented?

JonKnowsNothing • October 19, 2020 6:30 PM

@Jerry @Colin

re: They can just make their image recognizer recognize billboards and ignore them entirely.

If image less than 2 frames or 5 mSecs = ignore

Not a good idea for the USA. We have government billboards that provide information about traffic, dangerous road conditions and fire alert status.

We also have official signs that are cantilevered over the road way as well as a variety of fixed signs that are attached to poles, concrete abutments and overpasses. There are signs that blink and can be changed On The Fly by officials. Other signs that blink and change messages are moveable; placed at areas with temporary hazardous road conditions, temporary speed limit changes, temporary detours (often just an arrow).

Ignoring signs, setting a timing delay to “deliberately ignore” information, or ignoring alterations in road, speed and legal directives, would not be good plans.

It’s old news you can do similar “illusion attacks” on road signs like stop signs, yield, speed. Some gorilla tape on the road will do it too. They can also be directed at and reflected in the sensor covers.

These have been discussed in archived posts.

JonKnowsNothing • October 19, 2020 6:36 PM

@Anon Y. Mouse

re: Still waiting for an answer from autonomous vehicle advocates
for a simple question: by what mechanism will the police pull over
an autonomous vehicle? There are valid reasons to have this ability
besides traffic infractions, and it will be demanded by law enforcement
agencies and politicians. How will unauthorized use of this mechanism
be prevented?

afaik These options already exist in cars that have computer driven systems including OnStar or similar.

Computer car systems are more than accessible especially those that have Infotainment Systems. There are a number of paths to access.

LEAs are not going to put all their prizes on display until necessary.

Clive Robinson • October 19, 2020 6:38 PM

@ Phaete,

Try 2 hope this makes it.

They bith did.

With regards identifying a sign post, humans take their time 200mS is fairly fast.

But humans can usually tell a static image of a sign from a static sign when passing reasonably close.

In effect the static sign rotates against a background with perspective that continuously changes, but the static image does not. Thus the way the shape of the sign changes and it’s background is rather different from the image.

I’m guessing it is this that the Uni researchers are working on as a solution.

It’s not very difficult to think up as a concept but somewhat harder to code up.

And to be honest I don’t think it will be a totaly reliable fix. Because humans have evolved their recognition systems from before the time they were human and we still have problems with optical illusions and the like.

Whilst we are clever primates are we clever enough? The suggestion of throwing AI at the problem is beguiling to some, but in all honesty, I would not hold my breath on it happening any more reliably. Sure the illusions that trip AI up will be different to the ones that trip up humans, but I doubt that there will be any less illusions.

Jesse Thompson • October 19, 2020 9:53 PM

@Clive Robinson

In effect the static sign rotates against a background with perspective that continuously changes, but the static image does not.

Just bear in mind that nothing stops an attacker from A: gauging the average vehicle speed near the billboard, B: factoring in the position of the billboard relative to the road, in order to c: render the brief flash of imagery as a multi-frame video complete with moving background that approximates what a sign might look like were it in relative motion much closer to the camera.

So far as the researchers “trying to make the effect not fool human drivers”, that sounds like a waste of time. Anything sufficiently advanced to fool AI no matter how smart it’s made will also fool humans, and anybody trying to create a traffic jam is not going to selectively target autopilot when they can just as easily fool an entire road full of human motorists right along with them.

Adrian • October 19, 2020 11:17 PM

@Colin:

I think the problem with ignoring something identified for only a frame or two, is that a frame or two may be more important than the adjacent frames where something is not recognized. Motion blur, precipitation, temporary occlusions from other objects means you may only have a few frames here or there to detect an important object and determine its trajectory. The sooner you recognize an object, the more time you have to estimate risk and enact evasive maneuvers.

Although it’s not exactly the same thing, consider the case of the Uber self-driving car that killed the pedestrian. Based on my understanding of several articles I read about the investigation, the pedestrian was detected, but the system didn’t initially classify it as a pedestrian. For a few frames, it was an unknown object, then it was another vehicle in an adjacent lane, then it was a cyclist, then it was a pedestrian.

A problem with the software was that every time an object is reclassified, the trajectory information was thrown out and it took a few more frames to recompute a new trajectory from scratch. Apparently, the trajectory estimate is not based simply on observed location, direction, and speed, but also on models of how (and how fast) different types of road users typically move. Vehicles tend to keep to a lane, pedestrians tend to move perpendicular the roadway. Cyclists are typically faster than pedestrians.

By the time the pedestrian was recognized as a pedestrian that would be in the path of the car, the collision was imminent. Had the software heeded the early indication of “unknown object moving into the car’s path” it could have slowed, giving the system more time to collect data, classify it, and improve its estimate of the trajectories.

If instead, the system ignored the first couple frames where it spots an object because it might be a phantom or malicious projection, then it could unnecessarily delay detection of a potential collision.

Sure, ignoring a ten millisecond glimpse of a speed limit sign is much less important than ignoring a pedestrian for ten milliseconds, but I’m not sure such a system could reliably determine which glimpses can be be postponed and which require as much analysis as soon as possible.

SpaceLifeForm • October 19, 2020 11:47 PM

@ Clive

The motorists

It’s true

SpaceLifeForm • October 19, 2020 11:55 PM

@ Clive

The above was pure markdown, Previewed and it looked as expected.

Stuff disappeared after submit.

While it may be valid markdown, it may not be valid markdown-extra.

Or maybe the Preview corrupted it.

Clive Robinson • October 20, 2020 2:18 AM

@ Jesse Thompson,

Just bear in mind that nothing stops an attacker…

Yes I had thought about that, and it would work in the “one car” case and possibly for the “two to three car” case when they are very closely bunched up (tailgating).

But at the “safe breaking distance” at say 45mph I think that the optical effect would be sufficiently wrong to cause a motorist to recognize it as being wrong or a projected image. Obviously the closer the target car is to the bill board the worse the effect for a more distant driver. My back of an envelope projection drawing is shall we say lacking sufficient depth to say how bad and what seperation would be noticable…

But I think no matter which way you did it, it would be unwise to carry out as an attack anyway, the reason being “dash-cams”. Whilst there are not that many per 100k vehicles in the US and UK currently, in places like Russia where “shake-down” crime is rather more common[1] they are getting to the point of being a “fitted as standard” option much like car-radios once were.

I can only make a guess as to how an electronic bill board would look on a dash-cam but the chances are it would be a sufficient anommaly that it would end up on YouTube or similar as a lot of dash-cam footage of accidents and other weird stuff etc does[2] and thus the secret would be out.

[1] But shake-down attacks are happening else where a lot closer to home,

https://www.thecomet.net/news/dash-cam-footage-exonerates-4×4-driver-accused-of-trying-to-hit-jogger-near-stevenage-1-5235307

[2] This realy first came to my attention with all the dash-cam footage of a meteor in the Urals back in Feb 2013. Some of which has been compiled together,

https://m.youtube.com/watch?v=dpmXyJrs7iU

I wondered why there was so much of it, and a Russian friend in Moscow told me about the shake-down crime epidemic back then and he joked with me that his front and rear dash-cams had more than doubled the value of his car. But then more seriously said it was still cheaper than the insurance and other costs would be if he did get hit by a shake-down attack.

Clive Robinson • October 20, 2020 2:24 AM

@ SpaceLifeForm,

Yes, the “markdown” does apprar to be causing strange effects.

@ Bruce Schneier,

Do many people use “markdown” to make it worth having?

David • October 20, 2020 7:00 AM

Human vision is very sensitive to rapid movement at the edge of the field of view. These display billboards can be very distracting and probably cause accidents without any malicious intent

Phaete • October 20, 2020 10:46 AM

@Clive,

I would not hold my breath on it happening any more reliably

I fully agree.
I’m not sure why people expect object recognition and self driving cars to be fully matured. It’s probably a side effect of the “consuming ego” age we are in.

What will we expect more?
AI recognises if a person opens the door of the van infront and holds up a genuine traffic sign?
What about a remote controlled genuine sign popping up at the side of the road?
What if the a man in a van throws styrofoam or some other kind of padding infront of the selfdriving car? How does the car differentiate it from the neighbour’s poodle.

There are just exceptions where you cannot realistically expect the AI to correctly deal with.

This example is riding the wide line inbetween IMHO

It is also about just single camera systems.

If we are going to get this system of object recognition and selfdriving mature we can’t skimp out on sensors.

I would like to see a multitude of sensors, doppler systems, cameras (icl infra red) and good communication/computing between them.

JonKnowsNothing • October 20, 2020 12:23 PM

@Phaete @Clive @All

re: What will we expect more?

AI recognizes if a person opens the door of the van in front and holds up a genuine traffic sign?

What if the a man in a van throws styrofoam or some other kind of padding infront of the selfdriving car?

1, Will the AI recognize a man, woman, child, cis, trans person, fat, skinny, ethnic, aged? The bias is just huge see the current social media flup-up over similar photo poses of skinny vs plump with AI blocking one. Hint: AI likes skinny.

2, Real Life: driving down the highway and person moving had a mattress on the roof of the car tied with twine. Somewhere @50mph the mattress detached…

Real Life: driving down the same old highway to Silicon Valley, a truck carrying construction material traveling @65mph decided to jettison the load of 4×8 plywood onto the highway.

Real Life: same old highway to SV, a truck carrying highway paint dropped several 5gal buckets of it on the lanes.

Real Life: traveling along that fabulous section of road in SV pulling a horse trailer, the back doors of the trailer in front unlatched, leaving several horse derrieres one step away from a nasty encounter with the pavement. (everyone pulled over safely, the horses were fine, we re-loaded the horses in other trailers, the doors needed to be re-welded)

There is an assertion that the car AI can determine the difference after the event happens, it recognizes that the mattress is now in the road. There is no assertion that the AI notices the mattress starting to lift up on the roof of a car that is several cars ahead of yours, or that the AI will recognize horse butts about to hit the pavement.

Clive Robinson • October 20, 2020 1:45 PM

@ Phaete, JonKnowsNothing,

What will we expect more?

Yes much much more, we will expect “total safety” that no human can give.

Without appearing to be insensitive, death on the roads in the US is 30,000 or so each year, and nobody outside of those that knew them actually cares. (Joe Stalin’s “one death…” principle at work).

But a Hundred to two hundred people on a boat, plane, or train, can be world wide front page news for days and sometimes weeks (Air Malaysia and Air France flights that went missing).

We expect “perfect safety” when we pay to be transported, that is we require “A proffessional nut behind the wheel” not our ordinary everyday human fallible selves. Where humanity kidds it’s self as “the majority think they are better than average drivers” but some must obviously be wrong.

I’ve been known to say,

There is no such thing as an accident, they are all entirely predictable and avoidable, with sufficient sensing and time.

I have broken the toe on my left foot at the moment. A few days ago, I’d been moving some large storage batteries and had been wearing safety boots etc, that having just finished moving I was in the process of getting out of. There was a loud crash outside, that sounded like somebody throwing a brick through a window, and I moved towards it both fast and instinctively. In the proces I kicked a battery that had not been there just minutes before in just my socked foot… Yup appart from the crash caused by workmen next door dropping a glass door into my drive way, my broken toe happened by simple predictable physics based entirely on my actions both before and after the crash… And no knowing this does not make my foot hurt any the less : – (

Ulf Lorenz • October 20, 2020 2:03 PM

Stupid question: Why should an autonomous vehicle trust an image in the first place? There are way better sensors (e.g., Lidar) for detecting the situation on a road that do not require complex post-processing to deliver the data you are actually interested in. And a reasonable autonomous vehicle should have a map with the relevant signs. Otherwise it is bound to miss them, for example, when overtaking a truck.

I know that there are essentially cost reasons and Tesla, which pushes its assistance system as “self-driving”, but any vehicle that is fooled by such pranks is overhyped, not autonomous.

Thunderbird • October 20, 2020 2:14 PM

Clive, sorry to hear about your toe. I mean, really sorr–as in “my toe hurts thinking about it.” The “nice” thing about broken toes is there is nothing much they can do for them, either.

As for “markdown,” it seems that a lot of people use it in preference to learning a few HTML tags. Apparently the idea was that if your software supports HTML it frequently means it also supports Javascript which is useful to attackers, but it can support Markdown and you don’t have to worry about embedded Javascript attacks. Or something. Of course, the actual effect seems to be that you end up supporting both poorly, and since there are multiple competing standards for markdown it never works quite the way you think.

The usual problem with not liking an existing mechanism … we’ll create a new better one! But someone doesn’t like our mechanism! As they say, whoda thunkit?

Clive Robinson • October 20, 2020 7:35 PM

@ Thunderbird,

The usual problem with not liking an existing mechanism … we’ll create a new better one! But someone doesn’t like our mechanism! As they say, whoda thunkit?

Well, in engineering there is a saying,

“Standards are like toothbrushes, every one agrees it’s essential you use one, but nobody want’s to use yours.”

But there’s not that many engineers around to start with, and who listens to them anyway, they are a pesimistic bunch who don’t know how to have fun at the best of times (and I know, because I am one[1]).

Whilst flat caps, beards, pipes, real ale and pies are nolonger the hallmarks of an engineer, there still is a personality type tending to a certain reluctance to take risks and be somewhat thoughtful and introspective.

And before you ask yes I’ve a flat cap or two, a beard, a liking for the very occasional pint, and I’m known for not just eating but making pies, so call me “old school” if you like 😉

Oh and remember our host has a flat cap or two and a beard and I believe he has been seen with a pint in his hand. I don’t know if he likes pies but I’m told he used to do restaurant reviews.

[1] Back when I was of University age, there was a bit of a scandal at a well known UK University. Basically Venereal disease went rampent one year so much so it was considered by the health authorities as an epidemic. Some time later various stats about it became available. The highest spread at around half the students was the “liberal arts” and the lowest at zero recorded cases was “engineering”…

JonKnowsNothing • October 20, 2020 9:40 PM

@ Clive @ Thunderbird

re: The usual problem with not liking an existing mechanism … we’ll create a new better one! But someone doesn’t like our mechanism!

Oh that brought on an avalanche of memories….

I do not think there was ever a company I worked in, that did not have at least one of the “don’t like it, won’t use it” types. Sometimes there was more than one and you got dueling “don’t likes”.

Not a single manager, VP or CEO ever stuck their foot on the process; so spaghetti was introduced over and over and over into every project and code base.

The common mantra was they were going to make the code “clean” or “compact” or the current code was not to their “standard” (tabs vs space) and the worst was the code was not “efficient”.

I do not think any of these “don’t likes” ever improved anything and more than not made things worse.

iirc(badly) a rewrite of complex make file. System had huge cross dependencies and multiple passes to get all the objs before the final link. The rewrite omitted about 80% of the dependencies.

iirc(badly) a QA database update required programmers to actually document what they changed. Comments like “fixed bug” “WAI” were blocked so that they would indicate what they changed (vs just rechecking in the same code with a few tabs and spaces adjusted). One engineer decided that Engineering needed their own front end and wrote a UI that omitted important information. Leading to a QA database that was “less than useful” because the Engineers didn’t include any important tag fields just the one that said: Closed.

iirc(badly) the ubiquitous IF THEN ELSE massive Splitter file. I think nearly every company had at least one. A file so enormous that no one would even look at it. If a request came in for the Splitter, the answer was NO. Except the guy who decided to make it a career of rewriting it… it never got rewritten but the pay was good and no one bothered him for other stuff.

The Good Times in Silicon Valley when cases of jolt cola were delivered directly to the mini fridge next to the desk.

xcv • October 20, 2020 10:04 PM

@JonKnowsNothing

The Good Times in Silicon Valley when cases of jolt cola were delivered directly to the mini fridge next to the desk.

That’s High California.

Aside from that, “software engineering” has become a pseudo-scientific career-oriented discipline with the full blessings of Ivy League academia.

The reason for the spaghetti code is that “production” is delegated to junior level code monkeys and artificially and forcefully cut off from proper “design” stages of the workflow.

A strong ethic of propriety software only with draconian legal protections for patents, copyrights, trademarks, and corporate secrets in a heavy district of corporate-friendly law is an article of faith to the entire software engineering discipline.

All free and open source software is against their religion and any involvement in it at all is a heavy sin. Just ask Richard Stallman.

Garabaldi • October 21, 2020 3:46 PM

Humans manage to drive mainly using visual information, with a little help from haptics and audio. The information needed to drive is available from cameras. Trying to parse images will improve the state of the art, even though is not ready (and may never be ready) for public use.

There’s nothing wrong with throwing hardware at the problem to avoid the difficult bits, but there’s nothing that says throwing hardware at the problem will avoid the difficult bits. IMHO we don’t even know what the difficult bits are.

The types of improvements that are necessary (perhaps modelling the persistence of physical objects, even if you don’t know what they are, or become uncertain about what they are) should be useful even if the vision problem turns out to be too hard and additional sensors are needed.

Non-visual sensors will be spoofable in ways that are invisible to humans, in a way that sensors that use the same information humans use will not be.

Clive Robinson • October 21, 2020 4:35 PM

@ Garabaldi,

IMHO we don’t even know what the difficult bits are.

Yup, nor do I suspect we will ever know them all…

It’s part of the reason I said,

“Whilst we are clever primates are we clever enough? The suggestion of throwing AI at the problem is beguiling to some, but in all honesty, I would not hold my breath on it happening any more reliably. Sure the illusions that trip AI up will be different to the ones that trip up humans, but I doubt that there will be any less illusions.”

It’s the same logic as you have applied to hardware and sensors.

Even though they get smaller, lighter, and more capable with time, even exponentially so. The level of hardware and sensors required to reach the next fraction of a percentage closer to 100% safe will always out strip it. But there are limits that the laws of physics dictate for our current technology. We have gone as far as we can with sequential hardware and programming, thus we have no choice but to go down the parallel route, but at what level? And how do we stop the fractional decrease in ability each time we add another path?

Grey • October 21, 2020 11:31 PM

This problem will sure be fixed but toying with a blackbox would always create new one. People talk about the dystopian future but armed with data filled black boxes a dystopia is ready: up and working.

Schneier on Security

Split-Second Phantom Images Fool Autopilots

Comments

Leave a comment Cancel reply