Researchers trick tesla into going 85 mph autonomously with tape on speed limit sign

[spartaman64]

Quote

Hackers at McAfee Advanced Threat Research conducted the experiment.

They explain what they set out to do in a blog post:

McAfee Advanced Threat Research (ATR) has a specific goal: identify and illuminate a broad spectrum of threats in today’s complex landscape. With model hacking, the study of how adversaries could target and evade artificial intelligence, we have an incredible opportunity to influence the awareness, understanding and development of more secure technologies before they are implemented in a way that has real value to the adversary.

Quote

With that in mind, they decided to target MobilEye’s camera system since it’s deployed in over 40 million vehicles, including Tesla’s first-generating Autopilot vehicles, which were used for this specific experiment.

They decided to try to modify speed limit signs in ways that a human would be able to still know the limit, but the automated system could get confused:

Quote

Ultimately, they were able to make a Tesla vehicle on Autopilot accelerate by 50 mph over the limit:

The ultimate finding here is that we were able to achieve the original goal. By making a tiny sticker-based modification to our speed limit sign, we were able to cause a targeted misclassification of the MobilEye camera on a Tesla and use it to cause the vehicle to autonomously speed up to 85 mph when reading a 35-mph sign. For safety reasons, the video demonstration shows the speed start to spike and TACC accelerate on its way to 85, but given our test conditions, we apply the brakes well before it reaches target speed. It is worth noting that this is seemingly only possible on the first implementation of TACC when the driver double taps the lever, engaging TACC. If the misclassification is successful, the autopilot engages 100% of the time. This quick demo video shows all these concepts coming together.

Quote

McAfee confirmed that it disclosed its findings to both Tesla and MobilEye before making them public:

McAfee disclosed the findings to Tesla on September 27th, 2019 and MobilEye on October 3rd, 2019. Both vendors indicated interest and were grateful for the research but have not expressed any current plans to address the issue on the existing platform. MobilEye did indicate that the more recent version(s) of the camera system address these use cases.

In previous instances of vulnerabilities being exposed by white-hat hackers, Tesla has been fairly quick to fix them.

source: https://electrek.co/2020/02/19/tesla-autopilot-tricked-accelerate-speed-limit-sign/

 

Idk why im posting so much tesla stuff i blame google news and their algorithms. Some people have pointed out that on a public road where the car can also get the speed limit from gps data this probably won't be as much of a problem. Just another reason why you should pay attention with autopilot on. Also afaik this only affects older teslas.

[Guest]

To be fair, I would also do 85 if I saw it arbitrarily taped onto a sign.

[GDRRiley]

that first one at a quick glance I'd expect to get most people

[spartaman64]

1 minute ago, Den-Fi said:

To be fair, I would also do 85 if I saw it arbitrarily taped onto a sign.

I think this was what the sign looks like

[Guest]

Just now, spartaman64 said:

I think this was what the sign looks like

I know.

[PlayStation 2]

Is it gonna start stopping at Sop signs now?

[Taf the Ghost]

Weird trolling opportunities in SoCal...

[spartaman64]

2 minutes ago, handymanshandle said:

Is it gonna start stopping at Sop signs now?

 

if it was more octagonal probably

[rcmaehl]

What happens when you accept data without context.

$potentialspeedlimit = OCR.GetSpeedLimitSign()
If $potentialspeedlimit > 55 and Is.Road("Highway") then $speedlimit = $potentialspeedlimit
 

Heck we can add some additional checks...

$potentialspeedlimit = OCR.GetSpeedLimitSign()
If $potentialspeedlimit > 55 and Is.Road("Highway") and ($potentialspeedlimit - 20)  ≤ $speedlimit then $speedlimit = $potentialspeedlimit

[Sauron]

Can we really count sign vandalism as a problem? I mean, what if it were completely spray painted black? Would a human know how fast they can go in that case?

 

To improve the autopilot Tesla could make it so that if the uncertainty of the sign recognition is above a certain level the car will double check with GPS and a map before accelerating, or go with the lowest of the two speed limits (wouldn't want it to ignore a temporary slow down sign because the map wasn't updated in time).

[Sauron]

Just now, VegetableStu said:

is there a situation where the autocar has to refer to the camera for speed limit guidance rather than using map data of the country itself? o_o

Arguably, always. The signs you actually come across should have precedence over what the map says. With that said, if the car isn't sure of what the sign says and the map says 35 then it makes sense to drive 35 - it's not a crime to drive slower than the limit.

[Sauron]

16 minutes ago, VegetableStu said:

surely google maps (for example) has speed limit data for (almost) every road out there?

Yes, but it's not always up to date, particularly when there are temporary changes. On the other hand you can always trust signs (if they're wrong you can contest the fine, not so with GM).

[Master Disaster]

45 minutes ago, Sauron said:

Arguably, always. The signs you actually come across should have precedence over what the map says. With that said, if the car isn't sure of what the sign says and the map says 35 then it makes sense to drive 35 - it's not a crime to drive slower than the limit.

Yep.

 

As a human driver you should already be somewhat aware of the legal speed limit on a given road. If you saw a 50 sign in the middle of a suburb for example you'd know it probably wasn't correct. The car doesn't have that knowledge so it makes sense to include a fallback in the system. It seems like the obvious solution is to tell the car what speed limit signs actually exist around the world and instruct it to ignore any other numbers it encounters, heck they could include a database of images and have the car cross check the sign against a known image before it increases or decreases speed. The taking a slegdehammer to crack a walnut approach is to simply ask the driver if the sign as accurate before increasing speed.

[Tech_Dreamer]

in other news researchers used a Deer crossing sign to trick Tesla into thinking there was a Dolphin pool up ahead.

 

[Dylanc1500]

57 minutes ago, VegetableStu said:

surely google maps (for example) has speed limit data for (almost) every road out there? o_o i mean i'm with you for going to default street speed when unsure

There are many roads and state highways in the U.S. that they don't, however, with that said some of those roads you probably wouldn't want to use autopilot on.

[RonnieOP]

Tbh i didnt even think they worked off speed limit signs. 

 

My GPS somehow knows the speed limit on every road im on so i figured tesla would work like that using satilites or something.

 

I wish they would ban these autopilot features on public roads. Innocent people shouldnt be at risk because tesla wants their customers to be their test animals.

 

How many people have been in accidents over autopilot now?

 

Outlaw the shit until its ironed out. 

[Lady Fitzgerald]

1 hour ago, Sauron said:

...it's not a crime to drive slower than the limit.

Actually, in some places it is. Some have a posted minimum speed limit ,some may say you have to be going no less than a certain amount below the speed traffic is going, others may nail you for obstructing traffic, driving at an unsafe speed, etc.

[Salv8 (sam)]

1 hour ago, spartaman64 said:

Some people have pointed out that on a public road where the car can also get the speed limit from gps data this probably won't be as much of a problem

my car has this feature, it's a good safety feature but dear god is it annoying.

*you are over the speed limit by *5* kilometers*

I FUCKING KNOW!!!!! IM ON A STREET WITH NO ONE ELSE ON IT, WHO THE FUCK AM I GONNA HIT?!?!?!?!?!??!

[spartaman64]

10 minutes ago, RonnieOP said:

Tbh i didnt even think they worked off speed limit signs. 

 

My GPS somehow knows the speed limit on every road im on so i figured tesla would work like that using satilites or something.

 

I wish they would ban these autopilot features on public roads. Innocent people shouldnt be at risk because tesla wants their customers to be their test animals.

 

How many people have been in accidents over autopilot now?

 

Outlaw the shit until its ironed out. 

only the older teslas

[RonnieOP]

5 minutes ago, Salv8 (sam) said:

my car has this feature, it's a good safety feature but dear god is it annoying.

*you are over the speed limit by *5* kilometers*

I FUCKING KNOW!!!!! IM ON A STREET WITH NO ONE ELSE ON IT, WHO THE FUCK AM I GONNA HIT?!?!?!?!?!??!

Yeah. My gps blinks red when i go even 3 miles over the speed limit.

 

Drives me crazy seeing it blink. 

 

I know the speed limits 65mph. But going 75 im holding traffic...stop blinking at me lol

[Monkey Dust]

I find it curious how easy A.I can find some tasks that are borderline impossible for humans, but can get floored by others that we find trivial.  

[Bombastinator]

Bugs in this stuff are likely to keep on being found.  They’ll get harder though.  It’s how bugs seem to work, mostly.

This happened to UNIX too back in the day.  The OS is considered insanely stable not because it was bugless but because it’s bugs were found and stomped out.
 People are still finding bugs in intel stuff but they’re getting more and more esoteric.

[rcmaehl]

@spartaman64

 

Stolen from Reddit:
 

Quote

This isn't even a Tesla system. They're using the old MobileEye cars from 2016. MobileEye would need to fix it then Tesla would have to push a software update.

The current generation Tesla AP/FSD uses OpenStreetMap data for speed limits.

This seems very clickbaity in the fact that it doesn't mention Tesla hasn't used MobileEye's AP system in almost 5 years.

 

[Sauron]

21 minutes ago, Lady Fitzgerald said:

Actually, in some places it is. Some have a posted minimum speed limit ,some may say you have to be going no less than a certain amount below the speed traffic is going, others may nail you for obstructing traffic, driving at an unsafe speed, etc.

Sure, but those are edge cases. When there isn't a minimum speed sign then going slower when unsure is a good idea; when there is one then there isn't really a solution that covers all potential mistakes in a legal way, in which case it may be best to just stop on the side of the road and prompt the passenger for instructions.

[Lady Fitzgerald]

3 hours ago, Sauron said:

Sure, but those are edge cases. When there isn't a minimum speed sign then going slower when unsure is a good idea; when there is one then there isn't really a solution that covers all potential mistakes in a legal way, in which case it may be best to just stop on the side of the road and prompt the passenger for instructions.

No edge cases about it. Go to https://www.nhtsa.gov/sites/nhtsa.dot.gov/files/documents/summary_state_speed_laws_12th_edition_811769.pdf and scroll down until the pages stop being Roman numerals and are numbered with Arabic numerals. It then goes into detail what minum speed limits apply for each State in the SSA. Many States have (or had) posted minimum speed limits (AZ used to post minimum speed limits); others leave it to the LEO's discretion.