Autopilot safety stat cited by Tesla is seriously flawed, research group finds

It turns out that a heavily cited safety stat that Tesla received from the National Highway Traffic Safety Administration (NHTSA) back in 2017 isn't so accurate based on findings from a research group that got their hands on Tesla crash data.

Following a deadly Tesla crash involving Autopilot, Tesla's semi-autonomous advanced driver assistance system, a 2017 federal report found that Autopilot led to a 40 percent crash reduction. But last week a small research group finally obtained most of the data NHTSA used to reach that number and found the underlying data was severely flawed.

The discovery confirms what the group, Quality Control Systems Corp., and others like this report from Wired last May suspected. People had been skeptical of the data for a long time. Even NHTSA said last year the numbers weren't as solid as they had originally indicated. The QCS researchers wrote in their report, "Remarkably, NHTSA’s announcement was not accompanied by any of the data underlying this astonishing claim." So back in 2017 they requested the data and successfully fought for access.

Autopilot software and hardware has changed since the report came out, so this doesn't have much bearing on how Tesla cars function now. Tesla continues to earn high marks in crash safety tests. It's also been years since that data was collected. Tesla's since hit 1 billion miles on Autopilot. The number of Tesla vehicles on the road has ballooned in the past few years.

But now, measuring claims of increased safety on Autopilot and using features like Autosteer that keep the car in the lane isn't as clear-cut. QCS has shown that NHTSA made crash rate calculations using a Tesla data set with missing data and other flaws.

Ars Technica delves further into the math and problems calculating the crash rate if you want to understand the denominators and mileage calculations. The publication also points out other factors affecting safety, like Tesla driver age and socioeconomic status.

Tesla does voluntarily put out its own safety reports -- though those are pretty brief.

UPDATE: Feb. 13, 2019, 6:42 p.m. PST Tesla said in a statement about the QCS report that "given the dramatic increase in the number of Tesla vehicles on the road, their analysis today represents about 0.5% of the total mileage that Tesla vehicles have traveled to date, and about 1% of the total mileage that Tesla vehicles have traveled to date with Autopilot engaged." QCS focused on NHTSA data from only 5,714 vehicles.

A Tesla spokesperson also added that Tesla's own data analysis from 2 billion miles "shows that drivers using Autopilot were significantly less likely to be involved in an accident than those driving without using Autopilot.” Last quarter, Tesla found with Autopilot on drivers experienced one incident for every 2.91 million miles compared to those without Autopilot every 1.58 million miles.