Waymo says it built a better benchmark for comparing robotaxis to humans

Waymo has created a new computer model designed to more accurately answer a fundamental question: how does its autonomous driving software stack up against humans?

The Alphabet-owned robotaxi company, which developed the computer model of human driving capabilities in conjunction with TU Delft, published a research paper about it in Nature Communications on Wednesday.

Waymo said it expects the new model to be more accurate than the previous version it has used over the past several years. The new model was built using a framework called active inference — the theory that a driver is constantly imagining possible futures and taking actions to reach the safest, most predictable one.

Waymo said the new model will help it better understand how humans behave in crash scenarios that its robotaxis encounter.

“For decades, the automotive industry has used physical and virtual crash dummies to evaluate a car’s safety features, including its hardware and structural integrity,” Waymo wrote in a blog post on Wednesday. The new model, Waymo said, “evolves this concept, serving as a behavioral benchmark for autonomous driving systems able to realistically represent reasonable expectations on how a careful and competent human driver responds to traffic conflicts.”

A more accurate model of human driving behavior is table stakes for autonomous vehicle companies that need to understand and grade the performance of its robotaxis in crashes. And it comes at a critical juncture for Waymo, which is scaling to more cities and facing greater scrutiny from regulators and the public.

In January, when a Waymo robotaxi struck a child near a school in Santa Monica, California, the company relied on its previous computer model to claim that an attentive human driver would have made impact at around 14 miles per hour. The Waymo robotaxi hit the child at just 6 miles per hour, after decelerating from 17 miles per hour, and the company said she sustained minor injuries. (The crash is still under investigation by the National Highway Traffic Safety Administration and the National Transportation Safety Board.)

The biggest difference between this new model — which Waymo calls the Reference Driver — and its predecessor is that it is able to reproduce a human driver’s behavior in the run-up to a crash. Previously, Waymo’s models (and other industry models) focused on replicating “last-second, reactive” human maneuvers, according to the company.

The Reference Driver, meanwhile, can “simulate the internal ‘surprise’ a driver feels during a conflict, providing a more human-like benchmark for autonomous driving systems that was previously impossible to automate at scale,” Arkady Zgonnikov, an assistant professor at TU Delft, said in a statement.

Waymo says this new driver model can be adapted to model a “wide range of road user behaviors beyond collision avoidance,” and that it is better-equipped to be applied to “large test sets with thousands of scenarios.”

“The model can represent and evaluate numerous complex, real-world crashes in a virtual environment, identifying performance improvements with unprecedented speed and efficiency,” the company wrote.

Waymo wants others to collaborate on pushing the Reference Driver further, too. The company said Wednesday that it is making the research code for the model available under an academic, non-commercial license that allows it to be used for research, teaching, personal experimentation, and scientific publication.