Quantifying What-Ifs in Simulation

The Nuro Driver has completed over 1.2 million autonomous miles with zero at-fault incidents. We are proud of this achievement; however, it means we have limited real-world data on how the system behaves in the event of a collision.

Because only recreating crashes at our closed-course testing facility is expensive, we also rely heavily on simulation. Simulations allow us to safely and efficiently explore many scenarios that are too risky to test on the road. They also allow us to explore many different versions of each scenario, accounting for variations in the environment, in how the other vehicle acts, and in how the Nuro Driver responds.

To better understand all of the possible outcomes of a conflict, we’ve developed a framework called Fractional Collisions. Fractional Collisions allow us to estimate the likelihood and severity of a crash, giving us a probabilistic risk signal that reflects the spread of possible environments, real dynamics, and road interactions rather than a yes-no collision outcome for a scene.

With this approach, we can better compare AV behavior to humans, evaluate model releases, and ensure continuous safety improvements.

How It Works

Our methodology starts with sensor data—either collected by our Nuro Driver-powered vehicles or pulled from large-scale naturalistic driving databases. We simulate two-agent conflict scenarios between the Nuro Driver and another road user, where one agent initiates an interaction and the other responds.

For each conflict, we:

1. Detect and classify the conflict type from a taxonomy of options

2. Determine which agent is the initiator and which is the responder

3. Identify the responder’s point of reaction

4. Model plausible human responses using probabilistic counterfactual trajectories

Using probabilistic behavioral models, we simulate how the responder might have acted if timing, orientation, or decisions had been slightly different. We then analyze those alternate timelines to compute velocity differentials at collision, and apply established crash models to estimate injury or damage risk.

The Fractional Collision in a scene can be interpreted as the fraction of the human population that could have avoided the collision (in an agent-initiated conflict) or experienced the collision (in an ego-initiated conflict). Fractional Collisions can be numerically added across scenes to estimate total collisions.

This process allows us to extract dense and meaningful safety signals even when no actual collision occurred, and to compare responses and outcomes between human drivers and the Nuro Driver.

Validating with Real-World Data

To evaluate the accuracy of our approach, we tested it on over 300 reconstructed conflict scenarios sourced from two diverse datasets: the VTTI SHRP2 naturalistic driving study and Nexar dashboard camera footage.

We recreated each scene using recorded vehicle trajectories, simulating plausible alternate outcomes and computing Fractional Collisions, and we found that our methodology predicted Fractional Collisions within 1% error of SHRP2 ground truth. This high level of agreement demonstrates the reliability of Fractional Collisions as a proxy for real-world data and affirms the framework’s readiness for use in broader system evaluation.

Evaluating the Nuro Driver in Practice

Once validated, we applied Fractional Collisions to evaluate the performance of a production software release of the Nuro Driver. We did this across two types of conflict scenarios: agent-initiated (where the AV responds to another road user) and ego-initiated (where the AV initiates the interaction).

We evaluated agent-initiated conflict scenarios using two distinct sources:

• Synthetic reconstructions: Using the same dataset as our framework validation, we replaced the original human responder with a simulated Nuro Driver and compared the resulting Fractional Collisions. Impressively, the Nuro Driver showed collisions in 4x fewer scenes than the original human responses, and Fractional Collision severity was reduced by approximately 62%.

• 250,000 miles of open-loop sensor data: We also applied our methodology to proprietary sensor data collected from our test vehicles, re-simulated using the evaluated Nuro Driver software. We found that the Nuro Driver improved collision risk in 96% of the agent-initiated scenes.

These findings confirm that Fractional Collisions can meaningfully capture performance differences between human drivers and AV software under identical conditions.

In the same 250,000-mile simulation dataset, we also measured how the Nuro Driver performed when it was the one initiating the conflict. The simulated Nuro Driver-initiated interactions resulted in only 0.4 injury-causing and 1.7 property-damaging frontal or side Fractional Collisions across the entire dataset. This highlights the conservatism of the Nuro Driver’s planning behavior and its ability to avoid unreasonable risk.

Bridging Simulation and Reality

Simulation isn’t perfect—it’s affected by how we model the environment, vehicle dynamics, and agent behavior. One key advantage of the Fractional Collision framework is that it helps account for these imperfections by representing and analyzing the full range of possible outcomes using probabilities.

We’ve demonstrated the framework in terms of evasive agent behavior; however, its generalizability makes Fractional Collisions a powerful tool for quantifying and mitigating the gap between simulation and the real world.

Why It Matters

Fractional Collisions allow us to evaluate safety across a wide range of situations, not just the rare ones where a crash occurs. They help us:

• Understand how close we came to a collision

• Compare AV and human behavior in the same scenario

• Track software improvements across different releases

• Evaluate performance in simulation with confidence

As autonomous systems become more capable—and real-world incidents become rarer—these kinds of detailed, forward-looking metrics become increasingly important. They give us insight into edge cases, near misses, and system improvements that wouldn’t show up in traditional metrics.

Looking Forward

At Nuro, we believe the future of autonomy calls for a deeper, more forward-looking standard of safety—one that doesn’t just reflect outcomes, but also anticipates possibilities.

Fractional Collisions bring that vision to life, quantifying not only what happened, but also what nearly did.

By turning “what if” into actionable insight, we’re advancing the way safety is understood—and strengthening confidence in every decision our vehicles make.

Join Us in Shaping the Future of Mobility

If you’re passionate about solving complex problems and pushing the boundaries of technology, we’d love to hear from you. Join our team!

"Fractional Collisions: A Framework for Risk Estimation of Counterfactual Conflicts using Autonomous Driving Behavior Simulations" was presented at CVPR 2025’s Workshop on Data-Driven Autonomous Driving Simulation (DDADS).