LAMBDA: The Nuro Driver's Real-Time Language Reasoning Model

LAMBDA understands the high-level behavior and intention of the vehicles in front of the autonomous vehicle, and can reason about how it should react to them.

LAMBDA can understand important details about vulnerable road users, such as the pedestrian in the left scene that is walking a leashed dog, or the pedestrian in the right scene that is jogging.

Left: LAMBDA speculates the behavior of the oncoming vehicle when it is far away, and later correctly understands that it is yielding to the autonomous vehicle. Right: While the traffic light is green, LAMBDA correctly understands that the autonomous vehicle is remaining stopped because of the line of traffic.

LAMBDA can refer to road users by their unique IDs assigned by Nuro’s perception stack. On the left the response refers to the oncoming vehicle with ID 3139, and on the right the response refers to two cyclists with IDs 813 and 925.

On the left LAMBDA answers questions about a situation where the traffic light is green, but an oncoming vehicle is turning in front of the autonomous vehicle. On the right LAMBDA answers various questions about two pedestrians and their dogs that are crossing the street ahead.

LAMBDA can understand the underlying reason behind different driving decisions. On the left the autonomous vehicle is slowing down to give space to a vehicle reversing out of a driveway, and on the right it’s decelerating before a left turn to let an oncoming cyclist pass first.