Self

This article is part of our exclusive IEEE Journal Watch series in partnership with IEEE Xplore. The idea of letting a machine do the driving for you may put a lot of people off autonomous vehicles. But research could make it possible to backseat-drive an autonomous vehicle just as you might with a human driver. Self-driving cars carefully balance a host of parameters to ensure a smooth ride, including things like speed, acceleration, and the smoothness of turns. But human driving preferences can often vary depending on how much of a rush they’re in, whether they’re feeling carsick, or how busy the traffic is. These cars have a software component called the motion planner, which is responsible for choosing a safe and efficient path through traffic. The motion planner is normally tuned by engineers before the vehicles hit the road so that there’s little scope for passengers to adjust a vehicle’s driving style on the fly. But now researchers at the Delft University of Technology (TU Delft) in the Netherlands have developed a system that uses a large language model (LLM) to translate natural-language user requests such as “I am running late, go fast” into adjustments to a self-driving control system. The researchers posted their preprint on arXiv and are presenting the work at the IEEE Intelligent Transportation Systems Conference in September. LLMs Personalize Autonomous Driving The system doesn’t give users direct control over the vehicle’s driving decisions; it simply tunes the parameters of a safety-aware motion-planning algorithm, which helps to keep the vehicle’s behavior within safe bounds. And the system keeps the human in the loop by describing how it’s going to alter its behavior in nontechnical language, and by asking the passenger to confirm before making changes. When the system was tested in simulation, the researchers found it adjusted the speed and smoothness of driving in line with natural-language instructions. “The motion-planning problem is not only about reaching a place while avoiding collisions, it’s also how you do it,” says lead author Diego Martinez-Baselga, a postdoctoral researcher at TU Delft. “The motivation here is trying to make the way the autonomous car drives adaptable by end users easily, just by talking to the car.” Previous research has investigated the potential of using LLMs and video-language models (VLMs) to direct decision-making for self-driving vehicles, but the researchers deliberately targeted driving style instead. Using LLMs and VLMs to directly control vehicles faces several challenges, says Martinez-Baselga. These include relatively slow response times, which can make these models unsuitable for the fast-paced decision-making required in driving, and the fact that they can’t provide concrete performance guarantees in the way a deterministic motion planner can. Instead, the researchers used an LLM’s language and reasoning capabilities to translate fuzzy human preferences into something a vehicle’s motion planner can use. The system relies on a model predictive-path integral controller previously developed by the researchers, which identifies multiple paths the vehicle could take to reach its goal and then judges them on various criteria, including speed, steering angle, and collision probability. It then finds an optimal path that is a combination of the trajectories that scored best on those judging criteria. The team combined this with OpenAI’s GPT-4o-mini model to parse passengers’ natural-language suggestions and use them to tune how the controller chooses its path. The model is given the users’ prompt and a natural-language description of the scenario the vehicle is operating in. The description was handwritten by the researchers for the purposes of the study, but it could ultimately be provided directly by a car’s perception system, says Martinez-Baselga. The model doesn’t directly tweak the settings of the controller; it uses the prompt to rate the relative importance of the judging criteria the controller uses to assess trajectories. This rating is then used to adjust each criteria up or down either side of a safe baseline set by the researchers. So, if a user says they are feeling dizzy, the LLM will dial up parameters that encourage smooth steering and gentle acceleration to make the vehicle favor more sedate travel. Prior to making any changes, however, the model first presents the user with a natural-language description of the adjustments it plans to implement. The user can then sign off on the plan or make further suggestions. The system is also interactive, so the user can request further adjustments if the vehicle’s behavior doesn’t match expectations or the user‘s preferences change. Martinez-Baselga says this human-in-the-loop system allows the passenger to catch instances when the model misinterprets prompts. But it also helps deal with the inherent subjectivity of suggestions like “go faster” or the possibility that models don’t accurately describe changes they plan to make. In that case the passenger can simply follow up with additional prompts “as you would do if you were in a taxi or with a friend that is driving,” says Martinez-Baselga. The researchers tested the system in the popular self-driving simulator nuPlan in scenarios that involved merging onto a busy highway. Across eight different prompts, the system changed the controller’s parameters in ways matching user intent, with requests for a more comfortable ride dialing up smoothness and those indicating urgency leading to higher speeds. This isn’t the first time LLMs have been used to tune a self-driving car’s motion planner. Nicolas Baumann, a Ph.D. student at ETH Zurich in Switzerland, published research last year in which an LLM tweaked the parameters of a model racing-car controller, allowing the user to alter driving style but also give more concrete instructions like “reverse the car” or “maintain a specific speed.” The strength of the approach, says Baumann, is that separating the LLM from the main controller means that even if the model hallucinates, it can’t do anything dangerous. “You get the possibility of language interaction, but you can guarantee that it is going to be within the constraints of this classical controller, so you can bake in safety,” he says. However, setting these constraints requires considerable engineering work, he adds. And if you want provable safety, you need to go a step further, says Matthias Althoff, a professor of cyberphysical systems at the Technical University of Munich. His group built a system that gets an LLM to suggest driving decisions, but then uses a mathematical process to check them against traffic rules and predictions about the behavior of other road users. This makes it possible to verify their safety before committing to them, something the Delft paper doesn’t provide. “As with any LLM, it is not guaranteed that the result is correct,” says Althoff. “For that reason, we safeguard the decisions of the LLM in our works.” Edd Gent is a freelance science and technology writer based in Bengaluru, India. His writing focuses on emerging technologies across computing, engineering, energy and bioscience. He's on Twitter at @EddytheGent and email at edd dot gent at outlook dot com. His PGP fingerprint is ABB8 6BB3 3E69 C4A7 EC91 611B 5C12 193D 5DFC C01B. His public key is here. DM for Signal info.