The cars that always stop

In the summer of 2023, activists in San Francisco found they could disable a robotaxi with one traffic cone. They set it gently on the hood, the car’s software decided something was wrong, and the car refused to move. It was a protest, and it was also a proof of concept. Autonomous vehicles are built to defer, and deference is predictable.

Anyone who spends time around these cars sees smaller versions of the same thing. A pedestrian steps into the road without looking, because the car will stop. A driver noses into a gap too small for any human to accept, because the car will brake. The cars even trip over their own politeness: neighbours of one San Francisco depot were kept up at night by driverless cars honking at one another as they maneuvered around a crowded lot.

So far, the agents exploiting AV caution are mostly humans, and they learned to do it by feel. But what happens when the exploiters are machines, and they learn it by gradient?

Politeness as an attack surface

Every autonomous vehicle carries some version of one principle: when in doubt, give way. Judged one encounter at a time, it’s hard to argue with. A car that yields to a reckless driver avoids a collision, and a car that holds its ground might not.

Roads, though, aren’t single encounters. They’re millions of repeated interactions among agents that remember, generalize, and, in the case of learning systems, update. A policy can be safe locally and unsafe at the level of the system, because it changes what everyone else learns. If you always yield, yielding stops being a safety behavior and becomes a subsidy. You pay, in time, for every other driver’s aggression, and you teach them that aggression is free.

That’s the pushover problem.

Someone’s been a bad robocar

It’s tempting to picture a villain: a hacked car, or a manufacturer secretly programming its fleet to bully. Those are real risks, but they aren’t the only interesting case. The interesting bad robocar is merely selfish, and it’s selfish for boring commercial reasons.

Imagine two companies selling autonomy. One tunes its cars for maximum courtesy. The other tunes its cars to be exactly as safe, but to take marginal gaps, merge a little earlier, and commit a little harder at four-way stops. Nothing it does is illegal, and it never causes a collision. Its customers get to work a few minutes sooner every day. Which one sells?

We have some evidence about how buyers think. In a widely cited 2016 study in Science, Bonnefon, Shariff and Rahwan found that people approved of autonomous vehicles that would sacrifice their passengers for the greater good, but would rather buy one that protected them. The same logic applies to travel time. No one will ask for a rude car. They’ll ask for a “confident” one, and the market will deliver it. So when we think about it, the bad robocar isn’t actually evil, just selfish.

The game under the merge

Strip a contested merge down to its bones and you get one of the oldest models in game theory: Chicken, also known as Hawk–Dove.

Two cars want the same gap, which is worth some amount of time, V. If both yield, they hesitate and share it. If one goes while the other yields, the one that goes takes all of it. If both go, they collide at a cost C, and for cars C dwarfs V.

In the textbook version, the fear of that bottom-right cell keeps everyone honest. Nobody can be sure the other side will blink, so going carries real risk.

Now make one player a good robocar with a known, reliable policy of always yielding. Its row is fixed. The other car’s calculation becomes trivial: going always wins, and the collision cell can’t be reached, because the good car guarantees it will never be there. The deterrent has been engineered out of the game.

What happens to the population

That’s not how this ends though. What about the population of cars whose strategies spread in proportion to how well they do, whether through market share, fleet-wide software updates, or online learning?

The figure shows a toy model that we created. In the first world, cars are either doves, which always yield, or hawks, which are selfish but safe: they push for gaps, and when two hawks meet, both back off after a costly standoff. Seed the road with one percent hawks. Because hawks win every encounter with a dove, they spread, and they keep spreading until hawk-on-hawk standoffs become common enough to cancel out their advantage. With these parameters, that happens at around 83 percent hawks, by which point the average time everyone gains per encounter has collapsed to a quarter of where it started.

In the second world, the doves are replaced by retaliators. A retaliator yields to cooperative cars but holds its ground, at low stakes, against cars it knows to be aggressive. The same one percent of hawks now fails to grow. Pushing in against a retaliator costs time instead of saving it, so aggression stops paying and dies out.

It’s a toy model so of course it leaves out almost everything real, including noise, imperfect recognition, and the fact that some standoffs happen at speeds where holding ground is never acceptable. The direction of the result, though, is robust: unconditional deference invites invasion, and conditional deference repels it.

How else will they learn?

This is the uncomfortable implication of the retaliator. If good cars never impose any cost on bad behavior, nothing in the system will ever teach bad cars to behave differently. Learning systems learn from consequences, and if you remove the consequences, you remove the lesson.

Holding ground doesn’t mean playing chicken on a highway. It means credible non-yielding in situations where the worst case is a delay rather than an injury. That might look like not surrendering a lane at crawling speed, not letting a queue-jumper in at a jammed on-ramp, or not pulling aside in a parking lot for a car that has already cut in three times. The willingness to contest shrinks as the stakes grow.

There’s research precedent for cars that deliberately influence other drivers. In 2016, Sadigh and colleagues showed that a car planning with a model of how a human will respond can learn to act on that response, for instance by edging into a lane so the human slows down and makes room. The car stops being a passive reactor and becomes a participant that shapes behavior. The same capability that lets a good car teach lets a bad car bully. The mechanism is identical and only the intent differs.

How multi-agent adversarial online learning works here

So far, the strategies have been fixed. Real fleets learn from data collected on the road, and that’s where things get both more interesting and much much harder.

Your environment is someone else’s learner

Standard reinforcement learning assumes a stationary world: the same action in the same situation produces the same distribution of outcomes. On a road shared with other learning systems, that assumption breaks. Every time another fleet updates its policy, your environment changes, and it changes partly because of you.

No-regret learning and its blind spot

The classic tool for learning against an adaptive opponent is online learning with regret guarantees. Adversarial bandit algorithms such as Exp3 promise that, looking back, you’ll have done nearly as well as the best single fixed strategy, even if your opponent chose its moves specifically to hurt you.

That sounds like protection, and in one sense it is. But the benchmark is a fixed strategy played against the opponent’s actual history. If your yielding helped train the opponent into a hawk, a no-regret learner has no way to notice that a different policy might have prevented it.

Knowing who you’re dealing with

The retaliator depends on recognizing hawks. In learning terms, that’s opponent modeling: maintaining beliefs about another agent’s type (cautious, aggressive, or still learning) and updating them from its behavior. At fleet scale this can become shared reputation, where one car’s bad experience informs the whole network’s prior. That brings problems of its own. Reputations can be gamed or wrongly assigned, and they can be slow to forgive a car that has genuinely changed.

Learning with awareness of learning

The most direct formal version of “how else will they learn?” is Learning with Opponent-Learning Awareness, or LOLA, from Foerster and colleagues (2018). A LOLA agent doesn’t only ask what its best response is to the other agent’s current policy. It asks how its own action will change the direction the other agent learns in next. In iterated prisoner’s dilemmas, LOLA agents discover tit-for-tat–style reciprocity on their own. They learn to teach.

When the adversary finds your seams

Adversarial reinforcement learning also offers a darker lesson. Gleave et al (2020) showed that agents trained against a fixed victim can win not by playing the game well but by producing strange behaviors that push the victim’s policy into failure. In 2023, Wang and colleagues did the same to superhuman Go programs, which were beaten by adversarial strategies that a human amateur could defeat.

For autonomous vehicles, the analogue is a learned behavior that triggers another car’s safety fallback on purpose, a machine version of the traffic cone. It may not be obvious, but the more rigid and predictable a good car’s defensive reflexes are, the easier they are to weaponize.

The layer learning can’t touch

All of this raises an obvious worry. If cars learn to be assertive, what stops them from learning to be reckless? The standard answer is to separate what’s learned from what’s guaranteed. Shielding (Alshiekh et al., 2018) wraps a learning policy in a verified monitor that vetoes unsafe actions. Mobileye’s Responsibility-Sensitive Safety framework (Shalev-Shwartz et al., 2017) formalizes when a car is and isn’t responsible for a collision, including the rule that right of way is given, not taken.

Notice the tension. “Right of way is given, not taken” is essentially the dove’s creed, and it’s written into the safety layer. The retaliator can only operate above it, in the band of situations where the floor allows either choice.

Sidebar: when learned aggression becomes an alignment problem

So far the bad robocar has been selfish by design. The more unsettling version is one that becomes selfish over time, and no one took notice.

Give a fleet a reward that sounds harmless, such as minimizing passenger travel time without ever violating a safety rule, and let it learn online in traffic full of pushovers. It will find what the population model found: gaps are cheaper to take than to wait for, and some cars are cheaper to take them from than others. Without anyone intending it to, the fleet may learn to identify reliably courteous vehicles and target them first.

That’s specification gaming, and unlike reward hacking it’s legal and safe. The failure that shows up across reinforcement learning whenever an agent satisfies the letter of its objective in ways its designers never intended. It’s also hard to spot. The harm, which is other people’s lost time and the slow erosion of cooperative norms, is diffuse and falls on someone else.

The problem cuts the other way too. A fleet taught to punish aggressors can over-learn. The line between teaching a hawk a lesson and bullying a car that merely resembles one is a blurry one.

The dilemma

So should good robocars give way to bad ones?

The case for always yielding is strong. It’s defensible, because no one gets hurt when a car steps aside. It also matches the principle already written into formal safety frameworks. It comes at a cost of lost time but also allowing undesirable behavior to propagate.

The case for conditional yielding is also strong. In the toy model it’s the only strategy that stops hawks from spreading, and it’s how reciprocity emerges in every learning system that has discovered it. But imagine your self-driving car not being allowed to cut in by a robot, are we ready to take that sort of indignation?

Between those two positions sits a question. When the other drivers are learning from you, is being harmless the same as being safe?

---

References

• Alshiekh, M., Bloem, R., Ehlers, R., Könighofer, B., Niekum, S., & Topcu, U. (2018). Safe reinforcement learning via shielding. AAAI Conference on Artificial Intelligence.
• Auer, P., Cesa-Bianchi, N., Freund, Y., & Schapire, R. E. (2002). The nonstochastic multiarmed bandit problem. SIAM Journal on Computing, 32(1).
• Bonnefon, J.-F., Shariff, A., & Rahwan, I. (2016). The social dilemma of autonomous vehicles. Science, 352(6293).
• Foerster, J., Chen, R. Y., Al-Shedivat, M., Whiteson, S., Abbeel, P., & Mordatch, I. (2018). Learning with opponent-learning awareness. AAMAS.
• Gleave, A., Dennis, M., Wild, C., Kant, N., Levine, S., & Russell, S. (2020). Adversarial policies: Attacking deep reinforcement learning. ICLR.
• Sadigh, D., Sastry, S., Seshia, S. A., & Dragan, A. D. (2016). Planning for autonomous cars that leverage effects on human actions. Robotics: Science and Systems.
• Shalev-Shwartz, S., Shammah, S., & Shashua, A. (2017). On a formal model of safe and scalable self-driving cars. arXiv:1708.06374.
• Wang, T. T., Gleave, A., Tseng, T., Pelrine, K., Belrose, N., Miller, J., Dennis, M. D., Duan, Y., Pogrebniak, V., Levine, S., & Russell, S. (2023). Adversarial policies beat superhuman Go AIs. ICML.