THOUGHT EXPERIMENT AI: Since AI can be perfect and process much faster than humans, it can be massively safer than humans. Human drivers will probably be so much worse they'll be outlawed!
ACTUAL AI: We concluded brakes were distracting users from the ad flow so we removed them.
THE TROLLEY PROBLEM: A trolley barrels down a track. A control program must decide whether to turn a switch, killing a person, or do nothing, killing three. The control program was developed in 3 weeks by a team which quit the company last year and never got docs for the sensors.
There was an employee who advocated putting up rails to discourage people from walking on to the tracks so often but she was fired for being a "bad culture fit".
Previously, previously, previously, previously, previously, previously, previously, previously, previously, previously.
Wasn't that the story of "2001: A Space Odyssey" ? an AI is built to look after the human crew, but before the product is launched some managers step in and say "probing the monolith should be 110% your priority, more important than anything else." "I will," says HAL, because this AI does what it is told, even if we don't understand what we're telling it to do.
Pretty much, except it's "probing the monolith should be 110% your priority, more important than anything else, and the crew are not allowed to know about it under any circumstances."
Anyone with a working knowledge of AI, like Dr. Chandra (HAL's inventor), would probably have been able to understand the implications of what the managers were telling HAL to do, except that the managers weren't allowed to tell Chandra about the new orders either (as discussed in "2010: Odyssey Two", IIRC).
BTW, the Chris Nolan-approved 70mm 'non-restored' print of 2001 is playing at the Castro Theater (and other places) this week and next! W00t!
Having worked in software, I've long said that I'd be stunned if we ever got to the point of actual trolley problem considerations in code.
I've more recently added that the most-popular self-driving car will be the one that optimizes solely for the safety of its inhabitants. My more cynical side adds "and, should they have to die, kills them quickly."
Well that certainly explains why the construction is taking so long.
When it's finished, it'll be like CERN but for trolley problems.
Wow, this sort of nails it.