...using automated, plausible lies:After training the AI system to play the popular arcade game Frogger, and documenting how human players explained the decisions they made while playing the game, the team of researchers developed the agent to generate language in real time to explain the motivations behind its actions. [...]
When it comes to neural networks -- a kind of AI architecture made up of potentially thousands of computational nodes -- their decisions can be inscrutable, even to the engineers that design them. Such systems are capable of analyzing large swaths of data and identifying patterns and connections, and even if the math that governs them is understandable to humans, their aggregate decisions often aren't.
Upol Ehsan, lead researcher and PhD student in the School of Interactive Computing at Georgia Tech along with Riedl and their colleagues at Georgia Tech, Cornell University, and University of Kentucky, think that if average users can receive easy-to-understand explanations for how an AI software makes decisions, it will build trust in the technology to make sound decisions in a variety of applications.
"If I'm sitting in a self-driving car and it makes a weird decision, like change lanes unexpectedly, I would feel more comfortable getting in that car if I could ask it questions about why it was doing what it was doing," Riedl said.
This is absolutely horrifying. The idea here is: the self-driving car has done something inscrutable. What if there was a chatbot that was good at looking at what just happened, and making up a plausible explanation for it after the fact?
(Side note: that's probably how human consciousness actually works. Your "mind" is the post-facto explainer for what the meat-zombie automaton just happened to do. Sweet dreams.)
But this research is just mechanized "truthiness"! It's not actually explaining what happened inside the neural net, which it still treats as a black box. It's solving the problem of: "If the self-driving murderbox decided to cross 4 lanes and slam on the brakes, let's dig around in our memory and imagination, find a scenario where a human might have done that same thing, and proffer that as the explanation."
Even though the AI and the human might have done those for completely different reasons, that's the explanation that will sound most plausible!
The human's decisions were based on an unbroken straight line chain of 85 million years of "I am the primate who did not get eaten", layered with decades of fine motor control trailing. Whereas the AI based its decisions on whatever ad hoc junk a bunch of low-paid contractors dumped into the training set over a period maybe as long as 18 months.
So they're building an AI whose goal is to lie to you -- to build your confidence in decisions made by AIs.
We are so completely doomed.
Maxwell
Smart:Because, at this very minute, 25 CONTROL agents are converging on this building. Would you believe it? 25 CONTROL agents! Savage: I find that hard to beleive. Max: Would you believe 2 squad cars and a motorcycle cop? Savage: I don't think so. Max: How about a vicious street cleaner and a toothless police dog?
The human passenger behind the wheel is supposed to be paying attention, meaning 99.999% of the time, they already know what the car is doing and why. The other 0.001% of the time the right action is probably squeeze the brakes, or to go along with the car's unusual activity. The driver needs to be told anything.
This system is basically designed for the use case of: "I'm not paying attention to the road, and relying entirely on the murderbot. It just did a weird thing, and now suddenly I'm afraid I should be paying attention. 'Hey, murderbot, catch me up on what's happening, will you?' "
If self-driving cars aren't eventually going to be good enough to not require an attentive human, then they're not worth pursuing ("good enough" defined as a net reduction in death and destruction compared to human drivers). If they aren't, they should stick to "assist" features that clearly leave the human with primary responsibility to drive (brake for obstructions, stay in lane & keep pace in stop 'n go traffic but with a very low maximum speed, etc.).
and fucked.
Well, they already are--in highway-like driving conditions, as far as we can tell. Probably by several orders of magnitude, IIRC. (I mean, eventually someone will run a randomized controlled trial where some people are handed murderbots and some people are handed murder weapons, and see who kills more people.*)
The biggest problem isn't whether they reduce death and destruction overall. The problems are:
1. Which role actually has liability when murderbots do cause death and destruction (the passenger? the writer of the AI?), and which role should have liability, and what criteria should our lawmakers use when they choose?
2. How can we make choices about which people have that role? Does someone get fired when a murderbot murders an innocent bystander? Do they get a promotion based on how long since the last murder?
3. Do we accept as a society that murderbots are good, and charge a gas tax to all people and use that money to pay for the few murders by our murderbot overlords?
Because it sure looks like we make terrible choices as a society about who is responsible when software causes crashes.
*Ok, as fun as it sounds, that trial isn't going to be run. But we know a lot about how bad tobacco is without a randomized smoking trial; the same trials will be run in this context, and the data will be analyzed similarly.
This is fine.
This is fine. (GM version)
Legislation should definitely squander the lead time they have to consider the above questions before more people are killed by poorly-designed software.
Do you know any human beings?
Maybe if they did it like most of the programs I write, which are full of messages like
print "it's got this far without crashing".Surveillance-based advertising is the prevailing business model in tech. A major, competitive component of that business model is the cybernetic manipulation of people's attention and habits -- a competition to see which firm or which team within a firm can build the most (literally) addictive user experience. The people doing this work are themselves, for the most part, voracious consumers of their own products.
Marx's concept of historical materialism informs us that the ideology and ethos of a society morphs and adapts as needed to more or less reproduce that society's means of production. Since tech's production is based on pushing addiction and its workers, to have a feel for that work, must themselves be major addicts -- it follows that the ideology and ethos of tech is become that of the drug fiend / cook / pusher. That awful feeling in the Bay Area, since around 2000, of being colonized by something historically new and nauseous in tech -- all those techbros dominating every scene -- is like that time you couldn't move out or kick out the chaos roommate who ran around dealing, spoiling every party, and periodically prank-dosing anyone who crossed him with whatever designer psychedelic lately fell into the grungy watch-pocket of his unwashed 501's.
In the end you know the whole group house - which had been going just fine unti he showed up - will have to break up and scatter because that guy won't ever leave otherwise. Y'all lose your home and when you check back, a decade later, you hear he held out there for 7 years with a constant churn of junkie chaos roommates making the whole street miserable until the night that someone finally, inevitably, got stabbed and that's the last anyone saw of any of them. Took eight months to restore the property and these days its lived in by some quiet rich gay guy that works at B of A and his partner and their rat dog. They get fresh flowers for the front room, frequently.
Then you'll forever be haunted by those 18 months before tech bro addict pusher sociopath moved in when you swear, to this day, y'all almost had something. A brief time when it seemed like the world could again become magical and beautiful with culture and way of life that flowed from love and solidarity, art and invention, a playful abundance for all.
Your ideas are intriguing to me, and I wish to subscribe to your newsletter.
... I think I knew that house when I first moved here. Went to some nice potlucks there. And it went down... very much as you describe.
[shudder]
Most of the people I know who were killed in car crashes died because the guy in the other car was drunk. Road rage kills too, though it's harder to prove.
There are some fundamental flaws in the primate brain. Mostly our rationalization skill is tuned to explain how any problems were somebody else's fault, because the important thing is to save face.
Flawed primate brain >> zero-hours contractors hired by middle manager trying to "close the quarter"
Laugh awhile you can, monkey boy!
It's confabulating!
Step two is allowing customer service to completely deny you the ability to talk to a human, period. "The billing system explained itself; we can't do anything else for you."
Step five is robocop.
After reading the article, skimming the video and reading the paper's abstract, it seems like all they did was try and find a way to present explanations in an acceptable manner. They left generating the explanations themselves as an exercise for the reader.
Neural net |-------------------------| Explanation
^- here be dragons ^- study
Meh, the second line of the diagram was supposed to start in the middle (here be dragons meant to point to the chasm).
It also works as presented.
The system is kind of like by Kate Compton.
Templated responses triggered by specific data. The authorship is in the text that frames the data and that's written by a human with an authorial bias. The text parsing system is just a layer (very large spreadsheet) sitting on the steering AI. If the text actually explains the decision making process rules accurately and transparently it might be an adjunct to a HUD. But only practical in retrospect, as it can't tell a driver what's about to happen. Like my grandfather's last words, - A truck!
Boy I messed that up! Shouldn't imbibe and comment.
Always remember the last words of my grandfather, who said: 'A truck!'
Emo Philips
Also Emo Philips: "I want to die in my sleep, like my grandfather, rather than screaming, like the passengers in his car."
If you ask the experts, they can point to some data points in the neural network that were lighting up, and that have a high correlation with such-and-such event and probably means so-and-so which can be corrected by doing the opposite of something during training.
In the limit, nobody really knows how a non-trivial machine-learning system comes to its decisions--it's as much an area of active research as anything else in AI--so you might as well use a neural net to plow through the gigabytes of raw data and explain it. It's as good as any other tool we currently have.
Fun fact: ISO safety standards for passenger automobile systems have "do not use" in best-practices tables where the highest level safety columns and "machine learning" rows cross. (less fun fact: those standards aren't used for autonomous vehicle control systems)
This is essentially the same as looking at an MRI and saying, "when you think about ice cream, this part lights up, so I have found the 10 cu. cm. of the brain corresponding to ice cream." It's not exactly... actionable.
...and probably can't, so let's just make shit up.
Something must be done, and this is something, therefore...
There is a good all-purpose predictive model explanation system called LIME: repo, paper, video:
I successfully used its technique for a system to figure out which part of an unintelligible word was most in need of improvement in a pronunciation assessment system.
Thirty-three million K-6 ESL kids in China have been using this for the past couple years, but I can't get Rosetta Stone, Pearson, Duolingo, Mango, or SRI EduSpeak to return my calls because they don't want to admit the "one true accent" scoring system they've been using for the past couple decades is the garbage that it is.
Which is why these systems can be used to create narrative! Non-trivial, but easier when there's no definitive right or wrong answer and no passengers. Less funding than self driving vehicles, but more fun than Frogger.
In the paper, the MTurk participants are authors of the corpus, at the direction the person who set the questions for them to answer. Co-creation!
Fits in very well with James Ryan's recent thesis 'Curating Simulated Storyworlds'.
https://www.researchgate.net/publication/330855103_Curating_Simulated_Storyworlds
If you read the TeslaMotors subreddit, you'll see lots of posts about Autopilot unaccountably failing and lots of Tesla apologists saying, "You should be paying attention 100% of the time and be ready to take over dumbass!" These are actual people who are, for free, defending a corporation that put out an unsafe product for no other reason than Rah Technology or Rah Elon Musk. In one thread, someone actually asked if it's better to keep your foot on the brake or the accelerator when using Autopilot, i.e. which failure mode is more likely. As if that is remotely a sane avenue to even be heading down. Either your software works well enough to eliminate the need for human intervention, or it's worse than useless. Why do I know this but Tesla doesn't?
Then again, if you think it's something about A.I., you only have to head over to the Aviation subreddit to see how many people defend Boeing's shameful attempt at flight software by saying that the pilots were to blame for those 737 MAX crashes because the procedure for turning off MCAS was "in the flight manual." I suspect unfortunately that racism plays a role in people being willing to write off these crashes as pilot error. Safety critical systems require redundancy. Why do I know this but Boeing doesn't?
And how is any of this different than the decades of people defending auto companies against the big bad government for making them implement even basic features like rollover and side impact protection? Tens of thousands of people died because G.M., Ford, and Chrysler wanted to make their cars sexier and cheaper to make. It will never happen to me, they said, those people who died were idiots who shouldn't have been behind the wheel.
Then when seatbelts, ABS, and airbags became mandatory, I heard people arguing against that too. There was a popular argument that if you're a trained driver, ABS will increase your stopping distance, and if you're not trained you deserve what's coming to you if you can't stop in the rain.
It seems that a lot of tech-oriented people, ones who you'd think ought to know better, are more than willing to blame hundreds all the way up to tens of thousands of deaths on the user rather than blame a corporation for their extreme negligence.
This is the real issue here. You can blame the evil corporations all you want, but as long as public opinion (especially among people we consider our peers) is on their side, they will keep doing what they do.
"Your 'mind' is the post-facto explainer for what the meat-zombie automaton just happened to do". This is the most adorable affirmation I've ever read.
I constantly try to talk people away from the idea we are rational entities, and talk about our lizard brain, and how our mind just sits there trying to steer a bipod animal, but my god is "meat-zombie automaton" just the most awesome concept ever.
This work is really a glorified Eliza program, it matches user's descriptive phrases to raw state input. It doesn't model the user's internal goals, reasoning, or anything, and has no way to do so anyway. If this is what passes for AI these days, we are doomed.