History gives us the perspective to see what went wrong in the past, and to look for patterns, and check whether those patterns apply to the present and near future. And looking in particular at the history of the past 200-400 years -- the age of increasingly rapid change -- one glaringly obvious deviation from the norm of the preceding three thousand centuries -- is the development of Artificial Intelligence, which happened no earlier than 1553 and no later than 1844. [...]
Elon Musk has an obsessive fear of one particular hazard of artificial intelligence -- namely, the paperclip maximizer. [...] Unfortunately, Musk isn't paying enough attention. Consider his own companies. Tesla is a battery maximizer -- an electric car is a battery with wheels and seats. SpaceX is an orbital payload maximizer, driving down the cost of space launches in order to encourage more sales for the service it provides. Solar City is a photovoltaic panel maximizer. And so on. All three of Musk's very own slow AIs are based on an architecture that is designed to maximize return on shareholder investment, even if by doing so they cook the planet the shareholders have to live on. (But if you're Elon Musk, that's okay: you plan to retire on Mars.)
The problem with corporations is that despite their overt goals -- whether they make electric vehicles or beer or sell life insurance policies -- they are all subject to instrumental convergence insofar as they all have a common implicit paperclip-maximizer goal: to generate revenue. If they don't make money, they are eaten by a bigger predator or they go bust. Making money is an instrumental goal -- it's as vital to them as breathing is for us mammals, and without pursuing it they will fail to achieve their final goal, whatever it may be. Corporations generally pursue their instrumental goals -- notably maximizing revenue -- as a side-effect of the pursuit of their overt goal. But sometimes they try instead to manipulate the regulatory environment they operate in, to ensure that money flows towards them regardless. [...]
It seems to me that our current political upheavals are best understood as arising from the capture of post-1917 democratic institutions by large-scale AIs. Everywhere I look I see voters protesting angrily against an entrenched establishment that seems determined to ignore the wants and needs of their human voters in favour of the machines. The Brexit upset was largely the result of a protest vote against the British political establishment; the election of Donald Trump likewise, with a side-order of racism on top. Our major political parties are led by people who are compatible with the system as it exists -- a system that has been shaped over decades by corporations distorting our government and regulatory environments. We humans are living in a world shaped by the desires and needs of AIs, forced to live on their terms, and we are taught that we are valuable only insofar as we contribute to the rule of the machines.
Previously, previously, previously, previously, previously, previously, previously, previously, previously, previously, previously, previously, previously, previously, previously, previously, previously.
word.
Some related background on this "slow moving AI" threat:
Moishe Postone (U. Chicago) has pointed out - with a very simple argument, really - that the imperative to exhaust all natural resources and generally make life on earth impossible is "built in" to any society in which the capitalist mode of production prevails.
Why? Essentially, because (a) human progress keeps improving productivity meaning that more and more is made with less and less labor; (b) but at the same time, over time, more and more people must be employed or else there is no more growth, and the average "investment" takes place at a loss. Do the math:
The productivity gains mean that, on average, each person employed is processing greater and greater amounts of raw materials and energy. The growing number of persons employed mean that the absolute amount of raw materials and energy processed must rise. Every advance that makes production more efficient, paradoxically, plays out so as to accelerate the denuding of the planet.
You say you want a job? You planet killing monster!
So there's a thought. About this "AI" metaphor:
Same guy -- Postone -- has made an argument that relates to this "an AI has taken over" metaphor. It's a heavy topic.
Postone wanted to better understand why anti-semitism and National Socialism seemed to go hand in hand in Nazi Germany. He was puzzled by mysteries such as Nazis diverting badly needed military resources from fighting Soviets an diverted them for the sole purpose of accelerating the murder of Jews. Such a suicidal tactic for the Nazi army can't be explained by any simpleminded theory that the Nazis just picked Jews as a scapegoat for propaganda purposes. Clearly, even the top Nazis somehow really thought that Jews were their enemy and were in revolution against them. Why?
Here, Postone examined what Nazi anti-semites said about Jews. He found stuff similar to what the AI metaphor, above, says: A mysterious, almost super-human force dominates society and stays us from collective, rational, self-determination and self-regulation. It corrupts our government and all important institutions. It cheats our markets. This mysterious force is hell bent on accumulation for accumulation-sake. Personified as a class of Others (like the political class in the AI metaphor), the evil force insinuates its way into all the halls of power. It is the source of the growing misery of the honest, hard-working people. AI? Jewish question? Are these just two misleading names for the same, as yet unidentified thing?
Postone points out that the charges leveled against this alien force are nothing other than the emergent properties of capitalism itself -- in other words, the domination is not from Jews or an AI but is simply a side effect of the fact we ourselves insist on surviving by buying and helping to manufacture commodities. We have met the enemy, it is the core economic concepts in our way of life - the thirst for "honest work at decent pay". The phantoms that dominate society are none other than the consequences of our own mundane social relations.
In context of this analysis, what do we make of Stross fingering "the corporation" rather than wage labor itself? The corporate form emerges from three social developments: wage labor as the basis of subsistence; the division of labor - specialization; and the incompatibility of the market with the most advanced means of production.
That last point may seem counter-intuitive but it is not complicated: Internally to a corporation, production is mediated by systems of planning and delegation. Inside a corporation, the market does not exist. Outside of a corporation (up to some point in history), production is mediated by exchange -- by the market logic of buying and selling.
Competition among corporations naturally (and empirically, as a matter of history) leads to the evolution of super-structures - combinations of various sorts. What began in the late 19th century with trusts, cartels, and holding companies grew, over time, precisely because each of these higher levels of abstraction over an individual corporation had the very practical effect of replacing market mechanisms with overt, deliberate, conscious planning mechanisms. Less markets, more planning has been the trajectory of capitalism.
In fact, by the 1930s, the capitalist superstructure of planned production had nowhere left to go but to a merger with the state -- found in every fascist and every allied power. The fascist economic system was a global development.
Finally, where we are today: Neoliberalism and Trumpism both appear to have a substantial anti-statist, global element to them. This suggests that the nation state form has become too confining, and that planned production is now struggling to overthrow the state, and dominate in some as yet unforeseeable transnational way. Perhaps a name for what Trump and Clinton have in common might be "the death of liberal democracy".
"Corporations are like AIs" is therefore too simplistic though it would have been a spot on metaphor sometime in the late 19th century. It's an anachronistic metaphor since around 1930. Corporations have long been subsumed into much larger structures that encompass the financial branch, industrial branch, and government.
Stross goes on to argue that "the slow-moving [corporate] AIs" are, through things like internet advertising, starting to develop a theory of mind - as if this were some new development. As a matter of historical fact this is likely rubbish. We can spot early forms of such personalized theories of mind within the power structure as early as the development of the Confession (c.f. Foucault) and it surely took off in industrial form with the highly personal interventions of Fordism, around the dawn of the 20th century. Stross, for all his forward thinking, is still about a century behind.
Nevertheless, for his punchline, Stross envisions a number of ways to use "app addiction" to organize mass scale mutually predatory behavior among us consumers -- for fun and profit. He offers no "what is to be done" and the thrust is that these are scary things we ought to "think about". Underlying his argumentation is a sense that somehow we must confront "the slow moving AIs" by regulating away these threats.
That's a recipe for mystification and paralysis or tail-chasing.
Enter Postone:
The accelerating history to which Stross refers is uniquely characteristic of capitalist society. It results, as mentioned above, from the imperative for economic growth -- more jobs -- combined with the improved productivity, a steady increase in the average amount of raw materials and energy each job processes. Postone refers to accelerating history as a kind of accelerating treadmill.
In our society, on average, we must sell our time in order to be able to buy what we need to live. This is the simple, physically immediate, and human-scale computation process from which the "slow moving AI" superstructure emerges: we survive by buying and therefore live in a system where we sell our time to some production process, one way or another. This is the motive force that causes the monsters Stross conjures up.
What we must envision, then, is some rapid transition, on a global scale, to a society about which we know very little, mainly only this:
a) Nobody has a job.
b) Nobody needs money and there's nothing to buy.
That way lies freedom.
Shout out to the 3.9 Million membes of IG Metall and their fight for a 28-hour work week. And after 28, comrades: 21.
Sock and aw ... FUD(dy) duddy ... at least now I get the corp idea,without searching the interweb, that you guys are talking about hear in jwz land. Should I say company?
I don't know why anyone would expect deep insights from an author of escapist literature and this talk is definitely his shtick, straight to his favorite readers, but this is just so much recycled tech-futurism that there's no oxygen. He's huffing his own fumes. I mean, he trots out history, but it's to "spot recurring patterns in human behaviour that repeat across time scales outside our personal experience." As if the attempt to understand humanity, the development of human society and the objects that society manufactures is about training yourself on some corpus of data on "human behavior." That's a fetish if there ever was one.
I don't know why the tech-world loves this kind of talk, that brings in nothing outside of tech's own claustrophobic intellectual world. I mean I guess it's nice to see your world-view reinforced or something. But, Stross is never going to understand corporations or capitalism with the tools he's brought to the table. I think Accelerando is a great manic satire of the first tech boom, but Stross seems like he's started to believe his own shtick and that's not going to lead him into the future. The idea that corporate law could become self-propagating and ultimately self-aware (see: Accelerando) is a nice metaphor for lots of things, but it's not literally true doesn't explain anything about what has happened nor predict what will...
Also, it's all really convenient displacement for the insanely greedy actions of the people behind those corporations. It's more fun, even at CCC, to scare yourself with ghost stories about paperclips, than to realize that your officemate in grad school (see: Levandowski) has become a real-life monster of greed.
Cool ad hominem, bro'.
Stross has spent a lot of time trying to get away from being known as "the singularity guy" and early books like Accelerando in particular. I think this talk is a tongue-in-cheek way of using the language of the AI singularity to describe something he finds more interesting (and worrying).