The Unaccountability Machine: Why Big Systems Make Terrible Decisions—and How the World Lost Its Mind by Dan Davies
This excursion into nonfiction is brought to you by Ruthanna Emrys and her column “Seeds of a State”, particularly “Terrible Bosses, Evil Corporations, and Paperclip Maximizers: Dan Davies’ The Unaccountability Machine” in reactormag.com from February 24.
I’ve long been fascinated by systems and systems thinking – before I even knew it was called that – Kauffman’s Rules of Reality Hacking was my introduction. And after learning about Stafford Beer and the “lost” Massey lectures, I was doomed. That was fascinating and I’ve been sporadically pursuing the interest as resources became available. The Unaccountability Machine is an amazing work. It is an explanation of how we got into the polycrisis of the 21st century. It is also a mini biography of Stafford Beer which makes him more interesting than just his works. He was a psychiatrist without having a PhD or MD for one. He was a consultant then a guru. And generally a fascinating character, even without his insights on systems.
I had some big take aways
- Black boxes
- The purpose of a system is what it does (POSIWID).
- Accountability sinks.
- The Ricardian Vice
- All models are wrong but some models are useful
- The explanation of how we've arrived at the current poly-crisis.
- It comes down to designing systems with accountability sinks and a distinct lack of red handles and break points to interrupt them.
So what are they?
Black boxes are just that – we don’t know what goes on inside one, but we do recognize the inputs and outputs. And Beer strongly suggested that we stop trying to peer into them and just observe the outputs and make rules based around that. Where we often get into trouble is that we want to know how it works. This often means we break the system we’re trying to understand. Worse, we think we understand how it works (but we don’t), which can lead to catastrophic misunderstandings and failures as those in power make real the quote “It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.” How? By trying to tinker with the internal workings of a black box system that they, or even their experts, do not understand.
Descending immediately from this is the purpose of the system is what it does (aka POSIWID). No matter the high flying rhetoric or intentions of the individuals in a system, it’s actions that matter. Yes, the individuals can have their best intentions, but if the system they’re in doesn’t produce what they want, well…
Now accountability sinks. I love and loathe this term. It puts a name to something I’ve been trying to grasp for a while. Love because it defines it clearly – a buffer or shield to absorb negative emotions from decisions. And there’s no way to appeal the decisions by the system. I loathe it because now I see them everywhere. From phone trees, to automatic chats, to botched orders.
Which leads to a pithy little aphorism that I think ought to be everywhere “The fundamental law of accountability: the extent to which you are able to change a decision is precisely the extent to which you can be accountable for it, and vice versa.” Couple this with accountability sinks and no one can change anything. Beginning to sound familiar?
Then there’s the Ricardian Vice, which I did know beforehand (one of my best bosses is a PhD in economics and this is one of the things that drove him up a wall). What is it?
a) make a model of some feature of the economy, stripping away nearly all the complexity;
b) make a lot of simplifying assumptions, often questionable in terms of their empirical relevance;
c) show that their conclusion follows from their assumptions, which ought to be quite easy if they’ve made the assumptions strong enough;
d) act as if their conclusion has now been proved in the real world.
Or more succinctly, – you can solve the problem in a statistical model, therefore you can solve it in the real world where the data of the model came from!
However, “It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.” shows up again and again. Cory Doctorow calls it economism. Or the map is not the territory, and just because the map says you can get there from here, doesn’t mean that’s the reality.
Now, every model is wrong (or simplified), but some of them can be useful. Where most things begin to go wrong is that the models don’t have break points within them where the process can be halted. And all too often, they try to model a really complex black box, think they thus understand the black box. Also, a lot of the complexity stripped out of a model is people. Ones that can scream when things go wrong, or stop the line.
So how does this lead to the 21st century polycrisis of rising authoritarianism, climate, fragile supply chains, etc. etc.?
Well, take flawed understandings of black box systems, add in trying to force their behavior to do certain things without sufficient complexity (i.e. people), a lack of break points/red handles or allowing a response to people in pain, add in accountability sinks removing the power to change systems or ignore people screaming until they’re forming firing squads, and well, you get right wing authoritarians and nationalists who say they do and it all begins to look familiar.
Add in our human trait of inertia, we tend to leave them in place until something catastrophic happens. As a result, we let the systems make the decisions based on old assumptions and didn't move them until something catastrophic happens. One vivid example Davies provides is KLM and the shredded ground squirrels. There the system worked perfectly, until a pet supplier sent several hundred ground squirrels without the proper customs papers. Then it had to either return them (expensive) or use the industrial shredder. Which lead to outrage, and KLM no longer shipping animals for the exotic pet trade.
I mentioned one of the grand old men of management cybernetics Stafford Beer. I was introduced to him and his ideas by Graydon Saunders. And after sitting through a recovered recording of the Massey lectures, I got curious. Anyway, this went pretty far in filling in the background of Beer. At the tail end of World War II he joined the British Army as an officer from his PhD program and became an intelligence analyst in India, paying close attention to the alliances, enmities and outcomes of the various factions there. After the war wound down, he was placed in charge of a unit of the worst of the worst – illiterates, authority problems, and so on. The crazy thing is, he applied the same techniques he did for observing the factions in India and treated the men like black boxes. As a result, the behavior improved markedly. All it took was a letter from his PhD adviser and he became a psychologist without a degree.
After the war, he applied the same observations to businesses, resulting in massive organizational chart, a consulting business and media appearances.
Despite some SF authors and their fondness for Beer, Davies knocks some of the shine off Beer. He particularly goes over the famous, or infamous, time Beer was hired as a consultant for Chiles socialist Allende government to use management cybernetics with their economy.
Beer and Allende were ambitious. The possibility of using feedback and controls on a national scale – but they were guilty of the Ricardian Vice. They didn’t allocate enough channels of communication, or allow for the complexity of the real system and economy. A dozen teletypes and a command center weren’t anywhere near enough. This ties into early parts of the book and requisite complexity and variety to manage systems.
There's also the reason that Beer is so little known outside of a small circle – he was irregular. He never settled at a university long enough to become a thesis advisor and never had a student to really take up the torch of management cybernetics and adapt it to current technologies.
Davies also has sections on Milton Friedman and how he became so popular and executive compensation got so out of whack. They’re related. And he doesn’t blame Friedman though, just that Friedman was the man at the right spot and right time.
This is one of those books that pries open your mind and pours in new information and new ways to look at the world – like 1491, The User’s Guide to the Brain, Stolen Focus and others.
Highly recommended, especially to people interested in systems, management cybernetics, management and Stafford Beer.
by BravoLimaPoppa