All around me are familiar facesWorn-out places, worn-out facesBright and early for their daily racesGoing nowhere, going nowhere
And I find it kind of funny, I find it kind of sadThe dreams in which I'm dying are the best I've ever hadI find it hard to tell you 'cause I find it hard to takeWhen people run in circles, it's a very, veryMad world (mad world, mad world)
Children waiting for the day they feel goodHappy birthday, happy birthdayMade to feel the way that every child shouldSit and listen, sit and listenWent to school and I was very nervousNo one knew me, no one knew meHello, teacher, tell me, what's my lesson?Look right through me, look right through me
All around me are familiar facesWorn-out places, worn-out facesBright and early for their daily racesGoing nowhere, going nowhere
And I find it kind of funny, I find it kind of sadThe dreams in which I'm dying are the best I've ever hadI find it hard to tell you 'cause I find it hard to take(When people run in circles, it's a very, very)Mad world
Children waiting for the day they feel goodHappy birthday, happy birthdayMade to feel the way that every child shouldSit and listen, sit and listenWent to school and I was very nervousNo one knew me, no one knew meHello, teacher, tell me, what's my lesson?Look right through me, look right through me
Mad world
Snowflake apparently has a big cybersecurity problem but auditor PwC never warned anyoneFrancine McKenna
Who’s responsible for our accountability problem?
Who’s responsible for our accountability problem? From the algorithm that raises your insurance premium to institutional denials over state scandals, it’s a problem with deep roots
I was recently scheduled to present my Cautionary Talespodcast live on stage as a curtainraiser for a podcast conference. Talented voice actors, live sound effects and even a trombone would weave a dramatic soundscape for a full house. There was only one problem. Nobody seemed to have the authority to let me in.
The people issuing conference passes wouldn’t give me one — not unreasonably, since I wasn’t attending the conference proper and hadn’t registered to do so. They advised that I speak to “that lady there”. That lady there seemed puzzled: the conference wouldn’t officially open until tomorrow, so I didn’t need a pass. Just walk in, she said. The security guard had a different view. He had been given strict instructions that nobody gets in without a pass. Talk to the conference team, he told me. Nothing to do with us, they said — talk to security.
I realised I was facing what the writer Dan Davies has named an “accountability sink”, in which it was somehow nobody’s fault. In his recent book The Unaccountability Machine, Davies explains the basic logic of an accountability sink: decision-making power is removed from individuals you might want to shout at, and made instead by an algorithm or some distant committee both ignorant of and immune to your objections.
Everyone I spoke to insisted that they were powerless to act. And if you cannot change a decision, you cannot be held accountable for it. That’s the accountability sink at work.
“Computer says no” first became a comedy catchphrase 20 years ago, so although the accountability sink is a useful new term it describes an older problem. Just ask 440 luckless squirrels who in 1999 arrived at Schiphol airport in the Netherlands, en route from Beijing to Athens. Unfortunately, particularly for the squirrels, they had been shipped without the right paperwork.
What to do? It wasn’t legal to send them on to Athens. It wasn’t possible, apparently, to send them back to Beijing or to fix the paperwork. So the airport ground staff started up an industrial shredder normally used in poultry farms for killing male chicks and threw all 440 of the squirrels into it.
When this debacle became known to the public, it was universally agreed that shredding 440 squirrels had been The Wrong Thing To Do, but it was less clear who should be accountable. The basic problem, Davies explains, was that the decision to destroy paperwork-less animals had been turned into a matter of policy emerging from the Dutch department of agriculture. Policy is policy, not something that gets overruled by manual workers in airport sheds wearing protective gloves and standing next to an industrial shredder and a few crates of squirrels.
The policy itself was probably sensible. The problem was that there was no way to identify and deal with exceptional cases. The question, “What if the policy demands that we hurl hundreds of squirrels into an industrial shredder?” arose too late
Accountability sinks can have a legitimatepurpose.
When the pressure is on, it’s tempting to play favourites or take short-cuts, prioritising the noisiest complaints or putting off painful consequences. “Computer says no” may be a punchline, but the computer is a useful way to deflect awkward customers. It also often makes the right choice.
Consider the idea of an independent monetary policy, when an elected government sets an inflation target and then an unelected central bank has the task of trying to hit that target. It’s a sensible response to the fact that democratically elected governments always have a short-term incentive to stoke inflation, reaping the briefest of benefits followed by some serious headaches.
When the Bank of England hurts millions by raising interest rates, it can point to its inflation mandate. When people complain to the government, the government can say that interest rates are a matter for the Bank of England. It’s an accountability sink again, but monetary policy is almost certainly better as a result of the fact that somehow nobody is to blame.
Yet all too often accountability sinks are created for the simple, ignoble reason that nobody wants to be held accountable if they can avoid it. Even worse, the sink may be fictional, a mere scapegoat. A generation of British politicians learnt that whenever anyone complained about a policy, it was always easy to blame Brussels, or unelected judges, or health and safety gone mad.
Accountability sinks range from the algorithm that raises your insurance premium to the institutional denial that has delayed many reckonings for the British state: infected blood, persecution of sub-postmasters, Hillsborough, the Windrush scandal and so many others. Faced with such an enormous range of misfiring accountability sinks, is there anything useful we can say about what we could change to make things better?
Perhaps there is. The common thread here is an imperviousness to feedback. No doubt the people involved in the squirrel incident wondered if there might be a better way to deal with paperwork problems, but there was no line of communication open to the relevant ministry, unaware of the gruesome consequences. Without feedback, nothing ever gets better.
The world is a complicated place and humans are imperfect. Mistakes will be made. The question is, what happens after the mistake? Does news ever reach anyone with the ability to change things? If so, do they? Some accountability sinks exist for a good reason, some do not. Either way, if there is no way to learn from mistakes, the mistakes will eventually bury us all.
Dan Davies makes a compelling case for the use of Stafford Beer’s management cybernetics in the age of AI
Waiting to board a flight recently, I witnessed a stark example of a phenomenon at the heart of The Unaccountability Machine. A fellow passenger had been refused boarding and was pleading with the airline attendant. The attendant was obviously sympathetic — but it was down to company policy. There was unfortunately nothing she could do.
We were witnessing what Dan Davies calls an “accountability sink”: a situation in which a human system delegates decision-making to a rule book rather than an identifiable individual. If something goes wrong, no one is held to account.
The starting point of Davies’ entertaining, insightful book is that the uncontrolled proliferation of accountability sinks is one of the central drivers of what historian Adam Tooze calls the “polycrisis” of the 21st century. Their influence reaches far beyond frustrated customers endlessly on hold to “computer says no” service departments. In finance, banking crises regularly recur — yet few individual bankers are found at fault. If politicians’ promises flop, they complain they have no power; the Deep State is somehow to blame.
The origin of the problem, Davies argues, is the managerial revolution that began after the second world war, abetted by the advent of cheap computing power and the diffusion of algorithmic decision-making into every sphere of life. These systems have ended up “acting like a car’s crumple-zone to shield any individual manager from a disastrous decision”, he writes. While attractive from the individual’s perspective, they scramble the feedback on which society as a whole depends.
Yet the story, Davies continues, is not so simple. Seen from another perspective, accountability sinks are entirely reasonable responses to the ever-increasing complexity of modern economies. Standardisation and explicit policies and procedures offer the only feasible route to meritocratic recruitment, consistent service and efficient work. Relying on the personal discretion of middle managers would simply result in a different kind of mess.
The question, therefore, is not how to return to a world without non-human decision-making systems — it’s how to ensure that they’re open to feedback, able to adapt and improve. The Unaccountability Machine scrutinises two intellectual frameworks that aim to supply an answer. The first will be well known to most FT readers: economics. Davies judges it hopelessly deficient. Its highly abstract theories of how firms and markets operate are trivial, he argues. Worse, the classical postulate of the optimality of market outcomes positively feeds the accountability crisis by convincing managers and politicians that it is better to let the market decide.
Milton Friedman draws especial ire. Davies argues that his highly influential doctrine of shareholder value-maximisation spawned a pandemic of corporate accountability sinks; other objectives were unduly marginalised. Recent debacles at Boeing in the US and Thames Water in the UK suggest he has a point.
Instead, Davies argues, there exists a second, better way — one less familiar to most of us. This is “management cybernetics”, a framework for understanding the corporate and economic world that emerged from operations research, the use of mathematics to solve problems of human and technical resource management that began during the second world war.
Davies’ hero here is the management consultant, information theorist and philosopher Stafford Beer, whose heyday in the 1960s and ’70s saw him advise CEOs and governments around the world. Beer conceived of companies as forms of non-human intelligence — what the science-fiction writer Charlie Stross calls “very old, very slow AIs”. As such, solving their problems is a matter not of eliminating market failures, but of ensuring that the flow of information between deciders and decided-upon is kept in balance. It’s a kind of psychoanalysis for non-human intelligences, with Stafford Beer as Sigmund Freud.
Davies laments the combination of political backing and intellectual fashion that saw economics triumph instead. But with The Unaccountability Machine he provides an elegant new introduction to this intriguing road-not-taken in postwar social science, and makes a compelling case that in the age of AI its time has finally come.
So how about practical recommendations? Davies argues that in the corporate world, the most obvious one is to defang the leveraged buyout industry. Here, the cybernetic solution is to require private equity investors to guarantee the debts of takeover targets, sealing off the great accountability sink of limited liability.
Beyond this, though, Davies is coy about specifics. The most important thing, he says, is to dethrone mainstream economics. “You can’t have the economists in charge,” he writes, “not in the way they currently are.” Surveying the waning of economic orthodoxy around the world today, with free trade in retreat and industrial policy back in vogue, I’d say that wish is being granted. Perhaps management cybernetics will indeed finally have its day.
The Unaccountability Machine: Why Big Systems Make Terrible Decisions — and How the World Lost Its Mind by Dan Davies Profile £22, 304 pages
