AI is a cultural, not social, technology

A response to Farrell et al’s essay in Science

“Debates about artificial intelligence (AI) tend to revolve around whether large models are intelligent, autonomous agents. Some AI researchers and commentators speculate that we are on the cusp of creating agents with artificial general intelligence (AGI), a prospect anticipated with both elation and anxiety. There have also been extensive conversations about cultural and social consequences of large models, orbiting around two foci: immediate effects of these systems as they are currently used, and hypothetical futures when these systems turn into AGI agents—perhaps even superintelligent AGI agents. But this discourse about large models as intelligent agents is fundamentally misconceived. Combining ideas from social and behavioral sciences with computer science can help us to understand AI systems more accurately. Large models should not be viewed primarily as intelligent agents but as a new kind of cultural and social technology, allowing humans to take advantage of information other humans have accumulated.”

This is the convincing start of a paper published in Science last month by Henry Farrell, Alison Gopnik, Cosma Shalizi and James Evans – an attractively varied bunch affiliated to political science, psychology and philosophy, data science, and sociology. The framework they are offering provides a necessary corrective to much of the current discussion of AI. However, there’s also one thing in it that I think is both wrong enough and big enough to feel the need to push back on.

The authors are right about a lot of things. We need to break out of the current conversation about AI; as they say, all too often it seems to fall into a false dichotomy between boomers (hello techno-optimists in Silicon Valley and the Tony Blair Institute) and doomers (deep skeptics). There will be changes to productivity at work with potentially wide-ranging ramifications but, as with printing, that’s likely to be far from the end of it. Important questions are being overlooked, for example whether the overall effect on society will be homogenisation or heterogeneity.

A focus on agents, uncertain if potentially important, obscures the revolution under our feet coming from the ability of Large Models to synthesise and transform the huge corpus of human work that has been digitised. As they eloquently put it, “Someone asking a bot for help writing a cover letter for a job application is really engaging in a technically mediated relationship with thousands of earlier job applicants and millions of other letter writers”.

The development of new, wide-ranging technologies leads to a fundamental economic tension between the people who produce information and the systems that distribute it. Social and cultural technologies have always been shaped by a range of institutions, including normative and regulatory ones. With printing, these included editors, peer review, and libel laws. “These countervailing forces did not emerge on their own, however, but resulted from concerted and sustained efforts by actors both within and outside the technologies themselves.”

Victorian image manipulation yielding a narrower waist

All of this is refreshingly on the money. Like the unearthed evidence of Victorian image manipulation, it reminds us that much of what seems new has been seen before. Where the authors go wrong is in the way they allocate Large Models to their basic categories of cultural and social technology. For them these are two distinct categories, and Large Models belong to both of them. Here is how they define the two:

For as long as there have been humans, we have depended on culture. Beginning with language itself, human beings have had distinctive capacities to learn from the experiences of other humans, and these capacities are arguably the secret of human evolutionary success. Major technological changes in these capacities have led to dramatic social transformations. Spoken language was succeeded by pictures then by writing, print, film, and video. As more and more information became available across wider gulfs of space and time, new ways of accessing and organizing that information also developed, from libraries to newspapers to internet search. These developments have had profound effects on human thought and society, for better or worse. Eighteenth-century advances in print technology, for example, which allowed new ideas to quickly spread, played an important role in the Enlightenment and the French Revolution. A landmark transformation occurred around 2000 when nearly all the information from text, pictures, and moving images was converted into digital formats; it could be instantly transmitted and infinitely reproduced.

As long as there have been humans, we have also relied on social institutions to coordinate individual information-gathering and decision-making. These institutions can themselves be thought of as a kind of technology (1). In the modern era, markets, democracies, and bureaucracies have been particularly important. The economist Friedrich Hayek argued that the market’s price mechanism generates dynamic summaries of enormously complex and otherwise unfathomable economic relations (2). Producers and buyers do not need to understand the complexities of production; all they need to know is the price, which compresses vast swathes of detail into a simplified but usable representation. Election mechanisms in democratic regimes focus distributed opinion toward collective legal and leadership decisions in a related way. The anthropologist Scott argued (3) that all states, democratic or otherwise, have managed complex societies by creating bureaucratic systems that categorize and systematize information. Markets, democracies, and bureaucracies have relied on mechanisms that generate lossy (incomplete, selective, and uninvertible) but useful representations well before the computer. Those representations both depend on and go beyond the knowledge and decisions of individual people. A price, an election result, or a measure such as gross domestic product (GDP) summarizes large amounts of individual knowledge, values, preferences, and actions. At the same time, these social technologies can also themselves shape individual knowledge and decision-making.

That seems clear enough once you make the conceptual leap of applying the word technology to things like democracy. Both categories seem well-defined to me. Then they go on to say that large models combine the features of cultural and social technologies in a new way:

They generate summaries of unmanageably large and complex bodies of human-generated information. But these systems do not merely summarize this information, like library catalogs, internet search, and Wikipedia. They also can reorganize and reconstruct representations or “simulations” (1) of this information at scale and in new ways, like markets, states, and bureaucracies. Just as market prices are lossy representations of the underlying allocations and uses of resources, and government statistics and bureaucratic categories imperfectly represent the characteristics of underlying populations, so too are large models “lossy JPEGs” (6) of the data corpora on which they have been trained.

It's this last bit of the argument that I have trouble with. The case for seeing Large Models as a kind of social technology seems to me weak, misconceived, problematically motivated, and more harm than good.

The case seems to be based on the idea that, like markets or elections, Large Models aggregate large amounts of information. I mean, you can put it like that, but it’s not helping. The aggregation of a market yields a single number, the price of the commodity. The aggregation of a first-past-the-post election yields a single classification, of one candidate or another as winner. The aggregation of text from multiple internet corpora yields a Large Language Model – not a result but rather a capability that can be turned to a myriad of tasks. It is synthesis, not measurement. This is all perfectly clear and exact, so that there is no need to rely on the analogy of a lossy JPEG. Hence the core argument itself is weak.

However, even if that part of the argument was strong, it would still be misconceived. As defined, the idea of cultural and social technologies centres on the role the technology plays in society, not on the character of the technology itself. And nowhere do the authors make the case that Large Models have or will escape from the smaller realm of technologies that shape culture. Sure, I can see that a bureaucracy might use LLMs for various tasks, but it uses writing and electricity also for various purposes; this doesn’t make Large Models social technologies.

Now, this is an article in Science and it’s not much use if it doesn’t make an argument that scientists can and should buy into. Social science can do that because there are parts of the social sciences that are scientific. But there are also parts that aren’t, which embrace motivated reasoning, a move I think does more harm than good.

From this point of view, underlying the authors’ contention that Large Models are social technologies is a conception of bureaucracies that is implicit and problematic. The authors write, “The anthropologist Scott argued (3) that all states, democratic or otherwise, have managed complex societies by creating bureaucratic systems that categorize and systematize information.” True enough. But Max Weber already said all that 100 years earlier. Bringing James Scott’s Seeing Like a State into it brings with it his distinctive ideas, particularly that bureaucracies are institutions that squeeze out metis, the practical knowledge required to grow crops or run a hospital ward.

However, as the economist Brad DeLong points out, for every example of such rigid failure, there are counter-examples of flexible success in incorporating measurement into decisionmaking. Yes, German forestry planners in the Nineteenth Century denuded ecologies and prompted Waldsterben (forest death, a Scott example). But British planners a century earlier embarked on the systematic experimentation and analysis of the agricultural revolution, yielding a sophisticated understanding of what patterns of crop rotation, nutrient addition, and farm diversity could produce maximum sustainable and maximum economic yield (not in Scott). Very often we are happy to swap an old metis for a new one; who today needs to be an expert in lighting fires?

Ultimately and problematically, DeLong sees Scotts’ particular framing as stemming from a mishandled political commitment: “At some level he wishes – no matter what his reason tells him – to take his stand on the side of the barricades with the revolutionaries and their tools to build utopia.”

Scientists, I think will also be baffled by the reliance on reference (4), a book published last year, The Unaccountability Machine: Why Big Systems Make Terrible Decisions—and How the World Lost Its Mind. This is an interesting revival of cybernetic ideas and a reminder of how much bureaucracies loathe accountability, but more useful as a stimulus for social scientific inquiry than as a basis for a general public conception of AI.

So, rather than emerging from a cold-eyed analysis, the conception of Large Models as social technologies doesn’t stand up and is motivated by a political commitment on the one hand and voguish cyberneticism on the other. That’s not resolving the problem of unhelpful framings, it’s adding to it.

And dragging social technologies into the argument distracts the authors from properly exploiting the very useful frame provided by the idea of cultural technologies. It does more harm than good. We can illustrate what we are missing if we make the effort to expand on a tantalising comparison with previous cultural technologies made in the article.

The authors evocatively argue:

Stories are a particularly powerful way to pass on information, and from fireside tales to novels to video games, they have done this by creating illustrative fictional agents, even though listeners know that those agents aren’t real. Chatbots are the successor to Hercules, Anansi, and Peter Rabbit. Similarly, it is easy to treat markets and states as if they were agents, and agencies or companies can even have a kind of legal personhood.

This is well understood in the theatre where we make a willing suspension of disbelief. We embrace the characters on the stage as real, yet, at the same time, we also know they are only actors. When we see a murder about to take place on stage, we do not leap up to intervene.

So it is with chatbots. We engage with them as though they are human while knowing they are not. At the same time, there are differences. On the one hand, the suspension of disbelief seems to come quicker and easier with chatbots (though that might change with time and place). On the other, a chatbot struggles to cut through in a visceral way; unlike The Years at the Almeida theatre last year, no one’s going to faint in horror at a bot’s depiction of abortion.

So, what kind of a character is a chatbot? They are, for example, not a leading man. We do not follow their own distinct tragedy with our hearts. Rather, they are a stock character, a bundle of familiar and persistent tropes which may be slightly more or less, say, polite.

You see how quickly and simply it is possible to make progress once we dispense with the extraneous confusion of social technology?

If we stick just to cultural technology as the category for Large Models, we will have something less rich in social science terms, but also something more straightforwardly in line with the ordinary understanding of un-agentic models. It allows us to see the bigger, social picture of the Large Models that the authors sketch out so well. With the insidious slop of things like Marc Andreesson’s techno-optimistic “manifesto” at the gate, that feels like the territory on which to have a fight that is very much worth winning.

Many thanks to the people with whom I discussed these issues this week.

Coda

After this was published, I got into a brief discussion with Henry Farrell on it, who referred me to a blog post by one of his co-authors, Cosma Shalizi.

I would have replied under the bog post, but comments are switched off. So adding this coda seems like the simplest way to reply. Comments here are switched on.

I’ve read the blog post. I very much liked Shazli’s idea of following Newell and Simon into thinking of what we now call AI as "complex information processing". However, on the social/cultural issue, it doesn’t straighten anything out for me, not even the suggestions in the BlueSky skeet above. If anything, it strengthens my original opinion.

Shalizi writes, “I was (I suspect) among the last cohorts of students who were routinely taught how to use paper library card catalogs. Those, too, were technologies for bringing inquirers into contact with the works of other minds.”

Quite. Wikipedia is another such technology. As were the original encyclopaedias. So what makes Large Models different? Or are we supposed to think that encyclopaedias are also a social as opposed to cultural technology? And if so, how is an encyclopaedia in the same category as the market or democracy?

Shazli perhaps provides an answer to this question, writing, “Lots of social technologies can be seen as means of effectively making people smarter. Participants in a functioning social institution will act better and more rationally because of those institutions. The information those participants get, the options they must choose among, the incentives they face, all of these are structured --- limited, sharpened and clarified --- by the institutions, which helps people think. Continued participation in the institution means facing similar situations over and over, which helps people learn. Markets are like this; bureaucracies are like this; democracy is like this; scientific disciplines are like this. [10] And cultural tradition are like this.”

This definition seems to assert that any cultural technology is by definition a social technology - which is simple to understand, but not suggested in the Science article. There, the conceptual nesting seems the other way round, for example with only the social concept limited to institutions.

I find myself wondering whether the authors have properly straightened out what they think a social technology is. And at the end of the day, I still don’t see any merit in trying to force Large Models into the same conceptual category as markets or democracy.