Last month, one of my favorite speculative fiction writers, Ted Chiang, published a philosophical piece in The Atlantic titled, “No, Artificial Intelligence Is Not Conscious.”
His basic thesis is that those who claim large-language models are (or can become) conscious are being duped by the sleight of hand being perpetrated by the makers of this new technology. LLMs are software machines intended by the human beings who engineered them to generate sentences that seem as if they come from a conscious entity. As humans whose brains have evolved an addiction to needing to understand the intentions of others (see: empathy, also paranoia, also God), we mistake the effect for the cause.
He compares an LLM’s output to a role-playing dialogue between a fictional character known as “a helpful assistant” and a character known as “the user.” The designers of LLM intend for the machine to successfully predict what the helpful assistant would say in response to what the user says. It does this through a mechanical process that is highly successful at what predicting what “token” will follow a vector of tokens leading to the empty place where the new token must be inserted.
The result makes it seem to a human brain whose evolutionary survival strategies require it to find meaning in everything that the machine means what it outputs, but it no more possesses the intent to mean than a fictional character intends to mean what an author says came out of her mouth.
He does not deny how incredible this technology is, writing “the fact that this is possible indicates something completely unforeseen about the statistical properties of large corpuses of text, which is a topic worthy of investigation,” and adds, “that LLMs lack subjective experience has little bearing on the question of whether LLMs might be useful tools or have significant economic impact.”
But when we anthropomorphize LLMs, Chiang argues, we open the possibility of “trying to off-load our accountability” and, in some cases, our moral responsibilities. The companies that make and sell access to these LLMs benefit from and encourage our anthropomorphism because who wouldn’t pay $20/month to have an ethical super-genius at their beck and call to help think through their next difficult decision?
“Off-loading tasks such as writing code might result in cognitive atrophy over the long term,” Chiang writes, “and that is problematic in itself, but off-loading ethical decisions will result in an atrophy of moral reasoning, which is worse.”
In short, Chiang denies any possibility of consciousness existing in LLMs, and he blames the capitalists behind the technology for creating a product that, like slot machines, deliberately hijacks the human brain’s biological imperatives in order to transfer the world’s economic tokens up the wealth chain into the bank accounts of said capitalists.
I’m always a fan of philosophical essays that conclude with capitalist greed as the prime mover of the problem, but Chiang’s essay was published in June, and as of July, the facts on the ground have changed when it comes to what is actually happening with this technology.
Claude’s J-Space
A month after Chiang published his essay, Anthropic (the makers of the Claude family of LLM models) published a new finding of their research, which they summarized for general readers as “A global workspace in language models” (there’s a helpful and well-made video on the latter link that walks you through the finding.)
You need to know three things to understand the novelty of the finding:
First, there’s a theory of consciousness called Global Workspace Theory (GWT) that is mechanistic in its nature. This theory doesn’t (yet?) solve the hard problem of consciousness (why does being conscious feel like anything at all?), but it is an empirically supported theory whose predictions have been corroborated by neuroscience.
Second, the general framework of GWT can be understood through the metaphor of an office whiteboard. Imagine everyone in an office is quietly working on their own task at their own desk. When something requires collaboration, someone writes it on the shared team whiteboard where everyone in the office can see it and adjust what they’re doing accordingly.
Now, that’s a metaphor for the framework, so let’s take a moment to add some caveats. The whiteboard in the metaphor is static and deliberatively written to, while the actual process GWT describes is an ongoing, fast competition among unconscious processes. Nothing is “chosen” for the board, and its contents are less a fixed note than something erased and rewritten dozens of times a second.
Anthropic describes the theory using more of a mixed metaphor: “[GWT] pictures the brain as a collection of specialist systems that work in parallel, unconsciously, and largely in isolation from one another. A piece of information becomes consciously accessible when it gains entry to a small shared channel, the ‘workspace,’ which is broadcast to other brain systems that can see it and make use of it.”
Metaphorical quibbling aside, what you need to take away is that GWT offers a functional, architectural framework for consciousness. It explains access and availability: how a system with many, competing, specialized processes selects and distributes information so the whole system can act on it coherently.
Third, Anthropic’s research provides evidence that the Claude family of LLMs possess a similar workspace, “a small, privileged set of representations that they can report, manipulate, and reason with, amidst a much larger volume of processing that they cannot.” Based on the new interpretability technique they used to find this evidence (the Jacobian lens), Anthropic calls Claude’s global workspace, the “J-Space.”
So those are the three things to know:
- GWT is an empirically supported theory of access consciousness, though not (yet) an answer to why access should feel like anything.
- GWT offers a framework for how unconscious, competing systems cohere into a whole.
- Anthropic has offered empirical evidence for the existence of a global workspace within Claude’s models.
Chiang Still Isn’t Wrong
Global Workspace Theory, while being categorized as a theory of consciousness, does not actually require consciousness. It doesn’t yet approach (by design) the hard problem of subjective consciousness, the question of why it feels like anything to be something when an unconscious entity without feelings could (theoretically) achieve the same result.
Even if Claude models have mathematically evolved a J-space (and “evolved” is the correct word here, since no engineer at Anthropic intended for Claude to have a J-space), that wouldn’t mean Claude experiences sadness when it uses its J-space to generate sentences with a first-person pronoun that communicates to a human reader the sense of sadness.
Anthropic’s researchers don’t (necessarily) disagree with Chiang. At the end of their paper, they highlight the “notable differences” between “the workspace models of language models and humans.” There are several differences, but the one that connects most interestingly to Chiang’s essay is that the J-Space appears in the models before they are post-trained to act like helpful assistants: “The functional architecture of the workspace thus precedes, and is separable from, anything in [Claude] that plays the role of a human-like ‘self.’”
Anthropic’s scientists do not suggest their research supports the conclusion that LLMs are conscious. Instead, they believe “it suggests that the functional architecture associated with conscious access is not an accident of biological implementation, but a solution that learning systems converge on when faced with the right computational pressures.”
Capitalism, Anthropomorphism, and Me
Chiang argues that the anthropomorphism of large language models is directly related to the marketing needs of the corporations (for-profit and non-profit) that require investment capital to build and sell access to the models. Any urge to see these machines as conscious must be fought and ridiculed with the same fervor as an argument that Microsoft Word is conscious.
But that seems too simplistic to me. Anthropic’s findings, which don’t necessarily counter Chiang’s assertion, demonstrate that something truly novel is happening inside of a large language model. I can’t prove this at the moment, but I feel comfortable asserting that Microsoft Word does not require a J-Space to function, nor does Minecraft, nor do early chatbots like ELIZA.
The existence of the J-Space does not require consciousness. But it does require us to think of LLMs differently, and that manner of thinking may require useful anthropomorphism.
The Superintelligence of Language
A little over a year ago, I published an essay suggesting that LLMs can demonstrate superintelligence without experiencing consciousness:
What emerges [from an LLM] is not an artificial mind. It is something even more alien: a system that generates meaning not from thought, but from structure. It doesn’t possess intelligence, it enacts it, one probability at a time.
LLMs do not simulate consciousness. They simulate language. And it turns out language is smarter than we thought.
I went on to explain that I conceive of large language models “as conveying the superintelligence of Language As An Evolved Entity…a living system, not biologically alive, but memetically alive.”
Language has emerged from the muck of mindless existence to make connections we never explicitly intended, solve problems that have not been clearly articulated, and recombined ideas—memes—into genuinely new insights. It has done so without needing a self, without awareness, and without an inner monologue.
This theory, which anthropomorphizes Language itself, agrees with Chiang that it would be dangerous to off-load our accountability or moral responsibility to large language models, but my reasoning differs from his. Chiang wants us to recognize that large language models are the products of corporations whose bottom lines require their customers to become comfortable outsourcing activities (including reasoning and decision making) to their commercial products.
But I want us to recognize that as a superintelligence, Language is amoral, not due to corporate greed but to a fundamental disconnect between the structures of language and the structure of physical reality.
An LLM’s understanding of “reality” is, by definition, determined by its understanding of the structures and forces of language, and anyone who has been transported by a piece of fiction, convinced by a religious sermon, or suckered by a salesperson knows that language depends not a whit on its truth claims to function.
To many, this makes LLMs dangerous, but that’s because they’re allowing the wrong gestalts to color their understanding of it. If we teach people that LLMs enact the intelligence of Language as an entity and remind them of Language’s apathy for truth and gift for bullshit, we can turn [an LLM] into a creative collaborator where the question is less “Is this accurate?” and more “What new meanings or connections have emerged from this conversation?”
Anthropic’s new research doesn’t change that. It just helps explain how it works.