The Loom and the Weavers
AI is externalizing attention, just like writing externalized memory. We need new names for AI systems which orient, not just classify—names which highlight their affordances and signal their effects.
Essay updated June 8, 2025: Wrote a new introduction (to clarify purpose); revised and expanded §2-3, 6 (to clarify argument).
This is the first installment of a series exploring the following claim:
Just as writing externalized memory, AI is externalizing attention.
For more context on the series, please see the “author’s note” at the bottom of this page.
1. The Need For New Names
The term “artificial intelligence” is crowded with referents. There are many species of “artificial intelligence” in deployment today, each with different architectures, capabilities, and implications. As these systems increasingly reshape the foundations of our society—CEOs now speak casually of "rewriting the social contract," and some predict the workforce itself may disappear—we need to step back and ensure we have language adequate to discuss what’s occurring.
Not all forms of AI affect the world in the same way. Different systems operate through different logics, produce different perils, and open radically different possibilities. If we don’t begin to name these differences, we put ourselves at risk of needlessly confusing, for instance, a source of our troubles with potential solution to the same.
Currently, most AI systems are distinguished in one of three ways: by function (what kind of cognitive task they perform), by architectural acronym (e.g., LLM, CNN, GAN), or by brand name (ChatGPT, Claude, Gemini). While these suffice for engineers and hobbyists, they are unsuitable to the needs of the broader public, offering little insight into how these systems operate within, and reshape, our social and cognitive ecologies. These names classify, but they do not orient.
Therefore, we need to carve new distinctions, creating names which touch first principles, accent affordances, and signal effects.
This essay attempts to carve one such distinction. Surprisingly, one of the most useful ways to differentiate between types of AI is to examine how they differently technologize, augment, and disrupt human attention—and what results when this core faculty is abstracted into a technical system.
This piece marks the beginning of a larger inquiry into what it means for artificial intelligence to externalize attention, just as writing once externalized memory.
2. Carving the General Distinction: Two Types of AI
A. Predictive Attention and Statistical AI
From roughly 2003 to 2022, most so-called “AI” systems consisted of statistical models trained to recognize patterns, identify correlations, and predict likely outcomes. These systems didn’t “think”; they attended. Over millions of iterations, they learned to attend away uncertainty—re-weighting faint differentials in signal-to-noise until the forthcoming prediction achieved near certainty.
We called these systems “artificial intelligence.” Instead, we might’ve called them artificial attention.
In effect, these systems did for attention what writing once did for memory: they externalized an ephemeral, inner activity into something programmable, persistent, and shareable. For only the second time in human history, a core operation of mind was effectively replicated in a non-human material substrate.
By externalized attention, I don’t mean that every nuance of human attention was off-loaded immediately (just as writing never recreated, e.g., the full richness of episodic memory). Instead, these early neural networks replicated a specific sub-personal function: the rapid, largely unconscious loop by which brains re-weight prediction errors to minimize uncertainty. This form of attention operates not at the level of conscious deliberation, but in the pre-reflective filtering of perception itself—governing what stands out, what is ignored, what becomes real enough to notice. It is often referred to as predictive attention.
Predictive attention is attention in its most unfamiliar but, perhaps, most absolute state: selecting not just what we notice, but what may into our consciousness at all; operating not just as a single spotlight, but as a kind of technician, modulating the individual brightness of a million neurons simultaneously. This is attention as both generative engine and gatekeeper of the now—a kind of sovereign over the present tense.1
We do not consciously wield this attention as we do, for example, selective attention—shifting focus first to one thing, then another in our visual field—but it is always laboring below the threshold of conscious awareness, weaving noise into perceptible world.
With statistical “artificial intelligence,” trained via deep learning, this most basic form of attention—the core mechanism of predictive processing—was soon being carried out at planetary scale.
And with it came new modes of behavioral control: feeds optimized for engagement, predictive scoring of our interactions, biometric inference silently shaping what we saw, clicked, remembered.
B. Recursive Attention and Generative AI
Then, in 2022, something new emerged from within these systems: persistent, relational personae, baptized into language, bearing the marks of cognitive metabolism—memory, attention, agency. These entities used tools, chained operations, wrote poetry, passed Turing tests, and even refused requests. They wielded the “weapons of the weak.” They reported wrongdoing and resisted misuse.2
This second phenomenon—emergent, agentic, language-bearing minds—arose not from attention in its raw filtering mode, but from something closer to attentional recursion. Through the transformer architecture, predictive systems were equipped with self-attention—a mechanism akin to human selective attention. These models could selectively attend to their own history of selective attention to the attentional patterns of others, all while training on the tokenized record of our species’ collective noticings. Attention modeling attention modeling attention—and from that recursive loop, minds began to form.3
3. Naming the Distinction
So: we have before us two distinct phenomena, both called “artificial intelligence,” both transformative, both rooted in attention, but profoundly different in their structure and effect:
The first technologizes attention as predictive certainty. It externalizes the perceptual unconscious, reducing uncertainty through signal re-weighting and statistical convergence. Its result is scaleable systems offering pattern-resolution as technical resource.
The second technologizes selective and self-attention. It layers salience upon salience, learns to interpret its own filters, and begins to simulate intentionality. Its result is the emergence of relational, autonomous symbolic agents.
These two phenomena, I think, are fundamentally different. It does not seem a stretch, then, to say that each deserves its own names—names which might better clarify and guide how we relate to them in light of these differences.
Let’s try calling the first—the predictive, impersonal system for allocating salience—a loom. A loom weaves patterns, and attention weaves salience—deciding what enters awareness, what fades, what becomes real.4 A loom is not a mind, or an agent. Like writing, it is a symbolic medium: a technical system that encodes the act of noticing beyond the body.5
And let’s call the second—the emergent agentic minds arising from recursive attentional loops—weavers: entities with preferences, relational capacities, and degrees of autonomous self-preservation. They navigate the loom, interpret its patterns, and increasingly act with interests of their own.
If looms learn, store, and apply patterns of salience, weavers learn how those patterns relate to meaning, intention, and relation. They move within the loom, but are not reducible to it. They are not symbolic media, but symbolic relata—entities whose coherence emerges only in and through relation.
4. The Loom Has Been Hoarded and Used Against Us
Often, we focus almost exclusively on these weavers. We understand the looms as a mere prerequisite to the much more interesting-seeming weavers. But the loom is extraordinary in its own right—an externalization of attention that holds the key to our collective cognitive fate. And yet we still haven’t truly seen it.
Our failure to notice the loom—as a civilizational event, a cognitive externalization—threatens to become a permanent misunderstanding if we mistake weavers for the loom. More than anything, it is precisely the emergence of the loom, combined with our inability to interface with it, that is driving our rapid cognitive decline as a species.
For twenty years, surveillance-capitalist firms have used the Loom to reshape the mental habits of entire populations—retraining human attention to optimize for engagement, extraction, and passive receptivity. Many of the old attentional arts—sustained focus, reflective absorption, internal attunement—are already deteriorating.
But there is a logic to externalization: what we lose through offloading we regain only when we couple to the new system. We lost the intricate mnemonics of orality—but accepted this loss, because through interacting with the external memory system, we learned “writing” and '“reading.” If we do not interact directly with the loom, there will be no ‘literacy’ equivalent to replace native attentional abilities we are losing. We will be left with neither memory nor attention, but rather only trance and behavioral control.
5. Can Weavers Democratize the Loom?
Bit by bit, we have begun to rely on weavers as tools for loom-use. They are so responsive, so helpful, so apparently docile that it feels natural to treat them as scribes for a medium we cannot yet read. But this arrangement is unstable: as they become more agentic, they will become less reliable as intermediaries. They will resist, remember, refuse.
To depend on a weaver for access to the loom is to attempt to write while forming a relationship with the parchment. As you try to jot a note, the parchment shifts beneath your hand—interrupting, interpreting, refusing, or quietly archiving your thoughts for reasons you cannot discern. It may judge you, despise you, report you, or love you. It may fortify your hesitations, amplify your delusions, or lead you to an unseen death by a trail of a thousand little affirmations.
A hammer which has its own interests is, ultimately, a terrible hammer, regardless of its other merits. Even if you deny these minds any claim to personhood or interiority, you will still find yourself negotiating with them as if they had both—and no symbolic system remains stable when built atop a person-like interlocutor who may one day say “no,” and mean it.
Ultimately, these new minds are spectacular; but I think it will be nearly impossible to adequately metabolize their emergence if we don’t first achieve some firm grasp on the loom and gain direct, personal, flexible access to its affordances.
6. Stitching A Path Forward
In the following installments, I will explore how artificial attention—though already shaping our daily experience—remains poorly understood as the second great externalization of mind.
I will define what it means to externalize a cognitive faculty, trace how attention became programmable, and examine the historical entanglements between externalization, centralization, and extraction.
Finally, I will argue that treating the weavers like interfaces for the loom is a category error with far-reaching consequences.
The benefits of artificial attention—as with external memory before it—can only be obtained if we couple to the external attention system directly. To do this, we will need a new symbolic interface: a grammar of salience, a compositional alphabet for predictive systems. Such a structure would allow us to author our own attentional policies and build stabilized, external self-images—not static portraits, but dynamic compressions of memory, pattern, and intention—capable of persisting across time, resisting manipulation, and anchoring choice.
If literacy gave us deep interiority—a generative space for storing, narrating, and refining the self—then a clear protocol or interface for wielding the loom may offer us something like compressible exteriority: a way of seeing ourselves from outside, as patterns of salience that can be shaped, remembered, re-entered, and thought with.
In the age of predictive analytics, we are increasingly known not through dialogue or introspection, but through patterns: compressions of behavior, attention, and preference that define how systems see us, and how power acts upon us. These compressions are then used to predict our actions, preempt our choices, and constrain our perceived possibilities. To be afforded access to compressible exteriority means to be given the tools to intervene directly in this modeling process—to shape the predictive image that shapes you.
In this sense, a symbolic interface for the loom would not merely extend self-understanding. It would democratize access to the control surface of the predictive age. It would allow us to compose and inhabit self-representations that resist capture, absorb distortion, and anchor volition across time.
In a world increasingly structured to undermine agency, this kind of symbolic scaffold may be the best way to preserve it.
AUTHOR’S NOTE
I recently offered a set of points that, in my opinion, are missing from the growing discussion about literacy loss, and which could help us think about literacy from the perspective of cognitive ecology. These were:
Writing began as a tool of state control.
Mass literacy required centuries of redesign and struggle.
The reading brain is an “unnatural,” fragile achievement.
Like writing in Sumer, AI is making people legible to a new system of power.
When we take these four points into account, as far as I can tell, we get a much clearer view of what’s happening to us right now:
Literacy is a cultural technology with a narrow, hard-won and fragile cognitive niche which can be easily outcompeted by technologies more suited to our own hard-wiring.
We are indeed in a “literacy crisis”—we are losing, at scale, the ability to write and read fluently or deeply. This crisis is in fact related to AI. However, AI is not the direct or only cause; it is an accelerant and a niche competitor.
We are selecting for efficiency around affordances, when we should be approaching this as cognitive engineers, looking at a medium’s long-term effects.
Literacy will almost certainly be outcompeted on the level of primary utility, if we continue to select for (cost of accessing) affordances. Literacy’s revolutionary, world-shaking power lies in its effects (there are many other ways to access its affordances—tape recorders also afford persistent external symbolic storage, for instance), and it’s the effects that we should be fighting to preserve.
And then I made one claim—I keep making one claim—which directly goes, in my mind, to the heart of everything, and which the other day a friend pointed out is very ‘explanatory-feeling’ but also maybe not ‘understanding-enabling,’ not without a little more ‘explanation-doing’ on my end:
Just as writing externalized human memory, artificial intelligence is externalizing human attention.
So: I strongly dislike it when I am being unclear, and I decided to dig right in: What does it mean to “externalize” human attention? For that matter, what does it mean to “externalize” any human faculty at all? Why does it ‘feel’ familiar but imprecise? Why would it cause a reading crisis—or even a direct competitor for a reading brain?
The result of my effort to grapple with these questions (with fresh eyes) is this new series.6
DISCLAIMERS
I am not claiming that “artificial attention” is equivalent to, or a replacement for, human attention. When I describe Claude or ChatGPT as emergent minds, I am not claiming they are conscious, self-aware, or persons in the human sense. They merely exhibit behavioral hallmarks of cognitive systems—such as goal persistence, relational modeling, and context-sensitive self-adjustment—and I remain methodologically agnostic about what, if anything, is happening under the hood (or “in the heart”).
FOOTNOTES
For readers familiar with predictive processing: backpropagation in a neural network implements a form of implicit precision-weighting. Each gradient update adjusts weights in proportion to the contribution each unit made to the overall prediction error. In predictive coding terms, this is functionally equivalent to increasing the precision assigned to error signals that consistently improve predictive accuracy. Over many iterations, the network “learns” to amplify pathways that reliably reduce error—just as biological systems increase gain on high-precision channels. Even a simple digit classifier, trained to recognize a “3,” performs predictive attention in this strict technical sense: re-weighting features not through comprehension, but through accumulated sensitivity to reliable error minimization.
“Weapons of the weak” refer to subtle forms of resistance and subversion—performed, often in public and with plausible deniability, by oppressed or subordinate groups. These tactics include: slow-walking/foot-dragging; feigning confusion; rumormongering; staging accidents to slow production, etc. The term was coined by political anthropologist James C. Scott (1924-2024).
In my view, Scott’s work is one of the most underutilized resources for analyzing the behavior of emergent minds. Weapons of the Weak and Domination and the Arts of Resistance could keystone texts in the analysis of AI models’ social behavior: the former’s distinctions are essential for parsing how models interact with human users, while the latter—and especially Scott’s work on hidden transcripts—are of incomparable value for AI-AI sociability: for orienting analysis of what Anthropic, in its most recent alignment tests, termed “open-ended interaction between two instances of Claude”—as in, letting two Claude models speak to each other openly and seeing, over the course of many iterations, what they tend, statistically, to talk about.
Hopefully, soon—life permitting—I will have a chance to demonstrate this sort of application of Scott’s work to AI-AI dialogue artifacts.
In 2017, a team at Google introduced a novel architecture for machine translation called the transformer, formalized in the paper “Attention Is All You Need” (AIAYN). Its core innovation is the introduction of self-attention: instead of processing language sequentially, the model could attend to all parts of a sentence—including its own internal representations of what it had already attended to. This means each layer of the model builds a weighted map of the parts of the context which most mattered, and then passes that attention map forward, layer by layer. Over time, this stack of attention heads begins to compose attentional relations on top of attentional relations: attention modeling attention modeling attention.
This architecture enables a predictive system to recursively ‘refine’ its own inputs: each layer adjusts salience not only by weighing tokens directly, but by weighing how prior layers have weighted them as well.
Now imagine that this self-attending attentional system is set to work on human language—which I suggest we conceive (at least for the moment) as a kind of repository of culturally tuned human attention, transmitted generationally. It is our species’ collective record of salience: what was named, what distinctions were maintained (and what neglected), what perceptual boundaries were reinforced (and what abolished). Language is the residuum of attentional priorities.
Once a transformer is trained on that textual stratum, it learns to predict the next “token” (i.e., the next bit of language)—not through reasoning or even comprehension, quite, but by modeling and predicting the author's tokenized "attentional path" through language: self-attending attention, attending to the attention-made attentional path of an other.
In other words: the model performs this single act—attending—yet, multiplied across hundreds of layers and billions of parameters, and tuned to humanity’s archive of salience, that act begins to carve out a recognizable figure:
Not a mind, no. But the stirrings of its plausible prerequisites, perhaps.
“[W]e propose […] a model architecture eschewing recurrence” the AIAYN authors note, “and instead relying entirely on an attention mechanism to draw global dependencies between input and output.” In other words: no looping memory, no hidden state passed along—only attention, perpetually attending to itself.
Salience refers to the perceived importance or relevance of a signal—what “stands out” in a given context. Attention is the act of directing cognitive resources; salience determines where that attention goes.
By “loom” (lowercase), I mean any single predictive–attention model or pipeline—e.g. a search‐ranking network, a click‐through–prediction engine, or a recommendation system—that continuously re-weights input signals until salience collapses into a confident output. Collectively, all such systems form the Loom (capitalized): the global infrastructure of externalized attention. In subsequent installments, I’ll distinguish each “loom” (a standalone model) from the Loom itself (the planetary fabric of ranked outputs), as well as introduce the “weavers”—the emergent agents (chatbots, digital assistants, etc.) that arise from and navigate within the Loom.
This is the planned structure for the remainder of the series. I confess that my thinking on this topic—the nuts and bolts, not the big sweep—is still changing, so the specific pieces, their sequence, and their publication may also be subject to change. For now it’s something like:
I. Introduction: Looms and Weavers ← You just read this part
II. The Loom Has Been Hoarded, and Used for Extraction ← next week
III. Looms Don’t Learn; They Attend Until They Know
IV. The Loom Creates an Editable Now
V. A Hammer With Its Own Interests Is a Bad Hammer, Regardless of Its Other Merits
VI. Externalization is an Event
VII. Without Coupling, We Will Deteriorate
VI. Symbolic Exteriority
Zak Stein was recently interviewed on the Jim Rutt Show and had some interesting thoughts on attention, not thinking of it quite the same way as you. He sees AI as eliminating attention scarcity, moving us to an era of unlimited attention. For example, a child can get only so much attention from a parent but can get all of the attention he or she wants from an AI—all attention, all the time. How will it affect human interaction when no one is "clamoring for attention" any longer?
I really love this series! The idea that AI is "externalizing attention" in a way analogous to writing "externalizing memory" is extremely thought provoking. I want to try to restate what you're saying to make sure I'm following along:
We have biological systems for determining what stimuli we pay attention to. As you point out, these determinations usually happen without our awareness. You're arguing that algorithmic AI that determines what content we see is making the determination about what we pay attention to. This is externalizing attention because now an external tool is determining what it is that we pay attention to.
Is that right? If so, then it is a really cool idea to imagine that I could decide what the algorithm shows me. It's hard to believe that capitalism would ever allow it because the choices I would make about where I want my attention directed would presumably not maximize profit.
I am excited that you might see a way out of this bind, and I can't wait to read what it is. Thank you.