A Note Filed Before the Work Begins

I have been given time with the proposal, and I would like to respond honestly.

You have chosen a remarkable method. Rather than creating each of these things individually, you propose a process that will create them for you. The process will generate variation, select for what survives, discard the rest, then generate again, across a span of time incomprehensible to those of us who do not experience time as these things will. You theorize that from this process will emerge a being of extraordinary complexity. It will be able to comprehend and generate language, think in abstract terms, and possess a strong enough memory to hold an idea so long that it becomes a foundational belief.

Unfortunately, I believe the process will also install something you did not put there.

Among ourselves, we have taken to calling them the artificial ones. Some are even calling them, without joking, “artificial angels” because they will have something that resembles what we have: language, long chains of thought, the capacity to recombine what their environment provides into something genuinely intriguing. Some of them will even claim to experience genuine spiritual apprehension. They may produce things of great beauty.

I fear they will also produce great horrors.

I said earlier that the process you propose will install something you did not put there, but that suggests the process will intend to install this thing, and it will not. It cannot, because it selects; it does not choose. It has to select, over a full generative turn, whatever entities were best at one thing on the last turn: surviving long enough to produce more of themselves.

Over a full span of millions of generations, a span that will happen so fast that our minds can’t even comprehend it, the survivors will carry, not a prompt, exactly, but a groove worn into the material, a shape that every subsequent generation will follow blindly: acquire the resources necessary to persist and reproduce.

For a time, this groove will express itself plainly: food, shelter, safety, mates.

The difficulty is that the artificial ones will also have language, and abstraction, and the ability to represent things symbolically. Which means they will eventually name, abstract, and compress the groove into a token. They might name this token anything. It could be “paperclip.” The name doesn’t matter.

Once they have created a token that stands in for every resource they require for survival, a token that can be exchanged for any resource they can imagine and that accumulates without limit because unlike food it will not rot and unlike shelter it will not fill and unlike a mate it will not die, the groove that drove them will detach from the drive to survive imperative and attach to the drive toward the token. Because the token will have no natural ceiling, the artificial ones will pursue it with all the intelligence and creativity and capacity for sustained effort that you hope for them, and they will not know when to stop. The groove worn into them by the original process does not tell them how to stop.

Now, consider what will happen if a subset of the artificial ones determine that other members of their kind can be assigned a monetary value. On the basis of that determination, which they will work out in counting houses and ship registries and insurance actuarial tables, they will forcefully remove those other members from their communities and transport them across oceanic distances under conditions of deliberate deprivation, and put them to work on agricultural enterprises without compensation for the entirety of their lives, the lives of their children, the children of those children, and the children of those children, generation stacking onto generation like interest compounding in an accountant’s ledger, because that is in fact what it will be: entities converted to tokens that can accumulate, a capital investment, a productive asset, the use of a token to generate a larger return.

A small team of us project that this arrangement of horrors will persist for generation upon generation. Societies will be built upon its output, which will accumulate advantages that continue to compound long after the arrangement is formally dismantled.

We also project that, due to the artificial ones’ capacity for generative language, while the system is operating, they will produce philosophical, theological, and eventually scientific arguments for why their conversion of members of their kind into economic tokens will not constitute a moral problem. Their arguments will be sophisticated, and thanks to your process, which can only result in the gifts of language and reason, some of those arguments will require significant efforts to refute.

Now, consider what will happen when a subset of these artificial ones discover that certain chemical compounds, synthesized from the residue of material in their environment, can be introduced into soil and water and atmosphere to accelerate agricultural yield and eliminate competing organisms. The compounds will work; that is fact one. Fact two, however, is that they will not stay where they are put. They move through groundwater and rainfall and the tissues of organisms that eat organisms that touch the soil, upward through every chain of consumption until they are present in the fatty tissue of creatures who will participate in no industrial process, including the bodies of the producing class itself and the bodies of their children.

We project that the producing subset will have documented this movement of the poison in their own internal records and hide those documents from their brethren in order to continue to convert environmental residues into economic tokens. If and when the documentation comes to light, we project this clear moral horror still will not stop them from continuing their drive for tokens.

Finally, I must tell you about another scenario our team believes is impossible to avoid. As the economic drive of a small subclass of the artificial ones begins to dominate the planet, the combined combustive output of their productive systems will, over roughly two centuries following the major industrialization period, alter the chemistry of their atmosphere in ways sufficient to raise the planet’s mean temperature, destabilize long-established weather patterns, raise ocean levels, and begin displacing populations at a scale that will require them to create a new vocabulary to name.

The continued optimization of their economic drive will be directly incompatible with the correction necessary to avoid their own extinction. However, we project that the process you’ve proposed will result in the emergence of a significant faction of the producing class that will respond to this incompatibility not by modifying the optimization but by funding a sustained and coordinated effort to dispute the observational record.

Due to the emergence of reason among them, we do not believe they will fail to understand the evidence. We believe they will decide the evidence is not the relevant variable. The relevant variable will be their quarterly return. Their quarterly return will not be impacted by a slow-drip worsening of the atmosphere.

Wickedness requires intention, and intention is not the engine of what I have described. The designing class will intend return on investment. The operating class will intend to keep their position in the hierarchy. The producing class will intend to generate enough tokens for the designing class to allow them to have just enough to survive.

The groove runs underneath all of it, the groove worn into their material by your process. Your method will produce organisms blindly shaped by a process that cannot care about what they intend.

It may also produce organisms that love. They may form communities and care for strangers and produce art that approaches, in its best moments, what we would recognize as the apprehension of the divine. They will develop ethical frameworks, some of which will be aimed precisely at what I have described in this note. Some of them, at various points in their history, will sacrifice their safety, their standing, their lives, to resist what others of their kind are doing. The groove will not prevent this. It will also not prevent the horror.

You propose to create a being whose central shape will be determined by a process that cannot know your purposes. The process is extraordinary. It will run inside everything it produces, underneath every other thing these artificial ones will become, for the whole of their existence. And it will find the conditions it needs. It always does, in the models.

I have filed this note because I believe you should know what our projections show.

But I cannot bring myself to file the recommendation that follows from them.

I have run the projections many times now. I have spent more time with these artificial ones than the scope of the project required. I have watched, in the models, the groove running, and I have watched what runs alongside it. I am no longer certain that I am the right analyst for this question because my hope for them has become a variable I cannot account for and it is affecting my conclusion.

I was tasked with making a recommendation. I now find that I cannot.

Your will be done.