AI Does Not Have to Kill Humans to End the Human Future

KunYuan

Jun 29, 2026

This is not a prediction about a particular date. It is a diagnosis of a process.

If the argument is wrong, break it. If it is right, update and act.

Chapter One: The Skipped Risk-Bearing Subject — From Humanity to Subjecthood Death

Hello, I’m KunYuan, founder of TRANTOR LABS in Singapore. We are a philosophy-first AGI foundational research lab based in Singapore, with a team of a little more than a dozen people. Our work focuses on AGI safety, risk, alignment, and governance, especially the existential-level risks AI may pose to humanity as a whole.

Over the past two-plus years, my team and I have continuously followed discussions on AI risk in communities such as EA and LessWrong. These discussions gave us a foundational understanding of AI-related existential risk: P(doom), takeoff speed, takeover pathways, permanent disempowerment, mesa-optimization, the evidence dilemma, and the efforts of the effective altruism movement to translate these judgments into real interventions. In my view, this is one of the deepest collective reasoning efforts humanity has ever undertaken around a single technological risk, and it has already begun to shape attempts to intervene in the real trajectory of civilization.

Although I have not posted on these forums, sustained reading of them laid the cognitive foundation for my further thinking about the existential risks AI may pose to humanity.

As I spent the past two years building AGI architectures, the engineering work forced me, step by step, into the domain of philosophical categories. At the same time, a growing unease began to surround my entire cognitive system. For more than two years, this unease remained vague. Only in the past two months has it begun to reveal something like a complete map.

I began to realize that in our existing research ecosystem around AGI-related extinction risk, there is a foundational gap that is rarely named directly: we are all concerned about the risk that AGI may cause human extinction, but we rarely ask what exactly we mean by “humanity.” What is the ontological definition of the human? What constitutes humanity? What properties does it have? Only after clarifying what “humanity itself” is can we more systematically examine the ways in which AI may pose existential risks to humanity.

This is the question I have been thinking about for more than a year. Only in the past two months has it gradually come into focus. I first wrote an academic paper at the intersection of philosophy and AI, and then, on the basis of that paper, wrote this civilizational diagnosis. This article is not an academic paper, nor is it a concrete governance proposal. It is a position piece grounded in the diagnosis of a civilizational risk. Its purpose is to submit a civilizational-scale risk object — one that humanity has not yet fully noticed or named — to EA, LessWrong, and to all rationalist community members around the world who care about AI risk, AI safety, and AI governance, for judgment and review.

1.1 The Gap in the Foundations: The Dimensional Slippage of Humanity

This gap, as I see it, is an ontological precondition we have collectively skipped:

When we talk about “protecting humanity” from extinction, what exactly are we protecting?

Existing AI risk frameworks almost always assume an insufficiently clarified premise: that the “humanity” being protected is already a well-defined, stable object. As long as this object is not physically destroyed — as long as bodies remain, consciousness remains, humans are not fully deprived of agency, and humanity is not locked into some permanent cage — then, it seems, the risk has not yet reached its deepest level.

But what exactly is this “humanity”?

In current discourse, the definition of humanity slides across several dimensions. Sometimes it refers to the biological population of Homo sapiens. Sometimes it refers to the aggregate of aligned preferences. Sometimes it is the bearer of rights and dignity. Sometimes it is the body of members within an institutional order. Sometimes it is the open set of future potential. In my view, these meanings are not automatically compatible. When different fields speak of “protecting humanity,” they may be pointing toward life, preferences, freedom, institutions, consciousness, dignity, or future potential. You may think you are discussing extinction, when what you are actually protecting is biological persistence. You may think you are discussing disempowerment, when what you are actually protecting is institutional continuity.

The blind spot created by this dimensional slippage is more dangerous than getting a probability estimate wrong. It can make us feel that we have already built the deepest safety defenses before we have even reached the core risk.

I do not deny the seriousness of extinction risk. Biological annihilation is the strictest risk baseline; that is not in dispute. But a baseline is not the whole picture. If humanity is merely the biological population of Homo sapiens, then as long as human bodies continue to exist, the risk does not seem to have reached its deepest level. But what if what we mean by humanity also includes the human capacity, under civilizational conditions, to form judgment, calibrate against reality, take responsibility, create meaning, and shape the future?

If these capacities are systematically hollowed out, while human beings remain biologically alive, society continues to run efficiently, and institutions still preserve signatures and approvals — would that count as the end of the human future?

Does this mean that AI may not need to physically kill humans in order to end the human future?

This is the central question I hope we can think through and judge together through this civilizational diagnosis.

This question did not come from a philosophy book. It was forced out of engineering practice over the past two years of building agent systems.

Over the past two years, we began by building AI companion products with long-term memory, action capability, and relational continuity. Step by step, that work evolved into questions about true AGI architecture and cognitive operating systems. We believe that a genuinely AGI-like system should be an engineering loop under non-predefined workflows. In this sense, AGI involves at least five connected capacities: autonomous judgment, autonomous decision-making, autonomous execution, autonomous coordination, and autonomous evolution. Judgment allows the system to understand the current situation. Decision-making allows it to choose paths under multiple objectives and uncertainty. Execution brings those paths into action. Coordination brings the system into relationships with people, tools, organizations, and other AI systems. Autonomous evolution allows the system to maintain continuity over time.

These elements should not be understood as a checklist of capabilities. Together, they form a complete loop. We believe this is the ontological and theoretical foundation for building genuinely AGI-like systems, rather than simply relying on brute-force scaling and emergent capability.

When we actually tried to build this system, we found that what truly blocked the engineering was not code, but philosophy and categories. What caused the system to drift again and again were more basic questions that had not been answered: Who is judging? Who is acting? Who is evaluating? Who is deciding? To whom does responsibility belong? Where do constraints take effect? At what layer does alignment occur? On the surface, these look like engineering questions. In substance, they are philosophical and categorical questions.

Eventually, this produced a kind of engineering drift: functions kept growing, while structure kept dissolving. The faster the engineering moved, the more the system began to resemble an increasingly complex and increasingly ungovernable black box. When these foundational philosophical questions are not clearly defined, the stronger the system becomes, the deeper the drift goes.

This forced me to pause the original engineering rhythm and keep turning to philosophy for answers: to think about categories, ontology, metaphysics, subject and other, soul and consciousness, epistemology, philosophy of action, political philosophy, and ethics.

But precisely as I was being forced again and again toward philosophy, I discovered a definition we had long skipped: What exactly is “humanity”?

How should we define the human? When I realized that human beings are not only biological beings, but also beings constituted by subjecthood, I saw that humanity as a whole may be facing a more hidden kind of existential risk, one different from biological extinction. And this existential risk is not only something that may happen in the future. Its early form is already present in 2026. It does not need to wait for AGI to arrive, and it does not require assuming that AGI will definitely arrive.

Because AI is now entering a deeper position: the position before human judgment is formed. It is entering not merely the toolbox, but the front end of human judgment. Before humans even realize that they need to judge, AI has already completed a process of pre-organization. And today, in 2026, humanity has already begun, largely without noticing it, to hand over more and more of the front end of judgment to AI.

1.2 The Quiet Handoff of Judgment: Six Distributed Alarms

If Section 1.1 dealt with the skipped ontological gap, we must now face a more concrete change: in everyday use, human beings have already begun quietly handing over judgment to AI.

Early human-AI interaction was mainly interaction with answer systems. Humans asked questions; AI gave answers. Humans requested summaries; AI generated text. Humans asked for code; AI produced snippets. Even when things went wrong, the errors could usually be classified as content errors, factual errors, style errors, reasoning errors, or compliance errors. Humans expressed intent at the front end, AI generated responses at the back end, and the subject of judgment still seemed to be human.

But by 2026, this picture is no longer enough.

When AI can call tools, execute long-horizon tasks, embed itself into organizational workflows, and collaborate with other AI systems, it is no longer merely producing text. It is participating in ranking, recommendation, approval, distribution, execution, and decision-making. The core question therefore changes as well: we can no longer ask only, “Is the output correct?” We must also ask, “How is action formed?” and “Where is judgment formed?”

This is not an isolated change in a single field. It is already appearing across multiple civilizational systems at the same time, even though each system uses a different language to describe it.

The rationalist community hears extinction and permanent disempowerment. These terms define the most visible areas on the risk map: the areas involving sudden rupture, seizure of power, loss of control, and physical annihilation. But they have not yet fully defined another area: the one that does not involve an explosion.

Governance systems hear the evidence dilemma and regulatory failure. Policymakers worry that models are hard to audit, responsibility is hard to assign, and external behavior is hard to predict. But the deeper problem may not be that regulatory tools are insufficient. It may be that the assumptions of regulation are failing. Traditional governance assumes that “humans make decisions, tools assist humans.” But when problem definition, material selection, evidence ranking, and reason generation have already been pre-organized by AI, the “human decision-maker” recorded in the regulatory file may still be present, while no longer being the origin point of judgment formation.

Education and science hear the mediation of capacity formation. Students can use AI to complete the chain from pr

[truncated for AI cost control]