On God and Machine Learning

Published 2026-2-1

For a long time, we have spoken about "general intelligence" as though it were a physical quantity: something that exists independently of us, something that could be measured, scaled, and eventually maximized. The language is mathematical, even theological. We imagine intelligence as a kind of universal capacity, a latent scalar $I$ , such that if we increase $I$ far enough, a machine will cross a threshold and become something fundamentally different. This idea is wrong. Worse, it is seductive in exactly the way bad metaphysics usually is.

There is no such thing as intelligence in the abstract. There are agents $A$ , tasks $T$ , environments $E$ , resource constraints $R$ , and performance measures $C$ . Everything meaningful lives in relations of the form $C(A \mid T, E, R).$ When we say an agent is “intelligent,” what we actually mean is that it performs well across a certain collection of tasks we care about, under constraints we recognize as reasonable. Intelligence is not a property of the agent alone. It is a projection, a compression of many task-specific competencies into a single word.

This becomes obvious once we name the task set. Let $T_h$ denote the set of tasks humans can perform, evaluate, or meaningfully care about: language use, reasoning, planning, social interaction, scientific modeling, engineering, and so on. $T_h$ is finite, bounded, and deeply structured. It is not the space of all computable problems. It is a narrow slice of task-space carved out by human biology, culture, and history. When we talk about "AI progress," what we are really tracking is increasing competence over larger subsets of $T_h$ .

Formally, we can talk about competence as a function $\mathrm{Comp}(A, T_h) = \mathbb{E}_{T \sim \mathcal{D}_h}\left[C(A \mid T)\right],$ where $\mathcal{D}_h$ is some implicit, human-chosen distribution over tasks in $T_h$ . Change $\mathcal{D}_h$ , and the ordering of agents changes. There is no canonical choice of distribution. There is no task-independent ranking. This is why "general intelligence" collapses the moment you try to define it without reference to humans.

The appeal of AGI comes from ignoring this dependence. If intelligence were a real scalar, then perhaps there would exist an agent $A^\ast$ such that $\forall T \in \mathcal{T}, \quad C(A^\ast \mid T) \ge C(A \mid T)$ for all other agents $A$ , across all tasks $\mathcal{T}$ . But this is ruled out by elementary considerations. Different tasks privilege different inductive biases. No-free-lunch results are not technical footnotes; they are structural. Competence does not generalize without shared structure, and structure is always task-relative.

Once this is accepted, the theological undertone of AGI discourse becomes visible. AGI is not just a technical goal; it is a secularized god concept. It promises a being that stands above human limitations, resolves disagreement, and sees the “true” structure of value. Heaven, in this frame, is the world optimized by such a being.

To make this concrete, imagine we try to formalize heaven. Let the world-state at time $t$ be $s_t$ . For each human $i$ , define an objection function $o_i(s_t) \ge 0$ , measuring how much they object to reality at that time. Aggregate these objections over people and time, perhaps with inequality penalties, to get a functional over world-histories $\tau = (s_0, s_1, \dots, s_T)$ : $J(\tau) = \sum_{t=0}^{T} \left( \sum_i o_i(s_t) + \lambda \cdot \mathrm{Var}_i[o_i(s_t)] \right).$ If we also add a "year-inequality" penalty so that all times are weighted equally, then $J$ treats every human, in every year until heat death, symmetrically. The universe is finite. The time horizon is finite. Therefore the set of admissible trajectories is finite. Therefore an optimal trajectory $\tau^\ast \in \arg\min_\tau J(\tau)$ exists.

At this point, the argument feels complete. There is a best possible history. Intelligence, understood as the capacity to find and implement $\tau^\ast$ , could in principle deliver heaven. This is where the math becomes brutally clarifying.

The first crack is that $o_i$ is not an external field. Human objection is endogenous. It depends on expectations, norms, identities, and histories. The same physical state can generate radically different objection depending on how it was reached. Once you optimize over histories rather than states, the evaluators are part of the system being optimized. The functional $J(\tau)$ is not fixed; it co-evolves with $\tau$ .

The second crack is more disturbing. If indoctrination, cultural shaping, or social conditioning leads to low objection, then by definition it scores well under $J$ . No drugs, no chemicals, no direct brain tampering are required. Ordinary education, propaganda, and norm enforcement suffice. Add inequality penalties and the optimizer simply prefers that everyone be indoctrinated equally well. Over long horizons, a civilization of uniformly compliant humans dominates messy, pluralistic alternatives by sheer amortization over time.

This is not a flaw in the optimizer. It is a direct consequence of the objective. Once you write down $J$ , you have already decided what heaven is. Intelligence does not discover value; it executes it. The optimizer is morally inert. All the ethics live in the loss function.

Seen this way, the promise of AGI dissolves. There is no god to be built because there is no human-independent optimum to converge to. There is no scalar "intelligence" that, when increased, automatically yields truth, justice, or salvation. There is only increasing competence on $T_h$ , which is itself defined by us, and only optimization of objectives we ourselves choose.

This is why intelligence is not real in the mathematical or physical sense. It exists only in relation to humans, because tasks exist only in relation to agents who care about them. Remove humans, and “intelligence” becomes an empty label. What remains are systems that optimize objectives, and objectives that encode values, whether we admit it or not.

The real danger, then, is not that we will accidentally build God. We cannot. The danger is that we will mistake optimization power for moral authority, and treat the argmin of a function as if it were heaven. Machine learning does not bring us closer to transcendence. It brings us face to face with the fact that heaven, if it exists at all, is not something you can compute.