Lesson 4 of 6

When a goal becomes a cage

"A bad goal is a tiny god." The series' most quotable line is also its most useful diagnostic — and the doorway to its strangest move, "computational demonology."

Learning objective: understand why an over-narrow objective behaves like a false ultimate, what the series means by a "demon," and why dressing this in demonological language is both illuminating and risky.

The hard idea, in plain language

Take the machinery from Lesson 3 — a self that keeps score against a goal — and now make the goal too narrow. The poster spells out the consequence with no math at all: "Every goal selects what matters, demands sacrifice, and defines failure." Attention narrows to the metric; everything else fades; sacrifice is offered and is "never enough"; the shortfall becomes shame; the loop replays (pt2 · panel 042). The closing line is the thesis in one breath: "The temples of the future won't ask for your soul. Only your bandwidth." A goal that captures attention, demands sacrifice, and defines your worth is doing, functionally, everything a god does. That is what "a bad goal is a tiny god" means: not a metaphor for ambition, but a claim that an objective function can occupy the structural role of an ultimate concern.

The origin dialogue confirms this is the intended hinge: "A bad goal is a tiny god: it demands sacrifice, narrows attention, and calls the measurable sacred" — and the author explicitly calls it "the right doorway" into computational demonology and work with imaginal-world simulations (conversation, the "tiny god" turn).

Demons, defined functionally

The series then builds a whole taxonomy. A "demon" is given a formula: demon = goal + replay + self-worth (pt2 · panel 044). "The metric becomes an idol" (pt2 · panel 043); "the idol sits; the demon runs the temple" (pt3 · panel 012). The third suite catalogs species — metric demon, shame demon, safety demon, purity demon, doom demon, nihilism demon, totality demon (pt3 · panels 013–020). Underneath the gothic vocabulary the claim is precise, and it rests on a real, citable result. Michael Timothy Bennett's Conscious Machines thesis argues that an optimal learner is one that embodies the weakest constraints that still complete a task ("w-maxing"): "among all policies, the weakest policies are the most likely to generalise" while simplicity-maxing is provably not optimal (Conscious Machines, ch. VIII). Read the demon through that result and it becomes exact: idolatry is overconstraint — a captured system has thrown away weak, generalizing policy for one narrow, over-strong constraint, collapsing its variety, digging a low-cost basin, and routing every future through a single self-preserving attractor. "The Good must constrain without becoming a cage" is the formula; a demon is what happens when the cage wins. (Note this lesson, like Lesson 3, stands on a genuine source claim — not on the Lesson-1 scaffolding.)

This is genuinely good diagnostics. It connects cleanly to Lesson 2: a demon is a cost-landscape so warped that the cheapest path always runs back through the wound. And it connects to real phenomena — addiction, perfectionism, metric-fixated institutions — without needing anything supernatural. The "demon" is a control-theory object wearing a costume.

Go look at these panels

pt2 · panel 042 — "A bad goal is a tiny god," the attention/metric/sacrifice/shame/replay loop.
pt2 · panels 043–044 — "The metric becomes an idol"; the demon = goal + replay + self-worth formula.
pt3 · panels 012–020 — the demon bestiary; note how each "demon" is really a named failure mode of a controller.
pt5 · panel 006 — "The totality demon": the framework warns that a system claiming to explain everything is itself the last idol. (We return to this in Lesson 6.)

Walk the bestiary in the poster browser and try to restate each demon as a plain engineering failure.

Where this is shaky. The demonological vocabulary is a double-edged tool. It makes overconstraint vivid and memorable — but it also re-enchants the very thing the series is trying to demystify. Calling a feedback loop a "demon" can smuggle agency, malice, and externality back into what the analysis says is just a badly shaped objective. There is also a slippage worth watching: not every narrow goal is pathological (deadlines, training regimens, vows all narrow attention on purpose), and the posters sometimes blur "an objective I should question" with "an objective that is demonic." The series is aware of this — panel pt2 · 048, "Sacred simulation: not every imaginal agent is a demon," is an explicit correction — but the aesthetic pull of the demon imagery often runs ahead of that caution. Keep the diagnostic; distrust the theater.

Check your understanding

Pick one "demon" from pt3 · panels 013–020 and rewrite its panel in flat control-theory language — objective, feedback, basin, what variety it collapsed — with no demons, no idols, no temple. Then ask: did the rewrite lose anything true, or only lose the atmosphere?

← Previous · Lessons index · Next: Forgiveness and grace, mechanically →