There are nine geometries that matter. AI is built on only one of them, and it is the wrong one.

Pause there for a moment before looking at the table below. At first glance, it may seem like a parade of abstract symbols, the kind of thing people dismiss in an instant. Do not dismiss it.

Read it as you would read a report of your own mind, because that is what it is: a record of structures you already rely on, every waking hour, without ever naming them: when you turn your head toward a sound, when you reach around an obstacle to grab a cup, when you know that your left shoe will not fit your right foot, when you retrace the route that brought you here and remember the turns in the order they happened, you are relying on the very invariances, transformations, and path structures these geometries describe.

You do not need the math to do any of this. You need the math only to see it clearly, to understand how a truly intelligent engine would have to handle it, and thus to see that the machines everyone calls intelligent still cannot.

Behind the paywall: the Nine Geometries table that names exactly what AI is missing, the chirality infographic that explains why flat AI confuses cures with poisons, the financial case for the two companies already shipping what replaces the transformer stack, the architectural diagnosis that shows why scaling cannot close the gap, and new animations showing the geometry problem in motion… and more…

Read more

The LLM Era Is Over

The biggest names in AI have an aging problem, and they are trying to fix it by throwing more raw computation at it. Blatant mistake!

OpenAI, Google, Anthropic, and the rest have spent the last two years scaling inference-time compute: chain-of-thought prompting, search trees, verification loops, more tokens at test time. As I argued in previous articles, this approach has eliminated most of the surface hallucinations, the kind that embarrass you in a demo, while producing deeper structural errors that are far harder to detect and far more dangerous to trust.

And here is what most people are missing: the new LLMs sound smarter. They are not. They have simply learned to hallucinate with better grammar. And the data proves it: with every new release, the deeper hallucination rates are going up, not down.

One of OpenAI’s biggest recent models hit an almost mind-blowing 50% hallucination rate on their own SimpleQA benchmark. One in two answers, fabricated. The coherency is a mask. What is underneath is getting worse.

So there you go: the companies that dominated the first era of AI are becoming its dinosaurs, and they are too busy scaling to notice what the new-kids-on-the-block competitors have already understood:

intelligence is not a frozen function. It is a continuously updated probability distribution moving through a structured space. And in a real learning system, space is not the stage on which computation performs. Space is part of the script.

In quieter times, this would be a technical debt problem. An expensive one, but manageable. These are not quiet times. The window is closing because the architecture itself is hitting a wall that compute cannot push through, and for the first time, there are credible alternatives waiting on the other side.

A new class of AI architecture is now positioned for a serious overtake of the entire industry. Not by building bigger transformers. Not by training longer. By changing the space in which computation happens. The approach has no single brand name yet, but the technical foundation is clear: Fisher-Bayesian AI (see chart above). It replaces the flat Euclidean geometry that neural networks have assumed since the 1980s with the curved, information-theoretic geometry that probability distributions actually live in. It does not improve the existing paradigm. It obsoletes the mathematical surface on which that paradigm was built.

Let’s cut now the tech jargon and explore with a real-world example in an animation with two drones facing the same landscape with the same obstacles. However, one runs on a frozen AI trained like an LLM: it planned its route once before deployment and never updates, because for the LLM AI the world stops changing the moment you finish training. The other runs on a Bayesian brain: every sensor reading reshapes its map of the real world (you know, the one with mountains, valleys, and the kind of surprises that don’t care about your training data) in real time. Now watch what happens when a new obstacle appears mid-flight.

What follows is the proof: the names, the math, the battlefield map, and the animated visual arguments you will not find anywhere else. If you have read this far, you already know something is broken. The rest of this article shows you what replaces it.

Read more

Before the paywall drops, let me tell you what waits on the other side.

I will show you why the industry’s triumphal story is only half true, and why that half-truth is precisely what makes the moment so dangerous. I will separate surface hallucinations from deep hallucinations and show you why that distinction changes everything: how we measure progress, how we should think about AI risk, and why so many intelligent people are misreading what the current generation of models can and cannot do.

Then I will take you into the mathematics. Not vague opinion. Not futurist fog. The actual geometry of the problem. You will see why benchmark gains can keep rising while real progress bends, why the hidden wall is real, why the cost of pushing deeper grows much faster than the industry admits, and why paradoxically, even in its structurally broken state, AI is already strong enough to threaten elite work.

And then comes the part that matters most. I will show you where the machine still breaks, why those breaks are not random, and how those fault lines already sketch a professional map for mathematicians, developers, engineers, lawyers, researchers, and anyone whose work depends on structure rather than surface fluency.

In other words: on the other side of this paywall is not another AI article. It is an autopsy, a warning, and, if you are paying attention, a map.

Read more

A month ago I published Anthropic Is Killing Bitcoin. The rebuttals came fast. Exaggerated, they said. Alarmist. Irresponsible…you name it.

Then the evidence started arriving and not precise in in trickles. but in avalanches as you can check in the table below.

The evidence is not ambiguous. When the financial world, the most demanding, most unforgiving sector on earth, starts being reshaped by a technology, the experimental phase is over. Not winding down. Over. The consequences for your life and your job are already in motion whether you have accepted them or not.

And we do not mean just any corner of finance, but the business core that built its entire identity on eliminating middlemen, on offering an alternative to every convoluted, unfriendly, third-party financial provider in existence: the cryptocurrency universe. The one that promised to replace the old system, and is now being replaced itself. Its clay feet were always the wallets.

While I was writing about crypto dying, the replacement was already being built. Coinbase shipped autonomous AI wallets in February. Not a prototype. A protocol: x402, with 50 million transactions and Stripe already integrated. One month later, Sam Altman’s World project plugged human identity verification into the same pipes. Then Visa launched its Trusted Agent Protocol. Then Mastercard completed Europe’s first live AI-agent bank payment inside Santander’s infrastructure. In weeks, not years.

So the skeptics were wrong about the diagnosis. But I was wrong too. I was not pessimistic enough. 🤔

These are not experiments, these are production systems, and the largest payment companies on earth are racing to give AI agents wallets loaded with real money and full authority to transact on their own. The age of the human-operated wallet is not ending someday; it is ending now

You don’t believe me? Look at the numbers yourself: they tell exactly the same story. Crypto is losing its builders, its capital, and its price floor in the same twelve months that AI wallet infrastructure is exploding. One world is fading while the other is being born inside it.

What you just read is the autopsy. What follows is the surgery.

Never-seen-before animations show what no AI wallet whitepaper ever has: why flat space kills your money, and how a new geometry survives the same crash, frame by frame, in cinematic detail.

Subscribe to read the rest.

Read more

Google and OpenAI just made their last call. You will not hear that in a press conference, obviously. But read the architecture notes — here and here — behind their latest LLMs and the message is right there.

And guess what. They are still betting on the same trick they have been betting on for two years. You have probably already heard the names: inference-time compute scaling — chain-of-thought, tree search, reward-model verification.

If that is not enough, let it explore more branches. If that still is not enough, make it generate more intermediate tokens, score more candidate answers, verify more paths, and keep burning compute until the right answer finally appears.

That is where the quarter trillion goes.

15 models in 16 months is proof enough that something is rotten, and the smell is not coming from Denmark. 🙄

Look at the chart below: green for OpenAI, gold for Google. Each launch comes wrapped in marketing rhetoric. Each model is sold as a leap, yet under the hood it does the same thing as the last: thinking longer, searching harder, hallucinating more, and burning more money at inference time

Repeating the Same AI Seasons — and the Same AI Winters

The cracks are already visible. But to see them clearly, you need the wider frame… because without it, the present moment looks far more original than it actually is.

What the industry now calls inference-time compute scaling — chain-of-thought, tree search, reward verification — is a bundle of ideas from the 1940s and 1950s, repackaged under cleaner labels and sold back to you as a revolution.

The trick has worked before. More than once.

The two animations below compress roughly eighty years of AI history into a single recurring cycle. Once you see the rhythm, you will not be able to unsee it. AI does not advance in a straight line. It moves by seasons:

Breakthrough.
Explosion.
Limits exposed.
AI Winter.
Then the cycle begins again.

Watch the pattern unfold: from McCulloch and Pitts’ binary neuron in 1943, through the perceptron boom and the first AI winter, to the return of backpropagation, the second AI winter, and finally the deep-learning explosion that seemed — for a while — to have broken the loop for good.

But, infortunately It hadn’t

Now zoom into the moment that fooled everyone.

In 1969, Minsky and Papert proved that a single-layer perceptron could not solve even a basic XOR problem — one straight line cannot separate what needs separating. That critique helped trigger the first AI winter.

Then in 1986, Rumelhart, Hinton, and Williams showed the escape route: add a hidden layer, make it trainable through backpropagation, and the network can combine several straight cuts into a non-linear decision boundary.

The XOR problem was solved. The field celebrated. But here is the part nobody talked about at the celebration: backpropagation did not bend the space. It just learned to make more cuts inside the same flat geometry. The geometry never changed. Only the bill did.

Now watch the animation… you can see it clearly: backpropagation escapes the perceptron’s old XOR trap not by bending the geometry, but by learning enough straight cuts

The XOR problem was solved. But the cycle did not end there.

This is where the free ride ends. Everything above was only the runway. Below is where the real argument begins — with animations, diagrams, and peer-reviewed mathematics showing why Google and OpenAI’s reasoning push still runs on old search-era machinery, why models often hallucinate more as inference grows longer, and why Anthropic’s own technical trail keeps circling the same conclusion: the geometry underneath today’s AI is fundamentally wrong.

Subscribe now. No marketing BS. Just the math, the architecture, and the failure analysis — before everyone else catches on.

Read more

The New AI Land Nobody Saw Coming

While the rest of the AI industry spent the last three years in an arms race over who could build the best chatbot Anthropic was doing something very different.

They were building infrastructure.

Not a better LLM. Not a flashier chat interface. They flipped the revenue model inside out — 85% API, 15% chatbot — and built the plumbing underneath. So quietly that most of the industry didn’t realize what it was looking at until it was already running in production.

For years, Anthropic was seen as the company that couldn’t compete with ChatGPT, too niche, too expensive, too cautious, too obsessed with safety to matter. A second-tier chatbot maker losing the consumer war.

But nobody asked the obvious question: why would a losing company spend its time building governance frameworks, open protocols, and multi-cloud settlement rings instead of chasing subscriptions?

Because Anthropic wasn’t playing the chatbot game.

It was building something underneath it: something designed to make the chatbot game itself irrelevant. Meanwhile, OpenAI and Google kept squeezing a dried lemon: wringing the last performance gains out of the LLM paradigm and chasing diminishing returns.

So let me show you how Anthropic began to win the AI race and why you should care about it.

In fact, by the end of this article, you’ll understand why those moves are already changing the structure of the AI industry — and why those who recognize it early hold the best position in the global AI shift now underway.

The golden age of the LLM paradigm is over, the ground has already moved beneath it. You’ll see for yourself, from the solid arguments laid out in this article.

I’ll start by summing up Anthropic’s recent strategy in just six moves that that caught everyone off guard — and almost everybody completely misread. Not anymore for you.

Read the middle column first, then the right:

Anthropic Has Already Won the Game

Those six moves expose something most people still miss: the LLM was one piece. Not the centerpiece. Anthropic’s main focus has never been Claude the chatbot: it’s the AI infrastructure on which the future of AI runs, well beyond those fancy chat windows.

And that AI infrastructure is, at its core, a transaction infrastructure. You’ll see why this kind of infrastructure is crucial for the success and evolution of the true strong, trustable AI that we still don’t have.

To start realizing its importance, take history as an example: every technology that outlasted its hype cycle did it the same way. The internet was real in 1999 and still crashed… not because the technology failed, but because there was no infrastructure to settle value autonomously. It recovered only after transaction layers made value flow without a human in the middle.

And now with this novel AI infrastucture we are heading exactly to that spot right now. You can check it yourself: The models work. The capabilities are genuine.

Behind the paywall: The first global AI currency. The three degeneracies hiding inside every neural network alive — including Claude. A theorem from 1876 that proves your neurons have been crippled. The exact geometry that fixes it. Anthropic's own papers proving they're already looking for it. And why their infrastructure is one layer away from something nobody else has seen coming. Subscribe now and dont skip what brings you a unique advantage in the AI race

Read more

The Dawn of the Total Machine Economy

We are at the beginning of the total-machine economy — and you’re holding the wrong money.

You’re about to see a shift most people in crypto and AI are still missing.

While you’ve been watching Bitcoin prices and AI hype cycles, one company has been assembling financial infrastructure that operates beyond the architectural limits of Bitcoin, Ethereum, and everything else in the current crypto stack.

And yet most people around us don’t notice what’s happening. This revolution isn’t as loud as the ones before: it’s unfolding more quietly than that. Racks of machines coming online, one data center at a time. We’re still in the early stage but the growth is exponential — which is exactly why it’s so easy to miss.

What that looks like in practice takes about three seconds to understand. Just watch the animation below:

But this animation shows only the surface. Here’s the part that should worry every Bitcoin holder:

You’ve already been using this currency without realizing it.

This isn’t a currency that’s coming. It’s already running: processing billions in value daily, secured by physical infrastructure, priced and settled in real time.

At some point, someone will formalize it into an explicit AI coin — but by then, the early-adopter window will already be closed.

Let me show you how it reframes everything.

THE EXODUS

Let’s start by following the money as GPS to show us where the future will be: between cryptocurrency country roads and the new AI multi-infrastructure heavy highways.

TL;DR: Money is escaping the cryptocoin hype faster than it entered years ago. The promises of an economy built on cryptocoins never materialize; and in the meantime, new tech has already begun turning the entire crypto landscape into tech archaeology.

Don’t believe me? Then let me show you two charts that tell the whole story.

The Crossover

Look at these two charts below together. They’re telling the same story from opposite ends.

At the bottom: AI is swallowing everything else. In just two years, it moved from a quarter to half of all global venture funding. Half of every investment dollar on earth now flows into AI, while fintech, crypto, biotech, and SaaS fight over the scraps. And the trajectory shows no sign of slowing down.

At the top: the physical evacuation. Bitcoin miners are watching their revenue share collapse in real time. That crossover point on the chart isn’t decorative — it marks the exact moment when AI compute contracts became more valuable than Bitcoin mining on the very same hardware. Same racks. Same cooling systems. Same electricity bills… but taken over by AI guys paying more.

One chart shows where the capital is going. The other shows where it’s coming from. These aren’t two trends: it’s one migration, captured from both ends.

The capital didn’t disappear. It changed industries.

The Contracts and The Hedge

The first two charts showed you the flow. Now let me show you the receipts.

These aren’t projections. These are signed contracts — the names and amounts are in the chart below. What should stop you cold is who signed them: companies that used to mine Bitcoin, now locking themselves into AI infrastructure deals that stretch a decade or more.

These aren’t hedges. This is the exodus — the moment the people who built crypto started buying their tickets out of it.

And the bottom Figure 3 reveals something the contracts alone can’t tell you: where this is heading. The funding ratio between AI and crypto isn’t just wide — it’s widening. Every quarter, the gap grows. Project that curve forward two years and crypto venture funding doesn’t shrink gradually. It becomes an investment anecdote.

Now the betrayal is becoming explicit. Pantera Capital — the first US institutional fund built exclusively on blockchain: is pouring hundreds of millions into AI. Bloomberg reported that crypto VC funds are actively shifting away from tokens.

The people who built their careers on crypto are quietly moving their chips to the other table. When insiders build lifeboats with their own money, that’s not a cycle, it’s the end of the explosive cryptocoin Cambrian era.

Bye bye BTC

Still not convinced? Fair enough. Now let me show you why the money is leaving.

Imagine you own a data center. You’ve been renting it to Bitcoin miners for years. Then some AI investor guy walks in and offers you a decade-long lease at margins that make your crypto tenants look like they’ve been paying in loose change. A hyperscaler’s signature on a contract that prints money for fifteen years.

Which tenant do you keep?

That’s not a thought experiment. That’s what’s happening right now across every major mining operation on earth. And here’s what nobody’s talking about: they’ll still mint a new kind of coin — an AI-native one — pretty soon, while profiting from the infrastructure that makes all of AI run.

How? That’s what comes next.

You have read 20% of this article.

Behind the paywall: The autopsy of crypto’s last argument — and why AI already won. The four pillars that make Anthropic’s infrastructure a functioning currency. A one-to-one mathematical mapping you can’t unsee. The flywheel animation showing why this machine is unstoppable. The new AI Economy Stack. The protocol layer Bitcoin spent fifteen years failing to build. The four engineering gaps still left to close… and more…

The charts and animations in Part 2 are the ones you’ll want to screenshot.

This is where the math and tech gets interesting.

Read more

You Don’t Know How to Code with AI And the method that works is the exact opposite of what you’ve been taught

Vibe coding had its moment.

Around 2023, it felt almost magical. You described a problem in natural language, the AI answered with confident-looking code, and for a brief window it seemed like software development had finally been abstracted away. Business idea in, working system out — the dream of tech managers wanting to get rid of what they regarded as annoying developers. Easy peasy!

Then you got screwed. Wasted.

Not because you’ve been lazy. Not because your prompts aren’t clever enough. But because current AI systems cannot take a real business idea (messy requirements, edge cases, implicit constraints) and produce code that actually works.

What replaced the dream of your CEO is worse: a workflow that defeats itself.

Those upper layers in your tech company didn’t recognize it, but you did immediately, because you work with it daily.

You prompt. The AI generates code that looks fine — deep waters look still — sometimes even elegant. But the CEO isn’t beside you when you run it.

However, a closer look shows you the bad side of AI-generated code: the errors aren’t localized, they aren’t clustered . They’re scattered across the codebase. Innocent-looking lines sit next to logic traps, clean abstractions hide dangerous assumptions, and mixed in with harmless bugs you find code that isn’t just wrong but actively dangerous: security holes, data corruption paths, failure modes you would never have written yourself.

Congrats. You’ve run into what’s called semantic drift — and if you’re not a developer, here’s what the visual nightmare looks like:

You ask your AI copilot to fix a Module not found error and cool, the code runs now. Except — wait — now it’s Method undefined because the AI just made up an API that doesn’t exist. You ask the copilot to fix that, and suddenly SQL injection pops up because the AI built the query with string concatenation like it’s the early 2000s. You ask it to fix that too, and now you’ve got Token never expires, wow a security hole, wide open!

You close the copilot. You fix it yourself.

Your copilot isn’t stupid. It’s trapped. Current LLMs operate in flat embedding spaces where every direction costs the same( yeah, including the wrong ones). The path from working function to SQL injection nightmare costs exactly as much as the path to correct code: nothing. The model has no geometric reason to prefer one over the other. So it drifts. And you end up chasing after it.

That’s not an AI problem.
That’s a map problem.
The good news? maps can be redrawn.

And that’s exactly what we’re going to do, in a much easier way than the AI industry keeps patching without fixing the problem.

No, we don’t need to rebuild the AI from scratch, or fine-tune it, and much less throw more parameters at it. We’re going to change the terrain it walks on: give it a geometry where drifting toward bugs costs more than drifting toward correct code.

Same model and same infrastructure — but here’s the catch you need to pay attention to: we’ll use a different space where the AI operates and lives.

The Geometry That Actually Works

The solution? A curved toroidal space — the simplest geometry that gives us what we need: periodic structure and a well-defined notion of distance. On top of it, we place a Lyapunov energy landscape. Together, they let an AI copilot see a proper map: one where the path toward correct code is downhill in energy, and the path toward bugs is uphill.

And here's the surprise: Anthropic is taking notes — to the point of publishing research on curved manifolds inside Claude itself.

Note: In January 2026, Anthropic published When Models Manipulate Manifolds, showing that Claude represents information on curved manifolds with geometric transformations. They found the geometry. They’re observing it. What they haven’t done yet is engineer it — deliberately curving the space to make error paths expensive. That’s the lens we’re building here.

Yep, these are the same three error pairs we saw in the flat-space example earlier. The only change is the geometry: once we move to a curved space, the cost structure becomes visible… and the difference is impossible to miss.

Below: the equations, the architecture, and the cost breakdown. For paid subscribers.

Read more

TL;DR for the impatient:

1. Your intuition evolved for objects. However, AI is not an object. It’s a field.

2. Category theory looks rigorous but misses geometry. Algebra without metric is blind. cannot see failure modes.

3. Hallucinations are not bugs. They’re broken invariants in flat space.

4. Stop measuring accuracy. Errors have shape start measuring error structure instead.

5. Scale does not fix structure but amplifies whatever geometry you have. Flat geometry scales into nonsense.

6. Intelligence is geometric.If you don’t have structure, you don’t get intelligence. End of story.

Even the Best AI Engineers Keep Making the Same Mistake

Human brains evolved to track rocks, tools, animals, faces — things with edges. Things that push other things. This served us well for hundreds of thousands of years.

And most of us are still using that intuition in the wrong way for science and technology, including how we think about AI. Even highly skilled engineers are doing the same thing: trying to force square pegs into round holes, then failing to understand the errors in the AI systems they build.

This mismatch between intuition and science is not new.

Take, for example, the history of physics: heat, fluids, and turbulence looked like magic for centuries because we tried to understand them as things — substances that moved, pushed, and accumulated. But then we discovered that the interesting stuff doesn’t work that way and abandoned that picture of objects and things (see Figure 1 above).
Heat became a field.
Pressure stopped being a force you could point at and became a constraint in an equation.
Motion was no longer tracked object by object, but as transformations of a state distributed across space and time.

They are values distributed across space, constrained globally, governed locally. The moment we needed partial differential equations, our stone-age intuitions became a liability.

Paradoxically, even among most educated people is rare to find who have made this jump, you know, because of how we get through the educational system: by memorizing rote formulas and pattern-matching exam questions. That didn’t help our intuition fit reality. On the contrary, intuition for many educated folks stays broken, and curiously enough, it doesn’t matter for most jobs 🤔

But the few who did their homework right, they know that once that shift happened, the mystery vanished. The equations didn’t get simpler, but the intuition finally matched the phenomenon.

AI is now forcing the same adjustment, whether we are ready for it or not

AI Is Not a Thing That Thinks

This habit of seeing objects everywhere is deeply ingrained in most minds— even engineers’, which should come as no surprise given what we have already discussed.

Object-based intuition gets people into trouble — not just in programming (OOP) 😉. A neural network is imagined as a box: data goes in, stored knowledge is consulted, reasoning happens, and answers come out (see Figure 2, diagram 1).

This picture is wrong in every way that matters.

A neural network isn’t a container at all, it’s a state space, a geometry — and meaning doesn’t live inside the tokens or the weights the way we imagine it does.

It lives in the structure of transitions between states. The knowledge you think must be stored somewhere? You won’t find it anywhere you can point to, because it’s implicit in the shape of the manifold itself (see Figure 2 below, diagram 3).

Get this wrong, and you’ll spend years debugging the wrong AI. No surprise here, this pattern repeats in every new release of ChatGPT, Claude, Gemini, Grok. You name it: new patches, the same problems remain.

Mathematicians Are Also Wrong — Even with Category Theory

I know several mathematicians in AI who swear by category theory as the answer to nearly every AI failure… and they’re not fools. They may not be funny, social, or charismatic, but they are among the smartest people I know. Of course, they’ve seen the mess that passes for “theory” in most machine-learning papers — a mess we’ve already analyzed in previous posts:

• Ad hoc architectures justified by vibes.

• Vague claims about generalization that nobody can pin down.

• Benchmarks that prove nothing except that someone got lucky on a test set.

So, naturally, my mathematician friends tend to view category theory as a cleaner alternative (see Figure 2 above, diagram 2). Many of you know exactly what I mean, because a category in math is, after all, a precise combination of :

1. Objects as second-class citizens, understood primarily through how they participate in relationships.

2. Morphisms with composition and identities as first-class citizens, the structure-preserving transformations that do the real work.

3. Functors mapping between categories of representation spaces.

4. Commutative diagrams that actually mean something: statements you can write down and prove.

What follows is the technical depth. Available now for paid subscribers, free for everyone in 72 hours.

Yep, most of us agree: Category Theory at least forces you to ask the real questions: what structure is actually being preserved here? What transformations are legitimate? When can we say two architectures are genuinely equivalent and not just superficially similar?

But here’s the blunder — better than most is not the same as sufficient.

Category theory is algebra, not geometry, and that distinction matters more than most people realize. It tells you when two architectures compute the same function class, how transformations compose, which diagrams commute. So, pure computation? Yeah, that’s the pick any programmer could say. And therefore, due to its purely computational algebraic nature, category theory — the algebra at the core of our current AI — cannot tell you what any path through the network actually costs: there’s no metric baked in, no notion of distance, no way to distinguish a cheap transition from an expensive one or a safe region from a dangerous one.

So in the end, you still end up with AI hallucinations, even when using category theory.

Take this as a paradigmatic example: two networks can be categorically identical: same objects, same arrows, every diagram commuting exactly as it should, and yet one hallucinates constantly while the other produces reliable outputs (see Figure 4).

This is worth emphasizing: computational algebra (category theory) simply can’t see the difference. As long as composition, identity, and associativity hold, two neural networks can behave very differently and still look equivalent on paper. The morphisms compose, the functors preserve structure. And yet one works and the other doesn’t — and category theory has no way to explain why.

What Category Theory Gets Right (And Why It’s Still Not Enough)

To be fair, category theory does get one thing genuinely right — memorize this apparently innocent common word, because it is of crucial importance: naturality.

A construction is natural if it doesn’t depend on arbitrary choices — and this matters more than it might sound. Permute the hidden units? Same object. Reparameterize the weights? Same object. Change basis? Same object. If your “intelligence” disappears the moment you change coordinates, it was never intelligence in the first place. It was a coincidence in one particular basis, a mirage that evaporates when you look at it from a different angle.

But here’s the problem: naturality without geometry is like having a map with no scale. You know which cities are connected. You have no idea how far apart they are. You can prove that two routes are equivalent, but you can’t tell which one goes through a mountain range and which one follows the coast. The algebra guarantees they end up at the same destination — the geometry determines whether you arrive exhausted or refreshed, whether the journey takes an hour or a week, whether you survive at all (see Figure 5, below).

Patching OpenAI, Anthropic, Google, and Grok Won’t Fix Hallucinations — It Just Buries Them Deeper

By now, it should be clear that patching AI with the same wrong math doesn’t fix anything: it just makes the failures harder to see.

And despite all this evidence, the industry keeps chasing the same AI gold-rush dream: better data, tighter RLHF(reinforcement learning from human feedback), more guardrails, constitutional constraints — you know, whatever happens to be fashionable this quarter. Some of it helps a little. Mostly, it’s symptom-patching dressed up as progress, while the real problem stays exactly where it is.

Yep, those one involved unfortunately in that AI industry dynamic agree: the patches are making the disease harder to diagnose: for instance, RLHF makes models better at convincing humans they are correct — even when they’re wrong. The approval rate goes up, but the correctness rate doesn’t follow. Many failures simply weren’t expected, they passed internal safety checks without being noticed.

In other words, every new patch makes the model sound more fluent, more confident, more persuasive. The errors don’t go away… they just get buried deeper.

Here’s what hallucinations actually are, and it’s not what most people think.

They’re not lies. The model isn’t being lazy, rebellious, or confused about what you wanted. Hallucinations happen when the geometry of the embedding space doesn’t enforce meaning, as we saw in the previous section. Once you see it that way, it becomes clear why the usual fixes don’t work.

Think about it spatially for a moment. In whatever high-dimensional space the model uses to represent concepts, “dog” and “wolf” should live close to each other — they’re related, they share features, contexts where one appears often admit the other. Meanwhile, “dog” and “justice” should be far apart, because semantically they have almost nothing to do with each other. So far so good.

But this is where things really fall apart. In a flat Euclidean space, nothing stops a path from wandering through arbitrary points on its way from one concept to another. You can go from “dog” to “justice” in a straight line, and every step costs exactly the same. The metric doesn’t care that you’re moving through semantic nonsense. Every direction is allowed. Every transition is cheap. The space itself has no opinion about meaning.

And here’s the nasty part we run into over and over again: you try to correct your AI chat’s output, and it just ignores you. When you tell the model that was wrong, you’re penalizing a specific output, not reshaping the geometry that produced it.

The underlying space is still flat. The model can’t reason about the path it took — only the output. As a result, it either repeats the mistake or fails in a slightly different way next time. The root cause is unchanged.

At this point, there’s no need to ask what a hallucination is, geometrically. You already have the right intuition: it’s the model taking a path it should never have been able to take — if the geometry were right.

Yep, in a flat space, nothing prevents that. The model can move between unrelated concepts at no extra cost, because the space itself doesn’t say this path is wrong. Every direction looks equally valid.

With the right geometry, those shortcuts wouldn’t exist. The shape of the space would make nonsense paths expensive or impossible.

That’s also why the usual fixes don’t work. More data just fills the same flat space. More scale just repeats the same structure at higher resolution. These errors aren’t glitches: they’re consequences of how the space is built.

And yet — this is exactly where the industry is doubling down.

Scale Makes It Worse

Right now, the industry’s big bet( hundreds of billions of dollars’ worth) is that scale fixes everything. Ok we are already suffering more bloating LLMs: More parameters. More data. More compute, with the hope that continued scaling will make the problems go away.

That’s the wrong kind of linear thinking applied to a nonlinear system.

If the geometry is flat, scaling doesn’t fix anything; it just makes the problem bigger. You give the model more parameters to express the same broken structure at higher resolution. The hallucinations don’t vanish; they just sound smoother and more convincingly, but still wrong. The spaghetti gets longer. It doesn’t get straighter.

Summing Up

Physics learned this lesson the hard way: reality doesn’t care about your intuitions. It runs on constraints, invariants, and geometry. When physicists stopped thinking of heat as a substance and started treating it as a field, thermodynamics suddenly made sense. The mystery didn’t vanish because they got smarter — it vanished because their intuition finally matched the structure of the phenomenon.

AI is the same lesson, playing out again right now. The future isn’t bigger models. It’s models with the right geometry.

Not more data, but more structure.

Better intuitions, grounded in the right mathematics.

As this story has shown, intelligence isn’t something a system slowly accumulates through training. It’s something the geometry either allows or forbids.

Without structure, AI doesn’t gain intelligence. It gains cognitive trickery — something that becomes nastier and more disappointing with every new release. It’s time to play the right math game.

First, thank you.

In just a few months, A Mathematician Lurking in the Tech Underworld has grown to several hundred free subscribers—readers, commenters, and people who share the work.

That growth came with no marketing budget, no algorithm hacks, and no corporate backing.

It came from applying serious mathematics to the hardest problems in AI—and making those ideas accessible beyond a closed circle of private research dominated by a small technical elite.

Today I’m launching paid subscriptions. Here’s exactly how it works:

What stays free

• Every article’s core argument.

• The diagnosis.

• The what and the why.

If your goal is to understand what is broken in AI’s mathematical foundations—and how it can be fixed with the right mathematical and engineering tools—the free posts give you that, fully and openly.

What’s behind the paywall

The how.

How to apply the right mathematics and engineering to solve real AI problems—not just to understand them, but to fix them in production:

• Implementation insights for real production systems

• Code and algorithms

• Technical notes as they are developed—unfinished, exploratory, honest

• Priority responses to technical questions

This is the depth engineers need to actually apply these ideas, not just agree with them.

The price

$5/month or $50/year.

Many engineers expense technical publications through their company’s learning budget.

If that’s you, this is mathematics that may save your team from committing to the wrong AI architecture.

Founding members — $150/year

• Personalized, in-depth responses to your technical questions

• Named acknowledgment as a founding supporter of this research

Why this structure

I want the ideas to spread. Free readers get the full argument.

But turning ideas into working systems takes deeper work: original research, real implementations, code, and sustained technical exploration.

That work is what paid subscriptions make possible, and what paid subscribers gain access to.

Subscriptions keep this research independent.

No sponsors.
No institutional filters.
No pressure to soften the critique.

— Jose Crespo, PhD A Mathematician Lurking in The Tech Underworld

TL;DR (read this in 30 seconds)

• Today’s LLMs store meaning in flat space (vectors). Flat space lets the model slide between nearby ideas too easily. That’s one reason it drifts and hallucinates.

• The main solution to fragmentation is building a so-called unified map of meaning. Based on the currently developing AI toroid geometry, the most obvious and simplest solution is to use, inside the toroid, solenoid-style geometry that naturally combines:
  (1) Memory phase (where we are in a thought loop),
  (2) Taxonomy (what branch of meaning we’re in), and
  (3) Path-dependence (the path you took changes what you know next).

• The math object is real; the AI claim is a testable hypothesis.

• The development of this simplified technology will accelerate the evolution of today’s frustrating, limiting AI to the next level — the basic requirement for solid AGI: representation where semantic drift becomes expensive by design, not “fixed later by patches.”

Three concepts. You probably know them by different names

As Figure 1 shows, current AI treats these three concepts as separate engineering problems:
- different modules
- different geometries
- different code

Memory

Not RAM. We mean: what the model is currently carrying forward from the conversation. The cyan card in Figure 1— that circular arrow represents state that persists between turns. Current AI stores this as flat vectors. No phase. No structure. Just activation strength that fades.

Taxonomy

A fancy word for category tree.
Animal → cat → Persian cat.
Company → Apple → CEO → current CEO.
The purple card, that branching structure. Current AI handles this with hyperbolic geometry (Poincaré embeddings), completely separate from the memory system. Bolted on. Not unified.

Holonomy / path-dependence

Another fancy word for something you, if you’re a software developer, already know from debugging:

How you got here changes what happens next.

Same endpoint. Different route. Different state.

The orange card in Figure 1 shows it: two paths arrive at the same point, but they carry different histories. In LLM land: two chats can end with the same final question, but if one route wandered through Musk / SpaceX earlier, the model’s next answer can drift. Current AI? It ignores this entirely. No path memory. Just final position.

Memory. Taxonomy. Path-dependence. Current AI treats them as three separate engineering problems. That’s the mistake.

Look at Figure 1 one more time. Three cards. Three colors. Three separate systems. That fragmentation is the disease. The solenoid, added to the familiar toroidal geometry, is the cure.

The Geometry That Broke AI

Most LLMs store meaning as points in a huge vector space, like R^4096.

That sounds sophisticated. It is.
But it’s also… flat.

And flatness has a hidden cost:

• In flat space, “nearby” often means “shares features,” not “is the same thing.”

• In flat space, you can slide from one cluster to another without crossing a clear boundary.

Flat space is too permissive.

That’s the thesis.

Not LLMs are dumb.
Not training data is bad.
The map is wrong.

Three failures of Flatland (use your programmer intuition)

Look at Figure 2. Three cards, three bugs you already know — even if you’ve never heard them described this way.

1) Collision (name clashes)

See the first card: two circles labeled river and money overlapping in the middle, with a confused “?” where they meet.

Bank = riverbank vs bank = finance.

In flat embeddings, those meanings drift close because the surrounding text patterns overlap. Rivers have banks. Money goes in banks. The word appears in similar sentence structures. So the vectors converge.

The model keeps guessing: which bank did you mean?

Translation: it’s doing runtime inference on every occurrence because the representation didn’t hard-separate the branches. The red overlap zone in Figure 2— that’s where your model lives, perpetually confused.

2) Spaghettification (state contamination)

The middle card shows it: a path that starts cyan (cooking), gets contaminated by nuclear, and ends red (physics?!).

You’re talking about cooking. Someone says nuclear family. Now the word nuclear is activated in the representation, not because you wanted it, but because it appeared.

Later, the model is one weak anchor away from drifting into nuclear physics🤪

Translation: the internal trajectory is like a messy global variable.

Once something enters the state, it can leak anywhere. The tangled line in Figure 2 isn’t artistic license: it’s what attention patterns actually look like when context bleeds across topics.

3) Hallucination zones (illegal intermediate states)

The third card makes it visual: two green checkmarks (valid ideas) connected by a path, but the middle is filled with red X marks (word salad).

Between two coherent ideas in flat space, there are countless in-between points that decode to nonsense.

Flat space says: Totally fine… it’s just linear interpolation.

Language says: Word salad.

That is hallucination geometry: legal movement through illegal meaning. The green endpoints in Figure 2 are real thoughts.
The red zone between them? That’s where hallucinations live.

And flat space has no fence keeping the model out.

The killer line at the bottom of Figure 2 says it all:

Flat space is too permissive. Not LLMs are dumb. Not training data is bad. The map is wrong.

These aren’t three separate bugs. They’re three symptoms of the same geometric disease: The typical flat, projected, multidimensional ℝ⁴⁰⁹⁶ doesn’t encode the constraints that language enforces. The space permits what meaning forbids.

The Extended Torus as an Alternative to AI Flat Space: Two Dials, One Surface

We previously proposed, together with other researchers, representing conversation memory on a torus: basically a donut shape.

You don’t need topology to get the point — see Figure 3 above. Just imagine two circular dials:

• Dial A: reasoning phase (where we are in a thought loop)

• Dial B: topic phase (which thread we’re currently on)

This matters because a torus gives you something flat space doesn’t: phase + repetition.

In flat space (which is what every current LLM uses) visiting the same region three times just makes the activation stronger. A bigger number. That’s it.

The geometry doesn’t know you came back. It doesn’t count loops.

This is why LLMs lose track. You can repeat a constraint five times in a prompt, and the model treats it as seems important, not stated five times, violated zero times. There’s no counter. Just weight.

On a torus, the geometry records when you loop back. That’s called a winding number: an integer that counts completed cycles.
Visit once? n=1.
Three times? n=3.

And unlike activation weights, this number can’t drift.

Why? Because it’s an integer, not a floating-point score. You can’t half-complete a loop.
You can’t slide from 3 to 2.7 to 2.
The winding number is what mathematicians call a topological invariant — a quantity unchanged by continuous deformation: Wiggle the path, stretch it, compress it, as long as you don’t cut it, the count stays fixed.

Integers are immune to drift. Floats aren’t.

Think about the difference:

The user has circled back to budget constraints 4 times → a fact

Budget seems important → a vibe

Current LLMs give you vibes. The torus gives you counts.

Repetition becomes structure, not just emphasis. But the torus is incomplete: it tracks where you are in the loop, not which branch you’re on. That missing piece is taxonomy.

The missing piece: how do we encode “branches”?

Phase answers: where am I in the loop?
But taxonomy asks: which branch am I even in?

Persian cat isn’t just cat with more features.
It’s cat → subtype.

The core issue: A good system should make it costly to jump from one identity branch to another unless the text truly demands it.

Flat space doesn’t.

So we need a geometry where branches are native.

The Solenoid: a donut with infinite nested depth

This is a candidate shape for unifying a fragmented AI.

Today’s language models don’t live inside one clean geometry.

They juggle half-separate spaces for memory, categories, and reasoning state, then stitch them together with heuristics inside a similarity space that behaves mostly flat (dot products, cosine, smooth interpolation).

A solenoid offers a different move: one compact geometry that carries three things at once, simply because of how it’s built (nested windings), not because we glued three modules together.

• Phase: a built-in clock hand — where you are in the loop. (memory)

• Branch: a built-in address — which track you’re on, and how deep. (classification/taxonomy)

• Path: a built-in trip memory — how you got there still matters. (holonomy)

That coupling is the point: in a solenoid, cycle position and branch identity are not separate add-ons. They come as a package.

And that changes hallucinations.

It won’t abolish them in some absolute, philosophical sense. But it can kill the most common ones, the drift driven by collisions (different meanings sharing the same neighborhood) and spaghettification (too many near-parallel routes you can hop between for free).

In flat similarity space, branch switching is cheap: nearby is nearby, even when it’s the wrong nearby.

In a solenoid-like metric, branch identity and depth become part of distance. The deeper you commit, the more expensive it is to leave — not as a bolted-on penalty, but as a property of the geometry.

Watch the construction, it’s concrete, not a metaphor. A circle thickens into a torus; a winding wraps it; then a smaller wound copy nests inside; repeat… and the infinite limit is the solenoid. Every step is a concrete geometric operation. And the limit is precisely the kind of space where cycle position, branch address, and path-dependent state live together.

Now we’re on solid ground.
Not because this is mystical… because it’s structural.

The solenoid doesn’t give you a new trick.
It gives you a single representation where the three missing ingredients stop living in separate boxes:

phase (where you are),

branch (which meaning-track you’re on),

path-dependence (how you got there).

One object. One metric. One state.

The secret math hack behind the solenoid: the p-adic address.

Inside this solenoid story lives Zp​, the p-adic integers.

A p-adic number is like an infinite base-p digit string:

a0​,a1​,a2​,a3​,…

Read it as a path down a tree:

• a0​ chooses the main branch

• a1​ chooses the sub-branch

• a2​ refines again

And the distance between two addresses is basically:

How long do they share the same prefix before splitting?

Same first 5 digits? Very close.
Different first digit? Far.

That is taxonomy as an address system. No approximate tree embedding.
The tree behavior is built in.

A p=3 tree example:

Why This Could Reduce Drift (Bezos vs Musk Example)

CAT vs BEZOS? Trivial — first-digit split, maximally far.

Bezos vs Musk is harder. Same features: billionaire, tech founder, space company, media presence. Their vectors practically overlap.

So the model can silently slide:

Bezos → space company → Musk.

No alarm. No signal. Just a quiet interpolation through feature space. The model doesn’t know it changed subjects. It just… drifted.

Now watch what happens in a taxonomic address space.

Bezos and Musk are siblings — children of the same parent branch:

TECH BILLIONAIRE
    ├── BEZOS → Blue Origin
    └── MUSK  → SpaceX

They share a long prefix. They’re close. But they’re not the same node.

You can move between them — but it costs a branch change. You have to go up to the parent, then down to the sibling. That traversal is recorded. That cost becomes a signal.

So the model is less likely to drift unless the prompt demands a subject switch.

Flat space hides the switch. Taxonomy exposes it.

The GIF below shows the final branches where Bezos and Musk compete for attention:

How Attention Uses the Solenoid

Figure 6 makes this concrete. Normal transformers ask one question: How similar is this token to that token?

You can keep that exact mechanism. You don’t rewrite the architecture. You just change what similar means.

Instead of dot product in ℝⁿ — which measures feature overlap — use a score that respects both the phase cycle and the taxonomic tree that the solenoid carries:

That’s it.

You don’t need to call it a Mercer kernel. You don’t need RKHS theory. You don’t need a PhD in functional analysis. For attention, you only need a scoring function that respects the solenoid’s structure.

And because phase and taxonomy live in the same geometry (not two spaces glued together) the score is natural.
You’re not reconciling incompatible distance metrics.

You’re measuring distance in one unified space (our familiar toroid geometry ) that already encodes both where am I in the reasoning cycle? and which branch of meaning am I on?

The design goal is simple:

Make incoherent jumps expensive. Make coherent continuation cheap.

The geometry does the rest.

Why Unification Is the Key

AGI isn’t a bigger LLM. It’s not more parameters, more data, more RLHF. Those are necessary but not sufficient.

AGI requires a unified representation: one where memory, meaning, and reasoning aren’t stitched together after the fact, but emerge from the same underlying structure.

The solenoid offers exactly this:

• One geometry (not three bolted-together modules)

• One space where phase, taxonomy, and path-dependence coexist

• One representation where the failures that plague current AI become geometrically expensive instead of geometrically free

• Reversibility in AI tracking: today’s LLMs have no undo. No checkpoint. No way to return to the moment before things went wrong. The solenoid changes that — not by magic, but by geometry:
  

Drift becomes a detectable branch-crossing event. Contamination gets a geometric signature. The winding number becomes a drift odometer.

An intelligent system that can’t recover from errors isn’t intelligent — it’s brittle.
Human cognition doesn’t collapse when we notice we’ve gone off track.
We backtrack.
We revise.
We return to solid ground.

Let’s start bold:

Math theory didn’t fail AI. AI failed to understand Math theory,specifically the mathematics with direct impact on intelligent systems: category theory, topology, differential geometry, and the structures that bind them.

That distinction matters, because right now categorical AI is becoming a fashionable way to draw arrows, sound deep, and quietly erase the very things that make intelligence hard: time, memory, history, and paths.

If you’ve ever wondered why your AI pipeline behaves differently on Tuesday than it did on Monday, despite “nothing changing” — keep reading here we explains why. And it’s not a bug. It’s a fundamental architectural blindness that runs deeper than anyone wants to admit.

Let’s walk carefully. Buckle up.

1. What Category Theory Actually Gives You (And Nothing More)

At its core, category theory gives discipline.

Not intelligence. Not meaning. Discipline.

• Objects: states, spaces, types

• Morphisms: transformations

• Composition: how transformations chain

• Identity: what doing nothing really means

This matters. Without it, systems rot silently.

Category theory forbids magical glue. It forces interfaces. It makes incoherence illegal.

But let’s be precise about what it is:

Category theory is grammar, not literature.

Grammar doesn’t make you wise… but without it, you can’t speak coherently at all.

2. The First Mistake: Treating Categories as Just Functions

Most AI papers stop at the category Set:

• Objects = sets

• Morphisms = total functions

• Composition = function composition

This already commits a silent crime.

Functions are memoryless.

A function does not know:

• how you arrived at an input

• what happened before

• what curvature you traversed

It maps input → output as if it were a meme in your favorite spreadsheet:

f(x) = y

That’s it. No context. No journey. No scars.

But AI systems are nothing but accumulated history.

But we all know the obvious in this story:

Every token depends on context.
Every agent depends on interaction order.
Every trained model carries the sediment of its training path.

So when someone says our AI system is categorical — or, in a diminished sense, functional — what they usually mean is: We are just composing a bunch of stateless maps.

Calling a pile of stateless functions categorical is not depth. It’s cosplay.

3. Kernels: The Part of Reality Your Architecture Cannot See

Here is where most AI discussions collapse — because they never talk about kernels.

Mathematically, a kernel is simple:

The kernel is the space of differences that make no observable difference.

Formally, given a map p : A → B, the kernel is everything in A that collapses to nothing in B.

The kernel is what changes without anyone noticing , where “anyone” means “your measurement tool.”

And this is deadly in AI: the kernel is where the monsters hide.

Because AI systems are full of kernel directions:

• internal states that differ

• histories that diverge

• training paths that rotate the system

…while producing the same output.

Think about it. Two language models fine-tuned along completely different trajectories:

One saw toxic content early and was cleaned up.
The other was trained clean from the start.

They score identically on your benchmark. Same output. But internally? Completely different geometry. Different failure modes waiting to emerge. Different responses to prompts you haven’t tested yet.

The difference between them lives in the kernel of your evaluation map.

If your architecture cannot represent its kernel, it cannot know what it is blind to. You will see this much more clearly explained visually in Figure 1 below.

4. Quotients: When You Decide What “Counts as the Same”

Now comes the epistemological move.

Given a kernel, we form a quotient.

Not by deleting information — but by declaring differences irrelevant.

Two states are equivalent if their difference lies in the kernel.

This is not philosophy. This is exact mathematics.

A quotient is reality after forgetting.

AI systems do this constantly:

• Token equality (different Unicode representations → same token)

• Benchmark scores (wildly different models → same accuracy number)

• Task success metrics (different reasoning paths → correct answer)

• Embedding similarity (different meanings → same vector neighborhood)

Different internal worlds. Same equivalence class.

And the system cannot tell them apart.

Here’s the uncomfortable truth: every metric is a quotient map. Every evaluation collapses a vast space of internal states onto a thin line of numbers. The kernel of that map — everything your metric cannot distinguish — is where all the interesting failures hide.

When your production AI hallucinates confidently, when your agent suddenly destabilizes, when your equivalent models behave completely differently on edge cases: you’re seeing kernel directions assert themselves. Differences you declared irrelevant turning out to matter.

5. The Kernel-Quotient Trap

So we have two operations:

• Kernel: what your system cannot see

• Quotient: what your system refuses to distinguish

Together, they form our AI trap.

AI systems navigate enormous state spaces. To make them tractable, we quotient aggressively: we declare vast swaths of states equivalent for our purposes. Benchmarks. Metrics. Evaluations. All quotient maps.

But quotients don’t eliminate the kernel. They hide it.

The states you collapsed are still there, evolving, diverging, accumulating differences along their paths. You just can’t see them through your chosen lens.

Then one day, something emerges from the kernel. A behavior no one predicted. A failure mode no benchmark caught. A sudden capability or breakdown that was actually building for months in dimensions your quotient blinded you to.

This isn’t mysterious. It’s geometry. The kernel was always there. You just chose not to look.

Said another way: the kernel-quotient trap is written like a story where three characters start in the same place, and the narrator only tells you where they end up: never how they got there. They all arrived at 94.2%, the story concludes. But one walked a clean path. One spiraled through contamination. One is inches from the cliff. The ending looks identical. The journeys were not. And journeys determine what happens in the sequel. See the visual translation in the figure below:

6. Paths, Holonomy, and What Functions Cannot Express

Now we can state the core problem precisely:

Function composition erases paths. Reality remembers them.

When you compose functions f : A → B and g : B → C to get g ∘ f : A → C, you get a map from A to C. But you’ve lost something: the path through B.

In flat spaces, this doesn’t matter. All paths between two points are equivalent.

But AI state spaces are not flat.

They have curvature. And curvature means holonomy: the phenomenon where the path you take changes the outcome, even if you end up at the very same spot.

Let me make this bolder.

Imagine you’re carrying an arrow on the surface of a sphere. You start at the North Pole, arrow pointing toward London. Walk to London, keeping the arrow parallel — not rotating it, just maintaining its direction relative to your motion. Walk along the equator to Tokyo. Keep the arrow parallel. Walk back to the North Pole.

You’ve returned to your starting point. But the arrow? It points in a different direction than when you started.

The path rotated it. The geometry of the sphere induced a transformation that pure function from start to end cannot see.

This is holonomy. And it happens constantly in AI systems.

Two fine-tuning sequences that reach the same model parameters. Same endpoint. Different internal geometry. Different failure modes. Different responses to adversarial prompts. The path left traces that your function-only architecture cannot represent.

If your architecture cannot distinguish how a state was reached, it cannot reason about path-dependent phenomena. And path-dependent phenomena are everywhere in AI.

7. The DAG Illusion: Confusing Diagrams with Dynamics

Modern AI systems are drawn as flowcharts: directed acyclic graphs (DAGs):

• Nodes represent models, agents, or processing steps

• Edges represent data flow

• The whole thing executes in topological order

This looks categorical. You can squint and see objects and arrows. It might even satisfy the formal axioms.

But it captures nothing about dynamics.

A DAG has:

• No sense of how you traverse it, only that you do

• No accumulated errors along the path

• No memory beyond whatever state you explicitly pass

• No notion that taking a different route might yield different results

• No representation of its own kernel: the internal differences invisible to its outputs!

Here’s a concrete example. Two LLM agents negotiate a solution. Agent A proposes, Agent B critiques, Agent A revises.

Now reverse the order: Agent B proposes first!

Same agents. Same composition. Different outcome.

The DAG says these are equivalent pipelines. Reality laughs.

What’s happening? The path through the interaction space has holonomy. The order of traversal rotates the internal state in ways the DAG cannot see, because DAGs don’t have geometry. They’re flat. They’re Set in disguise.

8. The Functional Programming Crime

This isn’t just an AI problem. It’s a computational culture problem.

Even Haskell: the supposed cathedral of categorical purity — commits the same crime.

Haskell’s claim to categorical legitimacy rests on types, functions, and monads. Impressive vocabulary! But what category does Haskell actually implement?

Hask: the category of Haskell types and functions. Which is essentially Set with better marketing.

The monad, which allegedly handles effects, doesn’t fix this. A monad sequences computations along a linear chain:

action1 >>= \x -> action2 >>= \y -> action3

This traces a path, but the path has no geometry. Running the same monadic computation twice gives identical results: that’s the whole point of referential transparency. The path collapses to its endpoint.

What Haskell cannot express:

• Two different operation sequences that should be equivalent but aren’t (holonomy)

• Going around a computational loop and ending up somewhere unexpected (curvature)

• Operations that depend on how you arrived, not just where you are (path-dependence)

• The repeatedly mentioned kernel of a computation: what internal differences produce identical outputs

And everyone thinks this is categorical programming. It’s not. AGAIN it’s Set programming wearing a category theory costume.

9. Category Theory Already Solved This (Nobody Noticed)

Here’s the bitter irony.

Modern category theory( the mathematics developed since the 1960s) has extensive machinery for handling exactly these phenomena:

• Enriched categories: morphisms carry extra structure (cost, time, probability, distance)

• Higher categories: morphisms between morphisms, capturing transformations of transformations

• ∞-categories: infinite towers of higher morphisms encoding paths, homotopies, and loops

• Fibered categories: systematic handling of parameterized families of structures

• Sheaves and descent: local behavior constrained by global consistency

• Kan extensions: principled ways to extend functors along paths

These frameworks are explicitly path-aware. Holonomy lives there. Memory lives there. History lives there. Kernels and quotients are first-class citizens, not afterthoughts.

But almost no AI architecture uses them. 👀

We have the mathematical tools for path-dependent composition. We’ve had them for sixty years. And the AI community is still drawing flat flowcharts and calling them categorical.

10. What This Breaks in Practice

Ignoring path structure has concrete consequences:

Errors accumulate invisibly. Each step in a pipeline introduces small deviations. In flat composition, these are just noise. In curved spaces, they compound along trajectories and can land you somewhere completely unexpected.

Feedback loops become unpredictable. A system that monitors its own output and adjusts is traversing paths through its state space. Without geometry, you cannot predict where these loops converge — or if they converge at all.

Retries aren’t idempotent. Running the same agent interaction twice should NOT matter, but it does, because internal state has evolved along a path your architecture doesn’t track.

Autonomous agents become chaos. Multi-agent systems develop emergent behaviors — ruts, attractors, oscillations — that no one designed because no one modeled the geometry of their interaction space.

Debugging becomes archaeology. When you can’t trace how a state was reached, you can only examine artifacts. Was this hallucination caused by the prompt? The context window? The fine-tuning history? The moon phase? 😜Who knows. The path is lost.

Kernel effects ambush you. Your benchmarks look great… then production reveals failure modes that existed all along in the kernel of your evaluation. The quotient hid the problem until it couldn’t anymore.

This is why orchestration layers rot over time. Each patch adds another invisible path dependency. Eventually the system is a tangle of implicit holonomies that no one can map.

11. What Would Path-Aware AI Look Like?

This isn’t purely negative criticism. There’s a positive program here.

A genuinely path-aware AI architecture would:

1. Represent state spaces as geometric objects with meaningful topology — not just vectors, but manifolds or stratified spaces where curvature means something

2. Track trajectories explicitly, not just current states: know how you arrived, not just where you are

3. Define connections: rules for how to transport information along paths, so “parallel” operations stay consistent

4. Measure curvature: quantify exactly how much path-dependence exists in different parts of the system

5. Model kernels explicitly: know what your architecture cannot distinguish, so you’re aware of your blind spots

6. Use holonomy groups to characterize the space of possible history effects — what traces can paths leave?

Some fragments of this exist. Geometric deep learning touches on it. Topological data analysis hints at it. Homotopy Type Theory provides a programming foundation where types are spaces and equality is path.

But no one has built the full stack.

The tools are there: fiber bundles, principal connections, parallel transport, characteristic classes. Category theory provides the organizational framework — but the enriched and higher versions, not the 1940s fragment everyone uses.

12. Why This Happened

This architectural blindness isn’t accidental. It has roots:

Machine learning grew from statistics, not geometry. The founding metaphors were regression, optimization, probability distributions. These naturally live in flat vector spaces where all paths are equivalent.

Software engineering thinks in functions. The entire paradigm of programming — especially functional programming — is function composition. Categories of sets feel natural. Categories of spaces feel alien.

Higher category theory is genuinely hard. ∞-categories, derived geometry, homotopy type theory… these require years of serious mathematical training. It’s easier to slap categorical on a DAG and publish.

Kernels are uncomfortable. Admitting what your system cannot see is humbling. It’s easier to pretend omniscience than to map your blind spots.

There’s no immediate reward. A system with proper path-tracking doesn’t benchmark better on today’s metrics. The benefits show up in robustness, interpretability, and long-term stability — things that don’t win Kaggle competitions or inflate demo numbers.

So we collectively took the shortcut. And now we’re confused about why our AI systems behave like haunted houses — full of ghosts from paths nobody remembers taking.

13. The Accusation

Let’s say it plainly:

The problem is not that category theory is too abstract for AI. The problem is that AI — and functional programming, and most of computer science — uses a crippled, pre-1950s fragment of it.

As we have already seen: functions without paths. Composition without transport. Arrows without memory. Kernels without acknowledgment.

That’s not category theory. That’s cosplay.

It’s like claiming to use calculus while refusing to acknowledge derivatives. Technically you’re in the building, but you’ve locked yourself in the lobby.

The entire computational paradigm has internalized the Set-reduction so deeply that people mistake the costume for the substance. And we stop thinking and just stay comfortable in the lobby, ignoring the rest of the floors in our Category Hotel, as you see in the funny, somewhat cynical GIF below:

14. The Path Forward

As we again state, and unfortunately never tire of repeating: The real tragedy is that mathematics already solved these problems. The tools exist. They’re sitting in textbooks on differential geometry, algebraic topology, and higher category theory, waiting for someone to actually use them.

What would it take?

For AI researchers: Learn some geometry. Not just linear algebra: actual differential geometry. Understand what a connection is, what curvature measures, why parallel transport matters. Learn what a kernel tells you about your system’s blindness. Then look at your pipeline diagrams again.

For functional programmers: Acknowledge that Haskell isn’t the endpoint. Look at Homotopy Type Theory, where types are spaces and paths are first-class citizens. The next generation of programming languages will be path-aware. Be ready.

For everyone building AI systems: When your system behaves mysteriously: when fine-tuning order matters, when retries give different results, when agent interactions spiral unexpectedly — don’t dismiss it as noise. It’s signal. It’s the geometry you’re ignoring, asserting itself. It’s the kernel you never mapped, coming back to haunt you.

The question is whether AI will grow up and learn real mathematics… or keep playing with flat diagrams while wondering why nothing quite works.

SUMMING UP…

If your AI architecture cannot distinguish how a state was reached, then it cannot truly reason, learn, or adapt, no matter how many arrows you draw.

If your architecture cannot see its own kernel, it cannot know what it doesn’t know.

And no amount of categorical vocabulary will save a system that forgot geometry, history, and paths.

Category theory wasn’t misdesigned.

It was misused.

The paths are still there, curving through your systems, accumulating holonomy with every inference. The kernels are still there, hiding the differences your metrics cannot see. You can model them, or you can pretend they don’t exist.

But they’ll keep mattering, whether you acknowledge them or not.

The Split Is Already Here

Silicon Valley keeps buying more GPUs and scaling bigger transformers, waiting for AGI to finally emerge. It feels like progress, but something fundamental isn’t changing. The geometry underneath is wrong: flat Euclidean spaces pretending reality isn’t curved.

I’ve spent months tracing where models drift, break, and hallucinate. Same root every time.

The fix exists. It predates everything they’re building.

Let me show you.

The Old AI Lives in a Cartoon Universe

The Old AI — the endless parade of just scale Transformers prophets — believes the world is flat.

• Flat loss surfaces.

• Flat embedding spaces.

• Flat parameter landscapes.

• Flat reasoning steps.

Everything reduced to matrix multiplications on ℝⁿ. Everything solved by throwing more GPUs at the fog.

This worldview has one motto:

If it isn’t working, increase the FLOPs.

That worked… until it didn’t.

Those Transformers that power our current AI are basically neural networks with self-attention (plus some feed-forward layers and residual tricks) that let every token gossip with every other token in parallel.

That’s a big part of why modern AI scales so well: it’s brute force with better wiring.

But once we scaled that recipe to billions of parameters, the cracks showed up and every crack has a geometric signature:

• Hallucinations (confident interpolation to nowhere, with no option to say I don’t know): the model is lost in a flat, featureless region with no clear direction. You can spot this in the Hessian eigenvalues: specifically, a high condition number κ. Translation: too many directions look equally fine, so the model shrugs, picks one, and commits with full confidence to nonsense.

• Distribution shift failures (memorized patterns don’t transfer): the model got stuck in a narrow valley during training. The Hessian eigenvalue to watch here is spectral sharpness ε: when it’s high, it means the model latched onto a brittle, hair-trigger solution. Looks great on training data. Faceplants the moment real-world inputs drift even a little.

• Adversarial brittleness (small input changes cause large output swings): the model is balanced on a knife-edge — a saddle point. The telltale sign? Negative Hessian eigenvalues δ. That means unstable equilibrium: poke it with a tiny perturbation and the output goes careening off in wild, unpredictable directions.

These aren’t mysterious emergent behaviors: they’re receipts. Every failure mode has a return address. Here’s the transformer architecture that powers your chatbot, your copilot, your search engine, with the damage traced layer by layer:

The Current AI Paradigm Is Broken

So the transformers are broken, we just saw that. But here’s what nobody talks about: the math we use to study them is also broken. Two levels of failure, stacked on top of each other, both preventing AI from evolving beyond chat-toys into fully mature systems deployable in serious industrial environments.

Indeed, our mathematical analysis of AI is unnecessarily cumbersome and convoluted. Everyone analyzing AI today is doing what we can call mathematically — forward Riemannian thinking:

• Start in high dimensions (where the model lives)

• Try to understand local geometry (gradients, Hessians, attention patterns)

• Hope global structure emerges from local measurements

• Use dimensionality reduction (t-SNE, UMAP, PCA) to visualize — but these destroy the algebraic structure 💀

Result:
We can see local patches… but
We cannot see loops.
We cannot see closure.
We cannot see drift.

We’re blind to the very properties that determine coherence.

Instead, we need the contrary procedure. And you cannot imagine where this example comes from…

From music itself.

One moment — what are you saying? Music? 😄

Yes. Classical composers like J.S. Bach applied inverse Riemannian thinking to their masterpieces — the same logical procedure we need to tame the explosive dimensional growth of our current AI, which is kind of wild when you think about it.

• Start with global structure (does the loop close? does meaning return?)

• Represent it in a low-dimensional space with fixed topology (circle of fifths)

• Local curvature becomes visible as harmonic ambiguity

• Global flatness becomes audible as resolution

Result: We can finally answer the questions that matter:

Did the AI stay on track?
Did it return to the point?
Or did it wander off and hope you wouldn’t notice?

We can answer them in a simple space that doesn’t explode every time Silicon Valley releases a new model. ❌

But wait — don’t we have that already with the Hessian matrix? 👍

Good point. The Hessian eigenvalues (κ, ε, δ) do tell us about curvature. But here’s the difference:

The Hessian tells you curvature at a point. Is this step stable? Is this chord a singularity? That’s local information — valuable, but incomplete.

Bachian Holonomy tells you something the Hessian cannot: Does the whole path close?

Think about it: the Hessian might give you green lights at every step: low κ, stable ε, no negative δ, and you’d think everything is fine. But zoom out and the reasoning has drifted miles from where it started: several stable steps that lead nowhere.

Flip side: the path might pass through a singularity — a moment of genuine ambiguity, high κ, multiple valid exits — and still arrive home. The C+ chord in Bach’s progression is exactly this: a danger zone that resolves perfectly.

The Hessian sees the trees. Each one healthy? Check.

Bachian Holonomy sees the forest. Are you still in the same forest you started in? That’s the question that matters

The Geometry They Forgot

And here the practical consequence of our Bachian Holonomy: Nobody can visualize a billion-dimensional loss surface. But the old Bach will help us here by simplifying our work space where, regardless of the dimensions, size of our AI neural network, and number of tokens, we can capture the relevant curvature structure in a way that’s cognitively manageable and visually representable.

Fast forward to his Triple Concerto, BWV 1044. One of his most accessible works, and a perfect demonstration of inverse Riemannian thinking in action. Here’s the harmonic progression from the opening:

Am → E → Dm → C+ → F → G → C → E → Am

Watch what happens when we trace those chord roots on the circle of fifths:

.You see it? At each step, the harmony could branch multiple ways — multiple paths are valid. But when the progression hits the augmented chord (C+), something remarkable happens:

it becomes a singularity. Three equally legitimate exits. No obvious path forward. The local geometry is curved, ambiguous, multi-valued.

But globally? The progression must return to A minor. The listener’s ear agrees with the maths. Zero drift. Perfect closure.

Here you have it: inverse Riemannian thinking — sounding like music — in Bach’s harmonic space. The BWV 1044 concerto, first movement, chord progression from beginning to end.

Now we’re ready to pinpoint what inverse Riemannian structure actually is: the exact opposite of what textbook differential geometry describes:

Instead of building up from local flatness to a higher-dimensional manifold, we collapse higher dimensions down to flatland.

That reversal is exactly what we need to make sense of the dimensional explosion that comes with each new LLM: by projecting it into a space simple enough to see, hear, and think about.

But is this mathematically legitimate?

Yes. There’s a foundational result in algebraic topology — the basepoint independence theorem, that licenses the whole approach: In a path-connected space, any choice of basepoint yields an isomorphic fundamental group.

In plain english: it doesn’t matter which pitch you start from, or which word. The algebra of loops is intrinsic to the space itself. Start from C or A or F#. Start from “justice” or “bank” or “democracy.” The loop structure is the same.

This is why harmonic space can serve as a universal diagnostic for token space. Both have intrinsic loop algebras. Both are basepoint-independent. Both can be connected by a structure-preserving map.

We can’t claim full isomorphism: 12 pitches versus 50k+ tokens. But we can claim something almost as powerful: the algebraic structure that matters (closure, drift, path-dependence) is comparable across both spaces.

That’s the foundation for everything that follows.

And here’s the most important thing you have to remember: this is exactly what semantic coherence requires.

Think about it. When you reason through an argument, each step has multiple valid continuations. Local ambiguity is unavoidable — that’s what makes language rich. But globally, a coherent argument must return to its thesis. A story must resolve. A proof must close.

Locally curved. Globally flat.

Your Transformer doesn’t know this. It treats every direction as equally valid. It has no concept of must return. It drifts…and calls the drift creativity.

The Curvature AI Needs to Become AGI

Let’s take stock of where we are.

We’ve shown that current AI analysis is broken: forward Riemannian thinking that starts in high dimensions, hopes for global structure, and destroys what matters most when we try to visualize it.

We’ve shown that Bach solved this 300 years ago with inverse Riemannian structure: local curvature (ambiguity at each step), global flatness (must return to tonic).

We’ve shown that the circle of fifths — just 12 dimensions — captures the same algebraic structure as billion-dimensional token spaces, thanks to the basepoint independence theorem.

Now comes the payoff.

The circle of fifths doesn’t just represent harmonic relationships. It represents curvature itself.

When a progression returns to tonic with zero drift, the holonomy is trivial: flat geometry, closed loop, coherent reasoning. When it fails to return — when the comma accumulates, when meaning drifts — that’s measurable curvature. Detectable. Audible.

And here’s what ties it to AGI: this curvature corresponds exactly to what the Hessian eigenvalues capture in neural network training — the very numbers that tell us whether a model is learning, stuck, or lost.

The three critical eigenvalue signatures:

• κ (condition number): ratio of largest to smallest eigenvalue. How stretched is the local geometry? High κ means ill-conditioned: the optimizer zig-zags, progress stalls.

• ε (spectral sharpness): the largest eigenvalue. How steep is the curvature? High ε means sharp ridges: small steps or you overshoot.

• δ (negative eigenvalues): count of negative eigenvalues. Are you at a true minimum or a saddle point? δ > 0 means unstable: the model will eventually escape, but when? And to where?

Now, someone from Google might raise their hand: The Hessian is n×n. For a billion-parameter model, that’s a billion times a billion. You can’t compute that!

Correct. And completely irrelevant.

You don’t need the full matrix. You never did.

A Hungarian physicist named Cornelius Lanczos figured this out in 1950. His method extracts the dominant eigenvalues using iterative matrix-vector products. Complexity: O(n) per iteration. Basically free compared to a single forward pass.

Seventy-five years later? We now have Hutchinson’s trace estimator, stochastic Lanczos quadrature, randomized SVD. You can get spectral density estimates, top-k eigenvalues, condition number bounds: all at negligible computational cost.

Tools like PyHessian already prove this works at ImageNet scale. Today. Right now. (For the full technical breakdown, see my previous article: Three numbers. That’s all your AI needs to work.)

So the Hessian eigenvalues — κ, ε, δ — are computable and cheap. The trees are visible.

But here’s what Lanczos can’t give you: the forest.

Local curvature tells you about each step. It doesn’t tell you whether the whole journey makes sense. You can walk through several stable points and still end up lost. You can pass through a singularity and still return home.

That’s where holonomy comes in.

⚠️ Let me be absolutely clear about what we’re claiming here.

This is not metaphor. This is not music as inspiration. This is not poetry dressed up as science.

The circle of fifths is a mathematically legitimate diagnostic tool for AI.

The harmonic space has:

• Fixed topology (ℤ₁₂ — doesn’t change when you change the model)

• Physical metric (the fifth is a 3:2 frequency ratio: physics, not convention)

• Algebraic structure (loops, closure, path-dependence — same as semantic space)

• Basepoint independence (start anywhere, same loop algebra — proven theorem)

• Audible holonomy (you can literally hear whether a loop closes)

None of this is approximate. None of this is kind of like something else. This is homomorphism: structure-preserving correspondence between two spaces.

And without this kind of tool — without a way to visualize, diagnose, and audit reasoning coherence — AI will remain stuck exactly where it is NOW:

• Hallucinating with no warning system

• Drifting with no detection mechanism

• Failing at scale with no geometric insight into why

You cannot build AGI if you cannot see what your model is doing. You cannot deploy industrial AI if you cannot audit its reasoning. You cannot trust a system that cannot demonstrate coherence.

Current AI is flying blind through curved space with flat-world instruments.

The circle of fifths( yes, the same circle that music students learn in their first year of theory) provides what’s missing: a fixed, audible, algebraically-grounded reference frame for reasoning coherence.

This sounds absurd until you understand the mathematics. Then it sounds inevitable.

Yes — a Chord Can Equal an AI-Eigenvalue

When you’re training a neural network, the loss surface is like a landscape of mountains and valleys. The Hessian matrix tells you the shape of the ground beneath your feet — is it flat? steep? tilted? stable?

The eigenvalues of that matrix (κ, ε, δ) are just numbers that summarize that shape:

• High κ = you’re on a plateau where every direction looks the same. No clear path. The model guesses.

• High ε with negative δ = you’re on a knife-edge. Stable in one direction, but one wrong step and you tumble. Saddle point.

• All small, positive eigenvalues = you’re in a nice bowl. Settled. Stable. Home.

Now here’s the magic: chords feel the same way to your ear.

• An augmented chord (C+) sounds unresolved, floating, could-go-anywhere. Three equal exits. No pull in any direction. That’s high κ — the plateau.

• A dominant seventh (G7) sounds tense, unstable, demanding resolution. You physically feel the pull toward the tonic. That’s the saddle point — high ε, negative δ.

• The tonic (Am) sounds like home. Rest. Arrival. That’s the stable minimum — all eigenvalues positive and small.

This isn’t metaphor. It’s homomorphism — a structure-preserving map. The mathematical relationships between eigenvalues (stable/unstable/ambiguous) map directly onto the harmonic relationships between chords (resolved/tense/floating).

Same structure. Different encoding. One you compute. One you hear.

Yep, that harmonic space, both the Hessian signatures and the holonomy become perceptible:

The Hessian sees the trees — cheap, computable, local. The holonomy sees the forest — global, structural, audible.

Together, they give you complete geometric awareness.

What current AI lacks isn’t the math for local curvature — Lanczos solved that in 1950. What’s missing is the global picture: does the reasoning cohere? Does the loop close? Does the model know where it is?

The circle of fifths is not a metaphor for loss geometry. It is loss geometry — compressed to human perception.

Music is not the analogy. Music is the instrument.

And if we can hear it, we can fix it.

The Dual-Lens Architecture: Trees and Forest in One Framework

So how do we actually implement this?

We’ve established two complementary views of AI geometry:

• The Hessian eigenvalues (κ, ε, δ) — the trees. Local curvature at each point. Is this step stable? Is this transition well-conditioned? Are we on a saddle point?

• The inverse Riemannian holonomy — the forest. Global closure. Does the reasoning loop back? Does meaning drift? Does the path return home?

Current AI analysis uses only the first — when it uses anything at all. What we need is a dual-lens kernel that combines both.

The Pipeline

INPUT: Reasoning chain (tokens) or training trajectory (parameters)
↓
STEP 1: Compute local Hessian approximations (κ, ε, δ) at key points
↓
STEP 2: Map trajectory to circle of fifths via attraction-preserving Φ
↓
STEP 3: Measure holonomy — does the loop close?
↓
STEP 4: Sonify — convert to chord progression
↓
OUTPUT: Visual dashboard + audible diagnostic

The visual dashboard shows you the numbers. The sonification lets you hear the shape.

Coherent reasoning → Resolved cadence, return to tonic, closed loop, logically complete.
Drifting reasoning → Unresolved tension, comma drift: the path didn’t close, something’s off.
Hallucination → Never returns home, stuck on dissonance: structurally broken, obviously wrong.

Step 2 in Action: Mapping Tokens to the Circle of Fifths

Let’s trace a real reasoning chain through the pipeline.

The Input: A Reasoning Chain

An LLM is asked: “What is justice?”

It generates this reasoning chain:

“justice” → “law”
→ “courts” → “judges”
→ “decisions” → “fairness” → “justice”

The reasoning loops back to the starting concept. But did the meaning drift? Let’s find out.

Apply the Φ Map

Okay, here’s where the magic happens. We need to translate tokens into music. How?

Every token in an LLM lives in a high-dimensional space — 768, 12k, sometimes 32k dimensions. Each token is a point in that vast space, and tokens that mean similar things cluster together. Justice and fairness are neighbors. Justice and banana are not.

The Φ map does something simple but powerful: it takes each token and assigns it to one of 12 pitch classes on the circle of fifths.

The rule: tokens that are close in meaning get pitches that are close on the circle.

That’s it. Neighbors stay neighbors.

• Justice and fairness are semantically close → they land on nearby pitches (say, A and E — one fifth apart)

• Justice and banana are semantically far → they land on distant pitches (say, A and Eb — a tritone apart, maximum distance)

So when the LLM generates a reasoning chain like:

“justice” → “law” → “courts” → “judges” → “fairness” → “justice”

The Φ map turns it into:

A → E → B → B → E → A

Which becomes a chord progression:

Am → E → Bm → Bm → E → Am

Now you can play it. And hear whether the reasoning holds together.

Verdict: Resolved. The loop closes. Coherent reasoning.

Now Compare: A Hallucinating Chain

Same prompt, but the LLM drifts — following surface-level word associations instead of semantic meaning:

“justice” → “law” → “lawn” → “grass” → “green” → “envy” → “justice”

What happened? The model followed a phonetic bridge (“law” → “lawn”), then free-associated through “grass” → “green” → “envy,” then forced a return to “justice” with no logical connection.

This is semantic drift — one of the failure modes that leads to hallucination. The model isn’t lying about facts (yet), but it’s lost the thread. The reasoning has wandered off the path.

How does Φ know “lawn” is wrong?

It doesn’t look at spelling — it looks at meaning.

In the LLM’s embedding space, “law” and “lawn” are far apart. They sound similar to us, but the model knows they mean completely different things. “Law” lives near “justice,” “courts,” “legal.” “Lawn” lives near “grass,” “garden,” “mower.”

When the model jumps from “law” to “lawn,” it’s jumping across the embedding space: a semantic cliff. Φ translates that cliff into a harmonic leap: E → Db, almost a tritone.

Your ear hears that leap and says: “Something’s wrong.”

Φ didn’t create the error. Φ made the error audible.

You can see how the operator Φ works in these examples in the figure below:

The Path to AGI

Current AI has no self-awareness of its own geometry. It cannot sense when it’s drifting, when it’s stuck on a saddle point, when its reasoning fails to close.

A dual-lens kernel would give AI what it currently lacks:

• Local proprioception: sensing the curvature at each step

• Global coherence tracking: sensing whether the trajectory makes sense

• Early warning system:detecting drift before it becomes hallucination

This isn’t just interpretability. It’s the beginning of geometric self-awareness — the capacity to navigate curved semantic space without getting lost.

That’s not a feature. That’s a prerequisite for AGI.

The Closing Chord

Imagine an AI that knows where it is on its manifold. That tracks every rotation of meaning. That preserves coherence through cycles. That reasons in curvature instead of pretending the world is flat. That doesn’t drift, doesn’t fracture, doesn’t hallucinate: because its geometry forbids it.

An AI you can hear thinking correctly.

That AI is not a bigger Transformer. It is a new species.

Old AI can’t reach it, not with more layers, not with more GPUs, not with more cross-fingers and scaling prayers.

The gap is mathematical.

And now, finally, audible.

Bach solved this 300 years ago. The question is whether you’ll learn his math… or get left behind by those who do.

The split is here.

Choose your side.

If You Don’t Understand Eigenvalues, You Don’t Understand AI

What if I told you that three numbers — just three — could predict whether your AI will work or catastrophically fail?

No new architecture. No retraining. No infrastructure overhaul.

Just simple math. Math that’s been sitting in textbooks since 1950, waiting for someone to bother checking it.

These three numbers would have caught:

Your Tesla phantom braking for no reason.

GPT-4 citing a court case that doesn’t exist: submitted by a lawyer, who got sanctioned.

Your model, trained for three weeks, beautiful loss curve, shipped to prod, immediately falling on its face.

Same root cause. Every single time.

The Three Numbers Everyone Ignores

Nobody tells you about them. Not your framework. Not your coursework. Not the documentation, though they predict whether your training will work or blow up in your face. Whether inference will be stable or hallucinatory. Whether your minimum is real or a trap.

1. Condition number κ (λ_max / λ_min): Is your optimizer wasting compute zig-zagging? This number measures how “stretched” your loss landscape is. A high κ means canyon. You’re burning 90% of your compute going sideways instead of down (direction towards the error minimum)

2. Eigenvalue magnitude ε (|λ|_max): Did your model actually learn — or just memorize? Read on, and you’ll find a cute GIF below that explains it better than a whole series of equations..

3. Negative eigenvalue count δ (#λ < 0):
You might be convinced that the gradient from your AI framework is enough to finish training your shiny new network. That’s what you were told: the gradient says “we’re done, nowhere to go” because the slope is flat in every direction you’re measuring.

But what if your training is stuck at a fake minimum?
That’s what this killer δ number is for. As a preview:

• When δ = 0, you’re at a true minimum — a valley floor.

• When δ > 0, you’re on a mountain ridge.

Further below you’ll see an animated graphic that will clarify the questions you’re probably asking yourself right now.

And Yet — Believe It or Not — This Is Old Math

This math has existed since 1950. Before anyone had dreamed of a neural network.

Seventy-five years of mathematicians screaming into the void.

And still — right now, today, as you read this — not in PyTorch. Not in TensorFlow. Not in JAX. Not in Keras. Not in any standard training pipeline burning millions of dollars.

Three numbers that would tell you everything. Hidden in plain sight.

The biggest AI labs in the world? They’re watching loss curves. Crossing fingers. And calling it engineering.

OpenAI ignores them. Google skips them. Anthropic doesn’t even know they exist. Nobody checks.

Let it sink in. But keep reading, you haven’t seen the best part yet.

The Three Numbers Everyone Misses

So you’ve met κ, ε, and δ. Three numbers that predict everything. Three numbers that have existed since your grandparents were young.

Now ask yourself: where are they?

Not on your screen. Not in your logs. Not anywhere in the trillion-dollar AI industry.

Those three numbers aren’t theory. They’re a diagnostic panel that should exist… but doesn’t.

Instead, here’s what the most popular AI frameworks — PyTorch, TensorFlow, JAX, Keras — actually give you:

• Loss value ✓

• Gradient direction ✓

• Learning rate ✓

That’s it. That’s the whole dashboard.

Three numbers — but not the right three. Not κ (condition number). Not ε (eigenvalue magnitude). Not δ (negative eigenvalue count).

And it’s not just your framework. OpenAI’s internal tools? Same blind spot. Google’s infrastructure? Same gap. Anthropic’s training pipeline? Same missing panel. The entire industry is flying the same broken instrument cluster.

Training AI today is like flying a 747 with a speedometer, a compass, and vibes.

The loss is going down. Great — that means the error is going down, training looks good.

But why is it going down? Is the model finding a real solution, or just memorizing noise and digging itself into a hole it’ll never escape?

The eigenvalues would tell you. Your favorite AI framework’s dashboard won’t.

Figure 2b makes this obvious:

Where those three numbers come from

They live inside something called the Hessian: the second derivative matrix of your loss function. It encodes the complete curvature of your loss landscape. Every canyon, every ridge, every flat plateau. It’s all there.

And here’s the absurd part: the Hessian is fully computable. It’s sitting right there, screaming diagnostic data, and nobody’s looking at it.

Here’s how it works:

Look at the two boxes in the middle of the figure. On the left: what PyTorch shows you. Loss looks good. Gradient near zero. Learning rate set. Green light. Ship it.

On the right: what κ, ε, and δ reveal. Condition number through the roof. Sharpness off the charts. Two negative eigenvalues. You’re not at a minimum: you’re stuck at a saddle point pretending to be done. 💀

Same numbers. Same moment. Opposite conclusions.

OpenAI doesn’t compute this. Google doesn’t compute this.

Nobody running a several-hundred-thousand-bucks-a-month GPU cluster is checking eigenvalues.

They’re watching loss curves and you guessed right… praying!

But the Hessian is Too Heavy!

This is where someone from Google raises their hand.

The Hessian is n×n. For a billion-parameter model, that’s a billion times a billion. You can’t compute that. It’s mathematically insane !

Correct. And completely irrelevant.

You don’t need the full matrix. You never did.

A Hungarian physicist named Cornelius Lanczos figured this out in 1950 — back when “computer” meant a room full of vacuum tubes and an obsessive cross-fingering. His method extracts the dominant eigenvalues using iterative matrix-vector products. Complexity: O(n) per iteration. Basically free compared to a single forward pass.

Seventy-five years of progress since then? We now have Hutchinson’s trace estimator, stochastic Lanczos quadrature, randomized SVD. You can get spectral density estimates, top-k eigenvalues, condition number bounds — all at negligible computational cost.

Tools like PyHessian already prove this works at ImageNet scale. Today. Right now.

So why isn’t this in PyTorch? Why isn’t it in TensorFlow? Why isn’t it in JAX?

Because the people who understand spectral theory are in math departments writing papers nobody reads. The people building frameworks are shipping features that look good in demos. And the people training models are too busy babysitting loss curves to ask why they’re babysitting loss curves.

The math exists. The engineering exists. The will to connect them? Apparently not.

Seventy-five years. Still waiting…

What These Numbers Actually Tell You

Let’s make this practical, concrete, and — above all — visual, so you can build the right mathematical intuitions.

The Condition Number Disaster

The condition number κ (kappa) = λ_max / λ_min controls how hard your optimization problem actually is.

Here’s what nobody tells you: gradient descent convergence scales as (κ-1)/(κ+1) per iteration.

Translation:

• κ = 100? Roughly 100 iterations to cut your error in half.

• κ = 10000? Roughly 10k iterations for the same progress.

And κ = 10000 is common in practice. Not pathological. Not rare. Tuesday.

So what does high κ feel like?

Imagine rolling a marble down a valley. Nice round bowl? Marble rolls straight to the bottom. Done.

But high κ means your valley is a razor-thin slot canyon: walls a mile high, floor barely visible. Your marble doesn’t roll down. It pinballs off the walls. Left. Right. Left. Right. Burning energy going sideways instead of down.

That’s your gradient. Pointing exactly where calculus says: steepest downhill. Problem is, “steepest downhill” aims at the canyon walls, not down the floor toward the solution.

Those training runs that plateau for hours? Loss stuck at 0.0847, then 0.0846, then 0.0847 again? You tweak the learning rate. Nothing. You sacrifice a rubber duck to the ML gods🚮 . Nothing.

That’s high κ.

Your optimizer isn’t broken. It’s doing exactly what you asked. What you asked is just geometrically insane.

You’re burning compute fighting geometry your framework refuses to acknowledge exists.

The fix?

Preconditioning. Second-order methods. Adaptive optimizers that implicitly estimate curvature. The math exists. It’s been in textbooks since the 1960s.

But first, you’d need to know κ is the problem.

Your dashboard won’t tell you. So you’ll never ask. And your compute bill keeps climbing.

The figure below shows exactly what’s happening. Same starting point. Same destination. One path takes ~100 iterations. The other takes ~10000 — zigzagging off canyon walls while your GPU burns money.

Sharp vs. Flat: The Generalization Prophecy

Now let’s talk about ε (epsilon) — the eigenvalue magnitude |λ|_max.

This number answers the question every ML engineer secretly dreads: Did your model actually learn anything, or did it just memorize the test?

Picture your loss landscape as terrain. A flat minimum is a wide valley — you can wander around and the elevation barely changes. A sharp minimum is a knife-edge ridge — one wrong step and you’re tumbling into the abyss.

Small ε = flat minimum = good news.

When ε is small, your model found that wide valley. Production data comes in slightly different from training data: users type weird things, lighting changes, accents vary, and your model shrugs. Close enough. I got this.

That’s generalization. That’s what you’re paying for.

The GIF below shows what this looks like geometrically:

Now flip the script.

What happens when ε is huge? When your eigenvalues are screaming large numbers?

Well, when ε is large, your model has squeezed itself into a tiny, sharp crevice. Training loss looks perfect. Validation is “best run ever.”

Then real data arrives. Slightly different wording. Slightly different images. Slightly different anything.

And the model doesn’t perform a bit worse. It blows up:

• Confidence scores go crazy

• Predictions turn random

• Your alert-log system explodes at 2 AM. 💀

That’s a sharp minimum in action. The walls are so steep that the tiniest shift sends the loss rocketing. Your model wasn’t robust. It was brittle and faking it.

The GIF below shows the difference. Same horizontal shift. Wildly different outcomes.

As you see this is not philosophy. This is geometry you can measure.

Your model that crushed the benchmark and died in production? It found a sharp minimum. The eigenvalues would have told you… but nobody checked.

Saddle Points: The Silent Killer

Now for δ (delta) — the negative eigenvalue count, #(λ < 0). This one’s sneaky. This one lies to your face.

Your gradient hits zero.
Your loss curve goes flat.
Your framework prints: “converged.”
You relax.

But you’re not at an error minimum.
You’re at a saddle point ! 💀

A mountain pass where the terrain curves up in some directions and down in others.

But you might say: PyTorch shows my gradient is zero — shouldn’t that mean the optimization is done?
No, dear reader. No. It only means you’re balanced on a ridge, not that you’ve reached anything useful.

How common are saddle points?

Let’s do the math.

At any critical point, you can think of each eigenvalue as having a coin-flip chance of being positive or negative. For a true minimum, you need all of them positive. The rough probability? One-half raised to the power of your parameter dimension.

For a million parameters, that’s 1/ 2^(10⁶).

You have better odds of winning the lottery while being struck by lightning while a shark bites your leg. In high dimensions, almost every critical point is effectively a saddle. True minima are statistical miracles.

The good news: most saddle points are unstable. SGD’s inherent noise usually kicks you off eventually.

The bad news: “eventually” might be three weeks of wasted compute. Degenerate saddles — where eigenvalues hover near zero — create plateaus where the gradient whispers instead of speaks. Loss goes nowhere. You’re stuck, but you don’t know if you’re stuck or just slow.

δ > 0 would tell you instantly. One number. Saddle or not. But your framework doesn’t compute it.

The GIF below shows what this trap looks like:

The Gradient Flow Time Bomb

Oh, you thought we were done? It gets worse.

Your neural network isn’t one function. It’s a chain of functions — layer after layer after layer. And gradients have to flow backward through every single one during training. Like a game of telephone, except each person might whisper too quietly or scream too loud.

Each layer has a Jacobian matrix — the matrix of partial derivatives that governs how signals propagate. The singular values of these Jacobians determine whether your gradients survive the journey or die along the way.

Singular values > 1: The gradient gets amplified at each layer. By the time it reaches the early layers, it’s not a gradient anymore — it’s a bomb. Exploding gradients. Your weights go to infinity. Training crashes. NaN city.

Singular values < 1: The opposite disaster. The gradient gets squashed at each layer. By the time it reaches the early layers, it’s a rounding error. Vanishing gradients. Your early layers stop learning. They’re frozen while the rest of the network pretends to train.

Singular values ≈ 1: Goldilocks zone. Gradients flow cleanly from end to end. Every layer learns. This is why orthogonal initialization works. This is why spectral normalization exists.

But here’s the thing: these techniques were discovered by accident and applied as band-aids. Nobody monitors Jacobian spectra during training. Nobody watches the singular values drift. The diagnostic that would tell you: Layer 47 is about to kill your gradient flow 😁, simply doesn’t exist in any commercial framework.

Your network could be bleeding out internally, and the dashboard shows nothing.

See it for yourself:

And Here You Go: The Real Dashboard Your AI Framework Is Missing

This is the dashboard a serious AI framework should show you.

A mathematically literate training loop would compute lightweight spectral diagnostics at checkpoints and act on them:

• Hessian condition number exceeds a threshold → switch to a preconditioned method.

• Jacobian singular values drift away from 1 → apply spectral normalization.

• Negative eigenvalues appear → you’re at a saddle, perturb to escape.

Right now, no commercially available AI framework gives you these basic eigenvalue predictors. Instead, you’re hand-tuning learning rates and hoping…

Why Nobody Fixed This

Three reasons:

Scaling obscures mathematical sins. When throwing more compute at the problem works, nobody questions the foundations. This is temporary. Scaling laws plateau. When they do, the industry will suddenly need mathematics it never bothered to learn.

Disciplinary silos. The people who understand spectral theory work in applied math departments solving inverse problems. The people building AI frameworks took optimization and statistics. The Venn diagram overlap is nearly empty.

Abstraction debt. Implementing proper spectral monitoring requires infrastructure nobody wants to build. Everyone would benefit. Nobody wants to pay.

The Bill Comes Due

Here’s what’s missing in every production framework:

The industry has invested $100 billion in scaling AI.

The mathematical foundations remain incomplete.

Every unexplained training failure, every generalization anomaly, every model that worked in the lab and died in production — these are symptoms of mathematical pathologies that your tools cannot diagnose.

What You Can Do Now to go even beyond those big AI companies ?

Start here:

1. Learn what Hessian eigenvalues mean for your specific architecture

2. Monitor condition numbers during training — even crude estimates help

3. Check for sharpness before you ship — PyHessian exists, use it

4. Question everything when loss plateaus — you might be at a saddle, not a minimum

The math exists. It’s been waiting since 1950.

The only question is whether you’ll learn it before your next production failure teaches you the hard way.

ChatGPT confidently tells you about your dead grandmother’s favorite recipe. She’s not dead. You don’t have a grandmother named Ethel.

Your model? Trained for three weeks, beautiful loss curve, ships to prod, immediately falls on its face.

What the hell is going on?

Same root cause. Every single time.

Here’s the thing nobody tells you: there are three numbers that predict whether training will work or blow up in your face. Whether inference will be stable or hallucinatory. Whether your minimum is real or a trap.

Condition number. Ratio of largest to smallest Hessian eigenvalue. High means your optimizer is zigzagging through a canyon instead of descending a bowl.

Eigenvalue magnitude. Big eigenvalues = sharp minimum = your model memorized noise and will choke on real data. Small = flat minimum = actually learned something generalizable.

Negative eigenvalue count. If any are negative, you’re not at a minimum. You’re at a saddle point. Gradient says “we’re done here” but you’re stuck on a ridge.

This math has existed since 1950.

It’s not in PyTorch. Not in TensorFlow. Not in JAX. Not anywhere in your stack.

You know what your framework shows you? Loss value. Gradient. Learning rate.

That’s it. That’s the whole dashboard.

Why? Because the Hessian is huge — n² entries for n parameters. Except... you don’t need the full matrix. A Hungarian physicist named Cornelius Lanczos figured out how to extract the important eigenvalues in 1950. Linear time. Basically free.

Seventy-five years later, still not productized.

The industry scaled to $100 billion in compute. Nobody spent a weekend adding spectral diagnostics to the training loop.

I wrote the whole thing up: what the Hessian actually tells you, why frameworks ignore it, and the 75-year-old algorithms that could fix training failures before they happen.

→ The Missing Mathematics of AI

If you’ve ever watched a loss plateau for several hours wondering whether to kill the run or wait — this essay is the answer you didn’t have.

Send it to your ML team. Or don’t. Keep flying blind. Your call.

The Missing Mathematics of AI

Why Commercial AI Stacks Are Built on Incomplete Foundations

Jose Crespo, PhD

Modern AI frameworks—PyTorch, TensorFlow, JAX, and the ecosystem built upon them—implement a remarkably narrow slice of the mathematics available for learning systems. This article presents a rigorous analysis of the mathematical domains that govern inverse problems and identifies the specific gaps in commercial implementations. We argue that these omissions are not accidental but reflect deep structural blind spots in how the field conceptualizes machine learning. The consequences range from unexplained training instabilities to fundamental limitations in what current architectures can reliably learn.

1. The Taxonomic Position of Machine Learning

Before analyzing what commercial stacks lack, we must establish where machine learning sits within mathematics. This positioning is not merely academic: it determines which mathematical tools are appropriate and which pathologies are inevitable.

1.1 Machine Learning as Inverse Problem

Every supervised learning task has the same formal structure. Given observations y = F(θ, x) + ε, we seek parameters θ that explain the data. This is precisely the definition of an inverse problem: given the output of an operator, recover its input or parameters.

Jacques Hadamard formalized in 1923 the conditions under which such problems are well-posed: existence of a solution, uniqueness of that solution, and continuous dependence on the data. Virtually no machine learning problem satisfies all three conditions. Neural networks are overparameterized (non-unique solutions), loss landscapes have spurious minima (existence issues for global optima), and small perturbations to training data can produce radically different models (discontinuous dependence).

This classification has immediate consequences. Inverse problems require regularization: additional constraints that make the problem tractable. The entire history of successful regularization theory, from Tikhonov to total variation to Bayesian priors, applies directly to machine learning. Yet commercial stacks implement almost none of it.

1.2 The Hierarchical Structure of Relevant Mathematics

Inverse problems sit at the intersection of multiple mathematical domains, each contributing essential structure:

• Functional Analysis provides the framework for operators between infinite-dimensional spaces. Neural networks are approximations to such operators, and their properties—compactness, spectral decay, continuity—determine trainability.

• Operator Theory characterizes forward and inverse maps through their spectra. The singular value decomposition of linear operators generalizes to neural networks as the spectrum of the input-output Jacobian. Ill-conditioning manifests as clustered or decaying singular values.

• Differential Geometry describes the intrinsic structure of parameter spaces. Weight matrices often lie on manifolds (orthogonal groups, positive definite cones), and ignoring this structure forces optimization through unnecessarily difficult terrain.

• Category Theory provides the metalanguage for compositional structure. Forward operators are functors; inverse operators are adjoints. Composition of layers must respect diagram commutativity to preserve semantic meaning.

• Group Theory determines identifiability through symmetry analysis. Weight space symmetries (permutation invariance, scaling) create equivalence classes; learning should optimize over quotient spaces, not raw parameters.

• Probability Theory (Bayesian formulation) recasts regularization as prior specification. Every regularizer corresponds to a prior distribution; L2 regularization is a Gaussian prior, L1 is Laplacian, and geometric regularizers encode structured beliefs about parameter relationships.

2. Spectral Theory: The Invisible Scaffold

The most consequential omission in commercial AI stacks is spectral analysis. Every claim about generalization, stability, and trainability ultimately reduces to spectral properties of operators derived from the network architecture and data.

2.1 The Hessian Spectrum and Generalization

Consider the loss function L(θ) as a functional on parameter space. Its second derivative, the Hessian H = ∇²L, encodes complete local curvature information. The spectrum of H—its eigenvalues {λᵢ}—determines fundamental properties of training dynamics and learned solutions.

What Eigenvalues Reveal

Condition Number and Training Stability. The condition number κ = λₘₐₓ/λₘᵢₙ is the canonical measure of optimization difficulty. For gradient descent, the convergence rate scales as (κ−1)/(κ+1) per iteration. High κ means the loss landscape is “elongated”—steep in some directions, flat in others—forcing zigzag trajectories that waste computational effort.

This is not merely theoretical. The edge of stability phenomenon, empirically documented across architectures, confirms that gradient descent naturally finds and hovers at λₘₐₓ ≈ 2/η (where η is the learning rate). The optimizer self-organizes to the boundary of stability defined by the largest eigenvalue.

Important caveat: In practice, κ can be astronomically large (10¹⁴ or more in some studies), yet training still succeeds. This apparent paradox resolves because SGD noise effectively “regularizes” the spectrum: stochasticity helps escape ill-conditioned directions. The condition number constraint applies most strictly to full-batch gradient descent; mini-batch SGD operates under a softer regime.

Eigenvalue Distribution and Minimum Geometry. The bulk of the Hessian spectrum characterizes the geometry of the solution. Empirical studies (Sagun et al., 2016; Ghorbani et al., 2019) reveal a consistent bulk + outliers structure: most eigenvalues cluster near zero, with a few large positive outliers.

• Many small eigenvalues indicate a flat minimum—low curvature in most directions means the loss surface resembles a plateau rather than a sharp valley. Such minima tend to be stable under parameter perturbation and correlate empirically with better generalization.

• Many large eigenvalues indicate a sharp minimum, high curvature means small parameter changes produce large loss changes. Such minima correlate with poorer generalization, though the causal mechanism remains debated.

Theoretical nuance: The flatness-generalization connection is empirically robust but theoretically contested. Dinh et al. (2017) demonstrated that reparametrization can change eigenvalues without changing the function: a “sharp” minimum can be made “flat” by rescaling weights. The principled justification comes through PAC-Bayes bounds: small eigenvalues mean the loss changes little under parameter perturbation, yielding tighter generalization bounds. The correlation is real; the mechanism is subtle.

Negative Eigenvalues and Saddle Points. At any critical point (where ∇L = 0), eigenvalue signs classify the point’s nature (See Figure 2 above):

• All positive → local minimum

• All negative → local maximum

• Mixed signs → saddle point

In high-dimensional spaces, saddle points dominate overwhelmingly. A simple Bernoulli argument suggests the probability of a true minimum (all eigenvalues positive) decreases exponentially with parameter count. For networks with millions of parameters, essentially all critical points are saddle points:which is actually good news, since saddle points are dynamically unstable and SGD escapes them through noise.

Degeneracy at saddles: Most eigenvalues at practical saddle points cluster near zero (the degenerate case), with only a few significantly positive or negative. The “perfect saddle” with equal magnitude positive and negative eigenvalues is rare. This degeneracy creates plateaus( flat regions where gradients nearly vanish) explaining the loss stagnation commonly observed during training.

The Role of Batch Normalization

Batch normalization dramatically reshapes the eigenvalue structure. In non-normalized networks, large isolated eigenvalues (outliers) appear rapidly during training, and the gradient concentrates in corresponding eigenspaces. Batch normalization compresses these outliers and distributes the gradient more uniformly across the spectrum.

This is the spectral explanation for why batch normalization helps optimization: it reduces the condition number κ by compressing eigenvalue spread, enabling larger learning rates and more stable training. The internal covariate shift explanation originally proposed has been empirically challenged; the spectral explanation has held up.

No commercial framework computes, monitors, or exploits Hessian spectra during training. The Hessian is an n × n matrix for n parameters: billions of entries for modern networks. But spectral information doesn’t require the full matrix. Lanczos iteration, Hutchinson’s trace estimator, stochastic Lanczos quadrature, and randomized SVD can extract dominant eigenvalues, spectral density estimates, and trace with O(n) complexity per iteration.

2.2 The Jacobian Spectrum and Layer Health

Each layer defines a map fₗ: ℝ^(dₗ₋₁) → ℝ^(dₗ). Its Jacobian Jₗ = ∂fₗ/∂x governs gradient flow. The singular values of Jₗ determine whether gradients explode, vanish, or propagate stably through the layer.

The product of layer Jacobians gives the end-to-end gradient flow:

Jtotal=JL⋅JL−1⋅…⋅J1

If any layer has singular values consistently above 1, gradients explode exponentially with depth. If any layer has singular values consistently below 1, gradients vanish. Stable training requires singular values clustered near 1—a condition equivalent to each layer approximately preserving norm, which is precisely the definition of an isometry.

Orthogonal and unitary weight matrices achieve exact isometry. This is not a coincidence: it’s why orthogonal initialization works and why techniques like spectral normalization (constraining σₘₐₓ(W) ≤ 1) stabilize training. Yet these interventions are ad hoc patches applied without exposing the underlying spectral framework to practitioners.

2.3 The Picard Condition: When Inversion Fails

Inverse problems have a precise criterion for solvability. Given a compact operator K with singular value decomposition, the inverse exists and is bounded if and only if the Picard condition holds: the Fourier coefficients of the data (projected onto singular vectors) must decay faster than the singular values.

Translated to neural networks: if the data contains high-frequency components that the network’s effective kernel cannot represent (because corresponding singular values are too small), training will either fail to fit or produce unstable solutions that overfit to noise.

This is the mathematical explanation for several empirical observations: why networks struggle with high-frequency functions (spectral bias), why adversarial examples exist (inverting small singular values amplifies perturbations), and why certain architectures work better than others (their spectral properties match the data statistics).

3. Differential Geometry: The Curved Landscape

Commercial AI stacks treat parameter space as flat Euclidean space. Every optimizer—SGD, Adam, AdaGrad—assumes gradients live in ℝⁿ with the standard inner product. This assumption is mathematically false and practically costly.

3.1 The Intrinsic Geometry of Weight Spaces

Consider the constraints that weight matrices naturally satisfy:

• Orthogonal matrices (WᵀW = I) form the Stiefel manifold V(n,k), a curved subspace of ℝⁿˣᵏ. Enforcing orthogonality via Gram-Schmidt or QR decomposition after each gradient step is geometrically naïve—it projects onto the manifold rather than moving along it.

• Symmetric positive definite matrices (covariance matrices, kernel matrices) form a Riemannian manifold with non-Euclidean metric. The natural distance between covariances is the Fisher-Rao metric, not Frobenius norm.

• Probability simplices (softmax outputs, attention weights) have natural geometry given by the Fisher information metric. Euclidean gradients on logits induce non-uniform steps in probability space.

Ignoring intrinsic geometry forces optimization to work harder. Euclidean gradient descent in a curved space follows coordinate curves, not geodesics. The path from initial to optimal parameters becomes unnecessarily long, and the effective learning rate varies wildly across the parameter manifold.

3.2 Natural Gradient: The Correct Update

Shun-ichi Amari introduced natural gradient descent in 1998, recognizing that the parameter space of probability distributions has intrinsic Riemannian structure given by Fisher information. The natural gradient is:

θt+1=θt−η⋅G(θt)−1⋅∇L(θt)

where G(θ) is the Fisher information matrix (or more generally, the Riemannian metric tensor). This update moves in the direction of steepest descent in the geometry of the manifold, not in ambient Euclidean space (!)

Natural gradient has remarkable properties: invariance to reparameterization (the same curve regardless of coordinate system), optimal convergence in the neighborhood of minima (achieves the Cramér-Rao bound), and automatic adaptation to curvature.

3.3 Curvature Traps and Holonomy

The Riemann curvature tensor R measures how parallel transport around a closed loop fails to return to the starting vector. In parameter space, this manifests as holonomy: the accumulated rotation of the gradient direction as optimization traverses curved regions.

High curvature regions are training pathologies. The optimizer enters a curved tube where gradients point along the tube rather than toward the minimum. Progress stalls until random perturbations or momentum carry the trajectory out. Loss plateaus during training—universally observed but rarely explained—are geometric phenomena: the optimizer is navigating high-curvature submanifolds of parameter space.

Holonomy-aware optimization would detect these traps via curvature monitoring and apply geodesic corrections. No commercial stack implements this. The closest analogue—gradient clipping—is a crude intervention that clips magnitude without understanding geometry.

4. Category Theory: The Missing Compositional Semantics

Neural networks are compositional: layers compose to form networks, networks compose to form systems, systems compose to form agents. Yet composition in commercial stacks is purely syntactic: connect layer A to layer B if dimensions match. There is no semantic guarantee that composition preserves properties.

4.1 Functors and Adjunctions

Category theory provides the natural language for compositional systems. A category consists of objects and morphisms (arrows between objects) with associative composition. Neural network layers are morphisms: they map input spaces to output spaces.

A functor is a structure-preserving map between categories. The forward operator of a neural network is a functor from the category of data spaces to the category of representation spaces. It preserves composition: applying layer f then layer g is the same as applying the composition g ∘ f.

The inverse operator—backpropagation—is the adjoint functor. In the category of vector spaces with linear maps, the adjoint is transposition. For differentiable functions, the adjoint is the chain rule applied in reverse. Adjoint functors satisfy a universal property: they are the best possible inverses given structural constraints.

This perspective immediately explains why backpropagation works and suggests how to fix it when it doesn’t. If forward and adjoint functors don’t satisfy the adjunction equations (because of numerical error, non-differentiability, or approximation), gradient estimates become biased. Checking adjunction conditions could diagnose training failures before they manifest as divergence.

4.2 Natural Transformations and Equivariance

A natural transformation is a morphism between functors that commutes with their action. In neural networks, this appears as equivariance: if F is the forward map and G is a symmetry group action, then F(G·x) = G·F(x) means F commutes with symmetry.

Convolutional networks are equivariant to translation. Graph neural networks are equivariant to permutation. E(3)-equivariant networks (used in molecular dynamics) are equivariant to rotation and reflection. These are all instances of natural transformations in the categorical sense.

The categorical framework suggests a design principle: specify the symmetries (group actions) your model should respect, then construct the unique functor (network architecture) that is natural with respect to those symmetries.

4.3 Diagram Commutativity and Agent Orchestration

Consider a multi-agent system where agent A processes data and passes results to agent B, which queries external tools and returns to A. This is a diagram:

A→B→Tool→B→A

Diagram commutativity means: any two paths through the diagram with the same start and end points produce the same result.

If the diagram doesn’t commute, the system has race conditions, order dependence, or semantic inconsistency.

LangChain, AutoGPT, and similar agent frameworks compose operations with no commutativity guarantees. Tool calls may return different results depending on context accumulated from previous calls.

Memory updates may interfere with reasoning chains. The absence of categorical semantics makes these systems fundamentally unpredictable.

A categorically-grounded agent framework would specify diagrams declaratively and verify commutativity (or explicitly mark non-commuting squares as stateful operations). This is standard practice in programming language semantics and database theory; its absence from AI agent design is a mathematical oversight.

5. Regularization Theory: Beyond Weight Decay

Commercial stacks implement exactly two regularizers: L1 (Lasso) and L2 (Ridge/weight decay). Both are special cases of Tikhonov regularization with Φ(θ) = ||θ||ₚ. The space of possible regularizers is vast; these two points cannot represent it.

5.1 Geometric Regularizers

Regularizers can encode geometric structure:

• Total variation: ||∇θ||₁ penalizes parameter variation, promoting piecewise-constant solutions. Essential for image reconstruction, unused in neural network training.

• Sobolev penalties: ||∇ᵏθ||₂ penalizes high-frequency components, promoting smooth solutions. Controls the spectral bias toward low frequencies.

• Geodesic regularization: penalizes deviation from geodesics in parameter space. Promotes solutions that respect the intrinsic geometry of weight manifolds.

• Curvature constraints: bound the Riemann curvature of learned representations. Prevents the pathological geometries that cause training instability.

5.2 Spectral Regularizers

Regularizers can directly constrain spectra:

• Nuclear norm: ||W||* = Σσᵢ (sum of singular values) promotes low-rank solutions. The convex relaxation of rank minimization.

• Spectral norm: ||W||₂ = σₘₐₓ bounds the largest singular value, guaranteeing Lipschitz continuity. Spectral normalization approximates this.

• Condition number penalty: κ(W) = σₘₐₓ/σₘᵢₙ promotes well-conditioned layers. Directly addresses gradient flow stability.

• Spectral gap constraint: requires separation between leading eigenvalues. Ensures learned representations have clear low-dimensional structure.

5.3 The Bayesian Interpretation

Every regularizer corresponds to a prior distribution via the relation

Φ(θ) = −log p(θ).

This correspondence is exact:

The Bayesian perspective reveals that regularization is not merely a computational trick: it encodes beliefs about the structure of solutions. Choosing L2 asserts that parameters are normally distributed around zero; choosing L1 asserts that most parameters should be exactly zero. These are strong claims that may or may not match the actual structure of the problem.

6. Identifiability and Symmetry

A fundamental question precedes optimization: is the solution unique? If multiple parameter settings produce identical input-output behavior, which one will optimization find, and does it matter?

6.1 Weight Space Symmetries

Neural networks have intrinsic symmetries that create equivalence classes in parameter space:

• Permutation symmetry: For any hidden layer with n neurons, there are n! equivalent networks obtained by permuting neurons (and correspondingly permuting incoming and outgoing weights). A network with layers of width [784, 256, 128, 10] has 256! × 128! ≈ 10⁵⁰⁷ equivalent parameterizations.

• Scaling symmetry: For ReLU networks, if we multiply incoming weights to a neuron by α > 0 and divide outgoing weights by α, the function is unchanged. This creates a continuous family of equivalent solutions.

• Sign symmetry: For networks with tanh activation, flipping signs of incoming and outgoing weights simultaneously preserves the function. This doubles the number of equivalences.

These symmetries have consequences. Gradient descent in the full parameter space wastes capacity exploring equivalent solutions. Loss landscapes appear more complex than they are: many apparent local minima are actually the same solution in different coordinates. Model comparison (asking whether two trained networks learned the same function) requires symmetry-aware distance metrics.

6.2 Quotient Manifold Optimization

The principled solution is to optimize over the quotient space Θ/G, where G is the symmetry group. Points in the quotient represent equivalence classes of parameters, not individual parameter vectors.

Quotient manifold optimization is well-developed in Riemannian geometry. Algorithms exist for computing geodesics, parallel transport, and gradients in quotient spaces. These techniques are standard in computer vision (optimization over rotation groups) and signal processing (optimization over subspace arrangements). Their absence from neural network training is an implementation gap, not a theoretical one.

6.3 Symmetry Breaking for Identifiability

An alternative to quotient optimization is explicit symmetry breaking: adding constraints that select a unique representative from each equivalence class.

• Ordering constraints: require that neurons within a layer are ordered by some criterion (e.g., decreasing norm of incoming weights). Eliminates permutation symmetry.

• Normalization constraints: require that incoming weight vectors have unit norm. Eliminates scaling symmetry but changes the effective architecture.

• Sign conventions: require that the first nonzero element of each weight vector is positive. Eliminates sign symmetry.

These constraints are rarely applied because commercial frameworks don’t surface them as options. The result is that optimization explores a space orders of magnitude larger than necessary, and model comparison requires expensive alignment procedures that could be eliminated by construction.

7. The Gap Analysis: Commercial Stacks vs. Mathematical Necessity

We now have the vocabulary to perform a precise gap analysis. The following table maps mathematical requirements to their implementation status in major commercial frameworks.

8. Consequences of the Gaps

These mathematical omissions are not academic concerns. They manifest as concrete failures in production systems.

8.1 Training Instability

Without spectral monitoring, training failure is detected only after it happens—when loss diverges or gradients explode. The warning signs (deteriorating condition numbers, clustering eigenvalues, growing spectral radius) are invisible. Practitioners resort to learning rate schedules and gradient clipping as blind interventions, tuning hyperparameters without understanding what they control.

A spectrally-aware training loop would compute lightweight spectral estimates at checkpoints and adjust optimization strategy proactively. If the Hessian’s condition number exceeds a threshold, switch to preconditioned methods. If Jacobian singular values drift from unity, apply spectral normalization.

These are well-understood techniques applied reactively after trial and error; they should be applied automatically based on measured quantities.

8.2 Unexplained Generalization Failures

Models that achieve low training loss sometimes fail catastrophically on held-out data. Without geometric analysis, this failure is mysterious. With geometric analysis, the explanation is often clear: the model found a sharp minimum (large Hessian eigenvalues) that doesn’t generalize, or the learned representation has pathological curvature that amplifies distribution shift.

Sharpness-aware minimization (SAM) addresses this partially by explicitly seeking flat minima. But SAM is a heuristic; the principled approach is to constrain optimization to remain in regions of bounded curvature throughout training, not merely to prefer flatness at convergence.

8.3 Adversarial Vulnerability

Adversarial examples exist because neural networks invert small singular values, amplifying imperceptible input perturbations into large output changes. This is the Picard condition in reverse: the network’s effective operator has spectral components that grow rather than decay.

Lipschitz constraints (bounding the spectral norm of each layer) provably limit adversarial vulnerability. But current implementations apply these constraints layer-by-layer without accounting for how layer spectra combine through composition. The end-to-end Lipschitz constant can exceed the product of layer constants due to alignment of principal directions—a phenomenon invisible without compositional spectral analysis.

8.4 Agent Incoherence

Multi-agent systems built on LangChain or similar frameworks exhibit emergent pathologies: circular reasoning, inconsistent tool use, memory corruption, and semantic drift over long interactions. These are not bugs in the traditional sense: they are consequences of composing operations without compositional semantics.

A categorically-grounded agent framework would specify which compositions must commute and verify commutativity or fail explicitly. The current approach is to hope that empirical testing catches inconsistencies: a strategy that fails for the long tail of rare but catastrophic interaction patterns.

9. Why the Gap Persists: Structural Explanations

The mathematics described in this paper is not new. Inverse problems, Riemannian optimization, and category theory have decades of development. Why hasn’t this knowledge transferred to AI practice?

9.1 Scaling Obscures Mathematical Sins

The empirical success of large language models creates a selection effect against mathematical rigor. If brute-force scaling solves problems that principled methods would also solve, and scaling is funded while principled methods are not, the field optimizes for scale.

This is a temporary reprieve. Scaling laws plateau. Hardware improvements slow. Eventually the field will need to extract more capability from fixed compute budgets, and mathematical efficiency will matter. The question is whether the mathematical infrastructure exists when it’s needed.

9.2 Disciplinary Silos

Inverse problems expertise resides in applied mathematics departments, typically in the context of medical imaging, geophysics, or signal processing. Differential geometry expertise resides in pure mathematics or theoretical physics. Category theory expertise resides in logic, programming language theory, or pure algebra. None of these communities have significant overlap with machine learning engineering.

PhD programs in machine learning train students in optimization, statistics, and software engineering. The mathematical prerequisites do not include functional analysis beyond basic Hilbert spaces, Riemannian geometry beyond the definition of a manifold, or category theory at all. The expertise gap is structural.

9.3 Abstraction Debt

Implementing Riemannian optimization requires abstractions that don’t exist in commercial frameworks: manifold objects, tangent space representations, Riemannian metrics, parallel transport operators, geodesic solvers. Building these on top of tensor libraries is possible but requires invasive modifications that break existing workflows.

Category theory is worse: no mainstream programming language has native support for functors, natural transformations, or diagram verification. Implementing categorical structure requires either embedding in a dependently-typed language (impractical for production) or building custom verification tools (expensive and maintenance-intensive).

The abstractions that would make principled methods practical require infrastructure investment that no individual company has incentive to provide. This is a coordination problem: everyone would benefit from the infrastructure, but no one wants to build it.

9.4 Incentive Misalignment

Academic incentives reward novelty over foundations. Papers that introduce new spectral monitoring methods for existing architectures are less publishable than papers that introduce new architectures. The mathematical scaffolding is “not novel enough” for top venues, so researchers avoid building it.

Industry incentives reward shipping over correctness. A product that works 90% of the time is shippable; a principled system that works 95% of the time but takes twice as long to build is not. The 5% improvement doesn’t justify the investment, even though the failure modes in the remaining 10% vs 5% may be qualitatively different.

10. Toward a Mathematically Complete AI Stack

What would a mathematically principled AI framework look like? This section sketches the requirements without claiming implementation details.

10.1 Spectral Dashboard

Every training run should expose real-time estimates of: Hessian condition number and spectral density, layer-wise Jacobian singular value distributions, end-to-end Lipschitz constant estimates, and Picard condition violations (where data has components in spectral directions the model cannot represent).

These quantities should trigger automatic interventions: preconditioner updates when conditioning deteriorates, spectral normalization when singular values drift, architecture modifications when Picard violations persist.

10.2 Geometric Optimization Engine

The optimizer should understand parameter manifolds natively. Weight matrices with orthogonality constraints should be updated via Riemannian gradient descent on the Stiefel manifold. Probability distributions should be updated via natural gradient with Fisher-Rao metric. Quotient structures from symmetries should be handled automatically.

The API should accept manifold specifications declaratively: “these parameters lie on SO(n)” or “these parameters are equivalent up to permutation.” The optimizer should compute appropriate geodesics, retractions, and parallel transport without user intervention.

10.3 Regularization Library

Beyond L1/L2, the framework should provide: total variation and higher-order Sobolev penalties, nuclear norm and general Schatten p-norms, geodesic regularization for manifold-valued parameters, curvature bounds on learned representations, spectral constraints (condition number bounds, spectral gap requirements), and structured sparsity patterns from group theory (block-sparse, hierarchical-sparse).

Each regularizer should have an associated Bayesian interpretation exposed to the user, making explicit the prior assumptions encoded by the choice.

10.4 Compositional Agent Framework

Agent orchestration should be specified as diagrams with explicit commutativity requirements. The framework should verify at construction time that specified diagrams commute (for deterministic operations) or fail with informative errors explaining which paths diverge.

Stateful operations should be explicitly marked and tracked through composition. Memory updates should preserve declared invariants. Tool calls should satisfy contracts specified in typed interfaces with semantic content, not just syntactic shape.

11. Conclusion

Commercial AI frameworks implement a fragment of the mathematics that governs learning systems. The omissions are not random; they cluster around the mathematical domains that address instability, ill-posedness, and compositional semantics: precisely the domains needed to make AI systems reliable.

This article has catalogued the gaps: spectral theory for stability and generalization, differential geometry for optimization on curved spaces, category theory for compositional semantics, group theory for identifiability, and regularization theory for well-posedness. Each gap corresponds to a class of unexplained failures in current systems.

The gaps persist due to structural factors: scaling success that obscures mathematical necessity, disciplinary silos that prevent knowledge transfer, abstraction debt that makes principled methods impractical, and incentives that reward shipping over correctness.

Closing these gaps requires coordinated investment in mathematical infrastructur: the kind of investment that benefits everyone but no one wants to fund. The alternative is to continue building on incomplete foundations, accepting that certain classes of failures are endemic rather than solvable.

For practitioners: the immediate implication is that unexplained training failures, generalization anomalies, and agent incoherence are often symptoms of mathematical pathologies that existing tools cannot diagnose. The path forward is to develop fluency in the missing mathematics and apply it to understand (if not yet fix) the systems we build.

For the field: the call is to recognize that inverse problems are the natural mathematical framing for machine learning, and to import the century of theory developed for inverse problems into AI practice. The mathematics exists. The engineering challenge is to make it practical.

References

Hadamard, J. (1923). Lectures on Cauchy’s Problem in Linear Partial Differential Equations. Yale University Press.

Tikhonov, A. N., & Arsenin, V. Y. (1977). Solutions of Ill-Posed Problems. Winston & Sons.

Amari, S. (1998). Natural gradient works efficiently in learning. Neural Computation, 10(2), 251-276.

Engl, H. W., Hanke, M., & Neubauer, A. (2000). Regularization of Inverse Problems. Springer.

Absil, P. A., Mahony, R., & Sepulchre, R. (2008). Optimization Algorithms on Matrix Manifolds. Princeton University Press.

Stuart, A. M. (2010). Inverse problems: A Bayesian perspective. Acta Numerica, 19, 451-559.

Sagun, L., Bottou, L., & LeCun, Y. (2016). Eigenvalues of the Hessian in deep learning: Singularity and beyond. arXiv:1611.07476.

Dinh, L., Pascanu, R., Bengio, S., & Bengio, Y. (2017). Sharp minima can generalize for deep nets. ICML.

Ghorbani, B., Krishnan, S., & Xiao, Y. (2019). An investigation into neural net optimization via Hessian eigenvalue density. ICML.

Yao, Z., Gholami, A., Keutzer, K., & Mahoney, M. W. (2020). PyHessian: Neural networks through the lens of the Hessian. IEEE Big Data.

Bronstein, M. M., Bruna, J., Cohen, T., & Veličković, P. (2021). Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges. arXiv:2104.13478.

Cohen, J., Kaur, S., Li, Y., Kolter, J. Z., & Talwalkar, A. (2021). Gradient descent on neural networks typically occurs at the edge of stability. ICLR.

TL;DR

Modern AI desperately needs second derivatives where it actually hurts- self-driving, medical models, finance - but the usual way of getting them is so expensive and fragile you can’t use them on billion-parameter systems without everything collapsing under compute and tooling overhead.

This article shows how to fix that with dual/jet algebra + compositional JVP ∘ VJP:

• Hyperreals - think derivatives as ratios of infinitesimals, not limits-to-zero exam tricks

• Duals & jets - computable infinitesimals: ℝ[ε]/(ε²) → duals , ℝ[ε]/(εᵏ⁺¹) → jets, where derivatives and higher orders are exact algebraic coefficients, not finite-difference noise.

• Compositional autodiff - treat jvp and vjp as algebraic building blocks so JVP ∘ VJP gives you real second-order operators (H·v, local Taylor models) for roughly the cost of a couple of gradients, instead of O(n²) Hessian madness.

Yeah, PyTorch and JAX have grad, jvp, vjp.
But in practice, the overhead and tooling pain mean almost nobody uses second-order AD seriously at scale - it all gets treated like a give me a gradient button, not the dual/jet geometry engine it could be.

Payoff: With this approach, “normally expensive” second derivatives (HVPs) behave like cheap first derivatives, even on billion-parameter models - so you finally get real geometry and real mathematical guarantees to attack the actual hard problems: self-driving glitches, brittle medical models, financial blow-ups. No more hallucinations in first-order PyTorch/JAX vibes.

Why Your Calculus Is Lying to You

The limit-based calculus currently used in AI operates entirely in ℝ and thus fakes infinitesimals.

Cauchy and Weierstrass screwed up how we analyze continuous functions by replacing infinitesimals with limits:

f’(x) = lim[h→0] (f(x+h) - f(x))/h

Translation: “We can’t handle actual infinitesimals, so let’s pretend we’re getting close enough.”

In the 1800s? Sure, brilliant workaround. In 2025, when you’re computing derivatives across billions of parameters per second? Catastrophically obsolete.

Don’t believe me? Look at the plot below - and this is just a toy example. Real models? Much worse: absurd oscillatory calculus instead of the direct, easy solution you’ll learn in this story(dual numbers).

The limit-based calculus currently used in AI operates entirely in ℝ and thus fakes infinitesimals.

Cauchy and Weierstrass screwed up how we analyze continuous functions by replacing infinitesimals with limits:

f’(x) = lim[h→0] (f(x+h) - f(x))/h

Translation: “We can’t handle actual infinitesimals, so let’s pretend we’re getting close enough.”

In the 1800s? Sure, brilliant workaround. In 2025, when you’re computing derivatives across billions of parameters per second? Catastrophically obsolete. See the plot above

Here’s the irony: we’re finally circling back to what Newton and Leibniz actually wanted.

• Newton: Fluents/fluxions — calculus as flowing quantities with instantaneous rates

• Leibniz: True infinitesimals — algebra that captures the “infinitely small”

Both were right. The limit-based detour was a temporary hack because 19th-century mathematicians couldn’t rigorously define infinitesimals yet.

Three centuries later, YES WE CAN.

Hyperreals give infinitesimals a rigorous foundation. On machines, we use their finite engines — dual and jet algebras — to carry derivatives (and curvature) directly through computation.

No limits. No approximations. No lies.

This is calculus the way it was supposed to work, before we compromised because the math wasn’t ready yet.

Now it is. And your GPU deserves better than 200-year-old workarounds.

The Showdown: True Infinitesimals vs. Fake Limits

Time to step onto solid ground — number theory and algebra that actually work — and leave behind the 19th-century limit-based theater that’s been running your AI into the ground.

Forget the flat number line of the Real Numbers ( ℝ ).

That line was the foundation of the Archimedean empire — a world where everything had to be measured by stacking finite steps, where “limits” were invented to fake what infinitesimals actually are.

We don’t have to pretend anymore.

Welcome to the Hyperreal Numbers Universe (∗ℝ) — the long-overdue replacement for the broken real number system your AI is currently dying on.

Why Reals Can’t Cut It (And Why Your Models Keep Exploding)

The Real Numbers (ℝ) can’t represent infinitesimals or true infinities.

They flatten everything into one fragile, linear scale. No room for the infinitely small. No room for the infinitely large. Just a tight little rope you’re forced to balance on.

And that’s exactly why AI keeps breaking:

• Gradients vanish → model stops learning

• Values explode → NaN city, population: your training run

• You’re stuck oscillating between “too small to matter” and “too big to handle”

The Hyperreal field (∗ℝ) fixes this catastrophe.

It restores the full spectrum of scale — from the infinitesimally small (ε) to the infinitely large (ω) — and gives calculus back its missing architecture.

Stop for a moment and picture that:

A number system wide enough to handle every scale that nature — and intelligence — can throw at it.

Not “close enough.” Not “approximate.” Actually there.

That’s what your GPU deserves. That’s what hyperreals deliver.

The Galaxy View: Understanding Hyperreals Visually

The hyperreal universe isn’t a number line — it’s a galaxy cluster.

Each disk is an entire realm of magnitude, separated not by distance but by mathematical impossibility. You can’t walk from one to another. You either have access or you don’t.

Meet the three galaxies your AI needs:

Galaxy ε (left) — The Infinitesimal Cloud
Numbers smaller than any real, yet still not zero. This is where dual numbers live. This is where exact derivatives happen. Your current AI can’t access this galaxy… it fakes it with limits.

Galaxy ℝ (middle) — The Finite Realm
All ordinary reals and their infinitesimal halos (the Monads of 1, π, e, …). Your AI is trapped here, pretending the other galaxies don’t exist. Every real number pulses with its own infinitesimal cloud, values infinitely close but never identical.

Galaxy ω (right) — The Infinite Realm
The domain no finite number can ever reach. Unbounded scale without explosions. Asymptotic behavior you can actually use.

Critical insight: The dashed axis isn’t a ruler. It’s a logical ordering, not a geometric distance.

You can’t slide from ε to 1 by taking small steps. You can’t reach ω by adding big numbers. The Archimedean ruler — the idea that everything can be measured by stacking finite pieces — breaks here.

Geometry collapses. Only hierarchy survives.

This is why limit-based calculus fails: it tries to reach infinitesimals by making finite numbers smaller (h → 0). But infinitesimals aren’t “really small reals”… they’re a different galaxy entirely.

Hyperreal algebra gives you direct access to all three. No approximation. No pretending.

Your GPU deserves to compute in the full universe, not just the middle ring.

But That Row of Galaxies Was Only a Teaser

It shows where each realm sits, but not how they connect.

The Hyperreal Universe isn’t flat — it’s layered. Think less “number line,” more “onion of scales,” each world wrapped inside another.

Here’s the main idea: those distances mean nothing.

You can’t slide from ε to 1, or from 1 to ω, by taking steps. The Archimedean ruler — the idea that everything can be reached by adding enough finite pieces — breaks here completely.

Geometry collapses. Only order remains.

Chart 3 reveals the truth.

If Chart 2 showed the map, Chart 3 reveals the laws of hyperreal gravity — how the entire Hyperreal Universe holds together around its silent core: 0.

The galaxies stop lining up and start nesting. The infinitesimal cloud (Galaxy ε) sits at the center, surrounded by the finite realm (Galaxy ℝ), which itself is engulfed by the infinite galaxies (ω, ω², …)

Hyperreals in Action for Programmers

We’ve seen the shape of the hyperreal universe. Pretty galaxies, pulsating halos… all very philosophical.

Now let’s make it compute.

To turn that geometry into usable algebra, we need something programmable, differentiable, and verifiable. We must formalize what lives inside each “galaxy” of numbers.

1. The Definition That Newton Dreamed Of

Work inside ∗ℝ: the real numbers extended with non-zero infinitesimals.

Here’s the compact definition that took mathematics three centuries to rediscover!

ε ∈ ∗ℝ  such that  0 < |ε| < 1/n  for all n ∈ ℕ

0 limits. No guessing. Just a number smaller than any fraction of reals: a true infinitesimal.

This single idea resurrects Newton’s intuition and fixes the broken calculus that modern AI still runs on.

2. The Taxonomy of Fields Inside the Hyperreal Superset

(Or: the four number systems every serious AI engineer should secretly be using)

Let’s get practical.

Behind all that galaxy talk, we’re really dealing with four kinds of numbers; the ones that can actually change the way you build and train AI.

Each of these number systems — real, hyperreal, dual, jet — forms its own algebraic habitat, obeying different group, ring, and field laws. They’re not abstract nonsense; they’re layers of algebra that shape how your program handles motion, precision, and learning.

Think of them as four habitats in one mathematical ecosystem, each with its own physics: from the plain reals we’ve always used, to the hyperreals that define infinitesimals, down to the duals and jets that finally make all this computable.

One Universe, Four Number Forces

At the center sits the real numbers (ℝ) — the baseline arithmetic of every model and matrix multiply.

Extend them once, and you get the hyperreals (∗ℝ), the true mathematical universe that contains both infinitesimals and infinities.

Project a simplified, finite version of that same idea, and you arrive at the dual numbers (𝔻) and jets (Jᵏ) — the algebraic workhorses that make automatic differentiation possible.

In the set-theoretic view:

ℝ ⊂ ∗ℝ
ℝ ⊂ 𝔻
ℝ ⊂ Jᵏ

All four live in the same universe of structure… but their rules differ radically.

To Be or Not to Be Nilpotent

Here’s where the roads split.

Hyperreals (∗ℝ)

A field, perfectly ordered, no zero divisors.

That means no nilpotents — no “magic” numbers that vanish when squared or cubed.

Their infinitesimals ε are real elements in the algebraic sense:

ε² ≠ 0,  ε³ ≠ 0,  εⁿ ≠ 0  for all n

They’re tiny — smaller than any real fraction — but never zero.

Duals (𝔻) and Jets (Jᵏ)

These are rings, not fields, and they do contain nilpotents — elements that die after a few powers.

For duals:  ε² = 0
For jets:   εᵏ⁺¹ = 0

These nilpotents are algebraic doppelgängers of infinitesimals: not “real” in a field-theoretic sense, but powerful tools for derivatives. They gauge infinitesimal behavior just enough to automate calculus inside your code.

So the fact that nilpotents exist in duals/jets but are forbidden in the hyperreals is the deep structural divide.

• Hyperreals: true infinitesimals, no nilpotents

• Duals & Jets: nilpotent gauged infinitesimals, no field structure

Standard Part vs. Projection — How They “Return” to Reals

Now, how do we come back to normal reals after playing in these extended worlds?

In the hyperreals, we use the standard-part map:

st: ∗ℝ → ℝ

It collapses the infinitesimal halo:

st(x + ε·δ) = x  for any finite x and infinitesimal δ

That’s the real number your hyperreal was infinitely close to.

In duals and jets, there’s no notion of infinitely close, because these are algebraic rings, not ordered fields. So we use the simpler projection:

π₀: 𝔻 → ℝ  or  π₀: Jᵏ → ℝ

In plain English: just grab the value lane and ignore the derivative lanes.

Programmer shortcut:

• st(·) → “take the real shadow” (hyperreals)

• π₀(·) → “take the first component” (duals/jets)

Summary:

In ∗ℝ, infinitesimals are real field elements, with true proximity and a genuine standard part.

In 𝔻 and Jᵏ, ε-parts are nilpotent coefficients that vanish by projection — no infinite closeness, just algebraic bookkeeping.

Why This Matters for AI

Hyperreals give us the ideal semantics of calculus: the way derivatives and limits were meant to behave before we replaced them with fragile ε–δ tricks.

Duals and jets give us the machine algebra to compute those derivatives exactly, without symbolic headaches or numerical jitter.

When you lift a real input x into a jet:

x̂ = x + d₁·ε + (d₂/2!)·ε² + ...

and run your usual forward pass, you’re literally running your program over a truncated Taylor ring.

At the end, you project back:

• value in lane 0

• gradient in lane 1

• curvature in lane 2

• and so on

That’s how real code gets exact calculus.

So here’s the simple rule of thumb:

• Hyperreals explain calculus

• Duals and jets compute it

The first gives meaning, the second gives power.

Algebra: The Rules to Play the AI Game with Hyperreals

Alright, time to stop orbiting theory and touch the ground.

We’ve mapped the Four Number Forces: Reals, Hyperreals, Duals, and Jets. Now we build the algebra that actually makes them move.

This is where abstract structure turns into computation.

The Hyperreals give you the truth of calculus, infinitesimals included.

Duals and Jets give you the machinery, the algebraic engines that make that truth computable.

Think of it like this:

• Hyperreals are the blueprint

• Duals and Jets are the compiler

We’ll move step by step: from pure infinitesimals to first derivatives to curvature.

By the end, you’ll see that automatic differentiation — the core of modern AI — is nothing more than hyperreal algebra disguised as code.

1. Hyperreal Infinitesimals — The Conceptual Ground Truth

f’(x) = st((f(x+ε) - f(x))/ε)

Here, st(·) means standard part: it removes the infinitesimal tail.

We don’t simulate ∗ℝ directly: we build a finite algebra that reproduces its derivative behavior exactly.

2. Dual Numbers (k = 1)

Elements:

x̂ = x + d₁·ε   with   ε² = 0

Arithmetic:

(x + a·ε) + (y + b·ε) = (x + y) + (a + b)·ε

(x + a·ε) · (y + b·ε) = (x·y) + (x·b + a·y)·ε     # product rule appears

Constants lift as (c, 0).

Functions (first order)

The coefficient of ε is the directional derivative (JVP):

For smooth f:

f(x + d₁·ε) = f(x) + f’(x)·d₁·ε

Programming recipe:

1. Represent a scalar as (x, d₁)

2. Set d₁ = 1 to get ∂f/∂x, or use a direction vector v to get Jf(x)·v

3. Implement + and * once—everything else flows automatically

4. Cost: about 2–3× a normal forward pass

3. Jets (k = n ≥ 2)

Duals give you slopes. Jets give you curvature and higher-order awareness.

Elements:

x̂ = x + d₁·ε + (d₂/2!)·ε² + ... + (dₙ/n!)·εⁿ
with εⁿ⁺¹ = 0

Arithmetic (up to order 2):

(Ax, A₁, A₂) * (Bx, B₁, B₂) =
( Ax·Bx,
  A₁·Bx + Ax·B₁,
  A₂·Bx + 2·A₁·B₁ + Ax·B₂ )

Functions (order n):

f(x + d₁·ε + d₂·ε²/2!) = f(x) + f’(x)·d₁·ε + f’‘(x)·d₂·ε²/2! + ...

Each power of ε carries one derivative order.

It’s like a multi-lane Taylor engine where curvature, acceleration, and higher changes flow in parallel.

Why this matters:

• Jet² gives Hessian-vector products (HVPs) and Newton/TR steps without explicitly building a Hessian

• Higher-order Jets (J³, J⁴, …) give full Taylor expansions in one pass — vital for control, simulation, and meta-learning

So What Do I Actually Do With This on Monday?

Dual/jet algebra + compositional functors (JVP ∘ VJP) is one of the only viable ways to get exact, composable second-order math anywhere near billion-parameter, real-world AI.

The way PyTorch/JAX are actually used today is a tiny, first-order slice of that potential.

Right now PyTorch/JAX can do grad, jvp, vjp, even jvp(grad(f), …) – but:

• they’re treated in practice as give me a gradient buttons,

• second-order is bolted on as an afterthought (slow, awkward, mostly limited to small-scale experiments),

• almost nobody designs training, safety, or debugging around curvature because it’s too painful to use systematically on large systems.

Dual/jet algebra is the upgrade:

• Derivatives become exact algebraic objects, not some tensor the lib spat out once.

• JVP and VJP are understood as functors on that algebra, so JVP ∘ VJP is by construction a genuine HVP operator, not a fragile API trick.

Once you think this way, you can:

• localize second order (per block, per subsystem, per semantic direction), so you only pay the ~2–3× cost where it matters,

• and later bake jets into the backend (C++/CUDA with real jet types and fused kernels) so those same compositional operations stop being research toys and become everyday tools.

That’s the scaling story:

• Naive PyTorch/JAX usage: a first-order culture where second-order is technically possible but ergonomically and computationally painful on big, realistic models.

• Dual/jet-based, compositional design: Same asymptotic cost, but now you can point real second derivatives at the mess created by first-order-only vibes in self-driving, medical AI, finance, and other billion-parameter monsters, without your stack collapsing under PyTorch/JAX overhead and tooling friction.

Top 10 Essential References

On JEPA Architectures:

1.- LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architectures (September 2025)

2.- ACT-JEPA: Novel Joint-Embedding Predictive Architecture for Efficient Policy Representation Learning (April 2025)

3.- Point-JEPA: A Joint Embedding Predictive Architecture for Self-Supervised Learning on Point Cloud (February 2025)

On Transformer Attention Complexity:

4.- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures (October 2024)

5.- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (October 2023, still widely cited in 2024–2025)

On Toroidal/Topological Neural Networks:

6.- Toroidal Topology of Population Activity in Grid Cells (January 2022, Nature — still foundational for 2024–25 work)

7.- Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometry (June 2022)

On Dual Numbers & Automatic Differentiation:

8-. Dual Numbers for Arbitrary Order Automatic Differentiation (January 2025)

9.- Application of Generalized (Hyper-) Dual Numbers in Equation of State Modeling (October 2021 )

On LLM Hallucinations:

10.- Why Language Models Hallucinate (September 2025)

TL;DR

Most current AI runs on a flat, first-order OS: everything is a tensor in R^n, and training is driven primarily by first-order gradients. If you want anything richer—HVPs, curvature signals, better memory, you have to stack grad, jacfwd, jacrev, jvp, vjp and manage extra graphs, extra passes, and extra bugs. The result in practice: unstable training, brittle generalization, and ugly workarounds for hierarchy (hyperbolic “patches”) and memory (KV caches, ad-hoc recurrence).

This post proposes a new mathematical OS for AI: instead of differentiating the program, we differentiate the space. Internal states are jets on curved manifolds (hyperbolic for hierarchy, toroidal for phase/memory). A single pass on jet-valued states gives you value, gradient, and useful second-order information along the encoded directions as one object, and standard AD primitives (JVP, VJP, HVP) can be understood as consequences of the jet algebra and its functoriality. For AI engineers, this means:

• geometry and hierarchy built into the representation space,

• structured, low-interference memory from topology,

• higher-order training signals available by changing the state type, not by wiring up new AD pipelines.

Subscribe now

1. Introduction

Modern AI runs in what we will call the Euclidean first-order regime. Internal states are flat tensors in R^n, and training is driven almost entirely by first-order gradients in those coordinates. Contemporary automatic differentiation (AD) frameworks compute these gradients exactly on their primitives (up to floating-point error), but they do so as transformations on tensor programs— grad, jacfwd, jacrev, jvp, vjp —each of which can be viewed as defining a new graph in flat space. Large-scale optimization then unfolds on poorly conditioned Euclidean landscapes, and as models grow, we repeatedly see unstable gradients, tangled latent trajectories, memory interference, and brittle compositional behavior.

We treat these not as incidental engineering glitches, but as signs that the underlying mathematical substrate is misaligned with the phenomena we want to model.

From this viewpoint, hallucinations, gradient pathologies, “spaghettized” latents, and memory collisions are reinterpreted as consequences of forcing hierarchical, periodic, and higher-order structure into flat Euclidean embeddings while using almost exclusively first-order dynamics. By formulating differentiation and parameter updates at the level of jet functors, and by analyzing how curvature and topology shape the resulting flows, we argue that many weaknesses of current architectures arise from this flat, first-order foundation itself—and that a geometric, jet-based operating system for AI is a more appropriate starting point than further tuning of the existing one.

2. The Euclidean First-Order Regime

We begin by making precise what we mean by the Euclidean first-order regime that characterizes essentially all contemporary large-scale AI systems.

2.1. Representations as Euclidean tensors

Let X denote an input space (e.g. token sequences, images, or states), and let a model be a parametrized map:

In the next subsection we will see that the same Euclidean assumption is inherited by the learning dynamics: automatic differentiation computes exact gradients of these tensor maps in flat coordinates, and optimization operates on this Euclidean parameter and representation space using almost exclusively first-order information.

2.2. First-order gradient dynamics

Training consists in minimizing an empirical loss

and practical algorithms such as SGD, Adam, or their variants implement discrete approximations of this ODE.

In exact arithmetic, this gradient is mathematically exact: no finite-difference limits are taken, and the derivatives of the primitives are analytic. However, the dynamics of θt\theta_tθt​ under these gradients are governed by the geometry of a high-dimensional loss surface embedded in a flat parameter space RP\mathbb{R}^PRP.

Two structural features follow:

2.3. Geometric and numerical failure modes

Within this Euclidean first-order regime, well-known pathologies can be interpreted as geometric and numerical phenomena:

• Memory interference.
  In recurrent architectures and transformer key–value caches, different histories are encoded as trajectories in a shared Euclidean latent space. Without any topological notion of “distinct memories,” trajectories corresponding to different sequences can intersect, overlap, or collapse onto nearby regions, leading to interference and collisions in retrieval and generation.

• Brittle compositional behavior and hallucination.
  Since all structure is represented in a flat vector space and learning is driven by local first-order updates, there is no guarantee that compositional relations (e.g. hierarchy, nested scope, periodic structure) are preserved under extrapolation. The model may fit training statistics while mapping novel compositions to geometrically arbitrary regions of latent space, resulting in incoherent or fabricated outputs.

We do not claim that these phenomena can be uniquely or exhaustively explained by geometry alone, nor that curvature and topology are sufficient remedies. However, we take the above as evidence that the combination of (i) Euclidean representation spaces and (ii) purely first-order optimization constitutes a rigid and fragile foundation for intelligence at scale. The remainder of this work develops an alternative substrate—jets on curved manifolds—in which higher-order structure, curvature, and topology are built into the objects being optimized, rather than retrofitted as after-the-fact regularizers or architectural hacks.

2.4 The Hidden Geometry of Neural Networks (the smoking gun )


Up to now we have treated the Euclidean first-order regime as a design choice:

1. Representations as Euclidean tensors (Section 2.1):
   All internal states live in a single global coordinate chart R^n. No intrinsic structure—hierarchy, orientation, periodicity—is carried by the space itself.

2. Training as first-order gradient flow (Section 2.2):
   Optimization dynamics unfold in a flat parameter space R^P, driven almost exclusively by first-order derivatives computed in Euclidean coordinates.

3. Failure modes arising from this flatness (Section 2.3):
   Memory interference, unstable trajectories, brittle composition, and hallucinations emerge when complex structure is forced into a space with no geometry to support it.

What is less widely appreciated—and central to the foundational critique of this OS—is that even in this flat R^n setting, modern neural networks do not remain geometrically flat.
The geometry cannot be avoided.

Here are the mathematical “tools” for visualizing those hidden curvatures in current AI frameworks:

2.4.1. The Jacobian as a Transport Map

Now we take a single frozen block

From an optimization standpoint, these blocks are used to assemble gradients. From a geometric standpoint, they encode something much stronger:

In conclusion:

So as you see we have the collection V to be connected via the transport operator Gamma

where B again is the index set of sites (tokens, patches, slots, etc.).

This observation has been made explicit by Gardner (2024), but it also follows directly from the semantics of AD:
the representation graph becomes a connection graph, and the Jacobian blocks become its connection coefficients


2.4.2. Loops, holonomy, and curvature inside flat coordinates

Once transport maps exist, loops become meaningful.

Crucially, all of this happens inside the supposedly flat coordinates of R^n. There is no curved manifold in the architecture, no Riemannian layer, no special geometry—just the Jacobian of the block you already run.

The network pretends to occupy flat R^n,
but its own Jacobian-level transport shows that it behaves, operationally, like a non-flat connection on a bundle.

Here is where the model’s curvature is created: Jacobian transport around loops of tokens reveals non-trivial curvature in the network’s internal geometric structure:

For a much deeper mathematical explanation, I strongly recommend you read my other two articles about:

Holonomy I

Holonomy II

3.From emergent curvature to designed geometry: Jets and Curved Manifolds as a Computational Substrate

This is the point at which the foundations crack.

If every serious neural model already defines a connection and empirically exhibits curvature through its Jacobians, then the real architectural choice is not:

• Euclidean vs geometric methods,

but:

• accidental geometry vs designed geometry.

Having identified the structural limitations of the Euclidean first-order regime, we now introduce the central objects of our alternative architecture: jets on curved manifolds. These objects unify representation, memory, and differentiation within a single geometric–algebraic framework.

3.1. From points to jets

This representation encodes no intrinsic derivative or neighborhood structure; all higher-order information must be inferred implicitly through training, typically with only first-order gradients.

We instead represent an internal state not by a point but by a jet:

where M is a smooth manifold (hyperbolic or toroidal in our setting). A jet encodes:

• zeroth-order information (the point x),

• first-order information (the tangent vector),

• higher-order derivatives (curvature, torsion, etc., up to order k).

Jets can be realized algebraically using:

• dual numbers for k=1

• higher-order dual numbers / truncated polynomial rings for general kkk.

Thus, each internal state carries an intrinsic local Taylor model of itself.
This replaces the “pointwise” representation of modern networks with a much richer local geometric object.

3.2. Hyperbolic manifolds for representation

• exponential volume growth with radius,

• natural embeddings of trees into geodesics,

• concentration of distances that reflect semantic hierarchy.

3.3. Toroidal manifolds for memory

Recurrent and stateful components of neural architectures (e.g., attention caches, recurrent channels) must maintain distinct trajectories for different sequences. In Euclidean latent spaces, trajectories can cross, collapse, or interfere, causing memory collisions or blending of contexts.

• periodic structure,

• phase variables,

• winding numbers w∈Zmw \in \mathbb{Z}^mw∈Zm that serve as global, discrete invariants of recurrent dynamics.

A memory trajectory on a torus acquires a winding number vector that cannot be erased by smooth deformations. The pair

thus separates distinct histories even when their instantaneous positions are nearby.

3.4. Differentiation as an intrinsic algebraic operation

In standard deep learning, differentiation is executed by an external automatic differentiation engine operating on Euclidean tensors. In contrast, jets carry their own algebraic differentiation structure:

• The first-order jet implements the chain rule via evaluation on dual numbers.

• Higher-order jets implement higher-order chain rules via truncated polynomial algebras.

• The tangent and cotangent lifts induced by jets correspond to Jacobian–vector products (JVPs) and vector–Jacobian products (VJPs), respectively.

• Hessian–vector products (HVPs) arise from functorial compositions JVP∘VJP and VJP∘JVP, depending on mode.
  

Thus, differentiation is reinterpreted not as a separate numerical pass over a computational graph, but as a native operation in the jet algebra of the manifold.

3.5. A categorical formulation

Jets and their transformations naturally form a category:

This categorical viewpoint unifies representation, memory, and differentiation under a single mathematical framework. Every component of the architecture—layers, updates, recurrences, memory—becomes explicitly geometric and functorial.

Summary of Section 3

Jets on curved manifolds replace the Euclidean first-order substrate with a geometric object that:

• encodes higher-order local information via truncated Taylor expansions,

• organizes representation via hyperbolic structure,

• organizes memory via toroidal topology and winding invariants,

• internalizes differentiation via jet algebra,and

• supports functorial composition for higher-order derivatives.

This sets the stage for Section 2, where we formalize JVP, VJP, and HVP as categorical transformations and show how learning dynamics on jet-valued states differ from their Euclidean first-order counterparts.

4.Bringing all components together within the framework of a new AI Operating System

The previous sections treated the pieces of the proposal in isolation: jets on curved manifolds as state, hyperbolic spaces for representation, tori for memory, and categorical differentiation as the core calculus. In this final section we reassemble them explicitly as an operating system for AI, and show how familiar OS notions map cleanly onto geometric objects and functors.

4.1. From operating systems to differential geometry

Figure 5 (“From Operating Systems to Differential Geometry”) is a small Rosetta stone. The left column lists standard OS concepts; the middle column shows the corresponding object in our framework; the right column explains why each mapping has a precise mathematical meaning.

The point of the table is not metaphor. Each row replaces an OS-level abstraction by a well-defined geometric or categorical structure.

4.2. Geometric AI Operating System architecture

Figure 6 (“Geometric AI Operating System Architecture”) shows how these pieces stack into a full system.

• User space: AI processes, agents, applications.
  At the top sit “programs” in the usual sense: agents, training loops, planning routines. They no longer speak tensors directly; they make geometric syscalls against an abstract API.

• Categorical API layer (process interface).
  This layer exposes functorial operations and natural transformations (LiftToJet, PushForward, PullBack, ParallelTransport, etc.). From a programmer’s point of view, this is the interface for “differentiate this map”, “transport along that trajectory”, or “read/write from toroidal memory”.

• Geometric AI Operating System (core services).
  Inside the OS box sit three main managers:

  • Toroidal Memory Manager - implements long-term storage on M, with winding-number addressing and interference-aware memory layouts.

  • Hyperbolic Representation Manager - manages embeddings in H, ensuring that hierarchical structure is aligned with the ambient negative curvature.

  • Jet & Derivative Service - the “gradient kernel”: it maintains jet-valued states, executes jet lifts J^k(F), and returns algebraically exact first-order and selected higher-order information (up to order k and floating-point error) in a single semantic pass.

  All three share a common view of state: everything is a jet on a chosen manifold, not a bare tensor.

• Geometric kernel & type system.
  Beneath the managers is the kernel itself: the category C of manifolds, bundles, and jet types, equipped with its scheduler (composition plus jet propagation) and curvature-aware memory logic. This is where the mathematics from Sections 2 and 3 lives.

• Backend implementations.
  PyTorch, JAX, and similar frameworks sit below as (ideally) interchangeable backend implementations. To really live inside this stack, today’s tensor libraries must be mathematically upgraded: their autodiff engines extended to support k-jet bundles, functorial Jacobian/Hessian operators, and holonomy-aware updates, instead of the current program-bound gradient pipelines built around stacked grad/jacfwd/jacrev transforms.

• Hardware.
  Finally, GPUs/TPUs/CPUs are just the physical substrate. The OS shields geometric code from hardware details in the usual way: we can, in principle, change accelerators without rewriting the mathematical layer.

4.3. What this buys you in practice

For AI practitioners, this “new OS” view is not an aesthetic indulgence; it changes what becomes easy:

• Higher-order information by construction.
  Because state is jet-valued, one semantic pass yields value, gradient, and chosen higher-order terms along encoded directions. Instead of stacking nested grad / jacfwd / jacrev transforms and maintaining multiple shadow graphs (as oddly happens now with PyTorch, JAX, etc.), a k-jet carries all the required order-k information in that single pass.

• Geometry-aligned representations.
  Hyperbolic and toroidal manifolds are not post-hoc interpretations but native spaces in which states live. Hierarchy, periodicity, and phase are expressed in the geometry, rather than being laboriously rediscovered in a flat latent.

• Structured memory and recurrence.
  Toroidal memory with winding-number addressing provides a geometric notion of index and cycle, giving a principled scaffold for long-range, repetitive, or phase-sensitive patterns.

• Backend independence as a design goal.
  Modeling backends as functors from the geometric category to tensors makes it possible, in principle, to swap PyTorch for JAX (or a future system) without changing the abstract OS logic. The mathematics and the implementation are cleanly separated.

• A principled handle on curvature and path-dependence.
  Curvature is no longer an accidental artifact glimpsed through occasional Jacobian probes; it becomes a first-class part of the state and connection. Tools like Gardner-style curvature heatmaps turn into diagnostics of the OS itself, not just of an opaque black box.

In short, the geometric AI operating system takes the hidden structure we already see leaking out of current models—curvature, gauge behavior, path-dependent transport—and promotes it to the level of design. Instead of asking flat tensors plus a first-order engine to simulate curved, higher-order phenomena, we build those phenomena into the substrate and let ordinary programming sit on top.

Let’s take this slowly.

Holonomy sounds like a word only a geometer could love.
But the idea is very simple:

Move a vector around a loop.
If it comes back twisted, the world is curved.

That’s it.
Everything in your cladogram is just this idea getting sharper, cleaner, more abstract — until it finally lands in our neural networks.

We’ll walk the tree together.

Subscribe now

1. Einstein’s Curved Spacetime (1915)

Start at the root.

Einstein takes spacetime and says: it’s not a stage, it’s a field.
Mathematically:

Einstein is already living in a world where:

• paths matter,

• loops encode information,

• curvature is physical (gravity).

He doesn’t say holonomy group yet. But the seed is there:
vectors carried around loops don’t come back the same.

2. Cartan and the Upgrade to Bundles

Now geometers get hungry.

Einstein works mostly in coordinates. Cartan comes and says:
forget coordinates; use frames and bundles.

• Take your manifold M.

• Over every point, attach a little frame — a basis for tangent space.

• Collect all these frames: you get a principal bundle P→MP

A connection is now an object on this bundle telling you how to move frames along curves. You can still ask:

If I move a frame around a loop and come back, how is it rotated?

That rotation is no longer just an accident; it’s a structural feature of the connection.

• base = tokens / sites,

• fibers = representation spaces,

• connection = Jacobian transport.

He gave us the language for AI!

3. Holonomy Gets a Name

Next step in the cladogram: people stop hand–waving and give the loop-effect a precise identity.

Given:

• a bundle P→M with structure group G,

• a connection on it,

you fix a point p ∈ P and look at all loops in M based at its projection.

Each loop gives you a group element in G — “how the frame changed after going around.”
All those elements together form the holonomy group at that point.

This is holonomy in the modern sense:

Holonomy = the group generated by all parallel–transport maps around all loops based at a point.

Then Ambrose–Singer come in and show a beautiful fact:

• local: curvature tensor and its covariant derivatives

• global: holonomy group

This is the same logic we’ll apply to networks in AI:

• local data: Jacobians between sites,

• global data: products of Jacobians around loops.

If the loop map is not identity, the representation geometry is curved.

4. Berger and the Zoo of Holonomy Groups

Now that holonomy is a first–class citizen, geometers ask:

“What holonomy groups are actually possible?”

Berger answers with a classification:

• Generic Riemannian manifolds: holonomy SO(n).

• Special structures: Kähler, Calabi–Yau, hyperkähler, G₂, Spin(7) — each with their own holonomy subgroup.

Holonomy now does two things:

1. It detects curvature (Einstein’s game).

2. It classifies geometry (Berger’s game).

You look at the holonomy group and immediately know what kind of extra structure the manifold carries: complex, symplectic, exceptional, …

File this pattern away.
We’ll steal it for AI: holonomy as both detector and classifier of network geometry.

5. Physics and Gauge Theory: Loops Become Observables

Then physics returns.

Gauge theory reuses the same bundle–connection–holonomy package, but now the structure group is something like SU(2) or SU(3).

A gauge field is a connection.
A Wilson loop is the holonomy of that connection around a closed spacetime loop.

You can read it as:

“How much does the internal state of a particle rotate if I drag it around this loop in the gauge field?”

That rotation is measurable.
Holonomy stops being just a geometric curiosity and becomes a physical observable.

Again, the pattern we like:

• local: gauge potential and curvature,

• global: holonomy around loops,

• meaning: whether the field configuration is trivial or stores real, physical structure.

6. Modern & Discrete Holonomy

Fast–forward.

Geometry spreads into computation:

• Discrete differential geometry puts connections on meshes and graphs.

• You parallel–transport vectors along edges.

• You take products along cycles.

• Holonomy appears in graphics, shape analysis, geometric deep learning.

The space may be combinatorial, but the idea is identical:

edges = tiny transports,
cycles = loops,
holonomy = what you see after one tour.

We’re almost home.

7. Holonomy in AI: Jacobians as a Connection

Now we step into the red box of our cladogram above: Holonomy in AI Architecture.

This is no longer “just a derivative.”
It’s a transport operator between fibers.

All these Γij​ together form a discrete GL(d)-connection on your representation bundle.

Now you can play the Einstein–Cartan–Ambrose–Singer game inside the network.

Pick a loop of sites:

i→j→k→i

Compose the transport maps:

This is your holonomy operator for that loop.

You didn’t add any fancy manifold.
You didn’t force the model to live in hyperbolic space.
You just looked at what its own Jacobians are already doing.

The code pretends to live in flat R^n,
but the holonomy of its Jacobians exposes a curved, path–dependent geometry underneath.

That’s the punchline.

8. Why This Matters for “Flat AI”

Modern deep learning still reasons in a flat Euclidean mindset:

• tensors in R^n,

• gradients as simple vectors,

• loss landscapes sketched as nice bowls.

But the actual behavior of perturbations inside a large model is path–dependent, anisotropic, and often wildly tangled.

Holonomy gives you a language for this:

• Fibers: where representations live.

• Connection (Jacobians): how a nudge at one site becomes a nudge elsewhere.

• Loops: how information flows around motifs in the architecture.

• Holonomy: the net twist you get after a tour — the fingerprint of curvature.

Once you adopt this lens, “model failure” stops being purely a stochastic or optimization story and becomes a geometric one:

• exploding / vanishing directions,

• brittle generalization,

• weird trajectory dependence,

are all hints that the connection is badly behaved — too much curvature in the wrong places, or curvature aligned with the wrong subspaces.

Conclusions: From Einstein’s Spacetime to Our Transformer AI

If you zoom out, your cladogram reads like one continuous thought:

1. Einstein: curvature is physical, paths matter.

2. Cartan: connections and bundles are the right language.

3. Ambrose–Singer: curvature and holonomy are two sides of the same coin.

4. Berger & physics: holonomy classifies and explains real structure.

5. Discrete geometry: we can do this on graphs.

6. Deep learning: our models are graphs with Jacobians on the edges.

So the move is natural:

Treat a neural network as a discrete geometric object,
read its Jacobians as a connection,
compute holonomy along loops,
and let curvature tell you where the architecture is secretly twisted.

Einstein used this logic to understand gravity.
We can use the same logic to understand why flat Euclidean AI keeps breaking, and how to build architectures whose geometry actually matches the problems they’re meant to solve.

TL;DR — Why AI Isn’t Flat


1. Curvature emerges automatically in any nonlinear DAG.

2. Holonomy is the right diagnostic for model “twist zones”.

3. Euclidean embeddings are an illusion.

We store vectors in Euclidean tensors, but the network never treats them as living in a globally flat space. Because transport is curved, the “meaning” of a direction in embedding space depends on where you are and how you got there. The coordinates are Euclidean; the effective geometry induced by the Jacobians is not. “Flat embeddings” are a convenient lie.

4. AI needs explicit geometry (toroidal memory, hyperbolic hierarchy).

If geometry is unavoidable, we might as well choose the right one. Toroidal memory eliminates artificial edges and wrap-around glitches; hyperbolic spaces encode trees and hierarchies naturally. Making these structures explicit lets memory, recall, and abstraction live in geometries that actually match the tasks, instead of in accidental curvature produced by random stacks of layers.

5. A geometric AI OS fixes the accident-geometry problem.

Today’s AI discovers curvature by accident: opaque, emergent, and hard to control. A geometric OS flips the story. You design the state spaces (toroidal / hyperbolic), you specify the connection via duals and jets, and you treat Γ and H as first-class, auditable objects—not hidden side-effects of backprop. Curvature stops being a bug you diagnose after training and becomes an operating principle you engineer from the start.

Subscribe now

1. Holonomy Toolkit – Fibers, Jacobians, Transport Maps, and Loops

Let’s start with an easy, practical definition of holonomy:

Holonomy is what the network does to a tiny perturbation when you send it around a loop of sites and bring it back.

Want it a bit more concrete? Here you go:

Holonomy is what you get when you move a vector around a closed loop and see how much it has changed when you return to the starting point.

In geometric language, you have a space with a connection (rules for transporting vectors along paths). You pick a point, take a vector in its fiber, and parallel–transport that vector along a loop that begins and ends at the same point.

The resulting linear map — comparing the final vector to the original one — is, formally, the holonomy of that loop:

For a more detailed depiction of how holonomy theory was created over several generations of mathematicians, I strongly recommend reading my other article here.


And now the ingredients of the holonomy recipe:

2. The Holonomy in Action:

We begin with three sites i,j,k inside a single neural network layer.

the Jacobian block mapping infinitesimal variations at i to infinitesimal effects at j. These blocks exist for any modern deep-learning layer, because AD computes them implicitly via JVP/VJP.

We then form the holonomy operator along the discrete loop i→j→k→i:

If the induced connection on the discrete bundle is flat, the holonomy on every such small loop is trivial:

In general nonlinear architectures—attention, LayerNorm, GELU, gating—the Jacobian blocks do not commute. Algebraically this means, for typical sites,

so the effect of applying one transport and then the other depends on the order. This non-commutativity by itself is an operator property; its geometric consequence is path-dependence: moving a perturbation through “path 1 then path 2” is not the same as “path 2 then path 1”.

For our specific triangle i→j→k→i, that path-dependence shows up as non-trivial holonomy: there exists some perturbation v ∈ Vi​ such that:

So the loop returns to the same site i but not to the same vector. In short,

Which is exactly the discrete signature of curvature on that loop.

This phenomenon arises even though:

1. The underlying state space is Euclidean.

2. The architecture is a directed acyclic graph.

3. No explicit manifold structure is specified.

Curvature is therefore emergent, not designed. It is induced purely by the nonlinearity of the DAG and the resulting non-integrability of the local transport operators.

It happens even though your model lives in flat Rⁿ and has no cycles. Because the model’s nonlinearities bend the transport rules even when the coordinates are straight.

That’s the paradox the plot above reveals.

3. Why the Loop Exists in the Math but Not in the Architecture

Formally, the architecture is a DAG.
There are no computational cycles.

Yet the holonomy loop exists because differentiation produces linear transport operators that you can compose arbitrarily, independent of the forward DAG.

Let’s formalize:

For a fixed layer FFF, define its Jacobian as a block matrix:

This matrix defines a connection on the discrete set of sites.
The network architecture restricts which forward edges exist, but the Jacobian-level transport is independent of acyclic structure: it exists because AD defines Γij for all relevant pairs.

A loop in this connection is therefore a loop in operator space, not in the data-flow graph.
Holonomy is computed over compositions of differential maps, not over actual physical cycles of computation.

The result:
even a DAG can induce a connection with non-zero curvature.

4. Why Curvature Emerges Automatically

Mathematically:

Curvature appears whenever the transport operators fail to satisfy the discrete integrability condition:

In nonlinear DAGs, this fails generically because:

• nonlinear activations distort tangent directions,

• attention weights depend on content and therefore break symmetry,

• LayerNorm rescales each site differently,

• residual mixing causes non-commutativity of Jacobian blocks,

• depth compounds these non-integrabilities.

Thus curvature is not exceptional.
It is structurally expected.

As soon as each site interacts with others in a state-dependent, nonlinear manner, path-dependence becomes unavoidable.

5.Euclidean Embeddings Are an Illusion

Mathematically:

Even if the representation space is R^d, the connection induced by the Jacobians defines a non-Euclidean geometry on the bundle of site-wise fibers.

Thus the “geometry” relevant for signal transport is not the coordinate geometry of the ambient space, but the connection geometry induced by the architecture.

The embedding vectors sit in a Euclidean vector space,
but the model’s transformations define a geometry that is almost never Euclidean.

6. Why AI Needs Explicit Geometry (How Our AI-Operating System Fixes This)

Because curvature is emergent, uncontrolled, and opaque, it produces:

• semantic drift across layers,

• unstable gradients,

• inconsistent meaning of directions,

• poor memory retention,

• interpretability breakdowns.

A geometric OS solves this by:

• using toroidal memory for stable, boundaryless recall,

• using hyperbolic representation spaces for hierarchy and abstraction,

• defining a controlled connection via duals and jets,

• enforcing implementation-independent transport.

SUMMING UP…

Curvature stops being a side-effect and becomes an engineered property.

Right now, your model invents its geometry by accident.

Sometimes it works.
Sometimes it collapses.

A geometric OS says:

“No more accidental geometry.
We choose the curvature, we choose the memory topology,
and we choose how information moves.”

And suddenly AI becomes predictable, auditable, and mathematically grounded.