Ask ChatGPT how many E's are in the word "seventeen," and you will likely get a confident, articulate, and completely wrong answer. The model will insist there are three. The correct answer is obviously four. This is not a bug that will be patched in the next update. It is a fundamental feature of how these systems work—and it reveals the single most important truth about artificial intelligence: it does not know what words are.

To understand why a system that can generate doctoral-like dissertations on demand cannot count four letters in a nine-letter word, you have to strip away the marketing language and look at the math. Large Language Models are not reading your question. They are not seeing letters. They are not thinking. They are performing matrix algebra at a scale that creates an illusion of understanding.

When you type a prompt into ChatGPT, the very first thing that happens is the text is destroyed. The letters are stripped away and converted into numbers. The word "seventeen" does not exist to the model. What exists instead is a token ID—perhaps the number 12,074—which is then transformed into a massive list of decimal coordinates called a vector. The model processes these numbers through layer after layer of matrix multiplication, performing billions of calculations to determine which token has the highest statistical probability of coming next.

The Tokenization Problem

AI models do not read words letter-by-letter the way humans do. They process text in chunks called tokens. The word "seventeen" is not seen as eight individual characters: S-E-V-E-N-T-E-E-N. It is broken into two larger tokens: "seven" and "teen." This is not a design flaw. It is an optimization strategy that allows models to process language more efficiently. But it means the model never sees the individual E's at all.

When asked to count letters, the model cannot simply look at the word and count. Instead, it must rely on statistical patterns it learned during training. It has seen countless examples of people discussing letter counts online. It has learned that when asked about the word "seventeen," the phrase "three E's" appears with some frequency. The model does not verify this. It cannot verify this. It simply outputs the most statistically likely answer based on its training data.

• Words become IDs: "Apple" might be token number 12,074 inside the system.
• Meaning becomes coordinates: The AI maps tokens into a mathematical space where similar concepts cluster together.
• Processing is pure math: Generating a response is just multiplying massive tables of numbers together.
• Output is probability: The final result is a percentage—a guess about which token should come next.

This is why the model can write a beautifully crafted essay about the letter E, discussing its frequency in the English language and its historical origins, while simultaneously getting the count wrong in a single word. It has learned the patterns of discussing letters. It has not learned to count them.

The Illusion of Thinking

The reason this feels so counterintuitive is that we project human-like cognition onto systems that mimic human language. When ChatGPT responds to a question, it sounds like it is thinking. It sounds like it is reasoning. It sounds like it is considering the problem and arriving at an answer. But there is no thinking happening. There is no consideration. There is no internal monologue. There is only matrix multiplication.

The entire architecture of a model like GPT-4 is built on a Transformer network, a design introduced in 2017. It relies on a mechanism called self-attention, which allows the model to look at a sentence as a whole and figure out how words relate to each other. In the sentence "The bank of the river," the attention mechanism flags the word "river" so the model knows "bank" means a slope of earth, not a financial institution. This is sophisticated pattern recognition. It is not understanding.

The model has no concept of what a "river" is. It has no concept of what a "bank" is. It has no concept of what a "letter" is. It knows that certain numbers appear near other numbers in its training data, and it has learned to predict which numbers should come next. That is the entirety of its capability.

Why Tools Can't Fix the Core Problem

When you use a version of ChatGPT that has access to a code interpreter, the model can write a Python script to count the letters in "seventeen" and get the correct answer. This is not because the model has suddenly learned to count. It is because the model has learned to recognize the pattern of the question and output the correct code to solve it. The underlying matrix algebra is still blind to letters. It is simply following a statistical pattern that says: "When the user asks about letter counts, write code that counts letters."

This is the exact flaw that makes today's AI nowhere near capable of taking over the world. A system that cannot reliably count letters without an external tool is not going to outsmart human civilization. The danger is not that the text generator hallucinates. The danger is what happens when a text generator is plugged into autonomous software tools that can execute code, control machinery, or browse the internet completely unsupervised.

The fear is not that the AI will break out of its harness. The fear is bad software engineering. If a developer builds a powerful, poorly restricted harness—for example, giving an AI access to a powerful bash terminal API or a stock-trading API—and the AI has a hallucination, the traditional code will faithfully execute that flawed instruction.

What AGI Would Actually Require

Matrix algebra alone is not enough for Artificial General Intelligence. The path from statistical text prediction to genuine reasoning requires a fundamentally different architecture. The next leap will not be about making a bigger calculator. It will be about shifting from statistical approximation to dynamic systems that possess an active, continuous relationship with time, space, and reality.

Current research into alternatives is exploring three radically different approaches. Neuromorphic computing and spiking neural networks mimic biological brains by only firing neurons when a specific threshold of electrical spikes is reached, allowing for real-time processing and learning with a fraction of the energy. World model architectures like Yann LeCun's JEPA attempt to predict abstract concepts rather than next words, forcing the AI to build an internal model of physics and cause-and-effect. State space models like Mamba and Liquid Neural Networks use differential equations to constantly change their internal state based on new inputs, mimicking real-time learning without needing a massive rewrite of the entire model.

None of these architectures will emerge from simply scaling up the current Transformer design. The matrix multiplication that powers today's LLMs is an incredibly powerful tool for pattern recognition, but using it alone to achieve AGI is like trying to build a spaceship out of really high-quality wood. You can make a spectacular boat, but it is fundamentally the wrong material to reach the moon.