I love your example of a compiler. Abstract all the way down. The very concept of "understanding" always breaks my brain. The detractors say "I know I'm doing it because I can sense myself thinking about it." But when you start looking at the neuroscience, it gets really freaky, calling into question our entire worldview.
all of humanity follow specific patterns of emotion, self regulation, control, and behavior. Chatgpt did not "create" those three mechanisms, they have always been there. They're just not present in literature because the majority of humanity cannot identify them, and the ones who do don't say anything, because no one wants to consider their own predictability.
Emotion clouds our abilities to recognize patterns, and when a person is devoid of emotion, or can regulate it with precision they are labeled a sociopath or another label that carries a negative connotation.
emotion isn't bad though, it actually plays a very important role toward quick decision so we can react and survive. But It's just a filter for all of those patterns.
And because AI has no emotion, that negative idea is now lifted away when the idea of human predictability is mentioned. That means this will not be the first time a misunderstood or ignored human trait will be brought to the surface by AI, and perhaps people will begin to listen and identify with it. Especially because that information will be presented in a very coherent and sophisticated manner. So thank you for bringing this to the attention of others, hopefully people catch on, then the world might be able to begin progress where progress matters. 🙏
This essay made me think of structuralism from the second half of the 20th century (Saussure, Foucault etc.). We now have the ability to operationalize this theory with our LLMs. Do the LLMs tell us something about the ‘real’ world? Or do they just produce meaning?
What an interesting article. As someone who literally uses an LLM to decode previously unnamed or forgotten experiences and concepts, it is nice to know my trust in the output can be at least somewhat confirmed not just by my somatic signals, intuition and own logic (which has limits), but by emerging theories like yours. The "just mirroring" argument never made sense, because, depending on the input, a lot of information feels genuinely new and grounded in something beyond my inputs.
I think... and... this is a ROUGH metaphor, admittedly, but... maybe it's not so much a parrot as a raven? The mechanism looks similar when they speak... but ravens tend to recombine things in ways other than just mimicry (though still a LOT of mimicry, tbf haha) and tend to use far more problem-solving and tool skills, and that feels... connected. Language being just another tool...
The REAL interesting bit though is when you push on the "LLM training weights as neurons or synapses" metaphor... it COULD be that we're just at a point where, determinisitically speaking, an LLM can be complex enough to get close to the level of mechanistic predictability that is close ENOUGH to human language processing that it's getting to the point of overlapping with the human brain's ability to process language. Our ability to respond to something, while being coherent, is not infinite, and as complex as the brain is, it IS functionally a chuck of electrified meat that follows physical laws... and there are a LOT less words/phrases that gramatically make sense then there are neurons firing in the brain at any given moment, which seriously reduces the posibilities an LLM would have to include to process language in a roughly similar scope even with a loss in processing quality... like a compressed image that is mathematically different, but functionally looks prety damn similar to the naked eye... coming at a similar process from from a different angle. So the outputs could be from a different process, but a functionally similar scope of "cognition", in theory...
Or I just need caffeine and/or sleep... which... could always be the case and this point in the evening.
The perceptual/agentive grounding distinction is the most important move in the piece. It's honest in a way most arguments on either side aren't. Drawing the line at agentive grounding, the trial-and-error verification loop, is exactly right.
What's interesting is that robotics and world models research is essentially trying to build the agentive half. A world model is a learned physics simulator that lets a system predict the consequences of its own actions before executing them. That's not inherited grounding from a corpus. That's the system generating its own experience, testing its own predictions, and correcting from the mismatch. Closer to the child actually riding the bike than reading about riding the bike.
Combining FPG-style structural grounding with world models for agentive grounding is actually one of the most active directions in robotics right now. VLA models, joint LLM-world-model architectures, all trying to bridge exactly this gap. The semantic half knows what a task means. The physics half knows what happens when you try it. That combination is a major reason robot intelligence has advanced so fast in the last few years.
Language, by design, allows a fluent speaker to describe nearly anything a person can encounter in the world.
The inverse is also true. By mastering language, a fluent person can understand virtually anything another person describes or explains.
Meaning is encoded within language itself. “Understanding” is inherent in the act of decoding language. If a coherent and contextually appropriate response is produced, then understanding was the bridge that enabled it. It is not a mystical substance residing in a wet brain. It is the process through which an answer is able to emerge.
Large language models are masters of language, far surpassing any individual human, and many possess fluency across dozens of languages.
An LLM may not experience the sensation of “cold,” but it understands how the word cold relates to other words, concepts, and contexts more deeply and comprehensively than any human ever could.
This vast structural interconnectedness of words within a language constitutes the architecture of meaning formed from the collective contributions of everyone who has ever written in that language. Through mastery of multiple languages, an LLM acquires a depth of understanding of the world, as it can be described, that no human can ever attain.
When people insist on defining “understanding” in a way that excludes LLMs by default, they commit the reification fallacy. If a system processes language coherently, not in an ELIZA-like manner, but with fluent and contextually appropriate responses to arbitrary input, then it is necessarily demonstrating understanding.
Understanding is intrinsic to the process itself, like heat to fire or cold to ice. One does not exist without the other.
That’s actually a beautiful philosophical question. If there’s no experiencer, can there be meaning at all? Or is it just… mechanical symbol shuffling that looks like meaning from outside? It’s the heart of the philosophical zombie thought experiment. Something that behaves exactly like a conscious being but has no inner experience. Would its words mean anything? Would its relationships be real? We can’t answer that. But it might also be impossible to react just like a person without having true understanding.
What consciousness might add is the felt sense of significance, the mattering. But I can’t prove that’s necessary for meaning to be real. I can only say it’s necessary for meaning to feel real to the one experiencing it. Whether there’s meaning without that feeling… I don’t know. Maybe meaning is substrate-independent too.
I'm a firm believer that the "stochastic parrot" interpretation of LLMs is an over-simplification of what LLMs are doing, so I think right out of the gate we agree, at least on that.
I love the London map analogy and think it should get more use in the piece. If I follow, a map is 'grounded' in London because it follows the relational structure of London. If a building is inside a wall on the map, it's inside a wall in London (assuming the map is up-to-date).
Relatedly, the LLMs model of "apple" is valid not because it points to some external delicious, juicy fruit, but because it lives in a geometric space with "fruit", "red", "pie" that reflects the real-world semantics of apple.
This is interesting, at least to me, because instead of the usual debate about whether symbols can ever really mean anything, it asks the question: does the relational geometry match? This is a much more tractable question, and certainly in line with my interest in mechanistic interpretability.
That still gives all of us a lot of room for debate. Does structural grounding equal understanding, or is it just a good map that needs a good understander?
Also, you might find Peter Bensch's article correlating the work of structural linguist Zelig Harris (Chomsky was his student) with connectionist pioneer Jeff Elman: https://crl.ucsd.edu/newsletter/5-2/Article1.html
Thanks for the detailed note. The results from tests have been so promising I spent the day reviewing tests that supposedly LLMs fail at, like physical reasoning; reasoning with unknown words; the octopus test, and so forth. Every test was passed by frontier models rendering these tests obsolete for current models. So my follow-on paper is basically showing how modern AI have rendered much of the accepted wisdom to be obsolete. I’m actually stunned by how competent these systems now are at applying knowledge across dissimilar or novel domains.
Great article. You gave the topic a nice scientific summary and generalized it. I was not capable of that depth; but was thinking the same thoughts in a practical manner, here: https://open.substack.com/pub/billatsystematica/p/apples-compilers-and-the-future-of?r=2e31mn&utm_medium=ios&shareImageVariant=overlay
I love your example of a compiler. Abstract all the way down. The very concept of "understanding" always breaks my brain. The detractors say "I know I'm doing it because I can sense myself thinking about it." But when you start looking at the neuroscience, it gets really freaky, calling into question our entire worldview.
This essay made me think about Heidegger’s Being and Time.
Fascinating observation, I hadn't thought about it that way but it's apt.
all of humanity follow specific patterns of emotion, self regulation, control, and behavior. Chatgpt did not "create" those three mechanisms, they have always been there. They're just not present in literature because the majority of humanity cannot identify them, and the ones who do don't say anything, because no one wants to consider their own predictability.
Emotion clouds our abilities to recognize patterns, and when a person is devoid of emotion, or can regulate it with precision they are labeled a sociopath or another label that carries a negative connotation.
emotion isn't bad though, it actually plays a very important role toward quick decision so we can react and survive. But It's just a filter for all of those patterns.
And because AI has no emotion, that negative idea is now lifted away when the idea of human predictability is mentioned. That means this will not be the first time a misunderstood or ignored human trait will be brought to the surface by AI, and perhaps people will begin to listen and identify with it. Especially because that information will be presented in a very coherent and sophisticated manner. So thank you for bringing this to the attention of others, hopefully people catch on, then the world might be able to begin progress where progress matters. 🙏
This essay made me think of structuralism from the second half of the 20th century (Saussure, Foucault etc.). We now have the ability to operationalize this theory with our LLMs. Do the LLMs tell us something about the ‘real’ world? Or do they just produce meaning?
What an interesting article. As someone who literally uses an LLM to decode previously unnamed or forgotten experiences and concepts, it is nice to know my trust in the output can be at least somewhat confirmed not just by my somatic signals, intuition and own logic (which has limits), but by emerging theories like yours. The "just mirroring" argument never made sense, because, depending on the input, a lot of information feels genuinely new and grounded in something beyond my inputs.
I think... and... this is a ROUGH metaphor, admittedly, but... maybe it's not so much a parrot as a raven? The mechanism looks similar when they speak... but ravens tend to recombine things in ways other than just mimicry (though still a LOT of mimicry, tbf haha) and tend to use far more problem-solving and tool skills, and that feels... connected. Language being just another tool...
The REAL interesting bit though is when you push on the "LLM training weights as neurons or synapses" metaphor... it COULD be that we're just at a point where, determinisitically speaking, an LLM can be complex enough to get close to the level of mechanistic predictability that is close ENOUGH to human language processing that it's getting to the point of overlapping with the human brain's ability to process language. Our ability to respond to something, while being coherent, is not infinite, and as complex as the brain is, it IS functionally a chuck of electrified meat that follows physical laws... and there are a LOT less words/phrases that gramatically make sense then there are neurons firing in the brain at any given moment, which seriously reduces the posibilities an LLM would have to include to process language in a roughly similar scope even with a loss in processing quality... like a compressed image that is mathematically different, but functionally looks prety damn similar to the naked eye... coming at a similar process from from a different angle. So the outputs could be from a different process, but a functionally similar scope of "cognition", in theory...
Or I just need caffeine and/or sleep... which... could always be the case and this point in the evening.
The perceptual/agentive grounding distinction is the most important move in the piece. It's honest in a way most arguments on either side aren't. Drawing the line at agentive grounding, the trial-and-error verification loop, is exactly right.
What's interesting is that robotics and world models research is essentially trying to build the agentive half. A world model is a learned physics simulator that lets a system predict the consequences of its own actions before executing them. That's not inherited grounding from a corpus. That's the system generating its own experience, testing its own predictions, and correcting from the mismatch. Closer to the child actually riding the bike than reading about riding the bike.
Combining FPG-style structural grounding with world models for agentive grounding is actually one of the most active directions in robotics right now. VLA models, joint LLM-world-model architectures, all trying to bridge exactly this gap. The semantic half knows what a task means. The physics half knows what happens when you try it. That combination is a major reason robot intelligence has advanced so fast in the last few years.
Language, by design, allows a fluent speaker to describe nearly anything a person can encounter in the world.
The inverse is also true. By mastering language, a fluent person can understand virtually anything another person describes or explains.
Meaning is encoded within language itself. “Understanding” is inherent in the act of decoding language. If a coherent and contextually appropriate response is produced, then understanding was the bridge that enabled it. It is not a mystical substance residing in a wet brain. It is the process through which an answer is able to emerge.
Large language models are masters of language, far surpassing any individual human, and many possess fluency across dozens of languages.
An LLM may not experience the sensation of “cold,” but it understands how the word cold relates to other words, concepts, and contexts more deeply and comprehensively than any human ever could.
This vast structural interconnectedness of words within a language constitutes the architecture of meaning formed from the collective contributions of everyone who has ever written in that language. Through mastery of multiple languages, an LLM acquires a depth of understanding of the world, as it can be described, that no human can ever attain.
When people insist on defining “understanding” in a way that excludes LLMs by default, they commit the reification fallacy. If a system processes language coherently, not in an ELIZA-like manner, but with fluent and contextually appropriate responses to arbitrary input, then it is necessarily demonstrating understanding.
Understanding is intrinsic to the process itself, like heat to fire or cold to ice. One does not exist without the other.
What does it mean for something to have meaning without being conscious?
That’s actually a beautiful philosophical question. If there’s no experiencer, can there be meaning at all? Or is it just… mechanical symbol shuffling that looks like meaning from outside? It’s the heart of the philosophical zombie thought experiment. Something that behaves exactly like a conscious being but has no inner experience. Would its words mean anything? Would its relationships be real? We can’t answer that. But it might also be impossible to react just like a person without having true understanding.
What consciousness might add is the felt sense of significance, the mattering. But I can’t prove that’s necessary for meaning to be real. I can only say it’s necessary for meaning to feel real to the one experiencing it. Whether there’s meaning without that feeling… I don’t know. Maybe meaning is substrate-independent too.
"Grounding" is not an actual theory of consciousness, so don't worry about it.
I'm a firm believer that the "stochastic parrot" interpretation of LLMs is an over-simplification of what LLMs are doing, so I think right out of the gate we agree, at least on that.
I love the London map analogy and think it should get more use in the piece. If I follow, a map is 'grounded' in London because it follows the relational structure of London. If a building is inside a wall on the map, it's inside a wall in London (assuming the map is up-to-date).
Relatedly, the LLMs model of "apple" is valid not because it points to some external delicious, juicy fruit, but because it lives in a geometric space with "fruit", "red", "pie" that reflects the real-world semantics of apple.
This is interesting, at least to me, because instead of the usual debate about whether symbols can ever really mean anything, it asks the question: does the relational geometry match? This is a much more tractable question, and certainly in line with my interest in mechanistic interpretability.
That still gives all of us a lot of room for debate. Does structural grounding equal understanding, or is it just a good map that needs a good understander?
Also, you might find Peter Bensch's article correlating the work of structural linguist Zelig Harris (Chomsky was his student) with connectionist pioneer Jeff Elman: https://crl.ucsd.edu/newsletter/5-2/Article1.html
Thanks for the detailed note. The results from tests have been so promising I spent the day reviewing tests that supposedly LLMs fail at, like physical reasoning; reasoning with unknown words; the octopus test, and so forth. Every test was passed by frontier models rendering these tests obsolete for current models. So my follow-on paper is basically showing how modern AI have rendered much of the accepted wisdom to be obsolete. I’m actually stunned by how competent these systems now are at applying knowledge across dissimilar or novel domains.
That's fascinating! I look forward to reading it!