Github Readme

Hierarchy or Heterarchy? A Theory of Long-Range Connections for the Sensorimotor Brain: A Plain-Language Explainer

Read the paper: https://arxiv.org/abs/2507.05888

The neocortex is the seat of intelligence in humans. This layer of tissue is about 2-3 mm thick, and if unfolded, would be about the size of a large dinner napkin. Extensive anatomical mapping and physiological studies have revealed much of the wiring inside the neocortex.
Classical interpretations of this data suggest that regions in the neocortex are organized into hierarchical structures. Each sensory modality has a lowest-level region connected to sensors like vision, touch, and hearing, followed by several higher regions processing increasingly complex features.

This paper argues that while hierarchical connections exist between modality-specific regions, there are also many non-hierarchical connections. It also proposes a functional purpose for these non-hierarchical connections.

 

An overview diagram of all the connections we’ll be discussing in the paper! Many of these connections do not make sense when thinking of the brain as a strictly hierarchical system, which is why we propose to think of it as a heterarchical system with both hierarchical and non-hierarchical processing.

In order to understand why we’re making these bold claims, it’s also important to understand the functional units that these regions are composed of.

Columns Are Smart Mini-Brains

Imagine your brain is made up of thousands of tiny computers (columns), each peering through a keyhole at one part of an object, like a coffee mug. Even though each one sees only a small patch, it builds a full model of the object over time by sensing and moving around.

One of the key tenets of the thousand brains theory is that we believe that even columns in the lowest regions are, in fact, learning complete object representations, whereas classical theory holds that objects are only recognized higher up in the hierarchy. The Thousand Brains Theory proposes that a column can learn complete objects by integrating individual sensations over time to build full models. A column is much smarter than previously thought.

The Brain Translates on the Fly

Your senses report the world in your body’s frame, “to the left,” “above my hand,” etc., but your brain needs to represent things relative to the object. It’s like a GPS translating your location on Earth into a location inside a shopping mall.

The role of the thalamus is still a largely open question in neuroscience theory.  It is clearly highly connected to the cortex, and all sensory input enters the thalamus before being sent to the cortex. It was observed that a spike entering the thalamus, caused a single other neuron to fire, sending a spike to the cortex. These cells were thus called relay cells.  The cortex also projects back onto those relay cells, but those connections have a modulatory effect on the neurons that reach up to the next region.  

 

This paper provides an answer to the question of what is going on in this area of the brain. We propose that the thalamus is not just relaying sensory information to the cortex; it is transforming it into the reference frame of the object being sensed by the column. The feedback connection from the cortex to the thalamus informs it of what reference frame transform is required.

Here we see the thalamus converting from ego-centric coordinates to object-centric coordinates.

The Brain Doesn’t Think in Strict Hierarchy

The classical interpretation of hierarchy suggests that the lowest-level region senses only edges and small features. The next level up combines these edges into sets, and a higher region takes these sets and recognizes an object. But this appears to be, at best, an incomplete picture. There are many connections between columns within the same region, and sensory input is also fed directly into higher regions in the hierarchy, among other non-hierarchical connections.

 

Here we see the classical (but simplified and incorrect) interpretation of hierarchical connections in the neocortex. See the next figure for an alternate view.

You Don’t Need to Relearn What You Already Know

When you see an eye that belongs to the face of a dog, your brain doesn’t build a whole new mental model from scratch. Instead, it combines your existing knowledge of “dog’s face” and “eye,” linking them together on a point-by-point basis. This is possible because a column in region 1 and a column in region 2 are both observing the same area in space. Region 1 is modeling the small features of the eye, while region 2 is modeling how the larger features on the dog’s face, such as eyes, nose and ears, are arranged relative to each other. 

The ID of the detailed model of the eye in region 1 becomes a feature in the model in region 2. This way, increasingly complex compositional models can be represented, and previously learned components can be reused. Importantly, each component model is itself represented as a collection of features at locations.


We propose that hierarchy is used to represent compositional objects. Each region represents complete objects, and higher regions represent combinations of them. Importantly, each region receives sensory and motor input and also higher regions can receive direct sensory input.

Voting

Another important example of non-hierarchical structure is the long-range connections between columns within the same region. According to the Thousand Brains Theory, these connections enable columns to share their most likely hypothesis (MLH) about the object being sensed. When one column is uncertain, it can use input from other columns to resolve ambiguity and quickly reach consensus. 

Final Takeaway

The neocortex isn’t just a hierarchy, it’s more like a network of skilled collaborators, each one learning by interacting with the world, translating and composing their knowledge, and quickly combining what they already know to recognize new things.

If you want to build AI like this, stop thinking in static layers and start thinking in columns that explore.

At the Thousand Brains Project we are taking the ideas expressed in this paper and using them to create an implementation of these ideas.This initial implementation is called Monty.

Monty exists because we believe that Thousand Brains Systems will be the future of AI and robotics. If you’ve ever wondered what a neocortex-inspired learning system performs like, you’re in exactly the right place. We just published a pre-print demonstrating the unique capabilities of such a system based on this neuroscience theory: https://arxiv.org/abs/2507.04494 

 

The Thousand Brains Project is more than an implementation of an idea. It’s a collaborative, open research, open-source, non-profit dedicated to bringing about this new future. Our roadmap lays out the next set of challenges. Discussion on Discourse, detailed RFCs in the repo, and our documentation give you a voice in design debates and performance deep-dives and everything in between.

 

So whether you’re here to test Monty in a different setting, write documentation, or help steer the future of AI, the links that follow will guide you to getting involved.

Follow the Project

 

Roadmap Highlights

The live roadmap tracks everything that we plan to work on over the next few years; tasks are tagged by Monty component and capability so you can jump in where you add the most value.  

Tip: grab the planning spreadsheet linked at the top of the roadmap page for a view of what’s done, in-progress, and up next. https://thousandbrainsproject.readme.io/docs/project-roadmap 

Read more of our published papers here: https://thousandbrainsproject.readme.io/docs/further-reading#our-papers