Catching up with the human brain
In 1943, neurophysiologist and cybernetician of the University of Illinois at Chicago Warren McCulloch and self-taught logician and cognitive psychologist Walter Pitts published the foundations-laying article A Logical Calculus of the Ideas Immitent in Nervous Activity [307]. Building on Allan Turing´s work On Computable Numbers [308], McCulloch´s and Pitt´s paper set a path to describe cognitive functions in abstract terms showing that simple elements connected in a network can have a huge computational capacity. The computational model that is the core of the McCulloch-Pitts theory of neural networks stems from the idea that neurons in the brain can be considered as simple binary switches that either fire or do not fire, depending on the input they receive. Within the already lively community of biophysicists doing mathematical work on neural networks at that time, McCulloch and Pitts´ approach was novel in their use of logic and computation to understand neural, and thus mental, activity. Their contribution included [a] a formalism whose refinement and generalization led to the notion of finite automata (an important formalism in computability theory), [b] a technique that inspired the notion of logic design (a fundamental part of modern computer design), [c] the first use of computation to address the mind–body problem, and [d] the first modern computational theory of mind and brain [309, 310].
The McCulloch-Pitts theory of neural networks is a useful computational model that has been influential in the development of artificial neural networks. Emphasized their neurobiological aspects, artificial neural networks are modeled after the computational principles of the brain (as McCulloch and Pitts imagined them), with the specific aim of understanding and replicating human abilities. Such neural networks are being used for many machine learning tasks such as function approximation and pattern recognition – see for example [60,65,311]. However, these ideas no longer correspond to the state-of-the-art description of how the human brain works: the updated knowledge delivers an essentially different model. Since 1990s, Jeff Hawkins and the by him founded Numenta company develop neuroscience to dramatically increase performance and unlock new capabilities within the AI realm [298]. The field of their research is, first and foremost, Neocortex. The mantle of the large brain (telencephalon), also known as the cerebral cortex, phylogenetically the youngest part of the central nervous system, Neocortex can only be found in mammals to be fundamentally involved in functions such as perception, learning, memory, thinking, and others, unlike the „old brain“ inherited by Synapsida (and mammals) from Amniota reptiles in the Late Carboniferous period.
First identified by Mountcastle in 1957, the fundamental „processor“ of cognitive functions is Vertical Column (also called Cortical Column or Module) [312, 313], composed of six differntiated layers of neurons making together two to three mm of the human grey matter. There are roughly 150.000 Vertical Columns in Neocortex, interconnected mutually by dendrits and by axons that also, mediated by the cortical projection areas of „the old brain“, ensure the transmission of information from sensory cells, to locomotor apparatus, and the like. [t] This apparatus „thinks“ by creating – learning models of the objects encountered – a spatial, three- or more-dimensional object is the result – and [u] placing them into reference frameworks. [v] Learning runs in motion concerning the object to be learned, in other words, learnig happens diachronically; [w] movement and perceiving run repeatedly and continuously update the model – as opposed to one-time training and „freezing“ of todays AI models. [x] All senses that can are involved, within a breakthrough machine learning, novel types of sensors – radar as an example – can adjoin. [y] In our knowledge, there is not just one model for a particular object of reality: knowledge about particular object is distributed across many complementary models learned by many Vertical Columns. The structure of the Neocortex comprehending many models offers flexibility. [z] The key to making the many-model system work is „poll“ or „conciliation“. Each column works independently to some extent, but the extensive connections in Neocortex allow the columns to coordinate the perceiving.
The discrete nature of the Neocortex, or the complex of human cognitive processes, deserves emphasizing. At any moment in time, some neurons in the brain are active and others are not. The active neurons represent what we are currently thinking and perceiving. Our cognition and thinking is not in direct contact with the world around us, including our own body; all information comes to the Neocortex from the „old brain underneath“, in the knowledge of whose functioning we still have large gaps. Crucially, these thoughts and perceptions relate to the brain’s model of the world, not the physical world outside the skull. The world we live in is a simulation of the real world. This not only provides a state-of-the-art science´s proof of Descartes´ philosophical claim „cogito ergo sum“ but also shows importance of the comprehensiveness of the respective simulation and the at least three-dimensional spatiality of the models and their reference frameworks that are the platform on which the simulation takes place.
A deep mutual interconnectedness shows between three so far discreetly taken realms: spatiality and diachrony of thinking and learning, the spatial and diachronic performance of the state-of-the-art immersive virtual reality technology, and spatiality and diachrony of architecture. (In all the cases, the spatiality conceives three or more dimensions.) The first two spatialities and diachronies apply broadly, and the third renders specific for the field reviewed by this paper – architecture and development of the built environment. However, the specific concern turns out broad enough as soon as the ubiquity of architecture in terms of human life is considered.
Spatiality and diachrony of thinking and learning surveyed and elaborated by Jeff Hawkins and Numenta raise questions that remain sidelined in today´s race for ever-improving LLMs. Though a long way since, the current AI models still come from the McCulloch-Pitts essentials derived in the 1940s and recapitulated above under [a] to [d] above. However, these principles do not catch up with Hawkins´/Numenta model, as characterized by [t] to [z] above. Does this not mean a dead end, or rather a glass ceiling concerning development of AI models as practiced so far? Can the processes of tokenization, vectorization, and operation of two-dimensional, though in multiple layers arranged neural networks that „power“ today’s LLMs (and image- and video-processing models alike) cope with [t] three-plus-dimensional spatiality and [u] reference networks´ embeddedness, [v] diachrony and [w] continuous updating, [x] sensorial and [y] models´ plurality, and [z] „conciliation“ principle that altogether make learning and thinking run in the human Neocortex?
The so-much-needed answer to this question does not need to regard the McCulloch-Pitts theory or the up-to-date AI networks and algorithms as flawed. The model is the question: AI may be on a similar threshold to the next-level development as physics was when Euclidian geometry and Newtonian physics showed unable to respond to questions raised by the theory of relativity and quantum physics. In day-to-day life, we still follow Newton´s laws and dwell in Euclidian space; to learn the more tricky reality of the universe, we strive to get aligned with the Standard model and to put the theory of everything. Vice versa, without unraveling the problem of compatibility of the two models of thinking on the one hand and computing on the other, not only the AGI but, perhaps, also a fully potent architectural AI may remain just desire. As Abby Betrics of The Economist comments on the current state-of-the-art AI models: “[There is] no reason to believe…that this is the ultimate neural architecture.” [314].
Computing performance, impacting the environment
In 2023, MIT’s Schwarzman College of Computing launched a research delving into the opportunities and challenges of the computing age. Steven Gonzalez Monserrate’s case study addresses the environmental, social, and health impacts of skyrocketing computing power ubiquitous in contemporary life [315]. Until 2020, perhaps 2022, the main consumer of electricity for computing and the main producer of waste heat, burdening the environment, was the mining of cryptocurrencies. Since then, machine learning has taken over, and the acceleration has accelerated. With AI, energy consumption is skyrocketing and sustainability is at risk. The means of mitigation of the unwanted effects ought to comprise but not be limited to efficient model architectures, hardware innovations, efficiency policies and regulations, green data centers using renewable energy sources, measuring the environmental footprints of the services as a starting point for the impacts´ management and control, and – on the other hand – harnessing AI to address sustainability challenges – to optimize energy distribution, predict climate patterns, or enhance energy efficiency in buildings.
The Biden administration, too, wants to accelerate its conversations with big technology companies regarding how to generate more electricity — including nuclear power — to meet the massive energy demands of AI [316].
The hundreds of GPUs running AI foundation models, LLMs, and other applications consume tens of kilowatts of electric energy each, and about 80% of the energy converts into heat to burden the global climate. As a popular saying, the energy consumption of Google AI-search tools and LLMs copes with Ireland’s total consumption. Other elements of the performing AI ecosystem add on top: cloud servers that host the model and handle user requests, load balancers distributing incoming requests across multiple servers to ensure efficient utilization and prevent overload, networking infrastructure that needs to be robust enough to ensure seamless communication between servers, data centers, and clients; this includes routers, switches, and firewalls. Then come the storage systems for the models to rely on storage solutions for model checkpoints, training data, and other resources; distributed file systems manage large-scale data storage. Database systems store metadata, user profiles, and other relevant information; relational databases or NoSQL databases may be used. Monitoring and logging tools track system performance, resource usage, and errors; examples include Prometheus, Grafana, and ELK stack (Elasticsearch, Logstash, Kibana). Authentication and authorization services secure access, services like OAuth, JWT, or API keys authenticate users and authorize their interactions with the model. Further, docker containers and Kubernetes orchestration manage deployment, scaling, and updates of the model´s instances. Content delivery networks optimize content delivery by caching responses closer to users, reducing latency and server load. Backup and disaster recovery systems ensure data integrity and system resilience; and other components apply to contribute to the model´s reliable operation [317].
One way or another, despite all the advances in hardware, namely chip performance, AI developers as well as hardware producers are increasingly facing the challenge of optimizing the computing performance. Nvidia, as has happened to be a rule, has hurried up to lend a hand: its Blackwell is four times faster than the previous model Hopper. The Blackwell transformer engine utilizes fine-grain scaling techniques called micro-tensor scaling to optimize performance and accuracy enabling 4-bit floating point precision (FP4). This doubles the performance and size of next-generation models that memory can support while maintaining high accuracy. Optimized to work with shorter decimal numbers, Blackwell is at 20,000 FLOPS (floating-point operations per second) at the 4-bit precision [318].
A contribution to higher computing power at lower energy consumption and lesser environmental impact is revealed with the development of ternary computing. In the still-emerging field, ternary neural networks (TNNs) make deep learning more resource-efficient using only ternary weights and activations instead of traditional binary values. The models match full-precision transformers in terms of perplexity with the student TNN learning to mimic the behavior of a teacher network without using any multiplications. In addition, TNNs inherently prune smaller weights during training, making them more energy-efficient; ternary LLMs offer the potential for orders of magnitude faster inference and significantly improved power efficiency.
Samsung-backed researchers have debuted ternary semiconductor design, introducing a 1-bit LLM variant – BitNet b1.58. The 1.58-bit LLM defines a new scaling law and a recipe for training new generations of LLMs that are both high-performance and cost-effective. Furthermore, it enables a new computation paradigm and opens the door for designing specific hardware optimized for 1-bit LLMs [319].
Potentially much more powerful for certain tasks than traditional computers, quantum computers can process a vast amount of possibilities simultaneously. Quantum computing is a type of computing that uses quantum-mechanical phenomena, such as superposition and entanglement, to perform operations on data. While traditional computers use bits as the smallest unit of data (which can be either 0 or 1), quantum computers use quantum bits, or qubits, which can be both 0 and 1 at the same time. The state-of-the-art in quantum computing is developing. As for prospects, quantum computing has the potential to revolutionize various fields by handling complex problems that are currently unsolvable by classical computers. This includes areas like cryptography, drug discovery, financial modeling, AI of course, and more.
There are still many technical challenges to overcome before they will be ready for widespread use. Researchers are making progress in building more stable and larger systems of qubits, which are necessary for practical quantum computers. Along the way, there are multiple issues. In the field of error rate, the developers proceeded from an error rate of 8×10−3 to 10−5. This is still the beginning; Microsoft has set an error rate of 10−8, allowing some useful calculations on a quantum computer.
Quantum AI is another topic – so far nebulous and full of strong statements. It was very promising when for specific processes quantum neural networks needed less learning for the same quality of results than traditional networks. However now, Chinese scientists have presented experiments with optical neural networks using common light, not individual photons. Such optical neural networks behaved quite similarly to quantum ones, including their potential advantages [320]. Therefore, quantum computers will not have an easy time showing any advantage.
There is significant progress in quantum communication and security: 100 kilometers of quantum-encrypted transfer [321]. And of course, there are turmoils in business, investments, and politics concerning quantum computing R&D [322,323,324,325].
Not only computing strategies and hardware novelties are trying to enhance machine learning. Verses, a company that develops intelligent software systems based on the wisdom and genius of nature [326], proposes revolutionary neural network, learning strategy, and output-delivery technique (that seem inspired by Jeff Hawkins´ “A thousand brains” theory) to boost the computing power of its distributed intellignece model. Focus on biologically inspired distributed intelligence and transforming disparate data into coherent knowledge models are the starting points of Verses´ technology. Introduced in section (3) of this paper, patterned after natural systems and neuroscience, Verses´ flagship product – Genius operating system of thousands of intelligent agents minimizes complexity to deliver substantially smaller models that compromise neither computing power nor the performance quality.
