Neural Networks#

Neural networks are often introduced with an appealing analogy: they are inspired by the brain. The term “neural network” evokes images of neurons firing, synapses adjusting, and emergent intelligence arising from complex biological computation. At a glance, artificial neural networks (ANNs) seem to mimic this process. Each artificial neuron takes in signals, applies a transformation (like activation), and passes output along to the next layer. Connections have weights, analogous to synaptic strengths. Learning involves updating these weights, not unlike synaptic plasticity in biological systems.

This analogy has pedagogical value. It offers a rough intuition for how ANNs work: information flows, is transformed layer by layer, and leads to a decision. But beyond this basic conceptual scaffolding, the analogy quickly breaks down. Artificial neural networks and biological brains are fundamentally different in purpose, structure, and function.

Simplified Units vs. Real Neurons#

In ANNs, a “neuron” is a simple mathematical unit: it performs a weighted sum and applies a nonlinearity. In contrast, biological neurons exhibit complex electrical and chemical dynamics, including:

  • Spike-timing,

  • Nonlinear integration over dendritic trees,

  • Neurotransmitter diversity,

  • Local plasticity rules.

By comparison, an artificial neuron is a linear thresholding device—a pale abstraction of its biological counterpart.

Learning Algorithms Are Entirely Different#

The human brain does not train via backpropagation. There is no evidence that biological systems use global error signals or gradient descent over a fixed architecture. Instead, the brain likely relies on local learning rules (like Hebbian learning), reinforcement signals, and neuroplasticity shaped by development and experience.

Backpropagation, by contrast, is a centralized, mathematically-driven method requiring:

  • Global knowledge of the loss function,

  • Differentiable operations,

  • Repeated propagation of exact error signals.

This algorithm is not biologically plausible, even though it works well in practice for training deep networks.

Energy Efficiency and Architecture#

Brains are incredibly energy-efficient, operating at about 20 watts, yet supporting massive parallel processing across ~86 billion neurons. In contrast, modern neural networks often require gigawatts of compute during training, and vast datasets—orders of magnitude larger than what humans need to learn similar concepts.

Architecturally, the brain is highly recurrent and modular, with feedback loops and specialized subsystems (visual cortex, hippocampus, etc.). Most traditional neural networks are feedforward, though recent models (like transformers or recurrent neural networks) incorporate more flexibility.

Generalization and Robustness#

Despite their complexity, neural networks often generalize poorly outside their training distribution. They are sensitive to:

  • Small perturbations (adversarial examples),

  • Spurious correlations,

  • Shifts in data distribution.

Humans, in contrast, are remarkably good at transfer learning, abstract reasoning, and learning from very few examples. A child can learn a new concept from a single demonstration—something even state-of-the-art ANNs struggle with.

Interpretability and Transparency#

Brains are complex, but we can sometimes explain human decisions through introspection or reasoning. Neural networks, however, are often black boxes. Interpretability remains an open challenge, particularly in high-stakes applications like medicine or law.

Recommended Literature:

Bishop. Pattern recognition and machine learning. 2006. Sections 5.1. Feed-forward Network Functions, 5.2. Network Training, 5.3. Error Propagation and 5.5. Regularization in Neural Networks