12.4 The Artificial Neuron

12.4 The Artificial Neuron

The fundamental building block of a neural network

Although modern neural networks may contain millions or even billions of adjustable parameters, their remarkable capabilities emerge from the repeated application of one extremely simple computational unit: the artificial neuron.

Every neuron performs exactly the same mathematical operation.

It receives numerical inputs, combines them, applies a nonlinear transformation, and passes the result to the next layer.

Understanding this simple computation is the key to understanding how entire neural networks operate.

Surprisingly, the mathematics behind an artificial neuron is already familiar.

As we shall see, a neuron is essentially a linear regression model followed by a nonlinear activation function.

---

A neuron with a single input

Let us begin with the simplest possible neuron.

Suppose the neuron receives only a single numerical input (x).

This input might represent a biological property such as

• the hydrophobicity of a protein,
• its molecular weight,
• or the expression level of a gene.

The neuron multiplies this input by an adjustable parameter, called the weight,

$$w.$$

The output of the neuron therefore becomes

$$z = wx.$$

If the weight is large and positive, the input strongly increases the neuron’s output.

If the weight is close to zero, the input has little influence.

If the weight is negative, increasing the input decreases the neuron’s output.

The weight therefore determines how strongly the neuron responds to a particular biological feature.

Learning in a neural network largely consists of finding appropriate values for these weights.

---

Multiple inputs

Of course, biological systems are rarely determined by a single variable.

Whether a protein is transported into a chloroplast, for example, depends simultaneously on many sequence properties.

An artificial neuron therefore usually receives many inputs.

Suppose the neuron receives three biological features,

• hydrophobicity ((x_1)),
• N-terminal charge ((x_2)),
• amino acid composition ((x_3)).

Each feature is associated with its own weight,

$$w_1,\;w_2,\;w_3.$$

The neuron first multiplies every input by its corresponding weight,

$$w_1x_1,\qquad w_2x_2,\qquad w_3x_3.$$

These weighted contributions are then added together,

$$z = w_1x_1 + w_2x_2 + w_3x_3.$$

This weighted sum represents the combined evidence provided by all input features.

Features that are biologically important receive large weights.

Features that contribute little to the prediction receive smaller weights.

During training, the network automatically learns these weights from the available data.

---

The role of the bias

The weighted sum alone is usually not sufficient.

Just as linear regression contains an intercept, artificial neurons contain an additional adjustable parameter called the bias.

Including the bias yields

$$z = w_1x_1 + w_2x_2 + \cdots + w_nx_n + b.$$

The bias allows the neuron to shift its response independently of the input values.

Geometrically, it changes the position of the decision boundary without altering its orientation.

Without the bias, the neuron’s decision boundary would always pass through the origin of the feature space.

This unnecessary restriction would significantly reduce the flexibility of the network.

The bias therefore plays exactly the same role as the intercept in linear regression.

---

A familiar equation

At this point, the equation should look remarkably familiar.

Compare it with the linear regression model introduced in Chapter 9:

$$\hat{y} = w_1x_1 + w_2x_2 + \cdots + w_nx_n + b.$$

Apart from the notation, the two equations are identical.

The neuron first calculates a weighted linear combination of the input features.

If we stopped here, an artificial neuron would simply perform multiple linear regression.

This observation leads to an important insight.

The power of neural networks does not arise from the weighted sum itself.

The weighted sum is simply a linear model.

The crucial step comes next.

---

Why linear models are not enough

Suppose we construct a network consisting of several layers, but every neuron performs only this weighted sum.

Would adding more layers increase the expressive power of the model?

Surprisingly, the answer is no.

A composition of linear transformations remains linear.

No matter how many linear layers we stack on top of one another, the resulting network is mathematically equivalent to a single linear transformation.

Consequently, a deep network without additional nonlinearities could never represent relationships more complex than those already captured by linear regression.

This would defeat the entire purpose of deep learning.

The breakthrough comes from introducing one additional operation after the weighted sum.

Instead of passing the value (z) directly to the next layer, every neuron first applies an activation function.

This simple nonlinear transformation fundamentally changes the expressive power of the network.

It enables neural networks to approximate highly complex nonlinear relationships that cannot be represented by linear models alone.

---

The neuron as a two-step computation

We can therefore view every artificial neuron as performing two consecutive operations.

First, it calculates a weighted linear combination,

$$z = \sum_{i=1}^{n} w_i x_i + b.$$

Second, it applies a nonlinear activation function,

$$a = \phi(z).$$

The value

$$a$$

is then transmitted to the next layer of the network.

Thus, every neuron consists of two conceptually distinct components:

1. a linear model that combines the available information,
2. a nonlinear activation function that allows the network to learn complex relationships.

Remarkably, every modern deep neural network—from image recognition systems to AlphaFold—is built by repeating this simple computation millions of times.

---

From neurons to intelligent systems

At first glance, it may seem surprising that such a simple computational unit can ultimately recognize faces, translate languages, or predict protein structures.

The secret lies not in the complexity of individual neurons.

Each neuron performs only a straightforward mathematical calculation.

The extraordinary capabilities of neural networks emerge because enormous numbers of these simple units are connected into deep hierarchical architectures.

Understanding the neuron therefore reveals an important principle of modern artificial intelligence.

Complex behavior does not necessarily require complex individual components.

Instead, complexity can emerge from the interaction of many simple computational elements.

The remaining question is therefore not how neurons combine information—that part is already understood.

Instead, we must ask:

How should the activation function transform the weighted sum?

As we shall see in the next section, this seemingly small mathematical modification is what makes deep learning possible.

---

Conceptual Summary

An artificial neuron performs two simple operations. First, it computes a weighted sum of its inputs, mathematically identical to a multiple linear regression model with a bias term. Second, it applies a nonlinear activation function to this weighted sum before passing the result to the next layer. The weighted sum alone is only a linear model; the activation function provides the nonlinearity that allows deep neural networks to represent highly complex biological relationships.

---

Self-Check Questions

1. What are the two computational steps performed by every artificial neuron?
2. What is the role of the weights in an artificial neuron?
3. Why is the bias term important?
4. In what sense is the weighted sum equivalent to multiple linear regression?
5. Why would a neural network consisting only of weighted sums remain a linear model?
6. Why is the activation function essential for deep learning?
7. Why can highly complex behavior emerge from networks composed of very simple neurons?