10.2 Building Flexible Models

From straight lines to flexible functions

In the previous section, we argued that many biological relationships are too complex to be captured by a simple straight line. A natural response is therefore to build a more flexible mathematical model.

At first glance, this appears straightforward. Instead of fitting a line, we might fit a curve that follows the data more closely. If the relationship between hydrophobicity and protein stability bends, levels off, or changes direction, a curved function should provide a better description than a linear one.

This intuition is correct—but it immediately raises another question.

How can we construct models whose flexibility can be adjusted?

Rather than choosing between only a line or an arbitrary curve, we would like a systematic way of building models that range from very simple to highly flexible. Such models allow us to explore how increasing complexity affects predictive performance.

The key idea is surprisingly simple: complex functions can often be constructed by combining many simple functions.

Building complex models from simple building blocks

A useful analogy is the construction of a molecule.

Complex biological molecules are rarely created in a single step. Instead, they are assembled from simpler building blocks. Amino acids form proteins, nucleotides form DNA, and individual cells form tissues.

Mathematical models can be constructed in a very similar way.

Instead of describing a biological relationship with a single equation, we combine several simpler functions, each contributing one part of the overall prediction.

Mathematically, this idea can be written as

f(x)=\sum_{i=1}^{n} w_i h_i(x),

where

(h_i(x)) are simple basis functions,
(w_i) are weights that determine how strongly each basis function contributes,
and (f(x)) is the resulting predictive model.

The basis functions can take many different forms. They may be straight lines, polynomials, Gaussian functions, or localized functions that are active only within a small region of the input space.

Although the mathematical details differ, the underlying principle is always the same:

Complex behavior emerges by combining many simple components.

This idea will reappear throughout machine learning, including in artificial neural networks, where individual neurons act as simple computational units that collectively learn highly complex functions.

Increasing model complexity

The flexibility of a model depends largely on the number and type of basis functions that it contains.

With only a single basis function, the model is extremely simple. It may capture only the overall trend of the data.

Adding additional basis functions allows the model to represent increasingly subtle patterns.

Imagine trying to describe the outline of a mountain range.

A single straight line captures only the average slope.

Adding a few line segments allows major peaks and valleys to emerge.

Adding many short segments eventually produces a curve that closely follows every detail of the landscape.

Exactly the same idea applies to predictive models.

As additional basis functions are introduced, the model becomes increasingly flexible and can represent progressively more complicated biological relationships.

This flexibility is extremely powerful because it allows us to approximate relationships that would be impossible to describe using a simple linear model.

Piecewise models and splines

One particularly useful way of increasing flexibility is to divide the input space into several regions and describe each region separately.

Instead of fitting one global line through all observations, we fit several simpler functions that are connected smoothly.

These models are known as splines.

Conceptually, splines are easy to understand.

Imagine bending a thin strip of flexible wood through a series of points. The strip naturally forms a smooth curve that follows the overall trend without making abrupt changes in direction.

Mathematically, splines behave in much the same way.

The input space is divided into intervals, and within each interval a simple polynomial is fitted. The individual pieces are then joined together so that the overall function remains smooth.

The important point is not the mathematics of splines themselves.

Rather, splines illustrate a general principle:

Model flexibility can be increased gradually rather than all at once.

This allows us to construct models that are neither rigid straight lines nor completely unrestricted curves.

Flexibility is both a strength and a risk

At first sight, increasing model flexibility seems entirely beneficial.

A more flexible model can describe more complicated biological relationships and therefore usually fits the training data more closely.

However, this apparent advantage conceals an important danger.

Suppose we continue increasing the flexibility of the model.

Eventually, it may begin to follow not only the underlying biological relationship but also every small fluctuation caused by experimental noise and random biological variation.

The model then starts describing accidental details that are unique to the training dataset rather than general biological principles.

Ironically, a model that fits the training data almost perfectly may perform poorly when applied to new observations.

This is one of the central paradoxes of machine learning:

A model can become better at explaining the data it has already seen while becoming worse at predicting data it has never seen before.

Understanding this paradox is essential for building reliable predictive models.

It motivates one of the most important concepts in machine learning: generalization.

The remainder of this chapter is devoted to understanding why increasing model complexity eventually becomes counterproductive and how we can identify the level of complexity that provides the best predictive performance.

Key Concepts

Complex predictive models can be constructed by combining many simple functions.
Basis functions provide the building blocks for flexible mathematical models.
Increasing the number of basis functions increases model complexity.
Splines illustrate how flexibility can be increased gradually while maintaining smooth predictions.
Greater flexibility improves the ability to fit training data but also increases the risk of modeling noise rather than biological relationships.

Summary

Moving beyond linear regression requires models that can represent nonlinear biological relationships. Flexible models achieve this by combining many simple basis functions into a more expressive predictive function. While increasing flexibility allows increasingly complex patterns to be captured, it also introduces a fundamental trade-off: highly flexible models may begin to fit random fluctuations instead of the underlying biology. This tension between flexibility and predictive reliability lies at the heart of model selection.

Self-Check Questions

Why are many biological relationships difficult to describe using a straight line?
How can complex mathematical models be constructed from simple building blocks?
What role do basis functions play in predictive modeling?
Why does increasing the number of basis functions increase model complexity?
What is the central idea behind spline models?
Why can a more flexible model eventually become less useful for prediction?
Why does increasing model complexity naturally lead to the problem of generalization?