Section 7.6: Nonlinear regression

Problem: For a given class of functions ${\mathcal F}$ and a set of data $\{ (x_1,y_1), \ldots, (x_n,y_n) \}$ find a (the?) function $f$ of class ${\mathcal F}$ which minimizes the sum of squares of errors.

Linear regression is the solution to this problem where ${\mathcal F}$ is the class of linear functions.

Quadratic regression

A quadratic function is a function $f(x)$ of the form $f(x) = a x^2 + b x + c$ for fixed constants $a$, $b$, and $c$.

Note that taking $a = 0$ we see that a linear function is a special case of a quadratic function.

Large An example of quadratic regression

Find a quadratic function $f(x) = a x^2 + b x + c$ minimizing the sum of squares of errors for the data $\{ (-1,0), (1,2), (0,-1), (2,1) \}$.

Solution

$$\begin{eqnarray*} S(a,b,c) & = & (f(-1) - 0)^2 + (f(1) - 2)^2 \\ & & + (f(0) -... ...2 + (a + b + c - 2)^2 \\ & & + (c + 1)^2 + (4a + 2b + c - 1)^2 \end{eqnarray*}$$

Solution, continued

$$\begin{eqnarray*} \frac{\partial S}{\partial a} & = & 2 [(a - b + c) +(a + b +... ...+ (c + 1) + (4a + 2b + c - 1) ] \\ & = & 2[ 6a + 2b + 4c - 2 ] \end{eqnarray*}$$

Setting these equal to zero and performing some algebra, we find $a = 0$, $b = \frac{3}{5}$, and $c = \frac{1}{5}$.

Other forms of nonlinear regression

• Exponential regression: $f(x) = a e^{bx}$
• Power regression: $f(x) = a x^b$
• Logarithmic regression: $f(x) = a \ln(x) + b$
• Cubic regression: $f(x) = a x^3 + b x^2 + c x + d$
• Polynomial regression of degree $m$: $f(x) = a_m x^m + a_{m-1} x^{m-1} + \cdots + a_1 x + a_0 = \sum_{i=0}^m a_i x^i$

Multiple regression

Problem: Given a set of data $\{ (x_1, y_1, z_1), \ldots, (x_n,y_n,z_n) \}$ find a function $f(x,y)$ of two variables (of some specified class) best fitting the data.

When $f(x,y) = ax + by + c$ is taken to be a linear function, then this is the problem of multiple linear regression.