Consider binary classification, where the domain of the target variable is $\{0,1\}$. Multiple binary target variables can be learned separately.

The use of a linear function does not work well for such classification tasks; a learner should never make a prediction of greater than 1 or less than 0. However, a linear function could make a prediction of, say, 3 for one example just to fit other examples better.

A squashed linear function is of the form

where $f$, an activation function, is a function from the real line $[-\mathrm{\infty },\mathrm{\infty }]$ into some subset of the real line, such as $[0,1]$.

A prediction based on a squashed linear function is a linear classifier.

A simple activation function is the step function, $ste{p}_0(x)$, defined by

A step function was the basis for the perceptron [Rosenblatt, 1958], which was one of the early methods developed for learning. It is difficult to adapt gradient descent to step functions because gradient descent takes derivatives and step functions are not differentiable.

If the activation is (almost everywhere) differentiable, gradient descent can be used to update the weights. The step size might need to converge to zero to guarantee convergence.

One differentiable activation function is the sigmoid or logistic function:

This function, depicted in Figure 7.9, squashes the real line into the interval $(0,1)$, which is appropriate for classification because we would never want to make a prediction of greater than 1 or less than 0. It is also differentiable, with a simple derivative – namely, $\frac{d}{dx}sigmoid(x)=sigmoid(x)*(1-sigmoid(x))$.

The problem of determining weights for the sigmoid of a linear function that minimize an error on a set of examples is called logistic regression.

To optimize the log loss error for logistic regression, minimize the negative log-likelihood

where $\widehat{Y}(e)=sigmoid\left({\sum }_{i=0}^n{w}_i*X_i(e)\right)$.

where $\delta (e)=Y(e)-{\widehat{Y}}^{\overline{w}}(e)$. This is, essentially, the same as Equation 7.2, the only differences being the definition of the predicted value and the constant “2” which can be absorbed into the step size.

The $Linear\mathrm{\_}learner$ algorithm of Figure 7.8 can be modified to carry out logistic regression to minimize log loss by changing the prediction to be $sigmoid({\sum }_i{w}_i*X_i(e))$. The algorithm is show in Figure 7.10.

To minimize sum-of-squares error, instead, the prediction is the same, but the derivative is different. In particular, line 16 of Figure 7.10 should become

Consider each input feature as a dimension; if there are $n$ features, there will be $n$ dimensions. A hyperplane in an $n$-dimensional space is a set of points that all satisfy a constraint that some linear function of the variables is zero. The hyperplane forms an $(n-1)$-dimensional space. For example, in a (two-dimensional) plane, a hyperplane is a line, and in a three-dimensional space, a hyperplane is a plane. A classification is linearly separable if there exists a hyperplane where the classification is true on one side of the hyperplane and false on the other side.

The $Logistic\mathrm{\_}regression\mathrm{\_}learner$ algorithm can learn any linearly separable classification. The error can be made arbitrarily small for arbitrary sets of examples if, and only if, the target classification is linearly separable. The hyperplane is the set of points where ${\sum }_i{w}_i*X_i=0$ for the learned weights $\overline{w}$. On one side of this hyperplane, the prediction is greater than 0.5; on the other side, the prediction is less than 0.5.

Often it is difficult to determine a priori whether a data set is linearly separable.

When the domain of the target variable has more than two values – there are more than two classes – indicator variables can be used to convert the classification to binary variables. These binary variables could be learned separately. The predictions of the individual classifiers can be combined to give a prediction for the target variable. Because exactly one of the values must be true for each example, a learner should not predict that more than one will be true or that none will be true. Suppose we separately learned the predicted values for $Y_1\mathrm{\dots }{Y}_k$ are $q_1\mathrm{\dots }{q}_k$, where the $q_i\ge 0$. A learner that predicts a probability distribution could predict $Y=y_i$ with probability $q_i/{\sum }_j{q}_j$. A learner that must make a definitive prediction can predict the mode, a $y_i$ for which $q_i$ is the maximum.