The momentum for each parameter acts like a velocity of the update for each parameter, with the standard stochastic gradient descent update acting like an acceleration. The momentum acts as the step size for the parameter. It is increased if the acceleration is the same sign as the momentum and is decreased if the acceleration is the opposite sign.

To implement momentum, a velocity $v[i,j]$ is stored for each weight $w[i,j]$. Hyperparameter $\alpha$, with , specifies how much of the momentum should be used for each batch. The $update$ method for $Dense$ in Figure 8.3 becomes

The change in weight is not necessarily in the direction of the gradient of the error, because the momentum might be too great.

The value of $\alpha$ affects how much the step size can increase. Common values for $\alpha$ are 0.5, 0.9, and 0.99. If the gradients have the same value, the step size will approach ${\sum }_{i=0}^{\mathrm{\infty }}{\alpha }^i=1/(1-\alpha )$ times the step size without momentum. If $\alpha$ is 0.5, the step size could double. If $\alpha$ is 0.9, the step size could increase up to 10 times, and $\alpha =0.99$ allows the step size to increase by up to 100 times.

As well as handling canyons, momentum can average out noise due to the batches not including all the examples.

RMS-Prop (root mean squared propagation) is the default optimizer for the Keras deep learning library. The idea is that the magnitude of the change in each weight depends on how (the square of) the gradient for that weight compares to its historic value, rather than depending on the absolute value of the gradient. For each weight, a rolling average of the square of the gradient is maintained. This is used to determine how much the weight should change. A correction to avoid numeric instability of dividing by approximately zero is also used.

The hyperparameters for RMS-Prop are the learning rate $\eta$ (with a default of $0.001$ in Keras), $\rho$ (with a default of $0.9$ in Keras) that controls the time horizon of the rolling average, and $ϵ$ (defaults to ${10}^{-7}$ in Keras) to ensure numerical stability. The algorithm below maintains a rolling average of the square of the gradient for $w[i,j]$ in $r[i,j]$. The $update$ method for $Dense$ in Figure 8.3 becomes

To understand the algorithm, assume the value of $r[i,j]$ is initially much bigger than $ϵ$, so that $r[i,j]+ϵ\approx r[i,j]$.

• •

  When $r[i,j]\approx g^2$, the ratio $g/\sqrt{r[i,j]+ϵ}$ is approximately 1 or $-1$, depending on the sign of $g$, so the magnitude of the change in weight is approximately $\eta$.

• •

  When $g^2$ is bigger than $r[i,j]$, the error has a larger magnitude than its historical value, $r[i,j]$ is increased, and the step size increases.

• •

  When $g^2$ is smaller than $r[i,j]$, the value $r[i,j]$ is decreased, and the step size decreases. When a local minimum is approached, the values of $g^2$ and $r[i,j]$ become smaller than the magnitude of $ϵ$, so the updates are dominated by $ϵ$, and the step is small.

RMS-Prop only affects the magnitude of the step; the direction of change of $w[i,j]$ is always opposite to $d[i,j]$.

Adam, for “adaptive moments”, is an optimizer that uses both momentum and the square of the gradient, similar to RMS-Prop. It also uses corrections for the parameters to account for the fact that they are initialized at 0, which is not a good estimate to average with. Other mixes of RMS-Prop and momentum are also common.

It takes as hyperparameters: learning-rate ($\eta$), with a default of 0.001; ${\beta }_1$, with a default of 0.9; ${\beta }_2$, with a default of 0.999; and $ϵ$, with a default of ${10}^{-7}$. (Names and defaults are consistent with Keras.)

Adam takes the gradient, $d[i,j]/bath\mathrm{\_}size$, for the weight $w[i,j]$ and maintains a rolling average of the gradient in $s[i,j]$, a rolling average of the square of the gradient in $r[i,j]$, and the step number, $t$. These are all initialized to zero. To implement Adam, the $update$ method for $Dense$ in Figure 8.3 becomes:

The weight update step (line 7) is like RMS-Prop but uses $s[i,j]$ instead of the gradient in the numerator, and corrects it by dividing by $1-{\beta }_1^t$. Note that ${\beta }_1$ and ${\beta }_2$ are the names of the parameters, but the superscript is the power. In the first update, when $t=1$, $s[i,j]/(1-{\beta }_1^t)$ is equal to $g$; it corrects subsequent updates similarly. It also corrects $r[i,j]$ by dividing by $1-{\beta }_2^t$. Note that $ϵ$ is inside the square root in RMS-Prop, but is outside in Adam.

For convergence, it helps to normalize the input parameters, and sensibly initialize the weights.

For example, consider a dataset of people with features including height (in cm), the number of steps walked in a day, and a Boolean which specifies whether they have passed high school. These use very different units, and might have very different utility in predictions. To make it learn independently of the units, it is typical to normalize each real-valued feature, by subtracting the mean, so the result has a mean of 0, and dividing by the standard deviation, so the result has a standard deviation and variance of 1. Each feature is scaled independently.

Categorical inputs are usually represented as indicator variables, so that categorical variable $X$ with domain $\{v_1,\mathrm{\dots },v_k\}$ is represented as $k$ inputs, $X_1,\mathrm{\dots },X_k$. An input example with $X=v_j$ is represented with $X_j=1$ and every other $X_{j^{\prime }}=0$. This is also called a one-hot encoding.

The weights for the linear functions in hidden layers cannot all be assigned the same value, otherwise they will converge in lock-step, never learning different features. If the $w[i,j]$ parameters are initialized to the same value, all of the units at one layer will implement the same function. Thus they need to be initialized to random values. In Keras, the default initializer is the Glorot uniform initializer, which chooses values uniformly in the interval between $\pm \sqrt{6/(n_i+n_0)}$, where $n_i$ is the number of inputs and $n_o$ is the number of outputs for the linear function. This has a theoretical basis and tends to work well in practice for many cases. The biases (the weights multiplied by input unit 1) are typically initialized to 0.

For the output units, the bias can be initialized to the value that would predict best when all of the other inputs are zero. This is the mean for regression, or the inverse-sigmoid of the empirical probability for binary classification. The other output weights can be set to 0. This allows the gradient descent to learn the signal from the inputs, and not the background average.