A conditional probability distribution is a function on variables; given an assignment to the values of the variables, it gives a number. A factor is a function of a set of variables; the variables it depends on are the scope of the factor. Thus a conditional probability is a factor, as it is a function on variables. This section explores some variants for representing factors and conditional probabilities. Some of the representations are for arbitrary factors and some are specific to conditional probabilities.

Factors do not have to be implemented as conditional probability tables. The resulting tabular representation is often too large when there are many parents. Often, structure in conditional probabilities can be exploited.

One such structure exploits context-specific independence, where one variable is conditionally independent of another, given a particular value of the third variable.

Context-specific independence may be exploited in a representation by not requiring numbers that are not needed. A simple representation for conditional probabilities that models context-specific independence is a decision tree, where the parents in a belief network correspond to the input features and the child corresponds to the target feature. Another representation is in terms of definite clauses with probabilities. Context-specific independence could also be represented as tables that have contexts that specify when they should be used, as in the following example.

Another common representation is a noisy-or, where the child is true if one of the parents is activated and each parent has a probability of activation. So the child is an “or” of the activations of the parents. The noisy-or is defined as follows. If $X$ has Boolean parents $V_1,\mathrm{\dots },V_k$, the probability is defined by $k+1$ parameters $p_0,\mathrm{\dots },p_k$. We invent $k$ new Boolean variables $A_0,A_1,\mathrm{\dots },A_k$, where for each $i>0$, $A_i$ has $V_i$ as its only parent. Define $P(A_i=true\mid V_i=true)=p_i$ and $P(A_i=true\mid V_i=false)=0$. The bias term, $A_0$ has $P(A_0)=p_0$. The variables $A_0,\mathrm{\dots },A_k$ are the parents of $X$, and the conditional probability is that $P(X\mid A_0,A_1,\mathrm{\dots },A_k)$ is 1 if any of the $A_i$ are true and is 0 if all of the $A_i$ are false. Thus $p_0$ is the probability of $X$ when all of $V_i$ are false; the probability of $X$ increases if more of the $V_i$ become true.

A log-linear model is a model where probabilities are specified as a product of terms. When the terms are non-zero (they are all strictly positive), the log of a product is the sum of logs. The sum of terms is often a convenient term to work with. To see how such a form is used to represent conditional probabilities, we can write the conditional probability in the following way:

• •

  The sigmoid function, $sigmoid(x)=1/(1+e^{-x})$, plotted in Figure 7.9, has been used previously in this book for logistic regression and neural networks.

• •

  The conditional odds (as often used by bookmakers in gambling) is

  	$odds(h\mid e)=$	${\displaystyle \frac{P(h\wedge e)}{P(\mathrm{\neg }h\wedge e)}}$
  	$=$	${\displaystyle \frac{P(e\mid h)}{P(e\mid \mathrm{\neg }h)}}*{\displaystyle \frac{P(h)}{P(\mathrm{\neg }h)}}$

  where $\frac{P(h)}{P(\mathrm{\neg }h)}=\frac{P(h)}{1-P(h)}$ is the prior odds and $\frac{P(e\mid h)}{P(e\mid \mathrm{\neg }h)}$ is the likelihood ratio. For a fixed $h$, it is often useful to represent $P(e\mid h)/P(e\mid \mathrm{\neg }h)$ as a product of terms, and so the log is a sum of terms.

The logistic regression model of a conditional probability $P(X\mid Y_1,\mathrm{\dots },Y_k)$ is of the form

where $Y_i$ is assumed to have domain $\{0,1\}$. (Assume a dummy input $Y_0$ which is always 1.) This corresponds to a decomposition of the conditional probability, where the probabilities are a product of terms for each $Y_i$.

Note that $P(X\mid Y_1=0,\mathrm{\dots },Y_k=0)=sigmoid(w_0)$. Thus $w_0$ determines the probability when all of the parents are zero. Each $w_i$ specifies a value that should be added as $Y_i$ changes. If $Y_i$ is Boolean with values $\{0,1\}$, then $P(X\mid Y_1=0,\mathrm{\dots },Y_i=1,\mathrm{\dots },Y_k=0)=sigmoid(w_0+w_i)$. The logistic regression model makes the independence assumption that the influence of each parent on the child does not depend on the other parents. Learning logistic regression models was the topic of Section 7.3.2.

Noisy-or and logistic regression models are similar, but different. Noisy-or is typically used when the causal assumption that a variable is true if it is caused to be true by one of the parents, is appropriate. Logistic regression is used when the various parents add-up to influence the child.