To see why this is, note that the Hessian of is equal to

This matrix is equal to , where is the identity matrix and is the all-ones matrix, which is rank 1 and whose single non-zero eigenvalue is . Therefore, this matrix has eigenvalues of , as well as a single eigenvalue of , and hence is not positive definite.

On the other hand, any submatrix of size is of the form , but where now is only . This matrix now has eigenvalues of , together with a single eigenvalue of , and hence is positive definite. Therefore, the Hessian is positive definite when restricted to any variables, and hence is convex in any variables, but not in all variables jointly.

For many inference tasks, especially ones with either non-linearities or non-convexities, it is common to use particle-based methods such as beam search, particle filters, sequential Monte Carlo, or Markov Chain Monte Carlo. In these methods, we approximate a distribution by a collection of samples from that distribution, then update the samples as new information is added. For instance, in beam search, if we are trying to build up a tree, we might build up a collection of samples for the left and right subtrees, then look at all ways of combining them into the entire tree, but then downsample again to the trees with the highest scores. This allows us to search through the exponentially large space of all trees efficiently (albeit at the cost of possibly missing high-scoring trees).

One major problem with such particle-based methods is diversity: the particles will tend to cluster around the highest-scoring mode, rather than exploring multiple local optima if they exist. This can be bad because it makes learning algorithms overly myopic. Another problem, especially in combinatorial domains, is difficulty of partial evaluation: if we have some training data that we are trying to fit to, and we have chosen settings of some, but not all, variables in our model, it can be difficult to know if that setting is on the right track (for instance, it can be difficult to know whether a partially-built tree is a promising candidate or not). For time-series modeling, this isn’t nearly as large of a problem, since we can evaluate against a prefix of the time series to get a good idea (this perhaps explains the success of particle filters in these domains).

I’ve been working on a method that tries to deal with both of these problems, which I call probabilistic abstractions. The idea is to improve the diversity of particle-based methods by creating “fat” particles which cover multiple states at once; the reason that such fat particles help is that they allow us to first optimize for coverage (by placing down relatively large particles that cover the entire space), then later worry about more local details (by placing down many particles near promising-looking local optima).

To be more concrete, if we have a probability distribution over a set of random variables , then our particles will be sets obtained by specifying the values of some of the and leaving the rest to vary arbitrarily. So, for instance, if , then might be a possible “fat” particle.

By choosing some number of fat particles and assigning probabilities to them, we are implicitly specifying a polytope of possible probability distributions; for instance, if our particles are , and we assign probability to , then we have the polytope of distributions that satisfy the constraints , etc.

Given such a polytope, is there a way to pick a canonical representative from it? One such representative is the maximum entropy distribution in that polytope. This distribution has the property of minimizing the worst-case relative entropy to any other distribution within the polytope (and that worst-case relative entropy is just the entropy of the distribution).

Suppose that we have a polytope for two independent distributions, and we want to compute the polytope for their product. This is easy — just look at the cartesian products of each particle of the first distribution with each particle of the second distribution. If each individual distribution has particles, then the product distribution has particles — this could be problematic computationally, so we also want a way to narrow down to a subset of the most informative particles. These will be the particles such that the corresponding polytope minimizes the maximum entropy of that polytope. Finding this is NP-hard in general, but I’m currently working on good heuristics for computing it.

Next, suppose that we have a distribution on a space and want to apply a function to it. If is a complicated function, it might be difficult to propagate the fat particles (even though it would have been easy to propagate particles composed of single points). To get around this, we need what is called a valid abstraction of : a function such that for all . In this case, if we map a particle to , our equality constraint on the mass assigned to becomes a lower bound on the mass assigned to — we thus still have a polytope of possible probability distributions. Depending on the exact structure of the particles (i.e. the exact way in which the different sets overlap), it may be necessary to add additional constraints to the polytope to get good performance — I feel like I have some understanding of this, but it’s something I’ll need to investigate empirically as well. It’s also interesting to note that (when combined with conditioning on data, which is discussed below) allows us to assign partial credit to promising particles, which was the other property I discussed at the beginning.

Finally, suppose that I want to condition on data. In this case the polytope approach doesn’t work as well, because conditioning on data can blow up the polytope by an arbitrarily large amount. Instead, we just take the maximum-entropy distribution in our polytope and treat that as our “true” distribution, then condition. I haven’t been able to make any formal statements about this procedure, but it seems to work at least somewhat reasonably. It is worth noting that conditioning may not be straightforward, since the likelihood function may not be constant across a given fat particle. To deal with this, we can replace the likelihood function by its average (which I think can be justified in terms of maximum entropy as well, although the details here are a bit hazier).

So, in summary, we have a notion of fat particles, which provide better coverage than point particles, and can combine them, apply functions to them, subsample them, and condition on data. This is essentially all of the operations we want to be able to apply for particle-based methods, so we in theory should now be able to implement versions of these particle-based methods that get better coverage.

The point of this post is to construct an equality case for Chebyshev: a collection of pairwise independent random variables whose sum does not satisfy the concentration bound of Chernoff, and instead decays like .

The construction is as follows: let be independent binary random variables, and for any , let , where the sum is taken mod 2. Then we can easily check that the are pairwise independent. Now consider  the random variable . If any of the is equal to 1, then we can pair up the by either adding or removing from to get the other element of the pair. If we do this, we see that in this case. On the other hand, if all of the are equal to 0, then as well. Thus, with probability , deviates from its mean by , whereas the variance of is . The bound on this probability form Chebyshev is , which is very close to , so this constitutes something very close to the Chebyshev equality case.

Anyways, I just thought this was a cool example that demonstrates the difference between pairwise and full independence.

Problem: Suppose that we have a sequence of pairs of non-negative numbers such that and . Devise an efficient algorithm to find the pairs that maximize

Commentary: I don’t have a fully satisfactory solution to this yet, although I do think I can find an algorithm that runs in time and finds pairs that do at least as well as the best set of pairs. It’s possible I need to assume something like instead of just (and similarly for the ), although I’m happy to make that assumption.

While attempting to solve this problem, I’ve managed to utilize a pretty large subset of my bag of tricks for optimization problems, so I think working on it is pretty worthwhile intellectually. It also happens to be important to my research, so if anyone comes up with a good algorithm I’d be interested to know.

Theorem 1: If A and B are symmetric matrices with eigenvalues and respectively, then .

For such a natural-looking statement, this was surprisingly hard to prove. However, I finally came up with a proof, and it was cool enough that I felt the need to share. To prove this, we actually need two ingredients. The first is the Cauchy Interlacing Theorem:

Theorem 2: If A is an symmetric matrix and B is an principle submatrix of A, then , where is the ith largest eigenvalue of X.

As a corollary we have:

Corollary 1: For any symmetric matrix X, .

Proof: The left-hand-side is just the trace of the upper-left principle submatrix of X, whose eigenvalues are by Theorem 2 bounded by the k largest eigenvalues of X.

The final ingredient we will need is a sort of “majorization” inequality based on Abel summation:

Theorem 3: If and are such that for all k (with equality when ), and , then .

Proof: We have:

where the equalities come from the Abel summation method.

Now, we are finally ready to prove the original theorem:

Proof of Theorem 1: First note that since the trace is invariant under similarity transforms, we can without loss of generality assume that A is diagonal, in which case we want to prove that . But by Corollary 1, we also know that for all k. Since by assumption the are a decreasing sequence, Theorem 3 then implies that , which is what we wanted to show.

Note that . Intuitively, a small KL(p || q) means that there are few points that p assigns high probability to but that q does not. We can also think of KL(p || q) as the number of bits of information needed to update from the distribution q to the distribution p.

Suppose that p and q are both mixtures of other distributions: and . Can we bound in terms of the ? In some sense this is asking to upper bound the KL divergence in terms of some more local KL divergence. It turns out this can be done:

Theorem: If and and are all probability distributions, then

.

Proof: If we expand the definition, then we are trying to prove that

We will in fact show that this is true for every value of , so that it is certainly true for the integral. Using , re-write the condition for a given value of as

(Note that the sign of the inequality flipped because we replaced the two expressions with their negatives.) Now, this follows by using Jensen’s inequality on the function:

This proves the inequality and therefore the theorem.

Remark: Intuitively, if we want to describe in terms of , it is enough to first locate the th term in the sum and then to describe in terms of . The theorem is a formalization of this intuition. In the case that , it also says that the KL divergence between two different mixtures of the same set of distributions is at most the KL divergence between the mixture weights.

“Under what conditions will a set of polynomials be quadratically independent, in the sense that is a linearly independent set?”

I wasn’t able to make much progress on this general question, but in the specific setting where the are all polynomials in one variable, and we further restrict to just monomials, (i.e. for some ), the condition is just that there are no distinct unordered pairs such that . Arun was interested in the largest such a set could be for a given maximum degree , so we are left with the following interesting combinatorics problem:

“What is the largest subset of such that no two distinct pairs of elements of have the same sum?”

For convenience of notation let denote the size of . A simple upper bound is , since there are pairs to take a sum of, and all pairwise sums lie between and . We therefore have .

What about lower bounds on n? If we let S be the powers of 2 less than or equal to D, then we get a lower bound of ; we can do slightly better by taking the Fibonacci numbers instead, but this still only gives us logarithmic growth. So the question is, can we find sets that grow polynomially in D?

It turns out the answer is yes, and we can do so by choosing randomly. Let each element of be placed in S with probability p. Now consider any k, . If k is odd, then there are (k-1)/2 possible pairs that could add up to k: (1,k-1), (2,k-2),…,((k-1)/2,(k+1)/2). The probability of each such pair existing is . Note that each of these events is independent.

S is invalid if and only if there exists some k such that more than one of these pairs is active in S. The probability of any two given pairs being simultaneously active is , and there are such pairs for a given , hence such pairs total (since we were just looking at odd k). Therefore, the probability of an odd value of k invalidating S is at most .

For even we get much the same result except that the probability for a given value of comes out to the slightly more complicated formula , so that the total probability of an even value of k invalidating S is at most .

Putting this all together gives us a bound of . If we set p to be $\frac{1}{2}D^{-\frac{3}{4}}$ then the probability of S being invalid is then at most , so with probability at least a set S with elements chosen randomly with probability will be valid. On the other hand, such a set has elements in expectation, and asymptotically the probability of having at least this many elements is . Therefore, with probability at least a randomly chosen set will be both valid and have size greater than , which shows that the largest value of is at least .

We can actually do better: if all elements are chosen with probability , then one can show that the expected number of invalid pairs is at most , and hence we can pick randomly with probability , remove one element of each of the invalid pairs, and still be left with elements in S.

So, to recap: choosing elements randomly gives us S of size ; choosing randomly and then removing any offending pairs gives us S of size ; and we have an upper bound of . What is the actual asymptotic answer? I don’t actually know the answer to this, but I thought I’d share what I have so far because I think the techniques involved are pretty cool.

1. Exponential Families

An exponential family is a family of probability distributions, parameterized by , of the form

Notice the similarity to the definition of a log-linear model, which is

So, a log-linear model is simply an exponential family model with . Note that we can re-write the right-hand-side of (1) as , so an exponential family is really just a log-linear model with one of the coordinates of constrained to equal . Also note that the normalization constant in (1) is a function of (since fully specifies the distribution over ), so we can express (1) more explicitly as

where

Exponential families are capable of capturing almost all of the common distributions you are familiar with. There is an extensive table on Wikipedia; I’ve also included some of the most common below:

1. Gaussian distributions. Let . Then . If we let , then . We therefore see that Gaussian distributions are an exponential family for .
2. Poisson distributions. Let and . Then . If we let then we get ; we thus see that Poisson distributions are also an exponential family.
3. Multinomial distributions. Suppose that . Let be an -dimensional vector whose th element is and where all other elements are zero. Then . If , then we obtain an arbitrary multinomial distribution. Therefore, multinomial distributions are also an exponential family.

2. Sufficient Statistics

A statistic of a random variable is any deterministic function of that variable. For instance, if is a vector of Gaussian random variables, then the sample mean and sample variance are both statistics.

Let be a family of distributions parameterized by , and let be a random variable with distribution given by some unknown . Then a vector of statistics are called sufficient statistics for if they contain all possible information about , that is, for any function , we have

for some function that has no dependence on .

For instance, let be a vector of independent Gaussian random variables with unknown mean and variance . It turns out that is a sufficient statistic for and . This is not immediately obvious; a very useful tool for determining whether statistics are sufficient is the Fisher-Neyman factorization theorem:

Theorem 1 (Fisher-Neyman) Suppose that has a probability density function . Then the statistics are sufficient for if and only if can be written in the form

In other words, the probability of can be factored into a part that does not depend on , and a part that depends on only via .

What is going on here, intuitively? If depended only on , then would definitely be a sufficient statistic. But that isn’t the only way for to be a sufficient statistic — could also just not depend on at all, in which case would trivially be a sufficient statistic (as would anything else). The Fisher-Neyman theorem essentially says that the only way in which can be a sufficient statistic is if its density is a product of these two cases.

Proof: If (6) holds, then we can check that (5) is satisfied:

where the right-hand-side has no dependence on .

On the other hand, if we compute for an arbitrary density , we get

If the right-hand-side cannot depend on for any choice of , then the term that we multiply by must not depend on ; that is, must be some function that depends only on and and not on . On the other hand, the denominator depends only on and ; call this dependence . Finally, note that is a deterministic function of , so let . We then see that , which is the same form as (6), thus completing the proof of the theorem.

Now, let us apply the Fisher-Neyman theorem to exponential families. By definition, the density for an exponential family factors as

If we let and , then the Fisher-Neyman condition is met; therefore, is a vector of sufficient statistics for the exponential family. In fact, we can go further:

Theorem 2 Let be drawn independently from an exponential family distribution with fixed parameter . Then the empirical expectation is a sufficient statistic for .

Proof: The density for given is

Letting and , we see that the Fisher-Neyman conditions are satisfied, so that is indeed a sufficient statistic.

Finally, we note (without proof) the same relationship as in the log-linear case to the gradient and Hessian of with respect to the model parameters:

Theorem 3 Again let be drawn from an exponential family distribution with parameter . Then the gradient of with respect to is

and the Hessian is

This theorem provides an efficient algorithm for fitting the parameters of an exponential family distribution (for details on the algorithm, see the part near the end of the log-linear models post on parameter estimation).

3. Moments of an Exponential Family

If is a real-valued random variable, then the th moment of is . In general, if is a random variable on , then for every sequence of non-negative integers, there is a corresponding moment .

In exponential families there is a very nice relationship between the normalization constant and the moments of . Before we establish this relationship, let us define the moment generating function of a random variable as .

Lemma 4 The moment generating function for a random variable is equal to

The proof of Lemma 4 is a straightforward application of Taylor’s theorem, together with linearity of expectation (note that in one dimension, the expression in Lemma 4 would just be ).

We now see why is called the moment generating function: it is the exponential generating function for the moments of . The moment generating function for the sufficient statistics of an exponential family is particularly easy to compute:

Lemma 5 If , then .

Proof:

where the last step uses the fact that is a probability density and hence .

Now, by Lemma 4, is just the coefficient in the Taylor series for the moment generating function , and hence we can compute as . Combining this with Lemma 5 gives us a closed-form expression for in terms of the normalization constant :

Lemma 6 The moments of an exponential family can be computed as

For those who prefer cumulants to moments, I will note that there is a version of Lemma 6 for cumulants with an even simpler formula.

Exercise: Use Lemma 6 to compute , where is a Gaussian with mean and variance .

4. Conjugate Priors

Given a family of distributions , a conjugate prior family is a family that has the property that

for some depending on and . In other words, if the prior over lies in the conjugate family, and we observe , then the posterior over also lies in the conjugate family. This is very useful algebraically as it means that we can get our posterior simply by updating the parameters of the prior. The following are examples of conjugate families:

1. (Gaussian-Gaussian) Let , and let . Then, by Bayes’ rule,

Therefore, parameterize a family of priors over that is conjugate to .

• (Beta-Bernoulli) Let , , , and . The distribution over given is then called a Bernoulli distribution, and that of given and is called a beta distribution. Note that can also be written as . From this, we see that the family of beta distributions is a conjugate prior to the family of Bernoulli distributions, since

• (Gamma-Poisson) Let for . Let . As noted before, the distribution for given is called a Poisson distribution; the distribution for given and is called a gamma distribution. We can check that the family of gamma distributions is conjugate to the family of Poisson distributions.Important note: unlike in the last two examples, the normalization constant for the Poisson distribution actually depends on , and so we need to include it in our calculations:

  

  Note that, in general, a family of distributions will always have some conjugate family, as if nothing else the family of all probability distributions over will be a conjugate family. What we really care about is a conjugate family that itself has nice properties, such as tractably computable moments.

  Conjugate priors have a very nice relationship to exponential families, established in the following theorem:

  Theorem 7 Let be an exponential family. Then is a conjugate prior for for any choice of . The update formula is . Furthermore, is itself an exponential family, with sufficient statistics .

  Checking the theorem is a matter of straightforward algebra, so I will leave the proof as an exercise to the reader. Note that, as before, there is no guarantee that will be tractable; however, in many cases the conjugate prior given by Theorem 7 is a well-behaved family. See this Wikipedia page for examples of conjugate priors, many of which correspond to exponential family distributions.

  5. Maximum Entropy and Duality

  The final property of exponential families I would like to establish is a certain duality property. What I mean by this is that exponential families can be thought of as the maximum entropy distributions subject to a constraint on the expected value of their sufficient statistics. For those unfamiliar with the term, the entropy of a distribution over with density is . Intuitively, higher entropy corresponds to higher uncertainty, so a maximum entropy distribution is one specifying as much uncertainty as possible given a certain set of information (such as the values of various moments). This makes them appealing, at least in theory, from a modeling perspective, since they “encode exactly as much information as is given and no more”. (Caveat: this intuition isn’t entirely valid, and in practice maximum-entropy distributions aren’t always necessarily appropriate.)

  In any case, the duality property is captured in the following theorem:

  Theorem 8 The distribution over with maximum entropy such that lies in the exponential family with sufficient statistic and .

  Proving this fully rigorously requires the calculus of variations; I will instead give the “physicist’s proof”. Proof: } Let be the density for . Then we can view as the solution to the constrained maximization problem:

  

  By the method of Lagrange multipliers, there exist and such that

  

  This simplifies to:

  

  which implies

  

  for some and . In particular, if we let and , then we recover the exponential family with , as claimed.

  6. Conclusion

  Hopefully I have by now convinced you that exponential families have many nice properties: they have conjugate priors, simple-to-fit parameters, and easily-computed moments. While exponential families aren’t always appropriate models for a given situation, their tractability makes them the model of choice when no other information is present; and, since they can be obtained as maximum-entropy families, they are actually appropriate models in a wide family of circumstances.

Suppose that we want to maximize the Rayleigh quotient  of a matrix . There are many reasons we might want to do this, for instance of is symmetric then the maximum corresponds to the largest eigenvalue. There are also many ways to do this, and the one that I’m about to describe is definitely not the most efficient, but it has the advantage of being flexible, in that it easily generalizes to constrained maximizations, etc.

The first observation is that is homogeneous, meaning that scaling doesn’t affect the result. So, we can assume without loss of generality that , and we end up with the optimization problem:

maximize

subject to

This is where the trace trick comes in. Recall that the trace of a matrix is the sum of its diagonal entries. We are going to use two facts: first, the trace of a number is just the number itself. Second, trace(AB) = trace(BA). (Note, however, that trace(ABC) is not in general equal to trace(BAC), although trace(ABC) is equal to trace(CAB).) We use these two properties as follows — first, we re-write the optimization problem as:

maximize

subject to

Second, we re-write it again using the invariance of trace under cyclic permutations:

maximize

subject to

Now we make the substitution :

maximize

subject to

Finally, note that a matrix can be written as if and only if is positive semi-definite and has rank 1. Therefore, we can further write this as

maximize

subject to

Aside from the rank 1 constraint, this would be a semidefinite program, a type of problem that can be solved efficiently. What happens if we drop the rank 1 constraint? Then I claim that the solution to this program would be the same as if I had kept the constraint in! Why is this? Let’s look at the eigendecomposition of , written as , with (by positive semidefiniteness) and (by the trace constraint). Let’s also look at , which can be written as . Since is just a convex combination of the , we might as well have just picked to be , where is chosen to maximize . If we set that to 1 and all the rest to 0, then we maintain all of the constraints while increasing , meaning that we couldn’t have been at the optimum value of unless was equal to 1. What we have shown, then, is that the rank of must be 1, so that the rank 1 constraint was unnecessary.

Technically, could be a linear combination of rank 1 matrices that all have the same value of , but in that case we could just pick any one of those matrices. So what I have really shown is that at least one optimal point has rank 1, and we can recover such a point from any solution, even if the original solution was not rank 1.

Here is a problem that uses a similar trick. Suppose we want to find that simultaneously satisfies the equations:

for each (this example was inspired from the recent NIPS paper by Ohlsson, Yang, Dong, and Sastry, although the idea itself goes at least back to Candes, Strohmer, and Voroninski). Note that this is basically equivalent to solving a system of linear equations where we only know each equation up to a sign (or a phase, in the complex case). Therefore, in general, this problem will not have a unique solution. To ensure the solution is unique, let us assume the very strong condition that whenever for all , the matrix must itself be zero (note: Candes et al. get away with a much weaker condition). Given this, can we phrase the problem as a semidefinite program? I highly recommend trying to solve this problem on your own, or at least reducing it to a rank-constrained SDP, so I’ll include the solution below a fold.

Solution. We can, as before, re-write the equations as:

and further write this as

As before, drop the rank 1 constraint and let . Then we get:

,

which we can re-write as . But if is the true solution, then we also know that , so that for all . By the non-degeneracy assumption, this implies that

,

so in particular . Therefore, is the only solution to the semidefinite program even after dropping the rank constraint.

where maps a data point to a feature vector; they are called log-linear because the log of the probability is a linear function of . We refer to as the score of .

This model class might look fairly restricted at first, but the real magic comes in with the feature vector . In fact, every probabilistic model that is absolutely continuous with respect to Lebesgue measure can be represented as a log-linear model for sufficient choices of and . This is actually trivially true, as we can just take to be and to be .

You might object to this choice of , since it maps into rather than , and feature vectors are typically discrete. However, we can do just as well by letting , where the th coordinate of is the th digit in the binary representation of , then let be the vector .

It is important to distinguish between the ability to represent an arbitrary model as log-linear and the ability to represent an arbitrary family of models as a log-linear family (that is, as the set of models we get if we fix a choice of features and then vary ). When we don’t know the correct model in advance and want to learn it, this latter consideration can be crucial. Below, I give two examples of model families and discuss how they fit (or do not fit) into the log-linear framework. Important caveat: in both of the models below, it is typically the case that at least some of the variables involved are unobserved. However, we will ignore this for now, and assume that, at least at training time, all of the variables are fully observed (in other words, we can see and in the hidden Markov model and we can see the full tree of productions in the probabilistic context free grammar).

Hidden Markov Models. A hidden Markov model, or HMM, is a model with latent (unobserved) variables together with observed variables . The distribution for depends only on , and the distribution for depends only on (in the sense that is conditionally independent of given ). We can thus summarize the information in an HMM with the distributions and .

We can express a hidden Markov model as a log-linear model by defining two classes of features: (i) features that count the number of such that and ; and (ii) features that count the number of such that and . While this choice of features yields a model family capable of expressing an arbitrary hidden Markov model, it is also capable of learning models that are not hidden Markov models. In particular, we would like to think of (the index of corresponding to ) as , but there is no constraint that for each , whereas we do necessarily have for each . If is fixed, we still do obtain an HMM for any setting of , although will have no simple relationship with . Furthermore, the relationship depends on , and will therefore not work if we care about multiple Markov chains with different lengths.

Is the ability to express models that are not HMMs good or bad? It depends. If we know for certain that our data satisfy the HMM assumption, then expanding our model family to include models that violate that assumption can only end up hurting us. If the data do not satisfy the HMM assumption, then increasing the size of the model family may allow us to overcome what would otherwise be a model mis-specification. I personally would prefer to have as much control as possible about what assumptions I make, so I tend to see the over-expressivity of HMMs as a bug rather than a feature.

Probabilistic Context Free Grammars. A probabilistic context free grammar, or PCFG, is simply a context free grammar where we place a probability distribution over the production rules for each non-terminal. For those unfamiliar with context free grammars, a context free grammar is specified by:

1. A set of non-terminal symbols, including a distinguished initial symbol .
2. A set of terminal symbols.
3. For each , one or more production rules of the form , where and .

For instance, a context free grammar for arithmetic expressions might have , , and the following production rules:

• for all
• 
• 
• 
• 
• 

The language corresponding to a context free grammar is the set of all strings that can be obtained by starting from and applying production rules until we only have terminal symbols. The language corresponding to the above grammar is, in fact, the set of well-formed arithmetic expressions, such as , , and .

As mentioned above, a probabilistic context free grammar simply places a distribution over the production rules for any given non-terminal symbol. By repeatedly sampling from these distributions until we are left with only terminal symbols, we obtain a probability distribution over the language of the grammar.

We can represent a PCFG as a log-linear model by using a feature for each production rule . For instance, we have a feature that counts the number of times that the rule gets applied, and another feature that counts the number of times that gets applied. Such features yield a log-linear model family that contains all probabilistic context free grammars for a given (deterministic) context free grammar. However, it also contains additional models that do not correspond to PCFGs; this is because we run into the same problem as for HMMs, which is that the sum of over production rules of a given non-terminal does not necessarily add up to . In fact, the problem is even worse here. For instance, suppose that in the model above. Then the expression gets a score of , and longer chains of s get even higher scores. In particular, there is an infinite sequence of expressions with increasing scores and therefore the model doesn’t normalize (since the sum of the exponentiated scores of all possible productions is infinite).

So, log-linear models over-represent PCFGs in the same way as they over-represent HMMs, but the problems are even worse than before. Let’s ignore these issues for now, and suppose that we want to learn PCFGs with an unknown underlying CFG. To be a bit more concrete, suppose that we have a large collection of possible production rules for each non-terminal , and we think that a small but unknown subset of those production rules should actually appear in the grammar. Then there is no way to encode this directly within the context of a log-linear model family, although we can encode such “sparsity constraints” using simple extensions to log-linear models (for instance, by adding a penalty for the number of non-zero entries in ). So, we have found another way in which the log-linear representation is not entirely adequate.

Conclusion. Based on the examples above, we have seen that log-linear models have difficulty placing constraints on latent variables. This showed up in two different ways: first, we are unable to constrain subsets of variables to add up to (what I call “local normalization” constraints); second, we are unable to encode sparsity constraints within the model. In both of these cases, it is possible to extend the log-linear framework to address these sorts of constraints, although that is outside the scope of this post.

Parameter Estimation for Log-Linear Models

I’ve explained what a log-linear model is, and partially characterized its representational power. I will now answer the practical question of how to estimate the parameters of a log-linear model (i.e., how to fit based on observed data). Recall that a log-linear model places a distribution over a space by choosing and and defining

More precisely (assuming is a discrete space), we have

Given observations , which we assume to be independent given , our goal is to choose maximizing , or, equivalently, . In equations, we want

We typically use gradient methods (such as gradient descent, stochastic gradient descent, or L-BFGS) to minimize the right-hand side of (1). If we compute the gradient of (1) then we get:

We can re-write (2) in the following more compact form:

In other words, the contribution of each training example to the gradient is the extent to which the features values for exceed their expected values conditioned on .

One important consideration for such gradient-based numerical optimizers is convexity. If the objective function we are trying to minimize is convex (or concave), then gradient methods are guaranteed to converge to the global optimum. If the objective function is non-convex, then a gradient-based approach (or any other type of local search) may converge to a local optimum that is very far from the global optimum. In order to assess convexity, we compute the Hessian (matrix of second derivatives) and check whether it is positive definite. (In this case, we actually care about concavity, so we want the Hessian to be negative definite.) We can compute the Hessian by differentiating (2), which gives us

Again, we can re-write this more compactly as

The term inside the parentheses of (5) is exactly the negative of the covariance matrix of given , and is therefore necessarily negative definite, so the objective function we are trying to minimize is indeed concave, which, as noted before, implies that our gradient methods will always reach the global optimum.

Regularization and Concavity

We may in practice wish to encode additional prior knowledge about in our model, especially if the dimensionality of is large relative to the amount of data we have. Can we do this and still maintain concavity? The answer in many cases is yes: since the -norm is convex for all , we can add an penalty to the objective for any such and still have a concave objective function.

Conclusion

Log-linear models provide a universal representation for individual probability distributions, but not for arbitrary families of probability distributions (for instance, due to the inability to capture local normalization constraints or sparsity constraints). However, for the families they do express, parameter optimization can be performed efficiently due to a likelihood function that is log-concave in its parameters. Log-linear models also have tie-ins to many other beautiful areas of statistics, such as exponential families, which will be the subject of the next post.