September 7, 2015 Leave a comment
Consider a probability distribution on a space . Suppose we want to construct a set of probability distributions on such that is the maximum-entropy distribution over :
where is the entropy. We call such a set a maximum-entropy set for . Furthermore, we would like to be as large as possible, subject to the constraint that is convex.
Does such a maximal convex maximum-entropy set exist? That is, is there some convex set such that is the maximum-entropy distribution in , and for any satisfying the same property, ? It turns out that the answer is yes, and there is even a simple characterization of :
Proposition 1 For any distribution on , the set
is the maximal convex maximum-entropy set for .
To see why this is, first note that, clearly, , and for any we have
so is indeed the maximum-entropy distribution in . On the other hand, let be any other convex set whose maximum-entropy distribution is . Then in particular, for any , we must have . Let us suppose for the sake of contradiction that , so that . Then we have
Since , for sufficiently small this will exceed , which is a contradiction. Therefore we must have for all , and hence , so that is indeed the maximal convex maximum-entropy set for .