Moment generating functions and connections to other things

This post relates moment generating functions to the Laplace transform and to exponential generating functions. It also brings in connections to the z-transform and the Fourier transform.

Thanks to Brian Borchers who suggested the subject of this post in a comment on a previous post on transforms and convolutions.

Moment generating functions

The moment generating function (MGF) of a random variable X is defined as the expected value of exp(tX). By the so-called rule of the unconscious statistician we have

where f_X is the probability density function of the random variable X. The function M_X is called the moment generating function of X because it's nth derivative, evaluated at 0, gives the nth moment of X, i.e. the expected value of Xⁿ.

Laplace transforms

If we flip the sign on t in the integral above, we have the two-sided Laplace transform of f_X. That is, the moment generating function of X at t is the two-sided Laplace transform of f_X at -t. If the density function is zero for negative values, then the two-sided Laplace transform reduces to the more common (one-sided) Laplace transform.

Exponential generating functions

Since the derivatives of M_X at zero are the moments of X, the power series for M_X is the exponential generating function for the moments. We have

where m_n is the nth moment of X.

Other generating functions

This terminology needs a little explanation since we're using "generating function" two or three different ways. The "moment generating function" is the function defined above and only appears in probability. In combinatorics, the (ordinary) generating function of a sequence is the power series whose coefficient of xⁿ is the nth term of the sequence. The exponential generating function is similar, except that each term is divided by n!. This is called the exponential generating series because it looks like the power series for the exponential function. Indeed, the exponential function is the exponential generating function for the sequence of all 1's.

The equation above shows that M_X is the exponential generating function for m_n and the ordinary generating function for m_n/n!.

If a random variable Y is defined on the integers, then the (ordinary) generating function for the sequence Prob(Y = n) is called, naturally enough, the probability generating function for Y.

The z-transform of a sequence, common in electrical engineering, is the (ordinary) generating function of the sequence, but with x replaced with 1/z.

Characteristic functions

The characteristic function of a random variable is a variation on the moment generating function. Rather than use the expected value of tX, it uses the expected value of itX. This means the characteristic function of a random variable is the Fourier transform of its density function.

Characteristic functions are easier to work with than moment generating functions. We haven't talked about when moment generating functions exist, but it's clear from the integral above that the right tail of the density has to go to zero faster than e^-x, which isn't the case for fat-tailed distributions. That's not a problem for the characteristic function because the Fourier transform exists for any density function. This is another example of how complex variables simplify problems.