When Probability Meets Complex Analysis: Analytic Probability

当概率遇上复变：解析概率 苏剑林 · 2014-04-25 · Mathematics probability complex number generating function

Every time I see two seemingly unrelated branches of mathematics elegantly connected, I can't help but marvel at the magical beauty of mathematics. Long ago, when I saw how the method of generating functions links number theory with complex analysis to give rise to the wonderful subject of "analytic number theory," I was amazed by the elegance of the generating function method! Unfortunately, I never got around to properly writing up these ideas. Today, while reading Tsung-Dao Lee's Mathematical Methods in Physics, I saw him weave complex analysis and random walks together with what felt like divine craftsmanship, and once again I was struck with admiration. In the end I simply couldn't contain my excitement, so here I am writing about probability theory and generating functions.

Number theory combined with complex analysis gave rise to "analytic number theory." Following this logic, probability combined with complex analysis ought to give rise to something we might call "analytic probability" — but when I searched online, I found no such term in use. Nevertheless, I've decided to use this term in this post anyway. Although the name itself hasn't caught on, the method of analytic probability is actually not new at all — it can be traced back to the great mathematician Laplace and his work Analytic Theory of Probability. Even so, this elegant and beautiful method doesn't seem to have received the attention it deserves.

I think that even as a mere computational tool, the generating function method is a beautiful technique that deserves to be known as widely as possible among science enthusiasts, let alone among those in mathematics. more

Discrete Probability

Suppose the possible values of $X$ are discrete, $x_1,x_2,x_3,\dots$. For simplicity, let's assume here that the values $X$ we consider are all positive integers (non-integer discrete cases can be handled similarly). The number of possible values $X$ may be finite or infinite, and the corresponding probabilities are $p_1=p(X=x_1),p_2=p(X=x_2),p_3=p(X=x_3),\dots$. Then consider the generating function

$$f(z)=\sum_{n} p_n z^{x_n}$$

That is to say, the coefficient of the $z^{\alpha}$ term in $f(z)$ is the probability that $X$ takes the value $\alpha$.

For example, take $P(X=n)=2^{-n},n=1,2,3,\dots$; then the corresponding generating function is

$$f(z)=\sum_{n=1}^{\infty} 2^{-n} z^n=\frac{1}{1-z/2}-1$$

The generating function of a probability distribution has the following basic properties:

$f(1)=1$, which expresses the fact that the total probability over all possible values sums to 1, as it should;
the expected value of $X$ is
$$E(X)=\left.z\frac{d}{dz}f(z)\right|_{z=1}=\left.\frac{d}{d(\ln z)}f(z)\right|_{z=1}$$
The derivative above is called the "logarithmic derivative." By simply substituting in and applying the definition of expectation, one can derive the equality above.
Going a step further,
$$E(X^n)=\left.\left(\frac{d}{d(\ln z)}\right)^n f(z)\right|_{z=1}$$
If $f(z)$ and $g(z)$ are the generating functions of the probability distributions of two trials, then the generating function of the probability distribution after both trials is $f(z)g(z)$, which likewise follows directly from the definition.

For example, suppose in a single trial the probability of success is $p$, with success giving a value of 1 and failure giving a value of 0. Then the corresponding generating function is $p z^{1}+(1-p) z^{0}=pz+(1-p)$, and after $n$ trials, the generating function becomes

$$f(z)=[pz+(1-p)]^n$$

Expanding this expression, we get

$$f(z)=\sum_{i=0}^n C_n^i p^i (1-p)^{n-i} z^i$$

This tells us the probability, after $n$ trials, that $X$ takes on a given value:

$$P(X=i)=C_n^i p^i (1-p)^{n-i}$$

Moreover, one can readily compute

$$\begin{aligned} E(X)=\left.\frac{d}{d(\ln z)}[pz+(1-p)]^n \right|_{z=1}=\left.npz[pz+(1-p)]^{n-1} \right|_{z=1}=np\\ E(X^2)=\left.\frac{d}{d(\ln z)}\{npz[pz+(1-p)]^{n-1}\} \right|_{z=1}=np+n(n-1) p^2 \end{aligned}$$

and hence

$$Var(X)=np+n(n-1)p^2-(np)^2=np(1-p)$$

and so on. The above is the basic result concerning the "binomial distribution."

Continuous Probability

The method above can, in fact, be extended to continuous probability. Readers may wonder: the discussion above is built on Taylor series, and Taylor series are inherently discrete — how could this possibly extend to the continuous case? But this is precisely what's most wonderful about mathematics: once you break free of conventional thinking, an entirely broader landscape often comes into view. To this end, let us first define the following probabilistic setup. Suppose the possible values of $X$ range over any real number in $(-\infty,+\infty)$, with $p(x\leq X\leq x+dx)=P(x)dx$ — that is, $P(x)$ is the probability density. Following the same form of generating function as above, consider

$$f(z)=\lim_{dx\to 0}\sum_{x}P(x)z^{x}dx=\int_{-\infty}^{+\infty}P(x)z^{x}dx$$

We shouldn't be too surprised at replacing the sum with an integral, but a function of this form is not something we've encountered before. Still, since this is also a kind of power series, there's no reason we're obliged to use the notation $z$ — let's use a different notation instead, replacing $z$ with $e^{-i\omega}$. The discussion above still goes through unchanged, and we find that $f(z)$ becomes

$$F(\omega)=\int_{-\infty}^{+\infty}P(x)e^{-i\omega x}dx$$

This is precisely the Fourier transform of $P(x)$!

In this way, the Fourier transform acquires a vivid meaning in probability theory: it represents the generating function of a probability distribution. It follows readily that once we know the image under the Fourier transform, we can recover the original function, since the two contain the same information. The formula for recovering the original probability distribution from $F(\omega)$ is precisely the inverse Fourier transform:

$$P(x)=\frac{1}{2\pi}\int_{-\infty}^{+\infty}F(\omega)e^{i\omega x}d\omega$$

Correspondingly, the generating function (Fourier transform) of a continuous probability distribution satisfies a similar set of properties:

the total probability $F(0)=1$;
for the expectation, $E(X)=\left.i\frac{d}{d\omega}F(\omega)\right|_{\omega=0}$;
for the moments of various orders, $E(X^n)=\left.i^n\frac{d^n}{d\omega^n}F(\omega)\right|_{\omega=0}$.

Later on, we will try using the method of analytic probability to explore the problem of random walks (Brownian motion). We will see that studying random walks by this method turns out to be surprisingly, delightfully simple.