From Markov Processes to the Master Equation (A Derivation)

从马尔科夫过程到主方程（推导过程） 苏剑林 · 2017-10-06 · Mathematics probability master equation

The master equation is an important tool for modeling stochastic processes: it represents the differential form of a Markov process, and it's one of the core tools of our field. Put grandly, quantum mechanics, statistical mechanics, and so on are, in the end, nothing more than special cases of the master equation.

However, having read several works on the subject—such as A Modern Course in Statistical Physics, and my advisor's Stochastic Dynamics of Biological Systems—I found that these books are all rather vague about the derivation of the master equation. They focus on explaining the meaning of the result, but don't explain where the result actually comes from, which makes the derivation hard to find convincing. There's even a question on Zhihu, "How should one understand the derivation of the master equation for Markov processes?" that never got a satisfying answer, which confirms this is indeed a real gap.

Markov Processes

The master equation is used to describe Markov processes, and a Markov process can be understood as motion without memory: put plainly, the probability distribution at the next instant depends only on the current state, not on the history. Written as a probability formula (here we only consider the continuous case, so $p$ below is a probability density):

$$\begin{equation}\label{eq:maerkefu}p(x,\tau)=\int p(x,\tau|y,t) p(y,t) dy\end{equation}$$

Here the integral is taken over the entire space. The term $p(x,\tau|y,t)$ is called the transition probability: given that the system is at position $y$ at time $t$, it's the probability density of arriving at $x$ at time $\tau$. The physical meaning of this expression is fairly clear, so I won't dwell on it. more

Although $\eqref{eq:maerkefu}$ is quite intuitive, using it for modeling in practice runs into two difficulties:

1. To build a model we need to write down $p(x,\tau|y,t)$, but in practice it is very hard to directly write down a reasonable transition probability;
2. Even once $p(x,\tau|y,t)$ is written down, this is an integral equation, and our understanding of integral equations is far less developed than that of differential equations.

So it's worth deriving its differential form.

The Master Equation

Let us set $\tau = t+\epsilon$

$$\begin{equation}\label{eq:maerkefu-2}p(x,t+\epsilon)=\int p(x,t+\epsilon|y,t) p(y,t) dy\end{equation}$$

and consider the limit $\epsilon\to 0$, keeping terms up to first order in $\epsilon$. We then have

$$\begin{equation}\label{eq:yueqian}p(x,t+\epsilon|y,t)=\delta(x-y)+\epsilon \tilde{W}(x,y,t)\end{equation}$$

where we have used the fact that $p(x,t|y,t)=\delta(x-y)$, and

$$\begin{equation}\tilde{W}(x,y)=\left.\frac{\partial p(x,\tau|y,t)}{\partial \tau}\right|_{\tau=t}\end{equation}$$

Expanding both sides of $\eqref{eq:maerkefu-2}$ to first order in $\epsilon$ gives

$$\begin{equation}p(x,t)+\epsilon\frac{\partial p(x,t)}{\partial t}=\int \left[\delta(x-y)+\epsilon \tilde{W}(x,y,t)\right] p(y,t) dy\end{equation}$$

hence

$$\begin{equation}\label{eq:zhufangcheng-1}\frac{\partial p(x,t)}{\partial t}=\int \tilde{W}(x,y,t) p(y,t) dy\end{equation}$$

Making It More Convenient for Modeling

The reader may notice that $\eqref{eq:zhufangcheng-1}$ is not the familiar form of the master equation, and that's because it's inconvenient for modeling. Let's revisit $\eqref{eq:yueqian}$: note that we have

$$\begin{equation}\int p(x,t+\epsilon|y,t) dx = 1\end{equation}$$

Combined with $\eqref{eq:yueqian}$, this forces the constraint

$$\begin{equation}\label{eq:yueshu}\int \tilde{W}(x,y,t)dx = 0\end{equation}$$

The way modeling actually proceeds in research is the reverse of this: we first need to write down the form of the master equation, and only then solve it. In other words, to build a model using $\eqref{eq:zhufangcheng-1}$, we need to write down $\tilde{W}(x,y,t)$, and $\tilde{W}(x,y,t)$ must satisfy the constraint $\eqref{eq:yueshu}$—but it's very hard to write down some $\tilde{W}(x,y,t)$ out of thin air that automatically satisfies this constraint.

Fortunately, there's a trick that lets us get rid of this constraint. First, we write down an arbitrary function $W(x,y,t)$, and then consider

$$\begin{equation}\begin{aligned}&\int W(x,y,t)dx\\ =&\iint \delta(y-z) W(x,z,t)dxdz \quad (\text{next swap}x,z\text{position of})\\ =&\iint \delta(y-x) W(z,x,t)dzdx\end{aligned}\end{equation}$$

This lets us set

$$\begin{equation}\tilde{W}(x,y,t) = W(x,y,t) - \int \delta(y-x) W(z,x,t)dz\end{equation}$$

which automatically satisfies $\eqref{eq:yueshu}$, and the master equation finally takes the form

$$\begin{equation}\label{eq:zhufangcheng-2}\frac{\partial p(x,t)}{\partial t}=\int \Big[W(x,y,t)p(y,t)-W(y,x,t)p(x,t)\Big] dy\end{equation}$$

This is the form of the master equation we see in textbooks. At this point there's no special constraint on $W(x,y,t)$, so it can be conveniently used for modeling. As for how to interpret the physical meaning of $W(y,x,t)$, that's a matter for another time. A similar derivation carries over directly to the discrete-state master equation, so I won't repeat it here.

A Few Remarks

From the discussion above, we can see that the reason the master equation takes the form we usually encounter is that this form is more convenient for modeling—the interpretation of the result is, at heart, something imposed afterward, and shouldn't be mistaken for part of the derivation. Most textbooks, in deriving the master equation, focus on the physical interpretation of the result without elaborating on where the form of the equation actually comes from, and this is one important reason it's so hard to really understand. I hope this piece helps fill in that gap.