5 Continuous Markov chains 5.4 Detailed balance 5.6 Poisson processes

5.5 Use of the rate matrix

Theorem 5.5.1 (Backward and forward equations).

P^{\prime}(t)=QP(t)=P(t)Q.

Proof.

Differentiate $P(\alpha+\beta)=P(\alpha)P(\beta)$ wrt $\alpha$ , so

P^{\prime}(\alpha+\beta)=P^{\prime}(\alpha)P(\beta)

and set $\alpha=0$ and $\beta=t$ . Similarly differentiate wrt $\beta$ for the other result. ∎

Theorem 5.5.2 (Rates of change of distributions (useful for determining $\pi(t)$ )).

\pi^{\prime}(t)=\pi(t)Q.

Proof.

Differentiate $\pi(t)=\pi(s)P(t-s)$ wrt $t$ to get

\pi^{\prime}(t)=\pi(s)P^{\prime}(t-s)

and set $s=t$ . ∎

Remark.

(a)

This is often expressed as

$\pi(t+h)\approx\pi(t)+h\pi(t)Q$

or

$\pi(t+h)_{j}\approx\pi(t)_{j}+h\sum_{i}\pi(t)_{i}Q_{i\,j}.$
(b)

The equations written as $\pi^{\prime}(t)=\pi(t)Q$ are simultaneous differential equations which may be solved for the elements $\pi(t)_{j}$ .

Example 5.5.3.

The two state chain has

Q=\left(\begin{array}[]{cc}-\lambda&\lambda\\ \mu&-\mu\end{array}\right).

This only supplies one independent differential equation. However, as $\pi(t)$ is a distribution, we also have $\pi(t)_{1}+\pi(t)_{2}=1$ . The first equation gives

\pi(t)_{1}^{\prime}=-\lambda\pi(t)_{1}+\mu\pi(t)_{2}=-\lambda\pi(t)_{1}+\mu[1-% \pi(t)_{1}]=-(\lambda+\mu)\pi(t)_{1}+\mu.

To solve this, suppose first that $\pi(t)_{1}$ were a constant, $c$ , so that $\pi^{\prime}(t)_{1}=0$ ; then $\pi(t)_{1}=c=\frac{\mu}{\lambda+\mu}$ .

Next, consider in general that $\pi(t)_{1}$ is varying about $c$ so set $\pi(t)_{1}=c+r(t)$ . On substituting, $c$ cancels and

r^{\prime}(t)=-(\lambda+\mu)r(t)\Rightarrow\frac{dr}{r}=-(\lambda+\mu)dt% \Rightarrow\left[\log r\right]=\left[-(\lambda+\mu)t\right]

which leads to

r(t)=r(0)\exp[-(\lambda+\mu)t]\Rightarrow\pi(t)_{1}=\frac{\mu}{\lambda+\mu}+% \left[\pi(0)_{1}-\frac{\mu}{\lambda+\mu}\right]\exp[-(\lambda+\mu)t].

The solution to this differential equation is therefore

	$\displaystyle\pi(t)_{1}$	$\displaystyle=$	$\displaystyle\frac{\mu}{\lambda+\mu}+\left[\pi(0)_{1}-\frac{\mu}{\lambda+\mu}% \right]\exp[-(\lambda+\mu)t],$
	$\displaystyle\pi(t)_{2}$	$\displaystyle=$	$\displaystyle\frac{\lambda}{\lambda+\mu}+\left[\pi(0)_{2}-\frac{\lambda}{% \lambda+\mu}\right]\exp[-(\lambda+\mu)t].$

Provided $\lambda+\mu>0$ , $(\pi(t)_{1},\pi(t)_{2})\rightarrow\left(\frac{\mu}{\lambda+\mu},\frac{\lambda}% {\lambda+\mu}\right)$ .

Exercise 5.5.4 (Based on 2011 B3).

Young Hercules is fighting a two-headed hydra. Consider any small time interval, $J$ of length $\delta t$ . The probability that Hercules will chop off a head in the interval $J$ , is approximately $\mu\delta t$ . If the hydra has just one head then the other head will grow back; the probability that the head will grow back in $J$ is approximately $\gamma\delta t$ . If the hydra has no heads then it is dead and the fight ends. The hydra’s heads act independently; for each head the probability that it will kill Hercules in $J$ is approximately $\lambda\delta t$ . If Hercules dies then the fight ends.

(i)

Write down the rate matrix for the continuous-time Markov chain with the following states; 0) The hydra is dead, 1) the hydra has 1 head, 2) the hydra has 2 heads, 3) Hercules is dead.
(ii)

Write down a set of four differential equations for the components
( $\pi_{0}(t),\pi_{1}(t),\pi_{2}(t),\pi_{3}(t)$ ) of $\pi(t)$ .
(iii)

Assume that Hercules and each hydra head are evenly matched so $\mu=\lambda$ . Set $u(t)=\pi_{1}(t)+\pi_{2}(t)$ and $v(t)=\gamma\pi_{1}(t)-\lambda\pi_{2}(t)$ . Show that $u^{\prime}(t)=-2\lambda u$ and find a similar equation for $v(t)$ .
(iv)

As the fight commences the hydra has two heads. Solve the differential equations for $u(t)$ and $v(t)$ and hence show that

$\pi_{1}(t)=\frac{\lambda}{\lambda+\gamma}\left(e^{-2\lambda t}-e^{-(3\lambda+% \gamma)t}\right).$
(v)

Hence find $\pi_{0}(t)$ and show that the probability that Hercules kills the hydra is

$\frac{\lambda}{6\lambda+2\gamma}.$

(i)

$Q=\left(\begin{array}[]{cccc}0&0&0&0\\ \mu&-(\mu+\lambda+\gamma)&\gamma&\lambda\\ 0&\mu&-(\mu+2\lambda)&2\lambda\\ 0&0&0&0\end{array}\right).$

(ii)

$\pi^{\prime}(t)=\pi(t)Q$ so

$\displaystyle\pi^{\prime}_{0}(t)$	$\displaystyle=$	$\displaystyle\mu\pi_{1}(t)$
$\displaystyle\pi^{\prime}_{1}(t)$	$\displaystyle=$	$\displaystyle-(\mu+\lambda+\gamma)\pi_{1}(t)+\mu\pi_{2}(t)$
$\displaystyle\pi^{\prime}_{2}(t)$	$\displaystyle=$	$\displaystyle\gamma\pi_{1}(t)-(\mu+2\lambda)\pi_{2}(t)$
$\displaystyle\pi^{\prime}_{3}(t)$	$\displaystyle=$	$\displaystyle\lambda\pi_{1}(t)+2\lambda\pi_{2}(t).$

(iii)

$\displaystyle u^{\prime}(t)$	$\displaystyle=$	$\displaystyle\pi_{1}^{\prime}(t)+\pi_{2}^{\prime}(t)=(-2\lambda-\gamma+\gamma)% \pi_{1}(t)+(\lambda-3\lambda)\pi_{2}(t)=-2\lambda u(t).$
$\displaystyle v^{\prime}(t)$	$\displaystyle=$	$\displaystyle-(2\lambda\gamma+\gamma^{2}+\gamma\lambda)\pi_{1}(t)+(\gamma% \lambda+3\lambda^{2})\pi_{2}(t)=-(\gamma+3\lambda)(\gamma\pi_{1}(t)-\lambda\pi% _{2}(t))$
	$\displaystyle=$	$\displaystyle-(\gamma+3\lambda)v(t).$

(iv)

$\pi(0)=(0,0,1,0)$ . Solving the equations in (iii) gives

$u(t)=u(0)e^{-2\lambda t}\quad\mbox{and}\quad v(t)=v(0)e^{-(3\lambda+\gamma)t}.$

But $u(0)=\pi_{1}(0)+\pi_{2}(0)=1$ and $v(0)=\gamma\pi_{1}(0)-\lambda\pi_{2}(0)=-\lambda$ . Thus

$\displaystyle u(t)$ $\displaystyle=$ $\displaystyle e^{-2\lambda t},~{}~{}~{}v(t)=-\lambda e^{-(3\lambda+\gamma)t}.$

$\displaystyle\pi_{1}(t)$ $\displaystyle=$ $\displaystyle\frac{\lambda u(t)+v(t)}{\lambda+\gamma}=\frac{\lambda}{\lambda+% \gamma}\left(e^{-2\lambda t}-e^{-(3\lambda+\gamma)t}\right).$
(v)

$\displaystyle\pi_{0}^{\prime}(t)$ $\displaystyle=$ $\displaystyle\frac{\lambda^{2}}{\lambda+\gamma}\left(e^{-2\lambda t}-e^{-(3% \lambda+\gamma)t}\right)$

so

$\displaystyle\pi_{0}(t)$ $\displaystyle=$ $\displaystyle\frac{\lambda^{2}}{\lambda+\gamma}\left(-\frac{1}{2\lambda}e^{-2% \lambda t}+\frac{1}{3\lambda+\gamma}e^{-(3\lambda+\gamma)t}+c\right).$

By the initial condition $\pi_{0}(0)=0$ ,

$\displaystyle\pi_{0}(t)$ $\displaystyle=$ $\displaystyle\frac{\lambda^{2}}{\lambda+\gamma}\left(-\frac{1}{2\lambda}\left(% e^{-2\lambda t}-1\right)+\frac{1}{3\lambda+\gamma}\left(e^{-(3\lambda+\gamma)t% }-1\right)\right)$

$\displaystyle=$ $\displaystyle\frac{\lambda^{2}}{\lambda+\gamma}\left(\frac{\lambda+\gamma}{2% \lambda(3\lambda+\gamma)}-\frac{1}{2\lambda}e^{-2\lambda t}+\frac{1}{3\lambda+% \gamma}e^{-(3\lambda+\gamma)t}\right).$

Since state 0 is absorbing the probability the Hercules wins is the probability that the chain is eventually in state 0. Take the limit as $t\rightarrow\infty$ to see this is

$\frac{\lambda^{2}}{\lambda+\gamma}\times\frac{\lambda+\gamma}{2\lambda(3% \lambda+\gamma)}=\frac{\lambda}{6\lambda+2\gamma}.$

	$\displaystyle u(t)$	$\displaystyle=$	$\displaystyle e^{-2\lambda t},~{}~{}~{}v(t)=-\lambda e^{-(3\lambda+\gamma)t}.$
	$\displaystyle\pi_{1}(t)$	$\displaystyle=$	$\displaystyle\frac{\lambda u(t)+v(t)}{\lambda+\gamma}=\frac{\lambda}{\lambda+% \gamma}\left(e^{-2\lambda t}-e^{-(3\lambda+\gamma)t}\right).$

	$\displaystyle\pi_{0}(t)$	$\displaystyle=$	$\displaystyle\frac{\lambda^{2}}{\lambda+\gamma}\left(-\frac{1}{2\lambda}\left(% e^{-2\lambda t}-1\right)+\frac{1}{3\lambda+\gamma}\left(e^{-(3\lambda+\gamma)t% }-1\right)\right)$
		$\displaystyle=$	$\displaystyle\frac{\lambda^{2}}{\lambda+\gamma}\left(\frac{\lambda+\gamma}{2% \lambda(3\lambda+\gamma)}-\frac{1}{2\lambda}e^{-2\lambda t}+\frac{1}{3\lambda+% \gamma}e^{-(3\lambda+\gamma)t}\right).$

5.5 Use of the rate matrix

Theorem 5.5.1 (Backward and forward equations).

Proof.

Theorem 5.5.2 (Rates of change of distributions (useful for determining π⁢(t))).

Proof.

Remark.

Example 5.5.3.

Exercise 5.5.4 (Based on 2011 B3).

Theorem 5.5.2 (Rates of change of distributions (useful for determining $\pi(t)$ )).