4 Markov chains 4.3 Analysing Markov chains 4.5 Classification of states

4.4 Asymptotic and invariant distributions

Definition 4.4.1.

A homogeneous MC has an invariant distribution $\pi$ if

\pi=\pi P,

Therefore if $\pi_{t}=\pi$ then so also $\pi_{t+1}=\pi$ , …. In particular, if the initial distribution happens to be invariant, distribution of all $\{X_{t}\}$ remain same. For this reason, the invariant distribution is sometimes called the stationary distribution of the MC.

Method 4.4.2.

Calculation of invariant distributions can be done by directly solving

\pi P=\pi\ \mbox{or equivalently}\ \pi(P-I)=0.

When written out as a set of equations any one of them is a combination of the others, because they all sum to give $1=1$ , or equivalently $0=0$ . Thus any one equation is redundant and can be removed.

The equations are also homogeneous, i.e. given any solution (other than $\pi=0$ ), it is also a solution when multiplied by any constant. For that reason we have to solve the equation together with the condition that $\sum_{i}\pi_{i}=1$ .

Example 4.4.3 (How to obtain the invariant distribution).

(\pi_{1}\ \ \pi_{2})\left(\begin{array}[]{cc}1-a&a\\ b&1-b\end{array}\right)=(\pi_{1}\ \ \pi_{2})

gives the equation $a\pi_{1}=b\pi_{2}$ or $\pi_{1}=b\pi_{2}/a$ . Thus

\pi^{T}=\left(\begin{array}[]{c}\pi_{1}\\ \pi_{2}\end{array}\right)\propto\left(\begin{array}[]{c}\frac{b}{a}\\ 1\end{array}\right)\propto\left(\begin{array}[]{c}b\\ a\end{array}\right)\ \Rightarrow\pi^{T}=\left(\begin{array}[]{c}\frac{b}{a+b}% \\ \frac{a}{a+b}\end{array}\right).

The last step was simply to divide by the sum of the components of a solution proportional to that required.

This illustrates a useful approach to obtaining $\pi$ . First solve $xP=x$ or equivalently $x(P-I)=0$ by setting some element of $x$ to $\gamma$ . After solving for the whole vector $x$ , divide it by the sum of its elements to obtain $\pi$ .

Remark.

Invariant distribution may not always exist, that is, it may not be possible to solve $\pi=\pi P$ . A chain always has one (at least on a finite state space), and may have more than one invariant distribution. If $\pi_{a}$ and $\pi_{b}$ are two different invariant distributions, then another is $\gamma\pi_{a}+(1-\gamma)\pi_{b}$ (for $\gamma\in[0,1]$ ). In later sections we will discuss conditions for existence and uniqueness of the invariant distribution.

Consider example (ii) from 4.2.2.

The states of this chain are reducible into $\{1,4\}\cup\{2,3\}$ . Using the previous Example 4.4.3, the chain on states 1 and 4 has invariant distribution $(4/5,0,0,1/5)$ , and the chain on states 2 and 3 has invariant distribution $(0,2/3,1/3,0)$ . Let the total probability of starting in states 1 and 4 be $\gamma$ ; the invariant distribution of the chain is

\gamma(4/5,0,0,1/5)+(1-\gamma)(0,2/3,1/3,0).

Example 4.4.4 (Example 4.3.5 continued).

Find the invariant distribution for the no claims bonus.

The equation $x(P-I)=0$ gives, in this case,

\begin{array}[]{rrrrrrrrr}-\frac{2}{3}x_{1}&+&\frac{1}{3}x_{2}&+&\frac{1}{6}x_% {3}&&&=&0\\ \frac{2}{3}x_{1}&-&x_{2}&+&\frac{1}{6}x_{3}&+&\frac{1}{6}x_{4}&=&0\\ &&\frac{2}{3}x_{2}&-&x_{3}&+&\frac{1}{6}x_{4}&=&0\\ &&&&\frac{2}{3}x_{3}&-&\frac{1}{3}x_{4}&=&0\end{array}.

To solve this let $x_{3}=\gamma$ then $x_{4}=2\gamma$ , $x_{2}=\frac{3}{2}(x_{3}-\frac{1}{6}x_{4})=\gamma$ , and $x_{1}=\frac{3}{2}(\frac{1}{3}x_{2}+\frac{1}{6}x_{3})=\frac{3}{4}\gamma$ . So $x=(\frac{3}{4},1,1,2)\gamma\propto(3,4,4,8)\Rightarrow\pi=(\frac{3}{19},\frac{% 4}{19},\frac{4}{19},\frac{8}{19})$ .

Definition 4.4.5.

A homogeneous MC has an asymptotic distribution $\pi$ if $\pi_{t}\,\rightarrow\,\pi$ whatever the initial distribution $\pi_{0}$ .

Example 4.4.6 (Finding the asymptotic distribution).

Let $0\leq a,b\leq 1$ and

P=\left(\begin{array}[]{cc}1-a&a\\ b&1-b\end{array}\right)\mbox{\ \ and let\ \ }\pi_{t}=\left(u_{t}\ \ v_{t}% \right)=\left(u_{t}\ \ 1-u_{t}\right).

We find expression for $u_{t}$ as follows. Using $\pi_{t+1}=\pi_{t}P$ ,

u_{t+1}=(1-a)u_{t}+bv_{t}=(1-a)u_{t}+b(1-u_{t})=(1-a-b)u_{t}+b=\alpha u_{t}+b,% \,\,\,\alpha=1-a-b.

We use here just the first component of the matrix equation since $v_{t}=1-u_{t}$ .

Recall the technique for solving difference equations. The auxilliary equation $\theta-\alpha=0$ has solution $\alpha$ . Hence solution to the homogeneous equation $u_{t+1}=\alpha u_{t}$ is $u_{t}=A\alpha^{t}$ .

To find the particular solution to the non-homogeneous equation, try $u_{t}=c$ , a constant. Then $c=\alpha c+b$ implies, $c=b/(1-\alpha)$ . Hence the general solution is

u_{t}=A\alpha^{t}+b/(1-\alpha).

Substituting $t=0$ , $A=u_{0}-b/(1-\alpha)$ . Hence

u_{t}=(u_{0}-b/(a+b))(1-a-b)^{t}+b/(a+b).

Since $a,b\in[0,1]$ , $-1\leq 1-a-b\leq 1$ and so, provided $a$ and $b$ are not both 0 or both 1, the term $(1-a-b)^{t}\rightarrow 0$ as $t\rightarrow\infty$ . Therefore $u_{t}\rightarrow b/(a+b)$ and

\pi_{t}\,\rightarrow\left(\frac{b}{a+b}\ \ \frac{a}{a+b}\right)

whatever the value of $\pi_{0}$ . Observe that this is the invariant distribution of Example 4.4.3.

Next we consider some sufficient conditions under which a solution to $\pi=\pi P$ exists.

4.4.1 Detailed balance

Theorem 4.4.7.

A sufficient condition for $\pi$ to be the invariant distribution is given by the so-called detailed-balance equations:

\pi_{i}P_{ij}=\pi_{j}P_{ji}\mbox{ for all $i$ and $j$.}

(These equations do not always hold; but when they do they are much easier to solve. Also, $\pi_{i}P_{ij}$ can be interpreted as the flow from $i$ to $j$ .)

Proof.

Sum the left and the right-hand sides over $i$ :

If $\pi_{i}P_{ij}=\pi_{j}P_{ji}~{}\forall~{}i,j$ then

$\displaystyle\sum_{i}\pi_{i}P_{ij}$	$\displaystyle=$	$\displaystyle\sum_{i}\pi_{j}P_{ji}$
$\displaystyle(\pi P)_{j}$	$\displaystyle=$	$\displaystyle\pi_{j}\sum_{i}P_{ji}$
	$\displaystyle=$	$\displaystyle\pi_{j}.$

This is true $\forall~{}j$ so $\pi P=\pi$ . ∎

Example 4.4.8.

Use detailed balance to calculate the invariant distribution of the 4-state Markov chain with transition matrix

P=\left(\begin{array}[]{cccc}0.5&0.5&0&0\\ 0.25&0&0.75&0\\ 0&0.75&0&0.25\\ 0&0&0.5&0.5\end{array}\right).~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}% ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}% ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}% ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}% ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}% ~{}~{}~{}

Note that the detailed-balance equations hold automatically for $i=j$ and for those pairs of $(i,j)$ for which $P_{ij}=P_{ji}=0$ . Hence we only need to consider $i\neq j$ with $P_{ij}\neq 0$ , namely,

\begin{array}[]{lclclcl}\pi_{1}\cdot\frac{1}{2}&=\pi_{2}\cdot\frac{1}{4}&\mbox% {so}&\pi_{2}=2\pi_{1}\\ \pi_{2}\cdot\frac{3}{4}&=\pi_{3}\cdot\frac{3}{4}&\mbox{so}&\pi_{2}=\pi_{3}\\ \pi_{3}\cdot\frac{1}{4}&=\pi_{4}\cdot\frac{1}{2}&\mbox{so}&\pi_{3}=2\pi_{4}.% \end{array}

Hence $\pi\propto(1,2,2,1)$ , so $\pi=1/6~{}(1,2,2,1)$ .

Definition 4.4.9.

If the transition matrix satisfies detailed balance then the Markov chain is said to be reversible at equilibrium or, more simply reversible.

At equilibrium the system should behave the same going forward in time as it does going backwards in time so that someone observing a video of the stochastic process would be unable to tell whether or not it was being shown backwards.

Theorem 4.4.10.

Any transition kernel of the form

P=\left(\begin{array}[]{cccccccc}P_{11}&P_{12}&0&0&\dots&0\\ P_{21}&P_{22}&P_{23}&0&\dots&0\\ 0&P_{32}&P_{33}&P_{34}&\dots&0\\ 0&0&P_{43}&P_{44}&\dots&0\\ \vdots&\vdots&\vdots&\vdots&\ddots&P_{n-1,n}\\ 0&0&0&0&P_{n,n-1}&P_{nn}\end{array}\right)

satisfies detailed balance. Note that this is a matrix with $P_{ij}=0$ for $|i-j|>1$ .

We will prove a very similar theorem for continuous-time Markov chains (Theorem 5.4.2) and so will omit the proof here. However the reason why the theorem holds is that detailed balance requires exactly that

\pi_{i}P_{i,i-1}=\pi_{i-1}P_{i-1,i}~{}~{}~{}(i=2,\dots,N),

i.e. there are no other equations that need to be satisfied. Together with the fact that $\sum\pi_{i}=1$ we have $N$ equations in $N$ unknowns which can be solved.