# Continuous-Time Markov Chain

Example: {$$N_t : t\geq 0$$} is a Poisson Process with rate $$\lambda$$.

#### Basic Structure:

• {$$N_t : t\geq 0$$} is a $${\mathbb{N}}_0$$ valued random variable satisfying the Continuous Path almost surely assumptions.

Note: This condition guarantees that the Markov chain makes only finitely many jumps in any finite time interval. There are interesting examples (due to Blackwell) of processes $$N_t$$ that satisfy the Markov property ( next one) but make infinitely many jumps in every finite time interval.
• Set of Transition Matrices {$$P(t) : t\geq 0$$}
$$P_{ij}(t) : t\geq 0$$ denotes the probability of going from i to j in time t.
• Chapman Kolmogorov Equations $$P(u+v) = P(u)P(v)$$.
• Infinitesimal Generator Q such that $$P'(t) = P(t).Q$$ iff $$P(t) = e^{tQ}$$.

#### Theorem Chunk 1:

• Given a distribution $$\lambda$$ and a transition matrix P, there always exists a DTMC on finite state space with the transition matrix P and the initial distribution as $$\lambda$$ .
• Given a distribution $$\lambda$$ and a matrix Q with some conditions, there exists a CTMC on finite state space with an infinitesimal generator Q and initial distribution as $$\lambda$$ .
• Main Idea: Approximation of CTMC by DTMC at the dyadic rationals and using the continuous path property to show almost sure convergence.

Hitting Time : $$T_i$$ = inf{$$t\geq 0: X_t \neq i$$}

For discrete-time Markov Chain, $$T_i$$ = inf{$$t\geq 0: X_t \neq i$$} ~ Geo ($$1 – R_{ii}$$). Now, the continuous version of Geometric Distribution is Exponential Distribution.

#### Theorem:

$$q_{ii} = 0 \Longrightarrow$$ P({$$X_t = i | X_0 = i$$}) = 1.
$$q_{ii} < 0 \Longrightarrow$$ $$T_i | X_0 = i$$ ~ exp($$– q_{ii}$$)

Result: $$Q = XDX^{-1} \Longrightarrow e^{tQ} = Xe^{tD}X^{-1} = Xdiag(e^{td_1}, e^{td_2},…, e^{td_n})X^{-1}$$ where D = diag$$({d_1}, {d_2},…,{d_n}$$).

Corrollary: P ($$X_ {T_i} = j | X_0 = i) = \frac{- q_{ij}}{q_{ii}}$$) for i,j in the state space.

For DTMC, P ($$X_ {T_i} = j | X_0 = i) = \frac{R_{ij}}{1 – R_{ii}}$$) for i,j in the state space.

We consider a matrix $$R_{NxN}$$, define $$R_{ij} =$$

Fill in this gap.

#### Theorem :

(Finite State Space) i and j are distinct states. If $$q_{ij} > 0$$, then there is a c > 0, such that $$p_{ij}(t) > 0$$ for $$c \geq t > 0$$.

This can be generalized to a sequence of n states.

TFAE

• $$i = i_1 \rightarrow i_2 \rightarrow … \rightarrow i_n = j$$, i.e. for all t > 0 $$p_{ij}(t) > 0$$
• For some t > 0, $$p_{ij}(t) > 0$$
• There are $$i = i_1, i_2 , … , i_n = j$$, such that $$q_{{i_k}, i_{k+1}} > 0$$ for all k. This is defined as $$i \rightarrow j$$. i leads to j.

i.e. $$i \rightarrow j$$ iff for all t > 0 $$p_{ij}(t) > 0$$.

Go Top