# Continuous-Time Markov Chain

Example: {\( N_t : t\geq 0 \)} is a Poisson Process with rate \( \lambda \).

#### Basic Structure:

- {\( N_t : t\geq 0 \)} is a \( {\mathbb{N}}_0 \) valued random variable satisfying the Continuous Path almost surely assumptions.
**Note**: This condition guarantees that the Markov chain makes only finitely many jumps in any finite time interval. There are interesting examples (due to Blackwell) of processes \( N_t \) that satisfy the Markov property ( next one) but make infinitely many jumps in every finite time interval. - Set of Transition Matrices {\( P(t) : t\geq 0 \)}

\( P_{ij}(t) : t\geq 0 \) denotes the probability of going from i to j in time t. - Chapman Kolmogorov Equations \(P(u+v) = P(u)P(v) \).
- Infinitesimal Generator Q such that \(P'(t) = P(t).Q\) iff \(P(t) = e^{tQ}\).

#### Theorem Chunk 1:

- Given a distribution \(\lambda\) and a transition matrix P, there always exists a DTMC on finite state space with the transition matrix P and the initial distribution as \(\lambda\) .
- Given a distribution \(\lambda\) and a matrix Q with some conditions, there exists a CTMC on finite state space with an infinitesimal generator Q and initial distribution as \(\lambda\) .
**Main Idea**: Approximation of CTMC by DTMC at the dyadic rationals and using the continuous path property to show almost sure convergence.

**Hitting Time** : \(T_i\) = inf{\( t\geq 0: X_t \neq i \)}

For discrete-time Markov Chain, \(T_i\) = inf{\( t\geq 0: X_t \neq i \)} ~ Geo (\( 1 – R_{ii}\)). Now, the continuous version of Geometric Distribution is Exponential Distribution.

#### Theorem:

\( q_{ii} = 0 \Longrightarrow \) P({\(X_t = i | X_0 = i\)}) = 1.

\( q_{ii} < 0 \Longrightarrow \) \(T_i | X_0 = i\) ~ exp(\( – q_{ii}\))

**Result:** \( Q = XDX^{-1} \Longrightarrow e^{tQ} = Xe^{tD}X^{-1} = Xdiag(e^{td_1}, e^{td_2},…, e^{td_n})X^{-1}\) where D = diag\( ({d_1}, {d_2},…,{d_n}\)).

**Corrollary**: P (\(X_ {T_i} = j | X_0 = i) = \frac{- q_{ij}}{q_{ii}} \)) for i,j in the state space.

For DTMC, P (\(X_ {T_i} = j | X_0 = i) = \frac{R_{ij}}{1 – R_{ii}} \)) for i,j in the state space.

We consider a matrix \(R_{NxN}\), define \(R_{ij} = \)

Fill in this gap.

#### Theorem :

(Finite State Space) i and j are distinct states. If \(q_{ij} > 0 \), then there is a c > 0, such that \(p_{ij}(t) > 0 \) for \(c \geq t > 0 \).

This can be generalized to a sequence of n states.

**TFAE**

- \( i = i_1 \rightarrow i_2 \rightarrow … \rightarrow i_n = j \), i.e. for all t > 0 \(p_{ij}(t) > 0\)
- For some t > 0, \(p_{ij}(t) > 0\)
- There are \( i = i_1, i_2 , … , i_n = j \), such that \(q_{{i_k}, i_{k+1}} > 0\) for all k. This is defined as \(i \rightarrow j\). i leads to j.

i.e. \(i \rightarrow j\) iff for all t > 0 \(p_{ij}(t) > 0\).