Continuous-Time Markov Chain

Example: {\( N_t : t\geq 0 \)} is a Poisson Process with rate \( \lambda \).

Basic Structure:

  • {\( N_t : t\geq 0 \)} is a \( {\mathbb{N}}_0 \) valued random variable satisfying the Continuous Path almost surely assumptions.

    Note: This condition guarantees that the Markov chain makes only finitely many jumps in any finite time interval. There are interesting examples (due to Blackwell) of processes \( N_t \) that satisfy the Markov property ( next one) but make infinitely many jumps in every finite time interval.
  • Set of Transition Matrices {\( P(t) : t\geq 0 \)}
    \( P_{ij}(t) : t\geq 0 \) denotes the probability of going from i to j in time t.
  • Chapman Kolmogorov Equations \(P(u+v) = P(u)P(v) \).
  • Infinitesimal Generator Q such that \(P'(t) = P(t).Q\) iff \(P(t) = e^{tQ}\).

Theorem Chunk 1:

  • Given a distribution \(\lambda\) and a transition matrix P, there always exists a DTMC on finite state space with the transition matrix P and the initial distribution as \(\lambda\) .
  • Given a distribution \(\lambda\) and a matrix Q with some conditions, there exists a CTMC on finite state space with an infinitesimal generator Q and initial distribution as \(\lambda\) .
  • Main Idea: Approximation of CTMC by DTMC at the dyadic rationals and using the continuous path property to show almost sure convergence.

Hitting Time : \(T_i\) = inf{\( t\geq 0: X_t \neq i \)}

For discrete-time Markov Chain, \(T_i\) = inf{\( t\geq 0: X_t \neq i \)} ~ Geo (\( 1 – R_{ii}\)). Now, the continuous version of Geometric Distribution is Exponential Distribution.

Theorem:

\( q_{ii} = 0 \Longrightarrow \) P({\(X_t = i | X_0 = i\)}) = 1.
\( q_{ii} < 0 \Longrightarrow \) \(T_i | X_0 = i\) ~ exp(\( – q_{ii}\))

Result: \( Q = XDX^{-1} \Longrightarrow e^{tQ} = Xe^{tD}X^{-1} = Xdiag(e^{td_1}, e^{td_2},…, e^{td_n})X^{-1}\) where D = diag\( ({d_1}, {d_2},…,{d_n}\)).

Corrollary: P (\(X_ {T_i} = j | X_0 = i) = \frac{- q_{ij}}{q_{ii}} \)) for i,j in the state space.

For DTMC, P (\(X_ {T_i} = j | X_0 = i) = \frac{R_{ij}}{1 – R_{ii}} \)) for i,j in the state space.

We consider a matrix \(R_{NxN}\), define \(R_{ij} = \)

Fill in this gap.

Theorem :

(Finite State Space) i and j are distinct states. If \(q_{ij} > 0 \), then there is a c > 0, such that \(p_{ij}(t) > 0 \) for \(c \geq t > 0 \).

This can be generalized to a sequence of n states.

TFAE

  • \( i = i_1 \rightarrow i_2 \rightarrow … \rightarrow i_n = j \), i.e. for all t > 0 \(p_{ij}(t) > 0\)
  • For some t > 0, \(p_{ij}(t) > 0\)
  • There are \( i = i_1, i_2 , … , i_n = j \), such that \(q_{{i_k}, i_{k+1}} > 0\) for all k. This is defined as \(i \rightarrow j\). i leads to j.

i.e. \(i \rightarrow j\) iff for all t > 0 \(p_{ij}(t) > 0\).

Leave a Reply

Your email address will not be published. Required fields are marked *

Go Top