Sequential drug decision problems in long-term medical conditions: a case Study of Primary Hypertension Eunju Kim ba, ma, msc

2.4Mathematical description of sequential drug decision problems

SDDPs involve patients experiencing a series of health-related and drug-related events over a finite time. A finite set of mutually exclusive health states h∈H and drugs a∈A are defined. Note that a drug a defined here refers to a single treatment regimen, which could be a single drug (potentially with different doses), or a combination of more than one drug in circumstances where multiple drugs may be prescribed at the same time. Given a health state s_t∈H at time t, the selected drug x_t∈A has an impact on both the next health state s_t→s_t+1 and the costs/benefits associated with this transition.

As the health state transitions are not fully known in advance, a transition probability function can be used to represent the probability of the patients being in state s_t+1 at time t+1 if drug x_t is chosen for health state s_t at time t. For simplicity, the transition probability is often assumed to follow the Markov property, which assumes no memory. That is, future states s_t+1 depend only upon the present state s_t and x_t, not on the sequence of events that preceded it. Mathematically, such an SDDP can be defined as P(T, H, HS, A, SS, DS, Ω, p, r, f), as follows:

• 
  T=(t₁,t₂,…,t_n) represents the time dimension with n periods where possible changes in health state and drug use occur.
  
• 
  H represents a set of l possible health states H={h₁,h₂,…,h_l }.
  
• 
  HS represents the health state space, which is a set of possible health state combinations or disease pathways within the time dimension HS={θ=(s₁,s₂,…,s_n, s_n+1 )}, where s_t∈H. The number of potential health state combinations within T is:
  

, where all patients start with the same health state. Equation 2.1.

• 
  A represents a set of m possible drug alternatives A={a₁,a₂, …,a_m}.
  
• 
  A drug switch is when two different drugs (a_i and a_j where i≠j) are used sequentially in two adjacent time periods within T. Theoretically, the maximum number of drug switches that can possibly occur within T is n (i.e., if drug switch occurs between all adjacent time periods), but in reality the number of drug switches within T may be less than n if the same drug a_i can be used continuously for more than one time period whilst being effective and without any AE. In this definition, it is assumed that a drug can be continuously used for more than one time period, but a drug cannot be re-used if it has been previously used and replaced with another drug.
  
• 
  SS represents the search space, which is a set of possible drug sequences SS={π=(d₁,d₂,…,d_m)}, where d_i∈A. π represents one potential sequential treatment policy, which is the permutations of m possible drug alternatives within A. Once T is defined the number of possible drug sequences is:
  

, where m≤n,

, where m>n

Equation 2.2.

• 
  Decision space DS={π_sx=((s₁,x₁),(s₂,x₂),…,(s_n,x_n))} represents a set of health state transitions within T together with the associated drugs used for each health state, where s_t∈H and x_t∈A. Unlike the sequential treatment policy π=(d₁,d₂,…,d_m), the drug sequence used for each time period π_x=(x₁,x₂,…,x_n) allows repetition of the same drug depending on s_t (i.e., a drug can be used continuously for more than one time period if the drug is effective). Depending on the decision when to switch drugs within T, there are a set of variations of π_x={=(x₁,x₂,…,x_n)} for a specific π=(d₁,d₂,…,d_m). For example, considering three drugs 1, 2 and 3 for four time periods, one of the six possible sequential treatment policy π=(1,2,3) has 11 variations π_x during T as following:
  

π_x = {(1,1,1,1,1),(1,1,1,1,2),(1,1,1,2,2),(1,1,2,2,2),(1,2,2,2,2),(1,1,1,2,3),

(1,1,2,3,3), (1,2,3,3,3),(1,1,2,2,3),(1,2,2,3,3) and (1,2,2,2,3)}.

• 
  Ω represents the set of probability matrices ω for s_t+1, which depends upon either s_t and x_t, where the Markovian assumption is used, or (s₁,x₁,s₂,x₂,…,s_t,x_t) where non-Markovian assumption is used. The probability that θ=(s₁,s₂,…,s_n) happens for a given drug sequence π_x is:
  

,

where

Equation 2.3.

• 
  The reward function is the sum of the partial rewards of selecting drug x∈A for health state s∈H at time t:
  

Equation 2.4.

• 
  The objective function f(π) is the sum of all reward functions r(θ) for all possible health state combinations:
  

Equation 2.5.
In a maximisation problem, the optimal drug sequence π*∈SS satisfying f(π*)≥f(π) ∀ π∈SS is called the globally optimal solution of P(T, H, HS, A, SS, DS, Ω, p, r, f). Figure ‎2. illustrates possible health state transitions and associated drug uses of a simple SDDP diagrammatically where the Markovian assumption is made.

1) s_t represents the health state at t; x_t represents the drug choice at t; ω(s_t+1|s_t,x_t) represents the transition probability from s_t to s_t+1 where x_t is used; r(s _t+1|s_t,x_t) represents the partial reward where x_t is used for s_t at t; r(θ) represents the sum of the partial rewards of using drug π_x for a specific disease pathway θ=(s₁,s₂,…,s_n).

Figure ‎2.. Graphical presentation of an SDDP with the Markovian assumption