Sequential drug decision problems in long-term medical conditions: a case Study of Primary Hypertension Eunju Kim ba, ma, msc

FOR An infinite number of times. Randomly simulate the initial state st. FOR each step of episode Choose a from s using policy derived from Q-values, using the ε-greedy method. Take action a, observe st+1 and r. st←st+1 until st is terminal. END END

Figure ‎3.. Q-learning algorithm^[200]

3.6Application of the key optimisation methods on a hypothetical case

3.6.1Description of the simple hypothetical sequential drug decision problem

A simple hypothetical SDDP was used to allow full enumeration and to illustrate how some of the proposed optimisation methods can be applied. As this simple example was of low complexity, a successive semi-Markov decision-tree model was used to calculate the value of the objective function (see Figure ‎3.). Four optimisation methods – enumeration, DP, SA and RL (i.e., Q-learning) - were tested and compared in the context of computational efficiency, intensity and the quality of solution for SDDPs. DP was applicable due to the low complexity of the hypothetical case, whereas GA was not included because SA was enough to represent the nature of meta-heuristics in such a small problem. The Matlab version 7.6.0, R2008a (The MathWorks, Natick, Massachusetts, U.S.A) was used to develop both the evaluation and optimisation models.

Figure ‎3.. Decision-tree of the hypothetical simple SDDP

The problem was mathematically defined as follows:

• 
  The total follow-up period is divided into three 3-monthly decision-making periods T=(t₁,t₂,t₃). A maximum number of two drug switches are allowed.
  

• 
  There are three possible health states H={H_u,H_e,H_n}, where H_u represents the undesirable health condition, H_e represents occurring AEs and H_n represents the health condition under control. There are 27 possible disease pathways, where the initial health state s₁ is H_u and the subsequent health states s₂, s₃ and s₄∈H.
  

HS={ θ₁=(H_u,H_u,H_u,H_u), θ₂=(H_u,H_u,H_u,H_e), θ₃=(H_u,H_u,H_u,H_n),

θ₄=(H_u,H_u,H_e,H_u), θ₅=(H_u,H_u,H_e,H_e),θ₆={(H_u,H_u,H_e,H_n),

θ₇=(H_u,H_u,H_n,H_u), θ₈=(H_u,H_u,H_n,H_e), θ₉=(H_u,H_u,H_n,H_n),

θ₁₀=(H_u,H_e,H_u,H_u), θ₁₁=(H_u,H_e,H_u,H_e), θ₁₂=(H_u,H_e,H_u,H_n),

θ₁₃=(H_u,H_e,H_e,H_u), θ₁₄=(H_u,H_e,H_e,H_e), θ₁₅=(H_u,H_e,H_e,H_n),

θ₁₆=(H_u,H_e,H_n,H_u), θ₁₇=(H_u,H_e,H_n,H_e), θ₁₈=(H_u,H_e,H_n,H_n),

θ₁₉=(H_u,H_n,H_u,H_u), θ₂₀=(H_u,H_n,H_u,H_e), θ₂₁=(H_u,H_n,H_u,H_n),

θ₂₂=(H_u,H_n,H_e,H_u), θ₂₃=(H_u,H_n,H_e,H_e), θ₂₄=(H_u,H_n,H_e,H_n),

θ₂₅=(H_u,H_n,H_n,H_u), θ₂₆=(H_u,H_n,H_n,H_e), θ₂₇=(H_u,H_n,H_n,H_n) }.

• 
  Where a decomposition method is used (i.e., where DP and Q-learning are used), the health state space HS_t in each period is constructed as following so that the model considers the different transition probabilities depending on the disease history:
  

HS₁={H_u}

HS₂={ θ₁=(H_u,H_u), θ₂=(H_u,H_e), θ₃=(H_u,H_n) }

HS₃={ θ₁=(H_u,H_u,H_u), θ₁=(H_u,H_u,H_e), θ₁=(H_u,H_u,H_n),

θ₂=(H_u,H_e,H_u), θ₂=(H_u,H_e,H_e), θ₂=(H_u,H_e,H_n),

θ₃=(H_u,H_n,H_u), θ₃=(H_u,H_n,H_e), θ₃=(H_u,H_n,H_n) }

HS₄={ θ₁=(H_u,H_u,H_u,H_u), θ₂=(H_u,H_u,H_u,H_e), θ₃=(H_u,H_u,H_u,H_n),

θ₄=(H_u,H_u,H_e,H_u), θ₅=(H_u,H_u,H_e,H_e),θ₆={(H_u,H_u,H_e,H_n),

θ₇=(H_u,H_u,H_n,H_u), θ₈=(H_u,H_u,H_n,H_e), θ₉=(H_u,H_u,H_n,H_n),

θ₁₀=(H_u,H_e,H_u,H_u), θ₁₁=(H_u,H_e,H_u,H_e), θ₁₂=(H_u,H_e,H_u,H_n),

θ₁₃=(H_u,H_e,H_e,H_u), θ₁₄=(H_u,H_e,H_e,H_e), θ₁₅=(H_u,H_e,H_e,H_n),

θ₁₆=(H_u,H_e,H_n,H_u), θ₁₇=(H_u,H_e,H_n,H_e), θ₁₈=(H_u,H_e,H_n,H_n),

θ₁₉=(H_u,H_n,H_u,H_u), θ₂₀=(H_u,H_n,H_u,H_e), θ₂₁=(H_u,H_n,H_u,H_n),

θ₂₂=(H_u,H_n,H_e,H_u), θ₂₃=(H_u,H_n,H_e,H_e), θ₂₄=(H_u,H_n,H_e,H_n),

θ₂₅=(H_u,H_n,H_n,H_u), θ₂₆=(H_u,H_n,H_n,H_e), θ₂₇=(H_u,H_n,H_n,H_n) }.

The health states at t₄ are the terminal states from sequential drug use. No decisions are made at this stage.

• 
  There are three possible drug treatment options A={drug₁,drug₂,drug₃}, where drug₁ has smaller treatment effect but lower risk of AEs; drug₂ has moderate treatment effect and moderate risk of AEs; and drug₃ has larger treatment effect but higher risk of AEs. There are six possible sequential treatment policies, where the problem is not decomposed, as following:
  

SS={ π₁=(drug₁,drug₂,drug₃), π₂=(drug₁,drug₃,drug₂), π₃=(drug₂,drug₁,drug₃), π₄=(drug₂,drug₃,drug₁), π₅=(drug₃,drug₁,drug₂), π₆ =(drug₃,drug₂,drug₁) }.

• 
  Where the decomposition method is applied (i.e., where DP and Q-learning are used), the search space SS is equivalent to A. The feasibility of each drug for each state is considered by a penalty function, which forces the net benefit to 0.
  

SS₁ = SS₂ = SS₃ = A = {drug₁,drug₂,drug₃}.

• 
  It is assumed that a drug is switched to another drug in the case of H_u or H_e (i.e., treatment failure) and the same drug is continued in case of H_n (i.e., treatment success).
  

• 
  To consider the impact of disease history, it was assumed that the baseline risk of H_u increases by 5% after two successive treatment failures (i.e. for the patients who went through H_u-H_u-H_u in previous periods) and by 10% after relapse (i.e. for the patients who went through H_u-H_n-H_u in previous periods). For the patients who had a relapse, treatment effectiveness was also assumed to decrease by 20% for drug₁, 10% for drug₂ and 5% for drug₃. The data used to populate the model are presented in an Appendix 4.
  

• 
  The objective function to maximise the treatment net benefit was:
  

Equation 3.11.