Sequential drug decision problems in long-term medical conditions: a case Study of Primary Hypertension Eunju Kim ba, ma, msc

Chapter 6.Modelling sequential drug decision problem for hypertension: Implementation

6.1Chapter overview

This chapter includes an overview of the developed simulation-based optimisation model. Particularly, this chapter describes how various optimisation methods including enumeration, GA, SA and RL are implemented alongside the underlying evaluation model described in the previous chapter. The summary of the functions and pseudo-codes of optimisation algorithms used in the hypertension SDDP model are provided.

6.2Overview of the hypertension SDDP optimisation model

The hypertension SDDP optimisation model consists of a set of functions written in m-files, which is the name of script file in Matlab (see Table ‎6.). It starts with ‘OptModel’, which allows selecting different scenarios to test using one of the optimisation methods selected for implementation. The base model is designed to simulate a cohort of 60 year old male patients whose mean SBP was 173.5 mmHg. Every three months they experience one of three health states (i.e., Success, Failure or Death). There are 14 potential active treatment options including four single drugs and their two or three drug combinations. No treatment is included only for the purpose of model validation, but not included in sequential treatment strategies. The number of drugs used for the drug switching period is four (i.e., the maximum number of drug switches is three). The applied decision rule restricted the first-line drug to a single drug and the second-line drug to either a single or two-drug combination. Three-drug combinations were only allowed after the use of two-drug combinations.

Table ‎6.. Summary of the functions included in the hypertension SDDP model

Functions	Description
OptModel	Selects the scenario to test
TreeGenerator	Generates the health state space (i.e., a set of possible disease pathways).
PolicyGenerator	Generate the search space (i.e., a set of possible sequential drug treatment policies).
VasGenerator	Generates a matrix indicating which lines of drug treatment are used for a specific health state.
MTGenerator	Generates a matrix indicating whether the maintenance therapy is used and if so, how long it has been used.
TrtGenerator	Generates a treatment scenario for CVDs and DM.
Combinator	Performs permutation and combination with/without repetition.
EvModel	Calculates the value of the objective function where the decision space is not decomposed (i.e., where enumeration, SA or GA is used).
EvModelDC	Calculates the value of the objective function where the decision space is decomposed (i.e., where RL is used).
SBPmodelling	Simulates the SBP change after treatment using Monte Carlo sampling.
CVDmodelling	Computes CVD-related costs and effectiveness in long-term.
QRISK2	Predicts the 10-year CVD risk using the QRISK2 equation.
Data	Provides the relevant data.
Enu	Solves the problem using enumeration.
SA	Solves the problem using simulated annealing.
GA	Solves the problem using genetic algorithm.
RL	Solves the problem using reinforcement learning.
Validation	Runs the validation model, which assumes a drug is used continuously over the follow-up period.
PostHoc	Implements descriptive cluster analysis and statistical analyses (including t-test, ANOVA and multiple comparison) based on the enumeration results.

Different scenarios are tested depending on the selection of the parameters described in Table ‎6.. The options for the optimisation algorithm includes enumeration, GA, SA and RL. The objective function can be either to maximise total benefit in the long-term or treatment success rate in the short-term. The number of drug switching period is modifiable: however, it is not recommended to extend more than eight periods because of the computational time. The hypertension SDDP model also allows solving the decision problem in different gender and age groups with various level of SBP at the beginning. Two data sets of SBP lowering effect are included. Treatment scenarios for UA, MI, stroke, HF and DM can be either a set of recommended drugs (i.e., using CCBs, BBs, BBs, ACEIs, ACEIs for UA, MI, stroke, HF and DM, respectively) or randomly selected. AEs can be defined by either the discontinuation due to AEs or the prevalence of any unfavourable symptoms that might be caused by the antihypertensive drug. The number of PSA runs is modifiable; it was set to 100 in the base-case. Both costs and QALYs were discounted at 3.5%. Willingness-to-pay for a unit of QALY was assumed to be £30,000.

The purpose of the function ‘TreeGenerator’ is to generate a decision tree, which includes a set of possible disease pathways for the drug switching period. The number of possible disease pathways is dependent on the number of potential health states (i.e., three health states including Success, Failure and Death) and the selected drug switching period (i.e., four periods in the base-case). Where infeasible disease pathways (i.e., Success or Failure followed Death) are excluded, a total number of 31 possible disease pathways are generated as per Table ‎6..

The function ‘PolicyGenerator’ generates a search space that includes a set of sequential treatment policies having the fixed length of four (see Table ‎6.). The number of potential sequential treatment policies is dependent on the number of treatment options in each line of drug treatment (i.e., four options for the first-line, 10 options for the second-line and 14 options for subsequent lines of use), the maximum number of drug switching allowed (i.e., fixed to three in this study) and additional step-wise drug decision rules (e.g., three-drug combinations are only allowed after the use of two-drug combinations). 4,128 sequential treatment policies were generated with a combination of four integer numbers between 2 and 15 (see Table ‎6.). For example, a specific treatment policy using “Ds-BBs-CCBs-ACEIs/ARBs” was coded as “2-3-4-5”. Then, a policy number from 1 to 4,128 was assigned to the 4,128 policies, which were sorted in ascending order based on the coded number for the first-line drug (i.e., policy number 1 denotes “2-3-4-5” and policy number 4,128 denotes “5-11-15-14”). The complete list of 4,128 treatment sequences is presented in Appendix 7.

The function ‘VasGenerator’ returns a 31 by 4 matrix, where 31 is the number of possible disease pathways and 4 is the number of selected drug switching period (see Table ‎6.). The values in the matrix are an integer number between 1 and 4, which indicate what line of drug should be used on a specific health state. They are directly associated with the decision tree encoded like Table ‎6.. For example, for the disease pathway of “1-1-1-1-1” in the first row of the decision tree in Table ‎6., the values in the matrix generated by ‘VasGenerator’ will be “1-2-3-4”, where the first-line drug is replaced with second, third and then fourth-line drugs for the patient who failed to achieve the treatment goal successively during the drug switching period. 0 was used for Death.

The purpose of the function ‘MTGenerator’ is to create a 31 by 4 matrix that indicates whether the maintenance therapy is used and if so, how long it is used (see Table ‎6.). This matrix is also associated with the decision tree in Table ‎6.. ‘MTGenerator’ assigns 0 for Failure and an integer number between 1 and 3 for Success depending on the previous health states. If a patient achieves the treatment goal with the initial drug (i.e., second column and sixteenth row in Table ‎6.), the value in the same location of Table ‎6. is 1. If the patient consistently achieves the treatment goal in the third and fourth periods (i.e., 3rd column and 23th rows and 4th column and 26th row in Table ‎6.), the values in the same location of Table ‎6. are 2 and 3, respectively. These numbers facilitates the use of different levels of SBP lowering effect depending on the period in which a specific drug is used. If the current drug is replaced with a new drug in the second period, the SBP change in that period is determined based on the SBP lowering effect of the new drug in three months; otherwise, the SBP lowering effect is cumulative at 6, 9 or 12 months, depending on how long the current drug has been used.

Table ‎6.. Scenario selection to implement the hypertension SDDP model

Parameter	Scenario selection
Fixed parameters
The length of drug sequence	4
Cycle length	3 (months)
The number of health states	3
The number of potential treatment options	14
Additional decision rule	Stepwise drug selection
Variable parameters
Optimisation method	NaN = Validation model 'Enu' = Enumeration, which provides the exact solution. 'SA' = Simulated annealing, which provides an approximate solution. 'GA' = Genetic algorithm, which provides an approximate solution. 'RL' = Reinforcement learning, which provides an approximate solution.
The objective of the model	1 = To maximise the total treatment net benefit in long-term (until 100 years-old)* 2 = To maximise the treatment success rate in short-term (only in the drug switching period)
The number of drug switching period	4 = Total drug switching period is one year (i.e., 3 months x 4)* 6 = Total drug switching period is 1.5 year (i.e., 3 months x 6) 8 = Total drug switching period is 2 year (i.e., 3 months x 8)
Gender	1 = Male* and 2 = Female
Initial age	50, 60* and 70
Initial SBP	173.5 (SD 21.1)
SBP lowering effect	1 = Based on Wald’s and Wright et al’s systematic reviews* 2 = Based on ALLHAT and Dutch-TIA
Treatment scenario for UA, MI, Stroke, HF and DM	1 = A set of recommended drugs* 2 = Random selection
The risk of AEs	1 = Discontinuation due to adverse effects* 2 = Any unfavourable symptoms, which might be caused by the drug
The number of PSA repetitions	100
Discount rate	3.5%
Willingness-to-pay	£30,000

1) The values with * were used in base-case.

Table ‎6.. Matrix of the decision tree generated by ‘TreeGenerator’

Number	The combinations of potential health states
	t=1	t=2	t=3	t=4	t=4
1	1	1	1	1	1
2	1	1	1	1	2
3	1	1	1	1	3
4	1	1	1	2	1
5	1	1	1	2	2
6	1	1	1	2	3
7	1	1	1	3	3
8	1	1	2	1	1
9	1	1	2	1	2
10	1	1	2	1	3
11	1	1	2	2	1
12	1	1	2	2	2
13	1	1	2	2	3
14	1	1	2	3	3
15	1	1	3	3	3
16	1	2	1	1	1
17	1	2	1	1	2
18	1	2	1	1	3
19	1	2	1	2	1
20	1	2	1	2	2
21	1	2	1	2	3
22	1	2	1	3	3
23	1	2	2	1	1
24	1	2	2	1	2
25	1	2	2	1	3
26	1	2	2	2	1
27	1	2	2	2	2
28	1	2	2	2	3
29	1	2	2	3	3
30	1	2	3	3	3
31	1	3	3	3	3

1) 1, 2 and 3 represent Failure , Success and Death, respectively.

Table ‎6.. Matrix of the search space generated by ‘PolicyGenerator’

Policy number	Treatment sequence.
	First-line	Second-line	Third-line	Fourth-line
1	2	3	4	5
2	2	3	4	6
3	2	3	4	7
4	2	3	4	8
5	2	3	4	9
6	2	3	4	10
7	2	3	4	11
8	2	3	5	4
9	2	3	5	6
10	2	3	5	7
11	2	3	5	8
12	2	3	5	9
13	2	3	5	10
14	2	3	5	11
	. . .
4118	5	11	15	2
4119	5	11	15	3
4120	5	11	15	4
4121	5	11	15	6
4122	5	11	15	7
4123	5	11	15	8
4124	5	11	15	9
4125	5	11	15	10
4126	5	11	15	12
4127	5	11	15	13
4128	5	11	15	14

1) 1 represents no treatment; 2 represents Ds; 3 represents BBs; 4 represents CCBs; 5 represents ACEIs/ARBs; 6 represents Ds+BBs; 7 represents Ds+CCBs; 8 represents Ds+ACEIs/ARBs; 9 represents BBs+CCBs;10 represents BBs+ACEIs/ARBs; 11 represents CCBs+ACEIs/ARBs; 12 represents Ds+BBs+CCBs; 13 represents Ds+BBs+ACEIs/ARBs; 14 represents Ds+CCBs+ACEIs/ARBs and 15 represents BBs+CCBs+ACEIs/ARBs.

Table ‎6.. Matrix of the actual drug use generated by ‘VasGenerator’

Number	The position of the drug used at t in the treatment sequence.
	t=1	t=2	t=3	t=4
1	1	2	3	4
2	1	2	3	4
3	1	2	3	4
4	1	2	3	3
5	1	2	3	3
6	1	2	3	3
7	1	2	3	0
8	1	2	2	3
9	1	2	2	3
10	1	2	2	3
11	1	2	2	2
12	1	2	2	2
13	1	2	2	2
14	1	2	2	0
15	1	2	0	0
16	1	1	2	3
17	1	1	2	3
18	1	1	2	3
19	1	1	2	2
20	1	1	2	2
21	1	1	2	2
22	1	1	2	0
23	1	1	1	2
24	1	1	1	2
25	1	1	1	2
26	1	1	1	1
27	1	1	1	1
28	1	1	1	1
29	1	1	1	0
30	1	1	0	0
31	1	0	0	0

1) Where a specific drug sequence is given, 1 represents that the first-line drug is used for the corresponding health state; 2 represents that the second-line drug is used for the corresponding health state; 3 represents that the third-line drug is used for the corresponding health state; 4 represents that the fourth-line drug is used for the corresponding health state; and 0 represents that no drug is used for death.

Table ‎6.. Matrix of the maintenance therapy generated by ‘MTGenerator’

Number	The time period that maintenance therapy is applied.
	t=1	t=2	t=3	t=4
1	0	0	0	0
2	0	0	0	0
3	0	0	0	0
4	0	0	0	1
5	0	0	0	1
6	0	0	0	1
7	0	0	0	0
8	0	0	1	0
9	0	0	1	0
10	0	0	1	0
11	0	0	1	2
12	0	0	1	2
13	0	0	1	2
14	0	0	1	0
15	0	0	0	0
16	0	1	0	0
17	0	1	0	0
18	0	1	0	0
19	0	1	0	1
20	0	1	0	1
21	0	1	0	1
22	0	1	0	0
23	0	1	2	0
24	0	1	2	0
25	0	1	2	0
26	0	1	2	3
27	0	1	2	3
28	0	1	2	3
29	0	1	2	0
30	0	1	0	0
31	0	0	0	0

1) 0 represents no maintenance therapy (i.e., where either a new drug is started or death); 1 represents the start of maintenance therapy (i.e., a drug is continuously used in the next period); 2 represents the second period of maintenance therapy (i.e., a drug is continuously used in the next two subsequent periods); and 3 represents the third period of maintenance therapy (i.e., a drug is continuously used in the next three subsequent periods).
The function ‘TrtGenerator’ generates a 1 by 5 array that indicates which drug is used to treat patients with UA, MI, stroke, HF and DM. Depending on the scenario selected, it may generate a set of recommended drugs for each health state, which are [CCB, BB, BB, ACEI, ACEI] for UA, MI, Stroke, HF and DM; or randomly selected drugs excluding CCBs for HF.

The function ‘Combinator’ performs basic permutation and combination with/without repetition. Any function that involves numerical permutations or combinations such as ‘TreeGenerator’, ‘PolicyGenerator’, ‘VasGenerator’ and ‘MTGenerator’, employs the function ‘Combinator’. The m-file function developed by Matt Fig (2009) was modified for the hypertension SDDP^[361].

The function ‘EvModel’ (or ‘EvModelDC’) is the underlying evaluation model, which calculates the total net benefit of the sequential treatment policies. Given a specific sequential treatment policy generated by ‘PolicyGenerator’, this function provides total net benefit in the long-term or treatment success rate in the short-term. Where the decision space is decomposed, ‘EvModelDC’ provides the immediate reward from the transition between the current state and the next state. A number of sub-functions are implemented for ‘EvModel’ (or ‘EvModelDC’), which are ‘SBPmodelling’, ‘CVDmodelling’, ‘QRISK2’ and ‘Data’. Pseudo-code is provided in Figure ‎6..

For a group of patients who were in h at t, the function ‘SBPmodelling’ performs a Monte Carlo simulation to generate 1,000 random samples of baseline SBPs and treatment results after the consideration of SBP lowering effect. Given the mean SBP (and SD), which is the treatment result(s) in the previous period, and the drug used for the current state, this function returns the information required to decide the transition probabilities to the health states in the next period, which include the treatment success rate and the percentage of CVDs and AEs. The mean SBPs after treatment are saved for the controlled and uncontrolled, separately, and then used to generate the baseline SBPs depending on the health state in the next period. Pseudo-code is provided in Figure ‎6..

The function ‘CVDmodelling’ is the add-on Markov model, which calculates the CVD and DM related cost and effectiveness of each sequential treatment policy in the long-term. During the drug switching period, this function only considers those patients who have CVD or DM, whereas after the drug switching period, the function is applied to all surviving patients. Key data comes from the function ‘Data’. The 10-year CVD risk is calculated by the function ‘QRISK2’. Pseudo-code is provided in Figure ‎6..

‘Enu’, ‘SA’, ‘GA’ and ‘RL’, which represent enumeration, simulated annealing, genetic algorithm and reinforcement learning, were tested in this model. The following sub-sections describe the implementation of each method in detail. ‘Validation’ is included to validate the hypertension SDDP model internally and externally against the NICE hypertension model. ‘PostHoc’ implements various statistical analyses to provide a better understanding of the enumerated results.

1) Information about each functions and the full names for abbreviations are provided in Table ‎6..

Figure ‎6.. The structure of the hypertension SDDP model built by m-files