Sequential drug decision problems in long-term medical conditions: a case Study of Primary Hypertension Eunju Kim ba, ma, msc

Chapter 9.Conclusion

In the healthcare sector, selecting drug(s) among competing alternatives has been a key interest of decision-makers not just for clinical decisions, but also for resource allocation policy. Existing economic evaluations are normally concerned with evaluating drug alternatives provided at a specific point in a disease pathway, which could be first, second or third line drugs. However, decision-makers are also interested in which subsequent drug(s) should be used when the current drug needs to be replaced or complemented due to inefficacy, diminishing efficacy or AEs. These questions represent an SDDP that aims to identify a sequence of drugs along the disease pathway of a health condition with the objective of maximising the net benefits of treatment.

Studying the dynamic nature of SDDPs can be challenging because of the computational complexity caused by the potentially large number of drug sequences and disease pathways and the interdependence between the drug sequences and the disease pathways over time. Therefore, this research concerned the nature of SDDPs associated with computational complexity and suggested potential methods to solve SDDPs using a case study of primary hypertension.

In Chapter 2, a classification of model structures for the economic evaluation of SDDPs was proposed based on the taxonomy proposed by Brennan et al. The strengths and weaknesses of three main types of economic evaluation models – cohort-level decision-tree models, cohort-level Markov models and IEH including DES and ABS – were discussed in the context of modelling SDDPs. SDDPs were compared with - traveling salesman problems and job-shop scheduling problems to benchmark the well-known combinatorial optimisation problems for SDDPs. Different ways to define the SDDPs were discussed depending on the set of time defined and the decision-maker’s perspective. The SDDP was defined mathematically assuming that the given SDDP is a discrete-time Markov problem having a finite state/action spaces in a finite set of decision times. This indicated that the computational complexity of SDDPs mainly comes from the following factors: 1) the number of relevant health states, 2) the number of potential drug treatment options, 3) the number of times that a treatment change may occur, particularly where a time-sliced modelling approach is adopted, 4) whether the transition probability between health states depend on historic health states and drug uses and 5) relevant clinical-based rules to be incorporated (e.g., contraindication of certain drugs in the event of certain health states).

In Chapter 3, a systematic review was conducted to identify the potential the potential approximate optimisation methods to solve SDDPs. The studies classified as mathematical programming showed a limitation of that method to cope with large and complex problems and included simulation and/or a function approximation under the theoretical foundations in the traditional mathematical programming. For heuristic methods, three constructive methods, five single solution meta-heuristic methods and eight population-based meta-heuristic methods were identified. Based on the rank, RL, SA and GA were selected as the promising optimisation methods to solve SDDPs. The theoretical background and methodological application of the selected methods was explained. A hypothetical SDDP was used to show the feasibility of incorporating the proposed methods into an economic evaluation model and to identify some issues to be considered for the SDDP modelling in a real case. SA found exactly the same optimal solution with enumeration, although the advantage of SA in computational efficiency was not observed due to the small size of the decision problem. Classic DP identified the optimal solution, which had higher net benefits than enumeration. This can be explained by the stochastic scheme for choosing a next action where a problem is decomposed. This was also partially because of infeasible solutions as classic DP was limited to consider the medical history under the backward induction. Even though the applied one-step Q-learning identified a better solution excluding infeasible solutions, it required a considerable number of cases for convergence to the optimum and higher computational intensity to update the Q-values based on the newly observed data and to bootstrap the partial optimal solutions to construct the complete solution.

Chapter 4 was prepared to understand hypertension and pharmacological treatment of primary hypertension. An extensive literature review in previous economic evaluations of antihypertensive drugs in primary hypertension was included to understand the structures of previous CEA models and their limitations regarding the consideration of drug switching. Most previous models focused on the impact of an initial drug on long-term costs and health outcomes assuming that a patient continues with the drug for a defined time period regardless of the health state transitions. The main reasons that they could not consider drug switching were limited clinical or economic data of each drug and the complexity of modelling the interactive effects between various health states and drug choice over time.

The SDDP in primary hypertension was conceptualised following the mathematical description of the SDDPs in section 2.4. The time cycle was chosen to be three months, as the time to decide whether the drug is well-responded is fairly short in practice. As the NICE clinical guidelines on primary hypertension recommend a 4-step treatment algorithm, the maximum number of drug switching was assumed to be three after the initial drug was prescribed. Clinically and economically relevant health states – uncontrolled state without CVD or DM, controlled state without any CVD or DM, UA, MI, stroke, HF and DM – were selected based on a widely accepted underlying disease process of primary hypertension. Similar states with respect to transition probabilities or rewards were combined into a smaller number of aggregate states to alleviate the curse of dimensionality. Potential treatment options included four major single antihypertensive drugs – Ds, BBs, CCBs and ACEIs/ARBs – and their two or three-combinations. Dose-titration and drug switching within the same drug were not considered as a separate treatment option. Where the decision rule was based on step-wise treatment, the total number of sequential treatment policies was 4,128. For the transition probability, a semi-Markovian assumption was chosen for the short-term drug switching model because of the importance of considering the interactions between the disease pathway and sequentially used drugs in primary hypertension.

A simulation-based optimisation method, which identifies the optimal solution based on the outcome estimated in the underlying evaluation model, was used to build the hypertension SDDP model. The structure of the underlying evaluation model, data and key assumptions used to populate the model were described in Chapter 5 in detail. The underlying evaluation model has a form of successive decision tree that has an add-on Markov model. While the short-term drug switching model worked with a surrogate outcome modelling to allow drug switching based on SBP level, CV events and AEs, the long-term CVD model used the conventional RR approach to calculate the long-term cost and effectiveness of the sequential treatment policy. Chapter 6 described how various optimisation methods including enumeration, GA, SA and RL were implemented alongside the underlying evaluation model. The structure of the hypertension SDDP model built by m-files and pseudo-codes were provided.

Chapter 7 showed the outcomes of the hypertension SDDP model depending on the optimisation method applied, which were enumeration, SA, GA and RL. The optimal solution identified by enumeration was to start with ACEIs/ARBs, followed by Ds+ACEIs/ARBs, Ds+CCBs+ACEIs/ARBs and Ds+BBs+ACEIs/ARBs as second, third and fourth-line treatments. The total expected net benefit for this optimal sequential treatment policy was £330,080 (95% CI £330,013-£330,147). Considering the seven policies that were not significantly different to the optimal solution at 5% significance level, the optimal initial solution in primary hypertension is likely to be Ds or ACEIs/ARBs. The optimal second and third-line drugs were Ds+ACEIs/ARBs and Ds+CCBs+ACEIs/ARBs regardless the previous drug(s). The optimal fourth-line drug was not clearly observed. These results were robust to change in most variables used in sensitivity analyses: treatment objective, SBP lowering effect, the extension of drug switching period, the use of AE rates and random treatment scenario for CVD and DM, but relatively sensitive to the change in the patients’ initial SBP.

The difference between the results of the hypertension SDDP model and the recommendation of the NICE hypertension model demonstrates that considering the problem as a sequential problem makes a difference to the total net benefit compared with decision-making purely based on the cost-effectiveness of the initial drug. In addition, the cluster analyses showed that the cost-effectiveness of antihypertensive treatments can be affected by the subsequent drug use, particularly the use of a second-line drug. While no significant difference in total net benefit was found where 4,128 sequential treatment policies were divided by initial drug, a significantly better net benefit was observed where CCBs+ACEIs/ARBs, Ds+ACEIs/ARBs or Ds+CCBs were used as the second-line drug than where BBs+CCBs or BBs+ACEIs/ARBs were used as the second-line drug. Similar results were also found in the cluster analyses of the top 10% policies.

From the case study of primary hypertension, SA and GA proved their capability to identify the optimal (or statistically equivalent) solutions in a shorter time than enumeration. Their performance depends on the choice of a number of key parameters. The potential risk of premature convergence was observed where fast cooling schedule was employed for SA and where the population size was small or the population diversity was lacking for GA. However, using a slow cooling schedule or increasing the population size and diversity is directly associated with an increase in computational time. Therefore, prior tests are necessary to select key parameters with the aim of tuning the balance between diversification and intensification.

Compared with SA and GA, the quality of solution identified by RL was relatively less favourable. The quality of solution was improved where more cases were generated or where two or three-step future reward was considered than where one-step future reward was used. This may be because the total net benefit of each policy is much more affected by the long-term CVD model after the drug switching period than the short-term drug switching model in the current structure of the hypertension SDDP model. Therefore, the mechanism of updating Q-values based on the immediate reward or the reward from a certain future transitions could not fully consider the potential impact after the drug switching period. RL needs further investigation to improve the performance possibly by using more complicated RL methods or in a different structure of the underlying evaluation model.

The case study of SDDP in primary hypertension shows that various interacting trade-offs can be present in SDDP modelling. Firstly, the trade-off between the computational complexity and model validity was found in the underlying evaluation model. If the model considers more health states, treatment options and drug switching period and applies more complex transition rules considering the patient’s medical history, it potentially improves the model validity, but requires more time and effort to collect data and to build and evaluate the model. In contrast, if the underlying evaluation model is simple, it saves computational time and effort, but may miss something necessary for decision-making in SDDP. In the hypertension SDDP model, for example, drug switching was only considered for a limited period, which was four periods in the base-case. The uncertainty related to the drug switching period was assessed in sensitivity analysis. Although the optimal solution was now much different depending on the drug switching period, the sensitivity analysis indicated that the computational complexity increased considerably where the drug switching period was increased. Thus, it is important to improve the external model validity but also to keep such a large and complex model manageable where a large and complex problem is given like SDDP.

The trade-off between the computational complexity and model validity is also associated with the trade-off between the optimisation model and the underlying evaluation model. Given a limited time, increased computational time in the underlying evaluation model leaves less time to spend on the optimisation model. In particular, where a meta-heuristic is used, there is an implicit trade-off between the amount of search time and the quality of the solution. Where the search time was shorter (i.e., where fast cooling schedule was applied for SA and where the smaller number was selected as the population size for GA), it was more likely to trap at a local optimum. Therefore, the key parameters related to the diversification and intensification of meta-heuristics should be tuned through prior tests.

The hypertension SDDP model is a novel cost-effectiveness model, which involves clinically plausible drug switching rules for treatment success/failure, maintenance and contraindication. It also made a contribution by introducing a new type of dynamic and stochastic optimisation approach that applies approximate optimisation techniques to solve large and complex decision problems in HTA. In particular, methods to reduce the decision space and to efficiently solve large decision problems are promising research areas as more capable but complex modelling methods are introduced in HTA. Future research in developing the underlying evaluation model using individual-based models such as DES, improvements of heuristic algorithms and generating clinical data relevant to SDDPs will help to deal with the limitations of the current study and support better informed decision-making for SDDPs in HTA.