Sequential drug decision problems in long-term medical conditions: a case Study of Primary Hypertension Eunju Kim ba, ma, msc

List of tables

List of figures

Abbreviations

ABS	Agent-based simulation
ACEI(s)	Angiotensin converting enzyme inhibitor(s)
ADP	Approximate dynamic programming
AE(s)	Adverse effect(s)/event(s)
AF	Atrial fibrillation
ARB(s)	Angiotensin II receptor blocker(s)
BB(s)	Beta-blocker(s)
BMI	Body mass index
CCB(s)	Calcium channel blocker(s)
CEA(s)	Cost effectiveness analysis/analyses
CHD(s)	Coronary heart disease(s)
CI(s)	Confidence interval(s)
CKD(s)	Chronic kidney disease(s)
CMA(s)	Cost minimisation analysis/analyses
CRD(s)	Chronic renal disease(s)
CUA(s)	Cost utility analysis/analyses
CVD(s)	Cardiovascular disease(s)
D(s)	Thiazide-type diuretic(s)
DAE(s)	Discontinuation(s) due to AEs
DMARD(s)	Disease-modifying anti-rheumatic drug(s)
DAS	Disease activity score
DBP	Diastolic blood pressure
DES	Discrete-event simulation
DM	Diabetes
DP	Dynamic programming
ESH/ESC	European Society of Hypertension-European Society of Cardiology
GA	Genetic algorithm
HbA1C	Glycated haemoglobin
HDL	High-density lipoprotein cholesterol
HF	Heart failure
HR(s)	Hazard ratio(s)
HSE	Health Survey England
HTA	Health technology assessment/appraisal
IBM(s)	Individual-based model(s)
ICER(s)	Incremental cost-effectiveness ratio(s)
ITT	Intention-to-treat
JNC	The Joint National Committee on Prevention, Detection, Evaluation, and Treatment of High Blood Pressure
LDL	Low-density lipoprotein cholesterol
MDP	Markov decision process
MI	Myocardial infarction
NDP	Neuro-dynamic programming
NHS	National Health Service
NICE	National Institute for Health and Clinical Excellence
NP-hard	Non-deterministic polynomial-time hard
PSA	Probabilistic sensitivity analysis
QALY(s)	Quality-adjusted life year(s)
QoL	Quality of life
RA	Rheumatoid arthritis
RAS	Renin-angiotensin system
RCT(s)	Randomised-controlled trial(s)
RD	Renal disease
RL	Reinforcement learning
RR(s)	Relative risk(s)
SA	Simulated annealing
SBP	Systolic blood pressure
SD	Standard deviation
SDDP(s)	Sequential drug decision problem(s)
SE	Standard error
SLE(s)	SBP lowering effect(s)
SMDP(s)	Semi-Markov decision process(es)
SPLDT	Simulated patient-level decision-tree
SPLMM	Simulated patient-level Markov model
TC	Total cholesterol
TD	Temporal difference
TNB	Total net benefit
TNFb	Tumour necrosis factor-blocking agents
UA	Unstable angina

Chapter 1.Introduction

1.1Chapter overview

This study aims to set down the definitions of sequential drug decision problems (SDDPs) mathematically, to examine potential methods to identify optimal or near-optimal sequential treatment strategies in the context of a multiple drugs, multiple switches and multiple health states decision space problem and to discuss the performance of the proposed methods using a case study of primary hypertension. In the first chapter, SDDPs and the practical and methodological issues related to SDDPs are introduced. The optimisation methods used for sequential decision problems in other research areas are also introduced. These methods motivated this thesis and the research questions provided at the end of this chapter.

1.2Introduction to sequential drug decision problems

In the healthcare sector, selecting drug(s) among competing alternatives has been a key interest of decision-makers not just for clinical decisions but also for resource allocation policy^[1]. Existing economic evaluations are normally concerned with evaluating drug alternatives at a specific point in a disease pathway, which could be first, second or third-line drugs. However, decision-makers are also interested in which subsequent drug(s) should be used when the current drug needs to be replaced or complemented due to inefficacy, diminishing efficacy or AEs. These questions represent an SDDP that aims to identify a sequence of drugs along the disease pathway of a health condition with the objective of maximising the overall net benefits of treatment. A sequential decision process consists of a series of states and actions. In a dynamic system, which moves a sequence of states h₁, h₂, h₃,… at the times t = 1, 2, 3,…, a sequential decision consists of a sequence of actions a₁, a₂, a₃,…. The essential features of the sequential decision problem are that, given h_t at time t, the action a_t decides the subsequent state h_t+1 and the cost c_t and/or effectiveness e_t for the time spent in the state h_t. The action a_t could be a dose titration, replacing the current drug with a new drug or adding a new drug to the current drug. The overall treatment net benefit is defined as total health benefits expressed in the monetary units minus the total costs associated with sequential treatment.

The SDDP is most relevant for long-term health conditions where drug switching commonly happens. Taking the example of hypertension, 25.9-32.6% of participants in a major clinical trial took an additional open-label drug in one year due to lack of efficiency and safety concerns^[2]. In real practice, 50-60% of patients with hypertension experience some alteration of treatment regimen including a dose titration, drug switching or adding another drug to their initial prescription in the initial six months^[3]. In the longer term, the high drug discontinuation rate levelled off after one year regardless of the type of drugs (see Figure ‎1.)^{[2, 4]}. The main reasons for drug switches were either poor efficacy or adverse effects (AEs)^[3-6].[Hughes, 1998 #228]

1) Ds, BBs, CCBs, ACEIs and ARBs refer to different types of antihypertensive drug more fully defined in Chapter 4. The graph for "Others" was presented in the source from which this figure came from^[4]
Figure ‎1.. Long-term continuation rates of antihypertensive drugs

Although major clinical guidelines have recommended the pharmacological treatment algorithms for the subsequent treatments, the variation in their recommendations implies that there is no established standard of evidence-based optimal treatment sequence (see section 4.2.3 for details)^[7-10]. In addition, the recommendations from clinical guidelines are based on the piece-wise evidence but have not been tested in clinical trials or in economic evaluations. In the absence of robust evidence after initial treatment, the choice of the second or third-line drug among competitive drugs has been largely empirical and showed a random pattern^{[3, 4]}. Such non evidence-based decisions may reduce the net benefits of treatment and cause the inefficient use of constrained resources.

Several studies have been performed to compare the cost-effectiveness of a limited number of pre-defined drug sequences, particularly, where drug switching is commonly recommended because of the chronic and progressive features and concerns about potential resistance, tolerability and AEs (see Appendix 1 for the summary of an exploratory literature review, which was conducted to understand how sequential treatment policies for long-term medical conditions were modelled in health economic evaluation). These disease areas include rheumatoid arthritis (RA)^[11-21], cancer^[22-27], schizophrenia^{[28, 29]}, glaucoma^[30], and diabetes (DM)^[31-33]. The model structures were constructed to reflect the disease pathway or its pre-defined treatment pathway. In order to allow drug switching, most studies modelled a key drug switching point defined by non-response to treatment, the occurrence of AEs or other progressive events. When the drug switching point was reached, the next drug was given followed by the pre-defined treatment strategies. For example;

Welsing et al compared the 5-year cost-effectiveness of five sequential treatment policies including Tumour Necrosis Factor-Blocking agents (TNFb) and/or leflunomide (LEF) in the usual treatment for RA patients in the Netherlands^[12]. A Markov model was constructed with four health states defined by the Disease Activity Score (DAS). Patients with a high DAS score initially moved to another state depending on the results of a response assessment every three months. The patients, who did not respond to the initial treatment, switched to LEF for the patients who took usual treatment initially, or to usual treatment for the patients who took LEF or TNFb initially.

Cameron et al and Lux et al constructed a Markov model with four different treatment sequences of hormone-receptor-positive (HR+) advanced breast cancer, which included or excluded fulvestrant as either a second or third-line therapy^{[24, 26]}. Once patients experienced a progressive event, they moved to another treatment option up to five including best supportive care.

Bobes et al analysed the economic impact of most commonly used antipsychotic drugs - ziprasidone, olanzapine, risperidone and haloperidol - in schizophrenic patients^[28]. A Markov model was developed to simulate the AEs seen in the EIRE study (that is, Estudio de Investigacio´n de Resultados en Esquizofrenia; Outcomes Research Study in Schizophrenia)^[34]. Treatment started with one of four antipsychotic drugs. Depending on the type of AE, treatment was modified (i.e., decreasing dose or switching drug) as obtained from a local cross-sectional study and clinical trials previously published.

Payet et al defined the Markov states by alternative treatment options for glaucoma. Travoprost or latanoprosat were given to patients initially. If a patient failed to respond at the end of each cycle, the patient could 1) continue with the initial drug; 2) receive an additional glaucoma treatment; 3) switch to a new treatment; 4) undergo laser therapy or surgery; or 5) die. Treatment success rate and transition probability to alternative options were estimated from an observational study conducted to record the ophthalmologist's therapeutic decision within four weeks^[30].

To overcome the structural limitations of cohort-based models to consider patients’ time-dependent risk factors, some Markov models made attempts to add time-dependent variables and the interrelationship with health states and treatment decisions over time, using individual sampling and mathematical equations. For example, Brennan et al compared the cost-effectiveness of including or excluding etanercept to the most commonly used sequential treatment strategies of disease modifying ant rheumatic drugs (DMARDs) for the patients who failed with two DMARDs^[13]. The sequential use of DMARDs following disease progression was determined based on time-dependent variables such as response rates, mortality rates and Health Assessment Questionnaire (HAQ) score changes. In particular, modelling of HAQ score progression following initial response, non-response, ongoing success or withdrawal made it possible to present a patient’s medical history and the impact on costs and quality-adjusted life year (QALYs) over time.

Individual-based models (IBM) such as discrete-event simulation (DES) were employed for better description of the dynamic relationship between the disease pathway and individual patients’ time-dependent variables. Barton et al developed the Birmingham Rheumatoid Arthritis Model (BRAM) to compare the cost-effectiveness of different DMARDs sequences for the patients starting a DMARD^[35]. The most representative DMARD sequence, which was selected in a survey conducted in UK rheumatologists, was compared with two other strategies including etanercept or infliximab in the representative strategy. Each patient was assigned age, gender and initial HAQ score at the beginning of the simulation, and a drug was switched to the next drug in the case of HAQ increase, joint replacement or discontinuing the drug. The risk of joint replacement or death was dependent on HAQ, which was varied according to treatment. Time to competing events was calculated by the probabilities that each event occurs in an assumed constant time interval.

The Januvia Diabetes Economic (JADE) Model was designed to project the impact of different Glycated Haemoglobin (HbA1c) thresholds on long-term health outcomes for type 2 DM patients who have failed with metformin^[32]. The baseline profile included time-dependent risk factors, medical history and current treatment regimen. The first occurrence of DM-related complications was projected using equations of the UK Prospective Diabetes Study (UKPDS). Once a drug failed to reduce HbA1c below the threshold, the patient moved to the next treatment regimen. The model allowed up to six drug switches over a lifetime. Different sequential treatment regimens were used according to the patient’s HbA1c response, toleration and contraindications.

The main objective of these studies was to evaluate the overall cost and effectiveness of 1) including or excluding a specific new drug in/from routinely used treatment in practice, as a second or third-line therapy or 2) representative sequential treatment strategies selected based on survey or interview of expert groups^{[11, 24, 36]}; a separate micro-simulation generating the probability of treatment sequences^[29]; or clinical trials and retrospective observational studies^{[28, 30]}. Therefore these studies could show the improvement in the cost-effectiveness ratio across the competing sequential treatment strategies (i.e., local optimality), but do not present whether the strategy is the optimal strategy to reach a treatment goal. To the author’s knowledge, there has been no economic evaluation to address the global optimality of SDDPs in a long-term health condition.

The reason that the economic evaluation addressing the global optimality of an SDDP is rarely found is partially associated with the main interest of health technology appraisal (HTA). Due to the scarcity of resources in healthcare, economic evaluation was introduced as a means of promoting the efficient use of available resources. In the case of pharmaceuticals, particularly, economic evaluation has become a core procedure for pricing and reimbursement in many countries. The primary goal of economic evaluation is to help decision makers arrive at the best choices among competing claims on resources^{[37, 38]}. Therefore the cost-effectiveness evidence has been generated following the current guideline published by the HTA body, which limits the alternative treatments (sometimes treatment sequences) to the current best practice for a restricted population because of resource requirement, time to develop the model and to make a decision^{[7, 39]}.

From a methodological perspective, it has been argued that a methodological shift in economic evaluation is required from a static to a dynamic perspective to identify the optimal treatment sequence in long-term medical conditions^{[40, 41]}. However, studying the dynamic nature of SDDPs can be challenging because of the computational complexity caused by a potentially large number of drug sequences and disease pathways (i.e., the sequences of health states) and the interdependence between the drug sequences and the disease pathways over time^[42]. Another challenge is how to model SDDPs when experimental or observational longitudinal data of sequential drug use are not available. While data on the clinical effectiveness of first line drugs is relatively abundant, data on the effectiveness of subsequent treatment(s) after an initial drug fails and any potential interaction with concurrent drugs is scarce for many health conditions.

Several studies have developed a comprehensive policy model, which can simultaneously evaluate the cost and benefit associated with the prevention and therapeutic interventions across the entire disease pathway. Weinstein et al developed the coronary heart disease (CHD) policy model, which consists of the demographic-epidemiologic model for persons free of CHD, the bridge model for persons with new CHD and the disease history model for persons with a CHD history, to forecast the CHD incidence, mortality and cost of alternative preventive and therapeutic interventions^[43]. Recently, Tappenden et al proposed a methodological framework of “Whole Disease Modelling” in cancer treatment and prevention that covers the costs and effectiveness occurring in pre-clinical disease state, diagnosis and referral, treatment for early disease, follow-up for surveillance and treatment for metastases, using a case study of colorectal cancer^{[39, 44, 45]}. However, the comprehensive disease pathway model is a simulation model but not an optimisation model. It can provide the cost-effectiveness estimates for a larger part of complicated disease pathway but does not include a solution procedure, which finds the best solution to maximise the overall treatment net benefit. In particular, solving SDDPs involves testing a considerable number of possible treatment pathways to identify the optimal or near optimal solution. This means that an SDDP model will need to be incorporated with an optimisation technique, which could search the large decision space more efficiently. Hence this thesis tried to identify the potential optimisation methods to solve SDDPs efficiently and to discuss about the applicability of the key optimisation methods using a case study of primary hypertension.