Sequential drug decision problems in long-term medical conditions: a case Study of Primary Hypertension Eunju Kim ba, ma, msc

3.2.6Processing of the search result

The total number of retrieved hits from the four electronic databases was 10,517 including all the nine target papers identified by the pilot search. Table ‎3. shows the number of papers retrieved from four electronic databases. Full search strategies by database and the results of the varied combinations of search strategies are presented in Appendix 3.

The search results were transferred to EndNote together with abstract. After excluding 3,603 duplications, 6,914 potentially relevant studies were left. The title and abstract of these papers were manually reviewed to decide whether they meet the inclusion criteria. During this step, 6,211 studies were excluded because they did not address a stochastic sequential or multistage optimisation problem. 191 studies were excluded because they did not use a heuristic or optimisation method. Finally 512 studies that met the inclusion criteria were left. Figure ‎3. shows the flow chart of study selection.

Table ‎3.. The number of papers retrieved from four electronic databases by search strategy

		Database				Total^1)	Retrieval rate of the target paper^2)
	Search strategy	WOS	Scopus	Ovid	Embase
1	A	2,040,041	1,948,450	858,663	968,771	5,815,925	n/a
2	A'	53,400	50,124	9,409	10,861	123,794
3	A''	77,965	80,102	30,134	38,284	226,485
4	B	585,439	684,764	109,279	118,667	1,498,149
5	B'	172,268	233,930	25,113	25,061	456,372
6	C	2,064,178	2,222,123	961,438	1,157,020	6,404,759
7	A∩B	125,449	119,961	20,703	22,356	288,469
8	A'∩B'	6,011	7,653	248	254	14,166
9	A''∩B'	6,170	7,817	307	341	14,635
10	A∩B∩C	51,068	47,774	6,414	6,877	112,133	9/9
11	A'∩B'∩C	4,339	5,394	158	152	10,043	6/9
12	A''∩B'∩C	4,342	5,834	182	159	10,517	9/9

1) Duplication is not excluded.

2) The number of target papers retrieved in the search stage / the number of target papers.

3) The search strategy in grey is the final search strategy.

Figure ‎3.. Flow-chart of study selection

Based on the information in the title and abstract, these 512 studies were classified according to the heuristic and optimisation method(s) used. The methods were divided into two broad categories, which were mathematical programming and heuristics. For the classification, mathematical programming was defined as the exact algorithm as it guarantees to find the global optimal solution where the benefit or the constraints can be described as the functions of certain decision variables. In contrast, heuristics were defined as the approximate method, which identifies an optimal or near optimal solutions using a subjective heuristic rule.

In the review stage, heuristics were further divided into constructive methods, local search methods and other heuristic methods. The difference between constructive heuristic methods and local search methods defined in this study is the way to construct the search space. Constructive methods construct the decision space with partial solutions (i.e. potential individual drugs in the case of SDDPs) and combine the partial solutions of each sub-problem to construct a solution to the whole problem, whereas local search methods construct the decision space with complete solutions (i.e., potential drug sequences in the case of SDDPs). Local search methods were further distinguished between 1) single solution methods and 2) population-based methods depending on the number of candidate solutions used at the same time (see Table ‎3.).

Table ‎3.. The characteristics of each category

Classification	Characteristics
Approximate mathematical programming	Mathematically describe the problem with the objective function and/or the constraints. Find the optimum of continuous and differentiable functions using linear, integer, nonlinear or mixed programming method. Use the backward approach where DP is applied.
Heuristics (Constructive methods)	Organize the search space into a tree. Begin with an empty or partial solution. Construct the complete solution adding one by one.
Heuristics (Local search methods)	Start with one complete solution (Single solution method) or some complete solutions (Population-based method). Replace the current solution by a better solution iteratively in a defined neighbourhood.

Table ‎3. summarises the identified list of heuristic methods and optimisation methods that have been applied in the 518 identified papers. 228 studies used more than one heuristic method to compare with each other. 62 studies hybridised more than one heuristics method to improve the search procedure and the quality of solution. Therefore, one study may satisfy more than one category in Table ‎3..

Table ‎3.. Potential heuristic and optimisation methods for sequential decision problems

				SDP	SDP in Healthcare
(Approximate) Mathematical programming				283	7
	Dynamic programming Linear programming Mixed integer programming Integer programming Others			206 29 5 3 48	7 0 0 0 0
Heuristic method				402	5
	Constructive methods			23	0
	Greedy-based method Branch and bound Others constructive method			10 7 4	0 0 0
	Local search methods			129	5
	Single solution heuristics
		Simulated annealing Tabu search Variable neighbourhood search Guided local search Greedy randomised adaptive search procedure (GRASP)		12 6 5 1 1	1 0 0 0 0
	Population-based heuristics
		Evolutionary algorithm		22	0
			Genetic algorithm Differential evolution algorithm Evolutionary programming Estimation of distribution algorithms Memetic algorithm Scatter search (path-relinking)	12 2 1 1 1 1	0 0 0 0 1 0
		Particle swarm optimisation		2	0
		Artificial bee colony algorithm		1	0
	Other local search methods			86	4
	Problem-specific heuristics			290	0
Hybrid heuristics				62	0

1) SDP stands for sequential decision problem under uncertainty.

2) The numbers in dark rows represent the number of papers, which used the specified method, not the sum of the subgroups.

3.2.7Summary of the systematic review

1) Mathematical programming

206 studies, which were classified as DP, mostly used approximate dynamic programming (ADP), which includes simulation and/or a function approximation under the theoretical foundations in the traditional DP. Whereas DP is the standard methods to solve sequential decision problems in theory, it has been recognized that classic DP is limited in its suitability to directly treat large and complex sequential decision problems encountered in the real-world because of the curse of dimensionality and computational requirements^[51]. The systematic review also confirmed that classic DP on its own is considerably limited in its usefulness for large and complex real decision problems. This has motivated the development of ADP for solving large and complex sequential decision problems.

The most widely used ADP was RL, which is a simulation-based DP approach. Instead of computing or storing transition probabilities like DP, RL uses a stochastic approximation of Q-value, which is the expected value of each state-action pair for a certain period, within the simulation (this is further explained in section 3.5). Variations of this algorithm have been proposed for the transformation of Q-value^[123-126]. It also has been applied for discovering effective therapeutic regimens in clinical setting^[127-133]. Neuro-dynamic programming (NDP) is another name of RL where a Bellman’s value function is approximated using neural network architectures of simulated data^{[134, 135]}.

29 studies, which were classified as linear programming, attempted to alleviate the curse of dimensionality by fitting a linear combination of pre-selected basis functions to the Bellman’s value function^[136]. As this involved the DP procedure for optimisation, it was also called LP-based ADP or approximate linear programming^[137]. Various problem-specific methods for fitting the linear value function approximation have been proposed in the context of interactive multi-objective decision-making^[138], stochastic inventory/routing^[139], queuing control^[136], capacity of production line^[140], noisy prediction problem^[141] and fleet size and vehicle transfer problem^[142] and other large and complex MDP problems^[143-145].

Where sequential decision problems were formulated as integer or mixed integer programming, problem-specific heuristics were incorporated into the algorithms to search the decision space efficiently. Most heuristics used were based on “branch-and-bound” (see more information in the following sub-section)^[146-152].