Transitioning Regional Economies

110.E.4 Sensitivity testing

Sensitivity testing was conducted to see how much each region’s index values varied in response to:

creating the index using the single PCA approach or the nested PCA approach

changes in the regions included in the analysis

changes in the indicators used to assess adaptive capacity.

111.Comparing single PCA and nested PCA approaches

There are a number of differences and similarities between the PCA results for the two approaches. The correlation between the two indexes was 83 per cent. About 69 per cent of the regions that were in the least adaptive category under the single PCA approach were also in the least adaptive category under the nested PCA approach. For the most adaptive category, this was 60 per cent.

Figure E.6 charts each region’s index score on the nested PCA approach against their score on the single PCA approach. The 45 degree line indicates where regions would be positioned if their scores under both approaches were the same. Regions with lines extending upwards (downwards) from the 45 degree line have a higher (lower) score under the nested PCA approach than the single PCA approach. The horizontal and vertical grey lines indicate the category of adaptive capacity (least adaptive, below average, above average, most adaptive). Regions change adaptive capacity categories across single and nested PCA approaches if the line representing their index scores crosses one of the horizontal grey lines.

As an example, Barkly (represented by region A in figure E.6) has a line that extends upwards from the 45 degree line, but it does not cross a horizontal grey line, indicating that it remains in the same category (least adaptive) under both approaches. Central Tasmania (region B) has a higher nested PCA index score than single PCA index score, which takes it from the least adaptive category to below average. Greater Sydney (region C) has a lower nested PCA index score than single PCA index score, which takes it from the most adaptive category to above average. Overall, the regions that change categories are mainly in the above average or below average categories.

Figure E.6 Comparison of index scores under single and nested PCA

^aThe horizontal and vertical grey lines indicate the category of adaptive capacity (least adaptive, below average, above average, most adaptive).

Source: Productivity Commission estimates.

Comparing the maps of regions by adaptive capacity category under single PCA and nested PCA approaches (figures E.4 and E.5), there are notable similarities. For example, some remote areas (such as those in the Northern Territory, and the Kimberley region) have relatively low adaptive capacity across both indexes. Capital city regions tend to have relatively high adaptive capacity and regional areas of New South Wales and Victoria appear similar across both indexes.

However, there are some differences in mining and agricultural regions, due to the differences in weights attributed to mining and agriculture indicators in each index. In the single PCA index, mining has a negative weight, whereas in the nested PCA index, mining is omitted. Therefore, some regions with high mining employment (such as Karratha, Port Hedland – Newman and Mackay regions) appear as having at least above average adaptive capacity in the nested PCA index, whereas they were below average in the single PCA index. Agriculture has a larger positive weight in the nested PCA index compared with the single PCA index, which makes some agricultural regions (such as South Wheatbelt, Darling Downs–South West Queensland and Yorke Peninsula regions) appear to have higher relative adaptive capacity under the nested PCA index.

112.Changes in regions and indicators

The last two aspects of the sensitivity testing were done concurrently for the single PCA index through a bootstrapping technique. The true adaptive capacity of a region as well as the standard deviation of its estimated adaptive capacity is unknown, and both would be expected to be a function of the regions in the PCA as well as the variables included. The bootstrapping technique involves running the same analysis many times on multiple new samples of data that are constructed by random sampling with replacement from the initial dataset. These new samples have the same number of observations as the initial dataset. (This means that the new samples will likely have multiple observations of a particular region, while other regions may not appear in a particular sample at all.)

In the current analysis, 1000 bootstrap samples were formed, and the sensitivity of the index results to these changes in the sample of regions was examined. For each of the 1000 bootstrap samples, one indicator of adaptive capacity was removed from the analysis each time in order to assess the effect of small changes in the set of indicators on the index results. The removal of every indicator was tested in turn (that is, the sensitivity of the index results to each indicator individually was tested). As there were a total of 39 indicators, that means there was a total of 39 000 calculations of the index for each region using this bootstrapping technique.

Constructing indexes from PCA requires a degree of judgment in selecting the number of principal components to retain and examining whether the signs should be reversed (section E.1), but it is impossible to apply the same level of scrutiny to each PCA in each of the iterations of the bootstrapping analysis. Therefore a number of assumptions were made. It was assumed that decisions about the numbers of principal components to retain across all iterations are the same as in the calculation of the original single PCA index. It was also assumed that the main two indicators that contributed to the interpretation of each principal component were the same as in the construction of the original index, and signs of principal components were reversed where necessary, so that they were in the expected direction for those indicators.

The distribution of each region’s index values from the bootstrapping analysis was examined to see how sensitive the results were, and the 5^th and 95^th percentiles of each region’s distribution of index values were plotted. The chart for the single PCA approach is presented in figure E.7. It was found that many regions had particularly large intervals, indicating greater uncertainty in their index values and relative rankings.

Under the single PCA approach, the rankings of more remote regions tend to be more sensitive to changes in indicators, and they are more likely to change categories with the removal of various indicators. This is especially the case for Port Hedland – Newman (region A in figure E.7) and Karratha (region B) regions in Western Australia, which both have relatively high mining employment. Both of these regions are categorised as being below average in adaptive capacity, but would be placed in the least adaptive category based on their lower confidence limits, and in the most adaptive category based on their upper confidence limits.

Figure E.7 High uncertainty in the rankings of adaptive capacity Index values for each FER and their 90 per cent confident intervals, sorted from lowest to highest^a

^a The least and most adaptive regions are defined as those above and below one standard deviation of the mean index value of adaptive capacity across all regions. Regions are ordered based on their index value, where the whiskers represent the upper and lower 5 percentiles (90 per cent confidence intervals) of the region’s index value across bootstrapping analysis. Black dots represent the original index value.

Source: Productivity Commission estimates.

Regions in the least adaptive category tend to have positively skewed intervals, whereas regions in the most adaptive category tend to have negatively skewed intervals. For the exclusion of any indicator to dramatically increase (decrease) the score of a region identified as most (least) adaptive, the value for that indicator would have to be significantly worse (better) than most other regions’ values for that indicator. Regions identified as most (least) adaptive are unlikely to have many indicators with these properties (thus a short positive (negative) tail of their distribution).

Sensitivity testing results are discussed further in chapter 4. A spreadsheet of index scores for each region, as well as their 90 per cent confidence intervals based on the bootstrapping analysis, are provided as supporting materials on the Commission’s website.