Transitioning Regional Economies

116.Attachment A: Report on Productivity Commission Index of Adaptive Capacity

117.By Professor Robert Tanton

This report has been written by Professor Robert Tanton, who has worked on a number of similar indexes to that developed by the Productivity Commission for the report on Transitioning Regional Economies. These indexes have included the 2001 ABS SocioEconomic Index for Areas (SEIFA); and the National Centre for Social and Economic Modelling child social exclusion index, youth social exclusion index and the index of wellbeing for older Australians.

It is clear that the Productivity Commission consulted widely as they developed the index of adaptive capacity, and talked to experts in the field, as well as holding workshops to discuss the index between the draft and final reports.

The method used by the Commission is a standard method of creating summary indexes called principal components analysis (PCA). This method takes a number of indicators, and summarises these into a set of uncorrelated components (indexes). A number of summary indexes are calculated using this process, and the first summary index explains most of the variability in the data. Further indexes will explain smaller proportions of the variability, and there is therefore some judgement in deciding how many indexes to retain.

Principal components analysis has been used by the ABS for the SEIFA indexes (Australian Bureau of Statistics, 2013); the New Zealand indexes of deprivation (Atkinson, Salmond, & Crampton, 2014); the UK indexes of child deprivation (Bradshaw et al., 2008); the South African indexes of child disadvantage (Barnes, Noble, Wright, & Dawes, 2008); and many other indexes (Dinh, Freyens, Daly, & Vidyattama, 2017; Vidyattama, Pearson, Tanton, & Mohanty, 2017). Most of these indexes only retain the first component as the index, given that this component explains most of the variation in the original data.

As an example of the method used, the ABS SEIFA index includes all indicators in the model; the principal components analysis is run against all the indicators; indicators which aren’t strongly associated with the first component (the ABS uses a criteria of 0.3 for this) are removed; and as the first component explains most of the variability in the indicators, this is the only component used for the index.

More recently, Bradshaw et al (2008) used a theoretical framework to define a number of domains, and then estimated indexes for each of these domains. This domains based approach was used for the second set of NATSEM child and youth social exclusion indexes. Using this approach, indicators are placed in domains; and then principal components analysis is used to derive a single index for each domain. Again, only the first index in each domain is used; and indicators with low loadings are removed. The domains are then added using a log transformation, to derive the final index.

The Productivity Commission has used both these approaches, to enable a comparison between these two methods. However, the Commission has diverged from these approaches by using a number of indexes, and bringing together these indexes, rather than just using the first index.

The other step the Commission has conducted is a log transformation for some of the indicators. Given the PCA technique assumes a linear relationship between the indicators, this is a reasonable approach.

For the single index approach, the Commission had 39 indicators. This is a very large number of indicators to measure one concept (adaptive capacity), and it is therefore probably not surprisingly that the first index explained only 28% of the variability in the indicators. The 2011 ABS SEIFA index of Disadvantage has 20 variables with the final index including only 16 of these due to the removal of some indicators due to low loadings, and the first component (the final index used by the ABS) and explained 44% of the variability of the indicators (Australian Bureau of Statistics, 2013).

While it is possible to run PCA as a data mining approach on thousands of variables, the index that the Productivity Commission requires is one of adaptive capacity, so the selection of variables needs to start from a theoretical framework and select indicators based on this framework. This is not a data mining approach. This framework is outlined in the main report, and this report has informed the selection of indicators in the index (although I would question the inclusion of the large number of indicators in an index of adaptive capacity – this concern is outlined further below).

Because the Commission had so many indicators, and a low explanatory power of the first index, the first 5 indexes were used for the final index. This diverges from the normal procedure for developing these indexes, where the first index only is used. However, the eigenvalues and scree plot do suggest that more than the first index was important (again, this result was possibly driven by the large number of indicators), and therefore there is some judgement required to choose the number of indexes. Personally, my judgement would have been to take the first 2 indexes, possibly the third. With these first 2 indexes, 51% of the variance in the indicators is explained, similar to the 44% deemed appropriate by the ABS for the SEIFA indexes. The third component adds 13 percentage points to the explanatory power, so could have been included. The scree plot, a standard way of determining how many indexes to retain, is shown in Figure 1 and suggests that the first 3 components be kept – it levels out at the 4^th component.
Figure 1: Scree plot for Index of Adaptive Capacity

The fourth and fifth components only added 5 percentage points each to the explanatory power. Keeping all five components also meant no indicators were removed due to low loadings, which would have reduced the number of indicators.

The main problem that the Commission faces with keeping 5 indexes is that they only want one index, so how do they go from the 5 indexes to one index? The solution that the Commission has used is to weight each of the indexes by the % explained, standardize them, and then add them. This is a technique that has been used in other literature (Krishnan, 2010; Nicoletti, Scarpetta, & Boylaud, 2000). Both of these reports used varimax rotation to minimise any cross loadings (where one indicator will load strongly onto two indexes).

The problem with this technique for the Commission is that their final index had a high level of cross loading (see Table E.8). For example, looking at the indicator wkage, it loads onto the first index with a loading of 0.76; and loads onto the second index with a loading of 0.53. The first and second indexes have similar contributions to the final combined index (the first index is 0.28 and the second is 0.23), so these two nearly offset each other. Table E.9 shows the impact of this across all the indexes – the weight for wkage in the final index is 0.007.

Using a varimax rotation should reduce the crossloadings (as seen in the Krishnan and Nicoletti papers). Table 4 in the Krishnan paper shows no crossloadings; and Tables 7 to 10 in the Nicoletti paper shows no cross loadings. This means adding the indexes (with some adjustment to standardise and weight them) works better than when there are cross loadings.

The impact of not using a rotated matrix can be seen in Table E.9 in the report, where the final weights for each indicator are shown. The overall weights in the final index are very low – indicators in the SEIFA index and NATSEM indexes which only uses the first component are removed as low loading if they have a loading (weight) of less than 0.3. The weights in the final Productivity Commission index are all close to or less than 0.1. This then raises the question why bother including many of the indicators – for example, the indicator psychdistress had a weight in the index of 0, and agind had a weight of 0.002. These are not adding anything to the final index.

This issue has been discussed with Commission staff, who provided results for a varimax rotation, which showed more cross loadings than before rotation. This is an interesting result, and one that needs to be investigated further, as rotation should in theory result in fewer crossloadings. I expect this may have something to do with the number of indicators included.

In the nested PCA approach, the number of indicators in each domain was fewer, and the first index tended to explain the majority of the variation in the indicators. In all these domains, the Commission could be justified in only keeping the first index (scree plots are shown in Appendix 1 which certainly justify keeping the first index for human capital; financial capital; and social capital. Physical capital and natural capital were harder to identify the levelling of the scree plot). This first index explains a good proportion of the variation in the original indicators (remembering that the ABS keeps the first component of SEIFA with a % explained of 0.44).

Again, any indicators with correlations less than 0.5 onto the first index could be taken out. The index from the each domain can then be added, once they have been standardised. Noble’s log transformation is typically used for this (Noble et al., 2004) but other transformations, as used by the Commission, are suitable.

The Commission has then conducted extensive testing on the index, which is commendable. The results for Karratha and Port Hedland, which swapped from least adaptive to most, is concerning, and may be due to the method used that brings together the different components. The method has resulted in some very low weights, and a few indicators with higher weights (0.099 for selfhealth), which means a few indicators will have a large impact on the final index. The weight of 0.099 for selfhealth is 50 times the weight of 0.002 for agind. If only the first index had been used, the weight for selfhealth would have been 0.66 and the weight for agind would have been 0.66, so there isn’t the large 50 times difference between them – they directly offset each other.

One of the other outcomes of the approach of using multiple indexes for each domain is that the interpretation of the indexes is difficult. For example, the first social capital index is strongly associated with safety at night, discriminated, getsupport, homelessness and volunteering. This is a difficult mix to interpret – what is the link between all of them so that we can determine what this first index is measuring? The second index is slightly easier – discriminated and cultural acceptance may be about discrimination in an area.

Looking at the final maps, they seem to make sense in terms of areas being closer to cities having a greater capacity to adapt. Recent work at NATSEM has found this, using fewer indicators (9 in total); and bringing together three domains, but the first index from each domain. This paper was initially provided to the commission as a draft which brought together two indexes in each domain, using a varimax rotation so there were no cross loadings; however the final published version, after peer review, only used the first index, and used a total of 9 indicators, 3 in each domain (Vidyattama et al., 2017).

In terms of the clarity of exposition, I think the Commission has explained the method of PCA well, and has explained the concept of adaptive capacity. I think that describing the different indexes (what they represent) wasn’t as clear, but I think this is due to the number of indicators used, as outlined above.

Overall, the Commission has conducted a thorough investigation of the methods for deriving indexes, and has consulted widely. The results look reasonable, but I would probably argue that the approach taken has been overcomplicated and could be simplified by rethinking the conceptual basis for adaptive capacity, and reducing the number of indicators. 39 indicators in one index has meant that each index in the one index method, and each domain in the nested method, is difficult to interpret, and a number of indexes have come up in the analysis, which then need to be combined.

Using a more concise and targeted set of indicators would mean that cross loadings would be reduced, and a varimax rotation could be used to reduce these cross loadings further. Multiple indexes in each domain should also be easier to interpret, although I would suggest that the nested approach could use the first index in each domain only, looking at the eigenvalues.

If the Commission does consider that they have a strong theoretical basis for including all 39 indicators (and this is suggested in the report), then I would argue that there is reasonable justification for using the nested approach, and only including the first index in each domain and removing any indicators with weights less than 0.5 (the cutoff the Commission has used in the single PCA). This is based on the scree plots shown in Appendix 1. Some of the indicators in the single PCA, which don’t load onto the first three components (so don’t have a weight of 0.5 on any of the first three components) could also be removed, and the first two or three components (depending on the scree plot of the rerun results with low loading indicators removed), with this reduced set of data and a varimax rotation (tested to reduce cross loadings), could be used for this index.

Professor Robert Tanton

13 November 2017

Appendix 1: Scree plots for each domain