Evaluation of the ndis final Report Kostas Mavromaras, Megan Moskos, Stéphane Mahuteau, Linda Isherwood

Introduction

The NDIS aims to improve the quality of life and wellbeing of people with disability, their families and carers. A core task of the NDIS evaluation is to understand the degree to which the introduction of the NDIS has increased wellbeing. It is recognised by the NDIS evaluation that wellbeing is a very complex concept that can take different meanings by different people, even when they are faced with the same circumstances, and different meanings by the same person when faced with different circumstances or over time. It is also recognised by the NDIS evaluation that the academic and policy literature offers several different methods for measuring wellbeing, each of them with their strengths and weaknesses and their specific focus points and emphasis. Resolving these differences would be a major research task and lies well beyond the scope of this evaluation. The strategy underpinning the NDIS evaluation is to not rely on any single measure of wellbeing, but to collect information on a wide range of measures and at different points in time.

The evaluation collected wellbeing information using the well-known Personal Wellbeing Index (PWI) devised by the International Wellbeing Group (2013). We also collected information that is akin to the wellbeing information collected by the Organisation for Economic Co-operation and Development (OECD 2013) and information similar to that collected by the Household Income and Labour Dynamics in Australia Survey (see for example Qu, L., et al. 2012).

In this section we present quantitative and qualitative evaluation evidence to access the extent to which the NDIS is changing the wellbeing and quality of life for people with disability and their families and carers.

Wellbeing of People with Disability – Evidence from Large Scale Surveys

The quantitative data collected information on three separate measures of wellbeing: (i) wellbeing information based on the well-known Personal Wellbeing Index (PWI) devised by the International Wellbeing Group (2013); (ii) wellbeing information that is akin to what is collected by the Organisation for Economic Co-operation and Development (OECD 2013) often referred to as the psychological wellbeing (often called eudaimonic) index and (iii) wellbeing information similar to that collected in the Household Income and Labour Dynamics in Australia Survey index. We therefore do not rely on any single measure. Rather we examine all of the evidence to form a quantitative judgement as to the overall message that arises regarding the wellbeing and specific aspects of the NDIS.

6.2.1 Psychological (Eudaimonic) wellbeing measure - statements by people with disability about themselves and how they feel (OECD 2013)

• 
  This measure contains six questions on different aspects of people’s psychological functioning. It aims to measure a range of different concepts that are sometimes described as the ‘eudaimonic’ notions of wellbeing¹⁵. Respondents are asked to rate how they have been feeling this week on a scale of 0 to 10 (where 0 means ‘completely disagree’ and 10 means ‘completely agree’).
  
• 
  Appendix Tables A6.1 - A6.6 present the mean value of responses to each of the six questions that make up the psychological wellbeing block by broad disability type. The six measures are presented in this manner, as they are intended to be used independently by the evaluation in order to enable the separate investigation of different aspects of (psychological) wellbeing.
  
• 
  Overall, we see that the mean responses of the NDIS participants to the six measures range from 5.9 to 6.7 in wave 1 and 5.8 to 6.8 in wave 2.
  

Two broad comparisons are of interest:

• 
  First is the comparison between wave 1 and 2 levels of wellbeing. With the exception of the third and last wellbeing items (free to choose the things that I want to do myself and recovery time after adverse events), the mean responses show a modest improvement in the wellbeing of NDIS participants.
  
• 
  Second is the comparison between different subgroups among NDIS participants. Comparing wellbeing by broad disability type, NDIS participants with a mental health or psychosocial disability reported the lowest wellbeing and NDIS participants with physical and sensory disorders reporting the highest wellbeing.
  
• 
  The improvement in wellbeing between wave 1 and 2 holds for all main disability categories. The largest improvement is among NDIS participants with a mental health or psychosocial disability.
  

6.2.2 The Australia Unity Personal Wellbeing Index (PWI) - questions about how happy people with disability feel about things in their lives (International Wellbeing Group 2013)

• 
  Respondents are asked to report how they have been feeling this week on a series of seven questions, each corresponding to a particular life domain: standard of living, health, achieving in life, relationships, safety, community-connectedness, and future security. Answers are on a scale of 0 – 10 (0 is ‘very unhappy’ and 10 is ‘very happy’).
  
• 
  These questions have been designed to lend themselves to aggregated analysis, where a combined score across all component statements can be calculated, using a validated methodology. This allows more robust assessments of wellbeing to be formulated as it provides a cardinal measure of wellbeing, especially at the extremely low responses which have been found to be associated with depression.
  
• 
  Figures 6.1 and 6.2 below display the mean PWI for NDIS participants in wave 1 and 2 by broad disability type along with the corresponding 95 per cent confidence intervals. NDIS participants overall had a mean PWI of 63.5 in wave 1, and 64.3 in wave 2, both measures being well below the average PWI in Australia which fluctuates around 75.
  

Figure 6.1 Person with disability: Australian Unity PWI mean and 95 per cent confidence intervals by disability type wave 1 (Trial, Adults)

Figure 6.2 Person with disability: Australian Unity PWI, mean and 95 per cent confidence intervals by disability type wave 2 (Trial, Adults)

• 
  Australian Unity who are the architects of the PWI, advise that ‘there is a wide so-called normative range, between 60 and 90, as people are inherently different, some more positive, others less so. However if a person’s PWI is below 60 and stays that way over time, it is a signal their homeostatic response is failing and they are at risk of depression’ (Australian Unity, 2015: 5). When we consider the result by broad disability type we see that people with mental health/psychosocial disability have the lowest mean score on the PWI in both waves 1 and 2, well below the ‘minimum threshold’ defined by Australia Unity. NDIS participants with a neurological condition recorded a decrease in their PWI between wave 1 and 2, and in wave 2 also fell below the ‘minimum threshold’ defined by Australia Unity.
  
• 
  It will be essential for an ongoing independent evaluation and monitoring to assess whether the NDIS improves the personal wellbeing among these two vulnerable groups.
  

6.2.3 Sense of social connection - descriptions about how much support they get from other people (HILDA measure)

• 
  The final measure of wellbeing that we collect measures a person’s sense of social connection. Respondents are asked to report on a set of questions about their connection with others in the community, as an indicator of social capital and social inclusion. The questions are concerned with levels of perceived loneliness, sufficiency of friends and visitors, and the perceived level of help available. Respondents are asked to rate how they have been feeling this week with several aspects of their life on a scale of 0 - 10 (0 is ‘completely disagree’ and 10 is ‘completely agree’).
  
• 
  Appendix Tables A6.7 – A6.11 present the mean value of responses to each of the five questions that make up the ‘social connection’ responses for both waves 1 and 2, by broad disability type. The five measures are presented this way as they are intended to be used independently in order to investigate different aspects of social connection.
  
• 
  Responses range from 5.0 to 8.3 in both waves 1 and 2. NDIS participants report lower levels of agreement to the statement ‘I seem to have a lot of friends’. Comparatively, they report higher levels of agreement to the statement ‘I enjoy spending time with the people who are important to me’. There is very little evidence of change in ‘social connection’ between waves 1 and 2.
  
• 
  Comparing responses by broad disability type we see that people with mental health/psychosocial disability report lower levels of social connection on all measures in both waves 1 and 2.
  

6.2.4 The impact of the NDIS on wellbeing (people with disability)

In the following section we report on the estimated impact of the NDIS on the wellbeing of people with disability using each of the three different wellbeing measures presented. Before we look at these findings, it is worth reiterating that, unlike the analyses above using the ‘raw’ survey data, the estimations are able to control for many socioeconomic and demographic differences between sub-groups of participants.

The strongest indication for a positive impact of the NDIS on participant wellbeing was found when using the Australia Unity PWI (International Wellbeing Group 2013), and less so when the OECD and the HILDA measures were used.

Below is a summary of the important results from the multivariate analysis of wellbeing (using the PWI)

• 
  The PWI (Deakin wellbeing index) measure suggests that the NDIS has improved participant wellbeing by 3.04 points (with a 95 per cent confidence interval from 1.30 to 4.78; Appendix Table A6.12). However, the effect is modest with an estimated treatment effect of three points on the PWI scale which represents about a five per cent increase in overall wellbeing.
  
• 
  Our analyses also examined the difference in wellbeing between NDIS participants and the comparison group (Figure 6.3) The horizontal axis measures the wellbeing (which we see lies largely between 40 and 85) and the vertical axis shows how frequently each level of wellbeing is encountered. The figure indicates that NDIS participants (in red) experience greater levels of wellbeing than those not participating in the NDIS (in blue).
  

Figure 6.3 Person with disability: Distribution of estimated potential outcome by treatment status (Trial and Comparison, Aged 8+)

• 
  The analyses show that the impact of the NDIS on wellbeing differed according to NDIS participants’ disability type (see Figure 6.4). The NDIS was found to have the greatest impact on the wellbeing of people with a neurological condition. In contrast the impact of the NDIS on wellbeing is below the average for NDIS participants with mental/psychosocial, developmental/congenital and intellectual disabilities; for example, the estimated impact is as low as 1.5 per cent for people with intellectual disability.
  

Figure 6.4 Person with disability: Estimated impact of the NDIS on wellbeing by main disability type (Trial, Aged 8+)

• 
  The impact of the NDIS on wellbeing was found to vary with the NDIS participants’ age, with the youngest and the oldest benefiting the least from the NDIS (Figure 6.5). The positive impact of the NDIS on wellbeing increased with age up to the person with disability reaching their early to mid-40s. After this point, wellbeing was then found to fall at an increasing rate.
  
• 
  When looking at the wellbeing of NDIS participants aged 8-15 years, only a modest increase in wellbeing of 0.65 points was found, which is not statistically significant (Appendix Table A6.13).
  

Figure 6.5 Person with disability: Distribution of the estimated impact of the NDIS on wellbeing by age (Trial, Adults)

• 
  The duration that a participant has spent within the NDIS is recorded and is plotted below against the estimated NDIS impact (Figure 6.6). The impact of the NDIS on participants’ PWI did not seem to be impacted by the length of time people had participated in the NDIS. It should be noted that while a decrease in wellbeing for people who had been rolled out for the longest was observed, this may be due to smaller number of observations.
  

Figure 6.6 Person with disability: Distribution of the estimated impact of the NDIS on wellbeing by duration (months) of participation in the NDIS (Trial)

• 
  The impact of the NDIS on wellbeing was more positive for NDIS participants living in urban areas as opposed to those living in rural areas (Figure 6.7). There are many more people who are estimated to experience a decrease in their wellbeing (the part of the two lines that lies to the left of the 0 point on the horizontal axis) in the rural areas than in the urban areas.
  

Figure 6.7 Person with disability: difference in impact of the NDIS on wellbeing by urban vs. rural area

• 
  As shown in Figure 6.8, those NDIS participants who experienced unmet demand appear to also experience a lower impact on their wellbeing. It is not intended to assume a direct causal relationship here, but simply to remark that, after controlling for the individual characteristics of NDIS participants, those with lower wellbeing were more likely to also have problems with getting the supports they had funding for. This is a very complex relationship worth further investigation.
  

Figure 6.8 Person with disability: Difference in impact of NDIS on wellbeing by whether unmet demand is experienced or not

Overall, our analyses using the PWI measure of wellbeing, suggest that the NDIS has been making people with disability happier. However, we also have a stark warning of the inequalities that this overall positive impact of the NDIS brings with it in terms of age, type of disability, location and other. The quantitative data includes many such facets that could and should be studied in detail to guide the further development of the NDIS.

• 
  As with the previous estimations using the PWI, a positive impact of the NDIS on participant wellbeing was also found with respect to the OECD measurement tool. The estimated impact of the NDIS on wellbeing found with this measure was greater; with an estimated 6.3 per cent increase in participant wellbeing due to the NDIS.
  
• 
  The analysis of the impact of the NDIS on each component of this wellbeing measure shows slight variability depending on the question analysed, but, altogether, the differences are not statistically significant.
  
• 
  A positive impact of the NDIS on participant wellbeing was also found with respect to the HILDA measure of wellbeing. In this instance the estimated impact of the NDIS was similar to that found for the PWI.