Evaluation of the ndis final Report Kostas Mavromaras, Megan Moskos, Stéphane Mahuteau, Linda Isherwood

The Importance of the NDIS Evaluation Data and some Caveats

The final section of this introduction chapter outlines the strengths of the NDIS evaluation and the important contribution this evidence makes to the future implementation of the NDIS. We also highlight some of the challenges that were experienced (and mostly overcome) during the evaluation, along with the most important caveats that arose over the course of this work.

1.11.1 The importance of the NDIS evaluation for a healthy NDIS

This report concludes a robust, transparent and objective evaluation conducted by independent evaluators, which is essential for the credibility of the trial of the NDIS. By advising on the strengths and weaknesses of the NDIS trial implementation, the evaluation increases the scope for its effective long-term roll-out and management. The NDIS is a major social policy development that puts Australia at the forefront of international achievement and innovation regarding disability. By implementing a rigorous evaluation design, we provide the first independent, robust and comprehensive data to enable such an assessment. Moreover, this evaluation was conducted using a multidisciplinary approach that produced a wealth of well-evidenced and informed conclusions.

The most outstanding element of the evaluation is the uniqueness of the data it created: this is the only national resource which is both informed and independent of the performance of the NDIS from the point of view of people with disability and their families and carers. This evidence is critical for a transition from block funding support provision to individualised care and supports. The second outstanding element of this evaluation is that it provides a template for the further collection of independent data. The evaluation team were contracted to start a new national longitudinal data collection for people with disability and their families and carers, in a form that could be continued into the future. The NDIS survey can also be linked with other data sources, such as the NDIA administrative data, with the consent of participants (and over the course of the evaluation, the vast majority of respondents provided this consent). As a result, the evaluation has produced the first two waves of a very cost-effective longitudinal data set, which could be used as the springboard for a sophisticated and accurate monitoring tool for the performance of the NDIS during its national roll-out and beyond.

The integration of quantitative and qualitative evidence has enabled a better-informed data design and subsequent evidence interpretation. In addition, the use of an inclusive and accessible multi-method and multi-mode approach rather than a conventional survey design enabled the participation in the evaluation of a wide range of people with disability. As a consequence, the data collected in the evaluation is more informative and representative.

Many innovations were introduced during the course of the evaluation. An example of this was the use of three alternative quantitative wellbeing measures within the surveys supported by extensive qualitative evidence, in a manner that enabled us to determine the impact of the NDIS on levels of participant and carer wellbeing more fully and accurately. Another innovation was allowing the NDIS survey process to be completed by proxies for those parts that are easier to know objectively, but discouraging the involvement of proxies where opinions, perceptions and feelings were to be recorded. This differential treatment of proxy involvement was quantitatively implemented through the two-part questionnaires and further supported by introducing the gradual shift of the second part of the questionnaires into easier English and pictorial instruments.

The use and integration of multimethod and multi-mode data collections was necessary given the diversity of the groups of people who took part in the evaluation. The qualitative evidence has allowed more in-depth and detailed understanding of the quantitative research. The quantitative evidence meanwhile has allowed us to judge the overall prevalence of opinions and situations discussed in the interviews, especially for those who were very negative or very positive about the NDIS. The synergies that have been realised through the integration of our findings have been many and useful, and have created a higher level of understanding of the overall evaluation evidence.

1.11.2 Caveats

The complexity of this evaluation has given rise to several caveats, some of them remaining unresolved at the time of reporting.

Response rates of the provider surveys were too low for comfort during the second wave of data collection. Given that the instruments we used have been tried and tested with tens of thousands of responses from the aged care sector’s providers and workforces, in four different occasions in the last decade, we believe that the problem does not lie in the survey design, but in the perceptions of the sector about the usefulness of such an exercise in a time of such intense uncertainty and change. At the time of the first wave, the roll-out of the NDIS was just starting with limited impacts to date on the sector. As a consequence, many providers responded to our request to participate in the survey. However, at the time of data collection for the second wave of the survey (in mid-2017), providers were far less responsive. As we noted, the data instrument has previously been successfully developed and used by NILS in the aged care sector with very high response rates and numbers³. The disability support sector in 2017, however, was overwhelmed by change, over-researched in an inefficiently piecemeal manner (through the use of many short and different surveys and administrative data collections, which do not lead to major bodies of evidence). Unlike the aged care sector where these data collections have been guiding the development of the sector for more than a decade, key sector support was not provided (for example by the NDS encouraging their memberships) a lack that we believe has had a negative impact upon the size of the sample at wave 2. However, without going back to the field again, we will not know for certain why this exercise returned poorer results than we expected. The strength of the qualitative evidence on this front is a strong consolation and has been supporting the evaluation.

The bunched rollout of children by age group in SA, was not ideal for sampling. For the first stages of the evaluation, the information on eight to 15 year old people with disability was largely based on the non-SA part of the sample. Our statistical work, however, suggests that this problem has not impacted on the quality of the relevant results.

The use of administrative information provided directly by the NDIA in the form of administrative individual data had been written in the design of the evaluation and the necessary ethical approvals and individual consent was obtained by people with disability. However, as the evaluation progressed it was not possible to obtain such data and the design had to be altered. In October 2017 the evaluation was provided with some tabulations to construct the necessary population weights in order to ascertain the representativeness of the NDIS survey.

The sample for the comparison group was initially intended to be collated from records held by the Department of Human Services (DHS). However, this data was in many instances either out of date or incomplete, which questioned its present accuracy. This problem was overcome by adding a registration phase to the survey of people with disability and their family and carers. Additional questions were included in the registration phase in order to compensate for the lack of administrative data from the NDIA. This approach led to a much lower response rate but now that the data is at hand, we know that the statistical intervention has worked well.

During the course of the evaluation there were several other evidence collection exercises conducted by other organisations, some large and many small, principally due to the heightened public interest in the NDIS trial. In some instances the timing of such fieldwork activities overlapped and in some others the objectives were perceived by respondents to be very similar, often leading to avoidable interviewee burden and confusion. The evaluation believes that there is a strong public good argument for more coordination of such activities.

The NDIS rollout in the ACT picked up its speed later than we anticipated. This delayed wave 1c commencement as we waited for the numbers to build before commencing fieldwork. As a consequence, the conduct of a subsequent wave 2c was impossible within the time constraints of the evaluation. The possibility of conducting this follow-up wave would be available for at least another 12 months. If this were to be undertaken, it would provide valuable further information relating to both the ACT NDIS trial group and the comparison group sample, many of whom are likely to transition into the NDIS within the next year.

The overall design of evaluation of the NDIS in the NT had to be flexible in order to be suitable to the needs and respectful of the particulars of the populations to be studied and the nature of existing service arrangements, particularly within remote Aboriginal and Torres Strait Islander Communities. It also needed to ensure that people with disability, families, groups and communities were able to have input into discussions, consistent with local practice and the commitment to inclusive and ethical research that is culturally appropriate. The evaluation of the NDIS in the NT also had to take into consideration the objectives and timelines of the project in combination with weather, remoteness and other relevant factors including travel, communication and cultural events. It is important to note the complexity and contingencies that were involved in carrying out the NT study.