The NDIS Survey of people with disability, and their families and carers was designed to build information on the lives of people with disability, and their families and carers, and how these have been affected by the roll-out of the NDIS. The survey followed a two-wave longitudinal design and focused on information that assisted the evaluation to answer its core KEQs.
1.5.1 Design and development
The objectives of the survey design were as follows. First, to collect information on the changes that happened to NDIS participants in the trial sites (the trial group). Second, to collect information on the changes that happened to NDIS non-participants who were located in areas not included in the NDIS trial (the comparison group). The final objective was to compare the changes experienced by the NDIS participants with the changes experienced by the NDIS non-participants. Provided that the subjects in the trial and comparison groups are very similar in terms of their disability, the differences found between the experiences of the two groups can be attributed to the introduction of the NDIS.
Some of the core aspects of this data collection are explained below:
-
The survey was conducted in all the trial sites and also included a comparison group which comprised people with disability, and their families and carers who would have been NDIS participants if they were living within the trial sites. Following the NDIS trial design, the sample composition of the survey participants also differed by site.
-
The survey followed a mixture of opt-out and opt-in consent rules, designed to comply with the project’s ethics requirements. While these rules worked well in terms of protecting the ethical integrity of the project’s design, they came at the cost of reducing the final number of survey respondents. With the hindsight provided by the first wave collection, it is conceivable that this process could be simplified without loss of ethical integrity for future data collections.
-
The survey used a specially designed registration process to allow respondents to be approached in their personally preferred way. They were offered the possibility of a computer assisted telephone interview, an online response, a hard copy mail response, and a face-to-face interview. It is worth noting that an extremely small number of respondents chose a face-to-face interview. The registration process also allowed respondents to provide further information on their disability and to nominate their preferred family member or carer to be interviewed in the families and carers part of the survey.
-
The timelines of the survey followed the timing of the NDIS trial roll-out. The slower take-up of the NDIS in the trial sites was mirrored in the initial sample sizes of the survey. Due to the small initial numbers of NDIS participants, the survey’s first wave was divided chronologically into three separate samplings: the first one (referred to as wave 1a) happened at the end of 2014, the second (wave 1b) in the middle of 2015, with the third (wave 1c) in the second half of 2016. This third sampling was timed to coincide with the earliest opportunity when the NDIS participation numbers would be sufficient to enable the evaluation to reach its original targeted sample size and get as close as possible to the original composition by state and territory. However, this was done at the expense of not achieving the originally planned second wave sample size and of shifting the completion date of the evaluation to the end of 2017.
-
The survey was divided into two major linked but independently conducted components; the first one addressed people with disability and the second addressed their families and carers. The views of both were obtained through different questionnaires. Should they wish to, people with disability could answer their survey in private, as could their family members and carers.
Accessibility was one of the core objectives of the NDIS Survey. Extensive use was made of accessibility, disability and inclusion experts in order to design questionnaires that were as accessible as possible to complete.
-
In order to facilitate completion, the questionnaires were divided into two parts: Part A and Part B. Part A dealt mainly with factual, less subjective information. Where it was necessary to use a proxy, this proxy could provide this information on behalf of a respondent. Part A included information on disability, activities undertaken, supports before and during the NDIS, and socio-demographic indicators.
-
In contrast, Part B asked mainly subjective questions relating to opinions, assessments and perceptions, such as satisfaction and wellbeing. Here the survey design preferred to encourage the person with disability to answer, where they felt this was feasible and desirable. Part B questions included social, education and economic participation, wellbeing, aspirations, opinions and needs, and views on fairness and equity.
-
Part B was also offered in a pictorial easy English format and both versions could be completed either with or without the support of someone else. Whether such support was used or not was recorded and informed the analysis.
-
Questionnaires that were age-appropriate were designed and distributed accordingly. Children below the age of eight years were not asked to provide any direct information, their surveys were instead typically completed by their parents.
-
The diversity of the respondents’ categorisations was reflected in the fact that a total of 18 separate questionnaires were used in the survey, all targeted at and designed for specific sub-groups. Using the experience of wave 1, the number of questionnaire variants was halved for wave 2.
-
The first wave of the data asked all people with disability whether they would like to give their consent to the evaluation team to access their administrative data. About 86 per cent of them consented. The evaluation tried to obtain this data from the NDIA but were unable to do so, and the work that was planned for 2016 using this data was cancelled.
-
As these design elements have been reported extensively in the Initial Report and also discussed in many instances in the Intermediate Report, we do not repeat these discussions here, but refer the interested reader to these other documents.
1.5.3 Completed fieldwork with sample sizes and timing -
A pilot study was conducted and has been reported in the Initial Report. An outline of the timing of fieldwork for the surveys with people with disability and their families and carers are presented below in Table 1.2.
Table 1.2: NDIS survey fieldwork
People with disability
Wave
|
Fieldwork
|
Wave
|
Fieldwork
|
1a Trial
|
August 2014 to December 2015
|
1a Comparison
|
September 2014 to December 2015
|
1b Trial
|
January 2015 to December 2015
|
Trial only
|
Trial only
|
1c Trial
|
November 2016 to May 2017
|
1c Comparison
|
November 2016 to May 2017
|
2a/b Trial
|
June 2017 to November 2017
|
2a Comparison
|
June 2017 to November 2017
|
Families and Carers
|
Wave
|
Fieldwork
|
Wave
|
Fieldwork
|
1a Trial
|
September 2014 to December 2015
|
1a Comparison
|
October 2014 to December 2015
|
1b Trial
|
January 2015 to December 2015
|
Trial only
|
Trial only
|
1c Trial
|
November 2016 to May 2017
|
1c Comparison
|
November 2016 to May 2017
|
2a/b Trial
|
June 2017 to November 2017
|
2a Comparison
|
June 2017 to November 2017
| -
Fieldwork for wave 1 yielded for the trial sample 3,533 people with disability and 2,713 family members and carers and for the comparison sample, 2,139 people with disability and 1,738 family members and carers.
-
Fieldwork for family members and carers yielded lower numbers than that of people with disability, as the initial administrative data was not complete and extensive work was required to locate them. Also, there are some people with disability who have no identifiable family member or carer.
-
The whole of wave 1 sampling was in essence a census of all NDIS participants who had consented to take part in research at the time of sampling. Wave 1c focussed on the ACT and came later (as soon as the late rollout in the ACT permitted) with 838 people with disability and 861 carers for the trial group. Wave 1c also provided a top up for the comparison group of 466 people with disability and 204 carers.
-
The re-sampling of waves 1a and 1b (that is, the second wave of the longitudinal collection, referred to as wave 2a/b) commenced in 2017. The second wave was planned to have a shorter fieldwork duration, as a registration was not required. In order to allow the NDIS evaluation to be completed by the end of 2017, only the data collected by early October 2017 was used for this report. However, the fieldwork was allowed to continue to November 2017 in order to maximise the sample size for wave 2 (shown in Table 1.3).
-
Table 1.3 provides information about the total response and retention rates for people with disability and their families and carers in the trial and comparison groups achieved as of October 2017.
-
Wave 2 fieldwork was closed mid November 2017 with a total of 2,077 people with disability and 1,619 families and carers for the trial group and 900 people with disability and 605 families and carers for the comparison group. Wave 2c was cancelled in order to meet the completion date of the overall evaluation.
-
The final response rates for waves 2a and b for the trial group were 83 per cent and 87 per cent for people and carers respectively, close to the target set by the evaluation of 85 per cent. Responses for the comparison group (48 per cent and 40 per cent) were both much lower, which had been anticipated and factored into the design of the sampling by sending out a larger number of questionnaires to that group.
Table 1.3: Wave 2a&b responses and retention rates
|
Trial
People with disability
|
Comparison
People with disability
|
Trial
Carers
|
Comparison
Carers
|
Wave 1a and b
|
2,495
|
1,873
|
1,870
|
1,516
|
Wave 2a&b
|
2,077
|
900
|
1,619
|
605
|
Retention rate
|
83%
|
48%
|
87%
|
40%
|
1.5.4 Estimating the impact of the NDIS: An overview of the methodology
The analysis of the data collected was adapted to utilise fully the different sub-groups in the two samples (the trial and the comparison groups) and the added variation and richness found in the data. The methodology we use deviates from the textbook methodology described in the Evaluation Framework (2015) in order to adapt to and utilise in full the actual data obtained, as we describe in this section.
The evaluation sought to estimate the impact of the NDIS on outcomes for people with disability and their families and carers. We try to use non-technical language, where possible, to describe how this approach was used, but this is not always feasible. The survey data design and collection worked sufficiently well to allow the use of robust statistical methodologies to derive formal impact estimates and determine direct causal interpretations. This type of estimates are the gold standard of evaluation methodology as they allow us to state formally whether the NDIS can be shown to be causing change or not.
As the data structure and timing of the collections were dependent on the actual roll-out of the NDIS, the methodology was adapted to use the data at hand. A major advantage of the survey data collected for the evaluation is that we have a very rich set of personal, demographic and socioeconomic information both at an individual and family level, which helps us with the necessary matching operation for making statistically meaningful comparisons. Another critical advantage of the data is that the opportunities and frequency of individuals self-selecting to enter the ‘trial’ group was highly unlikely – so the distinction between a trial group and a comparison person is clear cut and easy to determine in our data. In technical language, we do not need to rely as much on methodologies that control for both unobserved heterogeneity and non-random selection into the trial group, as one would have to if weaker data and experimental design were used.
Technical box: Estimation the NDIS impact using large scale survey data
The way we conducted the impact estimations for the evaluation is explained below using the ‘choice and control’ Chapter 2 as our example. The impact of the NDIS on people with disability and carer perspectives of how much choice and control they can exert over their supports was estimated. The estimations used data collected from the following two survey questions:
- The amount of say the person with disability has over decisions on what supports he/she receives
- The amount of choice the person with disability has about where to obtain these supports
To estimate the impact of the NDIS, we first used matching techniques, to make people with disability in the NDIS trial and comparison groups comparable so that the outcomes of the NDIS participants could be properly evaluated. Following this, the average difference between the NDIS participants’ answers to the questions on choice and control and the answer they would have given had they not joined the NDIS was determined. Further, using the longitudinal dimension of the surveys of people with disability and families and carers, the incremental impact of the NDIS over time on those measures was estimated.
The analyses used both (i) a binary measure of choice and control, i.e. looking at the proportion of people with disability who reported having a lot of say/choice over their supports compared to those who did not, and (ii) a continuous measure of these outcomes, using all the points of the Likert scale answers to these questions.
Note that choice and control information, as with other outcomes such as support quality or supports being reasonable and necessary, was only elicited from adult people with disability (and not children with disability). However, the information was also elicited from the carers of people with disability across all ages. This allowed us to compare the estimated impact of the NDIS on the choice and control of adult participants as reported by people with disability themselves and their carers. Furthermore, the information collected from carers was used to estimate the impact of the NDIS on choice and control outcomes for child participants. It should also be noted that the magnitude of the estimated impact of the NDIS on choice and control was expected to vary depending on the type of measure used (i.e. binary or continuous).
Dostları ilə paylaş: |
