Evaluation of the Encouraging Better Practice in Aged Care (ebpac) Initiative Final Report

2.3Evaluation of the national roll-out projects

As noted, three of the eleven EBPAC projects were funded under the ACSIHAG program. Two of these involve the national roll-out of resources developed under Round 2 of EBPRAC:

(RC1) National Rollout of the Palliative Approach Toolkit – Queensland Health, and

(RC2) National Rollout of the Evidence Based Champions for Skin Integrity Program – Queensland University of Technology (QUT).

Both projects had similarities in their approach and were distinct from other projects. Each had a focus on the dissemination of evidence based resources (toolkits) to support high quality care in the residential aged care sector (one supporting end-of-life care and the other in wound management) and each had a train the trainer approach supported by the roll-out of national workshops.

As a result of the similarities between these projects a common evaluation methodology was developed. It was hoped that this would help to identify critical success factors that may inform future national rollouts of evidence-based materials and resources.

The delivery of each project involved three main developmental stages as highlighted below:

Stage 1: Workshop planning (evaluation questions include, what evidence base supported the delivery of the workshops and how were they advertised?)

Stage 2: Running workshops (evaluation questions include how the workshops were rated by the participants?)

Stage 3: Toolkit implementation (evaluation questions include to what degree were aspects of the Toolkit implemented in residential aged care?

Capturing data about Stage 1 (workshop planning) was primarily collected through the use of Evaluation Tool 4: Training materials evaluation questionnaire, Evaluation Tool 5: Project workshop log and Evaluation Tool 6: Project workshop notification/recruitment tool.

Capturing data about the workshops themselves and the implementation of the toolkit (Stages 2 and 3) was collected through a series of stakeholder interviews and surveys. Semi-structured interviews were used with key stakeholders to address issues relating to change management, knowledge transfer and context for implementation. These were carried out with RACF managers, quality managers and workshop participants within RACFs where the PA Toolkit and the CSI Toolkit have been implemented.

Evaluation Tool 7 also included the option of using an on-line survey approach as an alternative or in concert with stakeholder interviews. We felt that it was important to use both interviews and surveys as part of our evaluation methodology for the two national roll-out projects. The surveys were targeted at workshop participants and the interviews were targeted at RACF managers and quality managers.

Both the online survey tool and the interview protocol were developed in consultation with both the national roll-out projects.

National roll-out interviews

As mentioned above telephone interviews were used to capture data relating to change management, knowledge transfer and context for implementation. As it was not practical to interview stakeholders from each RACF represented at both the palliative care and CSI workshops, a stratified sampling approach was applied to capture data from a representative sample of RACFs. The sampling approach is summarised in the following section.

The interviews took place between April 2014 and December 2014. In total, 40 interviews were carried out for RC2 and 31 for RC1. A detailed report of these interviews can be found at Appendix 2 and 3.

National roll-out interviews sampling method

To obtain the required number of facilities to conduct telephone interviews for those who had received training from the RC1 and RC2 projects a sampling framework was developed (see section 5.1.1 in Evaluation Progress Report 4). The sampling framework ensured that a representative sample of 40 services was selected by stratifying for jurisdiction, location and size. Table provides an outline of the sampling framework.

Each state and territory was sampled separately, except for ACT and NSW, which were combined. Location was defined using the postcode of the facility and linking this to the Australian Standard Geographical Classification (ASGC) to determine if it was a metropolitan or rural/regional location.¹ For all jurisdictions, except NT and Tasmania, a facility’s location was considered Metro if the ASGC code was 0 (Major Cities of Australia) and Rural/regional otherwise. For NT and Tasmania the ASGC codes 0 (Major Cities of Australia) and 1(Inner Regional Australia) were used to identify Metro facilities. Service size was defined by the number of beds in the facility. A small service is defined as an RACF with less than 50 beds and a large service was an RACF with 51 or more beds.

Table Sampling methodology for telephone interviews

State/Territory:	number required (number sampled)	Location	Number required (number sampled)	Size of Service	Number required (number sampled)
NSW/ACT	8 (24)	Metro	4 (12)	Large: Small	2 (6) : 2 (6)
		Rural/regional	4 (12)	Large: Small	2 (6) : 2 (6)
Vic	8 (24)	Metro	4 (12)	Large: Small	2 (6) : 2 (6)
		Rural/regional	4 (12)	Large: Small	2 (6) : 2 (6)
Qld	8 (24)	Metro	4 (12)	Large: Small	2 (6) : 2 (6)
		Rural/regional	4 (12)	Large: Small	2 (6) : 2 (6)
WA	4 (12)	Metro	2 (6)	Large: Small	1 (3) : 1 (3)
		Rural/regional	2 (6)	Large: Small	1 (3) : 1 (3)
SA	4 (12)	Metro	2 (6)	Large: Small	1 (3) : 1 (3)
		Rural/regional	2 (6)	Large: Small	1 (3) : 1 (3)
NT	4 (12)	Metro	2 (6)	Large: Small	1 (3) : 1 (3)
		Rural/regional	2 (6)	Large: Small	1 (3) : 1 (3)
Tas	4 (12)	Metro	2 (6)	Large: Small	1 (3) : 1 (3)
		Rural/regional	2 (6)	Large: Small	1 (3) : 1 (3)
Total number of facilities: 40 (120)

A random sample of facilities was selected to fill the sampling framework. A list of facilities was obtained from the training records for the Pall Care and CSI projects. The CSI projects provided a list of the name, email contact and phone number of workshop attendees as well as their facility name, state and full address. Additional information of the facility bed numbers and main telephone number was obtained from the Residential Aged Care Australia website².

The Palliative Care Project only provided the facility name, state and postcode of workshop attendees. In view of this an additional database was purchased from A-ZGovBIZ³ that provided the facility name, full address, contact name and telephone number and bed numbers, which was linked to the original list using facility name and postcode.

For each facility a number was generated using the random number generator in Microsoft Excel and within each cell facilities were ordered by their number in ascending order. Starting from the top of each cell, facilities were selected for interview. This was repeated until the required number of interviews was achieved for all cells.

National roll-out surveys

As mentioned, online surveys were used to capture data relating to the delivery of both the RC1 and RC2 workshops and implementation of the respective toolkits.

Both survey tools were targeted at participants of the CSI workshops and were made available online using the SurveyMonkey®⁴ website. In total, 557 surveys were completed (278 of RC1 and 299 for RC2). A detailed report of these surveys can be found at Appendix 3.

Evaluation of remaining EBPAC projects

The remaining nine EBPAC projects worked with a consortium of aged care and health care services to implement a range of initiatives that aimed to promote better practice in the aged care sector.

These projects also had similarities with regards to their implementation methods. All were implementing a new model of care which was supported by the development of resources and targeted training.

Again, given the similarities between these projects a common evaluation methodology was developed to help identify critical success factors that may inform the future national rollout of evidence-based materials and resources in the specific areas being addressed by these projects.

Again, the delivery of these nine self-contained projects involved three main developmental stages as highlighted below:

Stage 1: Resource development (evaluation questions include, what evidence base supported the delivery of the initiative?)

Stage 2: Recruitment (evaluation questions include how were aged care staff supported to participate in the project?)

Stage 3: Implementation (evaluation questions include to what degree were aspects of the project incorporated into aged care practice?)

Capturing data about Stage 1 (resource development) was primarily collected through the use of Evaluation Tool 4: Training materials evaluation questionnaire. Additional data was captured through the site visits and project reporting.

Capturing data about Stage 2 (recruitment) was carried out through the projects progress reports and final report. Questions relating to implementation (Stage 3) were answered through semi-structured interviews with key stakeholders. The focus here was on collecting data relating to change management, knowledge transfer and the context for implementation. The interviews took place between April 2014 and December 2014 with members of the project team, consortium members, RACF managers and quality managers. The number of interviews is highlighted in Table .

Table Number of interviews carried out

#	Project	Number of interviews carried out	Number of interviewees
CL1	Bridging the leadership skill Gap	5	9
CL2	Clinical Mentoring: from evidence-base to outcomes for older people	8	12
CC1	Better practice for older people living with or at risk of chronic wounds in the community (wound management)	7	7
CC2	The Lifestyle Engagement and Activity Program (LEAP) for Life Project (social engagement and physical activity)	6	8
CC3	Home-based Preferred Music Listening Program (emotional well-being)	2 Interviews plus one focus group	15
CC4	Choices in CDC Aged Care	8	10
CC5	Building Better oral health Communities	5	7
CC6	Person-centred Dementia Support in the Community	4	7
RC3	The Sustainable Culture change in Residential Aged Care Project	9	13
Total		55	88