Wraparound Blog

New Research Study Sheds Light on Reliability, Validity, Potential Improvements, and Underlying Theory of the WFI-EZ

July 25, 2023 | NWI

By Jonathan Olson and Eric Bruns

University of Washington Wraparound Evaluation and Research Team and National Wraparound Implementation Center

Research consistently shows Wraparound care coordination promotes better residential, school, and mental health outcomes for youth with complex needs than services as usual (Olson et al., 2021). However, as we frequently discuss on blog posts, reports, and other transmissions from the National Wraparound Initiative (NWI), high-quality, model-adherent Wraparound service delivery is critical to assuring Wraparound provides this boost to child and family-serving systems “as usual.”

To help assure Wraparound quality and fidelity, the University of Washington Wraparound Evaluation and Research Team (UW WERT) has provided access to the Wraparound Fidelity Index, Short Version (WFI-EZ) for over 10 years. The WFI-EZ and other measures of the Wraparound Fidelity Assessment System (WFAS) aim to help Wraparound Provider Organizations (WPOs) assess the degree to which the core components of Wraparound are actually being used in practice. Over the past decade, more than 15,000 caregivers, care coordinators, youth, and Wraparound team members have completed a WFI-EZ survey.

Despite its widespread use, however, a published study of the psychometrics, reliability, and validity of the WFI-EZ has not previously been available – until now.

The Journal of Child and Family Studies (Bruns, Olson, Parigoris, Parker, & Walker, 2023) has published a new study examining the psychometrics of the WFI-EZ. The study had three aims:

  1. First, to determine if the underlying factor structure of the WFI-EZ actually aligns with the underlying structure that UW WERT has used to summarize WFI-EZ results; that is, the four key elements of the practice model promoted by the National Wraparound implementation Center (NWIC), plus a fifth focused on being data-informed.
  2. Second, to assess the reliability of the measure via internal consistency, agreement across caregiver and care coordinator versions of the measure, and a small test-retest reliability study.
  3. Finally, we assessed validity by comparing WFI-EZ scores for groups that should differ, such as WPOs with higher versus lower Wraparound climate or systems with and without training and coaching.

In the following paragraphs, we provide a brief overview of findings and their implications. For additional details, readers can also read a copy of the full paper as published in JCFS.

Methods

The WFI-EZ is a brief self-report survey, with versions for parents/caregivers, care coordinators, youth, and team members. All versions include a Section A with four items that assess fundamental elements of Wraparound practice: Presence of a team, a plan of care, regular meetings, and decisions driven by families. In addition, all respondents complete 25 items (Section B) that assess Wraparound fidelity. These items are organized into 5 subscales that correspond to the NWIC practice model: effective teamwork, natural supports, needs-based, strengths-driven, and data and outcomes-based.

Caregivers and youth complete four additional questions (Section C) that assess satisfaction with the Wraparound process. Finally, the caregiver and care coordinator forms include 9 items (Section D) that assess outcomes such as caregiver strain and resilience, and youth residential, school, and justice outcomes.

Study Methods

UW WERT analyzed data from 10,955 caregivers and 6,088 care coordinators within 243 WPOs in 25 states that were entered into WrapStat (the web-based data entry and management system for WFAS measures) and its predecessor, WrapTrack. Data collected between 2011-2021 were analyzed for this study.

In addition, 26 caregivers and care coordinators completed the WFI-EZ twice over two weeks to assess test-retest reliability. Finally, we used program-level data from 15 WPOs in one state to assess “known groups validity.”

Analyses included exploratory factor analysis (EFA) and paired-sample and independent-sample t-tests, as well as calculation of intraclass correlations (ICCs) and Pearson’s (r) correlations.

Results

WFI-EZ reliability and validity

As summarized below, results supported the reliability and validity of the WFI-EZ. However, analyses revealed some opportunities for improvement.

  • The total WFI-EZ index showed excellent internal consistency (alpha = .92), meaning that its items are well-related to an underlying construct (i.e., Wraparound fidelity). The original five subscales showed borderline to high (.54 to .84) internal consistency, indicating these subscales also generally can be used reliably, with some potential room for improvement.
  • Test-retest data showed excellent two-week agreement for caregivers (.92). However, reliability was only acceptable for care coordinators (.74).
  • Results of validity tests indicated that average WFI-EZ scores were significantly higher for groups that would be expected to be higher. For example, total WFI-EZ fidelity was significantly higher for youth who received services that met the basic definition of Wraparound (as assessed by Section A). Caregivers with higher satisfaction with their care and their progress (Section C) also showed significantly higher average fidelity ratings. Finally, caregivers who received services from WPOs with higher organizational readiness to implement Wraparound reported significantly higher average fidelity scores on the WFI-EZ.

Factor Structure of the WFI-EZ

Results from our EFA revealed four factors rather than five. Moreover, these factors did not completely align with the original subscales of the WFI-EZ, defined as per the NWIC model.

  • Although factors tapping into planning, teamwork, and natural supports were found, a fourth factor emerged that comprised 10 items from across the original five factors that seemed to relate to intermediate outcomes of Wraparound, including enhanced family assets and effective services and supports.
  • While some of these items were from the original Outcomes-Based subscale (e.g., “Wraparound has given me confidence as a caregiver”), others came from the Natural Supports (e.g., “Wraparound has helped us get more support from friends and family”) and Needs-Based (e.g., “Wraparound connected my family to community supports I found valuable”) subscales.
  • The internal consistency (reliability) of the new, empirically-derived subscales showed better internal consistency, test-retest reliability, and cross-informant agreement than the original subscales.
  • Finally, some WFI-EZ items did not load on any of the empirically-derived factors, and had low item-total correlations. These items could thus be characterized as not adding value to the WFI-EZ overall.

Implications for Wraparound Evaluation and Practice

Together, the findings of this study provide evidence that the WFI-EZ is a reliable and valid measure of Wraparound implementation and fidelity. As described above, the total index and the original theoretically-derived subscales show acceptable levels of internal consistency and test-retest reliability. The measure also results in higher average scores for groups as would be expected – e.g., WPOs with stronger organizational readiness and caregivers who reported experiencing better satisfaction with their family’s progress.

However, results suggest that there is also room to refine the WFI-EZ to make a good measure even better. Other study results help provide guidance for how best to use the WFI-EZ – for research, evaluation, and quality improvement.

Alternative ways of reporting WFI-EZ results

The current five domains of the WFI-EZ align with the practice model on which our NWIC colleagues train and coach. We were pleased to find that these subscales were adequately reliable and valid, and to be largely reflected in the results of the exploratory factor analysis.

  • At the same time, however, the subscales that were found via the EFA provide an intriguing alternative structure for WFI-EZ data that aligns with longstanding Wraparound theories of change. Walker and Matarese (2011) theorized that effective teamwork is critical to managing a planning process that results in intermediate outcomes such as enhanced natural supports, increased family assets and more effective services and supports. This aligns well with an alternative way of organizing WFI-EZ data that emerged from the EFA, which found factors related to planning, teamwork, natural supports, and intermediate outcomes. Taking this idea further, items from the proposed “Intermediate outcomes” domain can be further broken down into enhanced family assets and effective services and supports.
  • All these factors are assessed by the WFI-EZ, and all are proposed to lead to long-term outcomes for Wraparound-enrolled youth and families that are also assessed by the satisfaction and outcomes sections (C and D) of the WFI-EZ.
  • Thus, the current study reveals an intriguing alternative way of summarizing results. The figure below shows a way in which these study-derived factors may be organized. This framework may be explored in future research by UW WERT or others, and/or used by WFI-EZ collaborators to summarize their process and outcomes with data.

Figure 1. Theory for implementation and effectiveness of Wraparound as assessed by items and domains of the Wraparound Fidelity Index, Short Form (WFI-EZ)

Theory for implementation and effectiveness of Wraparound as assessed by items and domains of the Wraparound Fidelity Index, Short Form (WFI-EZ)

Click image to enlarge

Guiding evaluation planning

Among many other actionable findings, the study revealed that WFI-EZ data from care coordinators may be less reliable than for caregivers. This discrepancy is perhaps not surprising, given that parents and caregivers are immersed in only one Wraparound episode – their own – while care coordinators support many families over months and years, potentially reducing their recall – and thus, the reliability of their assessments. UW WERT typically recommends that, when faced with limited resources, Wraparound initiatives and WPOs prioritize data collection from caregivers, to assure their voice. These results provide another, data-informed, rationale for prioritizing caregiver report.

Revising the WFI-EZ to reduce burden and increase reliability

As described above, four WFI-EZ items did not load on the new factor structure. Removal of these particular items also resulted in better internal consistency of the overall measure. Other items were found to contribute little variance above and beyond other items, and thus may be considered duplicative. Thus, UW WERT is considering a possible revision to the WFI-EZ to increase reliability, which would have an added benefit of shortening the tool and reducing respondent burden.

Conclusion

Until such revisions are made, however, users of the WFI-EZ can be confident that the current version of the tool is a psychometrically sound measure of Wraparound fidelity and implementation, and that subscales presented in reports generated from WrapStat summarize the data in ways that are reliable and valid. UW WERT looks forward to continuing to analyze data from the many hundreds of WPOs that license the WFI-EZ and other measures of the WFAS.

We will continue to report back results, refine our measures, and use the data to better understand implementation and outcomes of Wraparound. It is only by helping you achieve your mission on behalf of youth and families that we can achieve ours.