Wraparound Blog

Wraparound Care Coordination Added to Inventory of Programs in Prevention Services Clearinghouse

January 28, 2022 | Eric Bruns

On January 25, “Intensive Care Coordination Using a High Fidelity Wraparound Process” (i.e., Wraparound or High Fidelity Wraparound) was added to the inventory of research-supported programs listed in the Prevention Services Clearinghouse.

The Clearinghouse was established by the Administration for Children and Families (ACF) to systematically review research and evaluation on programs and services intended to provide enhanced support to children and families involved in child welfare systems and prevent foster care placements. The goal of the Clearinghouse is to provide an objective source of information on evidence-based programs and services that may be eligible for funding under Title IV-E of the Social Security Act as amended by the Family First Prevention Services Act (FFPSA).

Following an approach required by statute, the Clearinghouse rates programs and services determined to have some level of research support as “Well-supported,” “Supported,” or “Promising.” Programs also can be evaluated as “Not Meeting Criteria” for research support.

In characterizing “the extent of evidence for the program or service,” Wraparound received a rating of “Promising.”

How Did the Clearinghouse Determine its Rating for Wraparound?

Promising programs are those whose research base is found to have “at least one contrast in a study that achieves a rating of moderate or high on study design and execution and demonstrates a favorable effect on a target outcome.” By comparison, to achieve a rating of “Well-supported,” reviewers need to find two “contrasts” (i.e., significant differences between groups) in one or more rigorous studies (i.e., with “moderate or high study design and execution”) and at least one study with sustained favorable effect of at least 12 months beyond the end of treatment. To achieve a rating of “Supported,” at least one contrast (significant difference) is needed in a rigorous study and at least one with sustained favorable effect of at least 6 months beyond the end of treatment.

Wraparound has been determined by other inventories and scholars to be evidence-based, based on the strength of at least 17 controlled studies (as found in our recent meta-analysis). So why did the Clearinghouse make its determination of “Promising”? Several specific statutory requirements for the Clearinghouse review process likely played a part:

1. First, given its focus on foster care and child welfare, the Clearinghouse only reviews studies where at least some proportion of participating youth were involved in the child welfare system. Because Wraparound is a model with broad applicability to youth with complex needs, many studies of Wraparound effectiveness were conducted in other child-serving sectors, such as mental health or juvenile justice.

2. Second, to assure consistency in its review process across many dozens of programs, the Clearinghouse defined certain criteria for level of rigor of research studies. Studies that do not meet these criteria cannot be the basis on which programs are determined “Well supported” or “Supported.”

   One relevant criterion for “Supported” is that study designs cannot be found to have a “Substantially Different Characteristics Confound.” That is to say, characteristics of the experimental and comparison groups must not substantially differ from one another. Because Wraparound studies are almost always conducted in “real world” service systems, this is a difficult condition to achieve, even in randomized trials. For example, a quasi-experimental study by Pullmann et al., (2006) reviewed by the Clearinghouse found that Wraparound youth experienced fewer offenses post-enrollment than the comparison group. However, the Wraparound and comparison groups differed by average age of youth and number of offenses at baseline. Specifically, the Wraparound group was older and had more previous offenses. Thus, even though youth in the Wraparound groups were probably more likely to have future justice offenses, strengthening the finding of Wraparound’s effectiveness, the study design confound ran afoul of Clearinghouse criteria. Several other Wraparound studies reviewed by the Clearinghouse featured similar design features. Thus, despite their findings being more likely to generalize to the “real world” (and often providing greater evidence of Wraparound’s effectiveness), the nature of Wraparound studies conducted in authentic service systems may reduce their rating of rigor as per the criteria of the Clearinghouse.

3. Finally, and perhaps most important, Clearinghouse criteria require durability of positive effects – 6 months post-discharge for “Supported” and 12 months post-discharge for “Well Supported.” Because Wraparound is focused on youth with very serious and complex needs and their families, enrollment periods are typically longer than interventions that focus on a more limited set of skills or goals. Furthermore, they vary in length, being individualized to the needs and progress of the youth and family. Thus, the durability of effect standard is more easily achieved for research on interventions that are brief and time-limited. Only one Wraparound study reviewed met both criteria for adequate rigor and assessed outcomes post-discharge (Coldiron et al., 2019). Unfortunately, this study was found to only assess outcomes 3 months (not 6 or 12 as required) post-discharge. Thus, in and of itself, this Clearinghouse standard probably meant that “Promising” was as good a rating as could be found for Wraparound, based on studies reviewed.

What Next?

Overall, the listing of Wraparound on the Prevention Services Clearinghouse is positive news. First, despite over 1,000 nominations of programs, Wraparound is one of only 50 or so to be listed in the Clearinghouse. Second, despite the above challenges of fitting its research base to the Clearinghouse criteria and process, Wraparound was found to be supported by evidence. Finally, states that seek to use Wraparound partly or wholly for its child welfare population now know of its status and what the rules are for including Wraparound in their Title IV-E plans.

Moving forward, there are two main implications of the Clearinghouse finding of “Promising” for Wraparound care coordination.

First, states that seek to fund Wraparound via Title IV-E will need to develop and describe a rigorous evaluation plan of Wraparound fidelity and outcomes. Because “Wraparound is worth doing well” (and often is not done well), we would argue such evaluation should be done with every large-scale Wraparound initiative. Wraparound also is implemented very differently in every state service system. As such, this requirement is far from unreasonable. Furthermore, the field now has a greater array of strategies for continuous quality improvement and evaluation than ever. These include fidelity measures (such as the Wraparound Fidelity Assessment System) and web-based software (such as the WrapStat system) that support rigorous evaluation of fidelity and outcomes. Also, states that organize their systems of care and Wraparound initiatives via care management entities (CME) will have additional, research-based methods to collect and use data that can meet FFPSA / Title IV-E criteria.

Second, now that we know what is lacking in the Wraparound evidence base (at least per criteria of the Clearinghouse), we also now know what we need to do to improve Wraparound’s Clearinghouse research support rating. States that choose to invest in Wraparound may consider seeking opportunities to partner with researchers to conduct evaluations with adequate rigor to meet Clearinghouse standards, such as by comparing Wraparound outcomes to comparison groups of similar youth, and collecting youth outcomes data at least 6 months post-transition. Although such levels of evaluation rigor won’t be necessary or possible in all state Wraparound rollouts, we should work together to find such opportunities where possible.

In the coming weeks, the NWI will issue further blogs and updated guidance documents on how Wraparound-implementing states can meet requirements for FFPSA evaluation plans, and tools and systems that can support evaluation. While the finding of Wraparound as “Promising” on the Clearinghouse may feel like mixed news, this news also provides clarity on next steps and an array of new, exciting challenges for us to work together to achieve.