Update on SMART-Wrap: A Text-Based Fidelity and Outcomes Monitoring for Wraparound
December 10, 2022 | Eric Bruns
Earlier this year, we reported that the University of Washington Wraparound Evaluation and Research Team (UW WERT) and 3C Institute received funding from the National Institute of Mental Health (NIMH) to develop and test a “mobile Routine Outcomes Monitoring” (mROM) system for Wraparound that collects data from parents, caregivers, and youth using text-based surveys.
SMART-Wrap (Short Message Assisted Responsive Treatment for Wraparound), aims to provide a low-burden, reliable approach for obtaining feedback from caregivers and youth enrolled in Wraparound. Data will be available to care coordinators, supervisors, and managers for use in monitoring fidelity, satisfaction, and outcomes for use in planning for individual families, supervision, and quality assurance for the entire Wraparound program. An initial phase of effort engaged over 70 experts from the National Wraparound Initiative (NWI) community on how the SMART-Wrap System would work, and to identify and develop wording for survey items.
Initial Results and Responses
Working with two Wraparound provider organizations – Region 6 Services of Nebraska and Lutheran Community Services Northwest of Washington State – we conducted an initial round of usability testing with 12 parents and caregivers and five youth served by six care coordinators. Results found that:
- Text messages with linkage to SMART-Wrap surveys were opened 86% of the time and surveys took an average of approx. 30 seconds for youth and caregivers to complete.
- Responses to user surveys were very positive:
- Over 90% of caregiver and youth testers “agreed” or “strongly agreed” that the SMART-Wrap system was easy to use.
- 100% of testers “strongly agreed” that they were “satisfied with the system” and 100% reported it did not take too much of their time.
- SMART-Wrap obtained a mean score of 82% on the System Usability Survey (SUS), a widely used measure of technology acceptability and usability. This far exceeds the 68% benchmark for usability established via research on the SUS.
- Focus groups with parents and youth provided critical feedback for improvement on:
- SMART-Wrap items – parents said satisfaction and fidelity items should specify the time period for feedback, because care coordinator turnover may mean they had varying experiences over time.
- Item wording – youth said to simplify items.
- Response scales – youth suggested using emojis
- Timing of sending surveys – Parents and youth both suggested the system should send texts at more convenient times
- Setting up for success – Parents and youth both recommended care coordinators should help families set the system up in their phone as part of the engagement phase, to explain its purpose and make sure they recognize the sender.
Specific Quotes from Parents
- “I liked doing this. It was quick and easy.”
- “Sometimes I almost disregarded the texts because I wasn’t paying attention and almost mistook them for spam.”
- “It was simple and straight forward I didn’t have any issues at all.”
- “Just knowing that my team is there and if I need help or have question to issue, it helps having other people point of view.”
- “I think it could be a valuable resource if they decide to use it.”
- “To the point and fast to communicate”
Data collected showed reasonably good variability across the 11 SMART-Wrap engagement, satisfaction, fidelity, and outcomes items. See the table below for how responses looked by item for one user testing site.
Next Steps for the SMART-Wrap System and Opportunities to Get Involved
Development of Dashboard and Reporting System
A dashboard and reporting system are now being developed for Wraparound provider organizations that will present data in “real time” to care coordinators, supervisors, and managers, to help them respond to individual families who may be struggling and provide effective support to all families based on data.
UW WERT and NWI are now seeking additional SMART-Wrap pilot test sites for a second round of testing from February – June 2023. If you are interested in learning more about the SMART-Wrap system and/or being a pilot test site, we invite you to join us for a webinar on Tuesday January 17 at 2pm ET / 11am PT.
On the webinar, we will provide an overview of the system and how it works, present initial results, and describe how results are informing improvements. We will then present details on how your Wraparound program or initiative can participate in a new round of testing, expectations of sites, and incentives for participants, and answer questions. In the meantime, you can also go to the website for UW WERT to find a summary for interested SMART-Wrap pilot sites.
Have Questions or Want to Be a SMART-Wrap Pilot Site?
Table 1. SMART-Wrap Survey Items and Frequencies from Pilot Test*
Not at All
|1. My care coordinator, Wraparound team and I are all on the same page about what we are working on together.||2||7|
|2. I feel confident my care coordinator will help my family meet our needs||1||8|
|3. My youth and family have made progress since starting Wraparound||1||6||2|
|4. I am satisfied with how the Wraparound process is working for my family and me so far||1||9|
|5. My youth and family have a team of people that work with us to meet our needs||2||8|
|6. My youth and family helped create a written plan of care that includes our needs and strategies to meet those needs||3||7|
|7. Our Wraparound plan of care is based on the strengths, needs, and preferences of my youth and our family||2||8|
|8. Since starting Wraparound, my youth has made progress that has positively affected their quality of life||8||3|
|9. Participating in Wraparound has helped our family’s quality of life||4||7|
|10. I feel confident I can support and care for my child, now and in the future||6||3|
|11. Participating in Wraparound has increased my confidence that I can manage future challenges that may come up for my family||3||6|
*Note that youth/young adult survey items are phrased somewhat differently to align with their developmental level and perspectives.