Wraparound Blog

Update on SMART-Wrap: A Text-Based Fidelity and Outcomes Monitoring for Wraparound

December 10, 2022 | Eric Bruns

Earlier this year, we reported that the University of Washington Wraparound Evaluation and Research Team (UW WERT) and 3C Institute received funding from the National Institute of Mental Health (NIMH) to develop and test a “mobile Routine Outcomes Monitoring” (mROM) system for Wraparound that collects data from parents, caregivers, and youth using text-based surveys.

SMART-Wrap (Short Message Assisted Responsive Treatment for Wraparound), aims to provide a low-burden, reliable approach for obtaining feedback from caregivers and youth enrolled in Wraparound. Data will be available to care coordinators, supervisors, and managers for use in monitoring fidelity, satisfaction, and outcomes for use in planning for individual families, supervision, and quality assurance for the entire Wraparound program. An initial phase of effort engaged over 70 experts from the National Wraparound Initiative (NWI) community on how the SMART-Wrap System would work, and to identify and develop wording for survey items.

Initial Results and Responses

Working with two Wraparound provider organizations – Region 6 Services of Nebraska and Lutheran Community Services Northwest of Washington State – we conducted an initial round of usability testing with 12 parents and caregivers and five youth served by six care coordinators. Results found that:

• Text messages with linkage to SMART-Wrap surveys were opened 86% of the time and surveys took an average of approx. 30 seconds for youth and caregivers to complete.
• Responses to user surveys were very positive:
  • Over 90% of caregiver and youth testers “agreed” or “strongly agreed” that the SMART-Wrap system was easy to use.
  • 100% of testers “strongly agreed” that they were “satisfied with the system” and 100% reported it did not take too much of their time.
• SMART-Wrap obtained a mean score of 82% on the System Usability Survey (SUS), a widely used measure of technology acceptability and usability. This far exceeds the 68% benchmark for usability established via research on the SUS.
• Focus groups with parents and youth provided critical feedback for improvement on:
  • SMART-Wrap items – parents said satisfaction and fidelity items should specify the time period for feedback, because care coordinator turnover may mean they had varying experiences over time.
  • Item wording – youth said to simplify items.
  • Response scales – youth suggested using emojis
  • Timing of sending surveys – Parents and youth both suggested the system should send texts at more convenient times
  • Setting up for success – Parents and youth both recommended care coordinators should help families set the system up in their phone as part of the engagement phase, to explain its purpose and make sure they recognize the sender.

Specific Quotes from Parents

• “I liked doing this. It was quick and easy.”
• “Sometimes I almost disregarded the texts because I wasn’t paying attention and almost mistook them for spam.”
• “It was simple and straight forward I didn’t have any issues at all.”
• “Just knowing that my team is there and if I need help or have question to issue, it helps having other people point of view.”
• “I think it could be a valuable resource if they decide to use it.”
• “To the point and fast to communicate”

Data collected showed reasonably good variability across the 11 SMART-Wrap engagement, satisfaction, fidelity, and outcomes items. See the table below for how responses looked by item for one user testing site.

Next Steps for the SMART-Wrap System and Opportunities to Get Involved

Development of Dashboard and Reporting System

A dashboard and reporting system are now being developed for Wraparound provider organizations that will present data in “real time” to care coordinators, supervisors, and managers, to help them respond to individual families who may be struggling and provide effective support to all families based on data.

Pilot Testing

UW WERT and NWI are now seeking additional SMART-Wrap pilot test sites for a second round of testing from February – June 2023. If you are interested in learning more about the SMART-Wrap system and/or being a pilot test site, we invite you to join us for a webinar on Tuesday January 17 at 2pm ET / 11am PT. 

On the webinar, we will provide an overview of the system and how it works, present initial results, and describe how results are informing improvements. We will then present details on how your Wraparound program or initiative can participate in a new round of testing, expectations of sites, and incentives for participants, and answer questions. In the meantime, you can also go to the website for UW WERT to find a summary for interested SMART-Wrap pilot sites.

Have Questions or Want to Be a SMART-Wrap Pilot Site?

To express interest in being a SMART-Wrap pilot site or ask questions, please feel free to contact us at wrapeval@uw.edu and/or ebruns@uw.edu.