DESCRIPTION

The WFI-EZ is a brief, self-report survey that measures adherence to Wraparound principles as well as client satisfaction and youth outcomes. It can be administered online, via pen/paper, or via interview, and takes about 10 minutes to complete.

There are four versions of the WFI-EZ: One for caregivers, care coordinators, youth, and Wraparound team members. All versions include items that assess if the fundamental elements of Wraparound practice are present (4 items), as well as implementation quality (25 items). Implementation quality items are further organized into five key elements that correspond to the National Wraparound implementation Center (NWIC) practice model: Effective teamwork, natural supports, needs-based, strengths-driven, and outcomes-based. Caregiver and youth versions of the survey also assess client satisfaction (4 items), and caregiver and care coordinator versions also assess youth outcomes (9 items). Each version of the WFI-EZ is also available in Spanish. Data result in quantitative summaries of Total Fidelity, Key Element Fidelity Scores, Satisfaction, and Outcomes. 

RELIABILITY AND VALIDITY

Results from a recent study found the WFI-EZ to be a reliable and valid assessment of Wraparound fidelity. The study analyzed data from 10,955 caregivers and 6,088 care coordinators who completed WFI-EZs between 2011 and 2021 and concluded that the WFI-EZ showed excellent internal consistency and two-week test-retest reliability for caregivers and acceptable reliability for care coordinators. The study also showed that WFI-EZ scores were related to known indicators of high-quality Wraparound implementation as expected. For more information on the methods and results of this study, please click here. 

DESCRIPTION

The TOM assesses adherence to standards of high-quality Wraparound as observed by trained external evaluators or local supervisors during team meeting sessions. The TOM consists of 36 indicators, organized into seven subscales, five dedicated to key elements of Wraparound included in the WFI-EZ (per the National Wraparound Implementation Center), one that evaluates meeting attendance, and one that assesses facilitation skills. Working alone or in pairs, the trained raters indicate whether or not each indicator was present during the Wraparound team meeting. The TOM can also be completed via remote observation or after the fact through recordings. Ratings are translated into a score for each subscale, as well as a total fidelity score for the session overall. 

RELIABILITY AND VALIDITY

A series of three validation studies provide data on the reliability and validity of the TOM. The first examined data drawn from 1,078 Wraparound team meetings that were submitted by 59 unique sites. The research team found high levels of internal consistency, meaning that individual items reliably assessed the latent construct of Wraparound fidelity. A second study examined TOM scores for pairs of trained TOM observers who both observed the same 23 team meetings. Results of this analysis indicated substantial agreement between raters. Interesting, agreement was strongest when both raters were external to the provider organizations that comprised the sample. The final study in this series focused on the validity of the TOM. In this analysis, the research team examined associations between TOM scores and those from the Wraparound Fidelity Index across 47 Wraparound sites. Findings were mixed with a large correlation between TOM and WFI-4 measures at the program level, but smaller correlations at the site level. Together, these findings provide evidence that the TOM is a reliable and valid indicator of Wraparound fidelity.

For more information on the methods and results of these studies, please click here. 

DESCRIPTION

The DART assesses the quality and fidelity of Wraparound planning and implementation via review of plans of care, meeting notes, rosters, and outcome measures found in youth/family records. The tool consists of 50 items across six sections: Timely engagement, key elements, safety planning, crisis response, transition planning, and outcomes. The key elements section is further divided into five fidelity subscales: Meeting attendance, driven by strengths and families, natural and community supports, needs-based, and outcomes-based process. The DART is completed by trained reviewers who are not directly involved with services for the families whose documentation is being reviewed. This can be done either locally within each provider site, or via a more centralized process undertaken by individuals outside of the direct provider site. DARTs take about 45-60 minutes to complete, and data results in quantitative summaries of Key Element scores (aligned with the WFI-EZ and TOM 2.0), and subscale-level scores.  

RELIABILITY AND VALIDITY

A recent study was conducted on the psychometric properties of the DART based on data drawn from 731 reviews in six states. State-level data suggest relatively consistent patterns of scores across states with most experiencing success in providing needs-based support but struggling to engage natural and community supports. General state-level data patterns were similar across the DART and WFI-EZ, which provides evidence of concurrent validity of the DART. Inter-rater reliability was also found to be high across DART subscales, though there was evidence ratings diverged on several items in 3 of 9 subscales examined. Such findings underscore the importance of proper training of DART raters and double-scoring DARTs until raters demonstrate consistency in scores.

Please click here for a more detailed summary of this study.

DESCRIPTION

The IOTTA is a self-report survey designed to assess perceptions of the quality and impact of training and/or technical assistance (TA) efforts. The measure includes two forms: A post-event form completed by participants immediately following a training or completion of TA, and a follow-up form typically completed 6 to 8 weeks later. The immediate post-event survey is designed to assess participant satisfaction with various aspects of a training or TA event, the importance of the training or TA goals, the anticipated level and type of impact the training or TA will have, their current mastery of the training concepts, and their anticipated mastery 6 to 8 weeks in the future. The long-term follow-up survey asks participants to rate their current mastery of the training/TA content, the amount and type of practice-level change that resulted from the training/TA, drivers and barriers to integrating the information and skills into their practice, and the overall "worthwhile-ness" of the training. The IOTTA can be administered as a web-based or paper and pencil survey, and both versions of the form take about five minutes to complete.  

RELIABILITY AND VALIDITY

A recent study looked at the reliability and validity of the IOTTA by examining IOTTA data from two separate sources: A sample of 9,330 behavioral health providers who participated in a Wraparound training hosted by the National Wraparound Implementation Center (NWIC), and a sample of 900 professionals who received training or TA by the Northwest Mental Health Technology Transfer Center (MHTTC). Exploratory factor analyses (EFAs) of both versions of the IOTTA suggested items grouped together as expected, and follow-up analyses suggested strong internal consistency of each factor, meaning the items within each factor were related to one another. Evidence for concurrent validity was also found, showing IOTTA scores from two months post-training correlated with similar items from another scale. For construct validity, post-event IOTTA ratings were found to predict two-month follow-up outcomes as expected. Furthermore, IOTTA ratings of two-month impact were higher for trainees who received coaching and consultation after the initial training. These results indicate the IOTTA is measuring the constructs it was designed to assess, including likelihood of impact of training/TA immediately after, and actual impact two months after.

For a more detailed summary of methods and results from this study of the IOTTA, please click here. 

DESCRIPTION

WrapSTAR provides an external, objective method for how Wraparound is working within a community or agency using a wide variety of information from a range of WFAS tools, allowing for a comprehensive approach to accountability and quality improvement across all implementation drivers for Wraparound. Thus, the WrapSTAR process goes above merely measuring a site’s adherence to the Wraparound model (aka, “fidelity”), and also assesses key implementation drivers rooted in years of implementation science research, as well as community and system supports found to be essential for sustaining a Wraparound initiative. This holistic approach allows for a robust understanding of not only how, but why Wraparound is functioning within an organization or community, leading to greater insight about what can be done to improve outcomes for youth, families and staff. The WrapSTAR process typically takes place over a few months. 

All organizations that complete the WrapSTAR process are recognized by the National Wraparound Implementation Center (NWIC), which may help a provider organization demonstrate the strength of its services and commitment to continual quality improvement to stakeholders, such as families, funders, and community leaders.  

To inquire about WERT facilitating a comprehensive evaluation using the WrapSTAR framework, contact us at wrapeval@uw.edu.

DESCRIPTION

The CSWI is a 40-item research and quality improvement tool developed by Janet Walker, PhD, and is based on the Framework of Necessary Conditions described by Walker, Koroloff and Schutte (2003). The tool is meant to measure how well a local system supports the Wraparound process, which makes it somewhat unique from other WFAS instruments, which measure fidelity to the practice model for an individual child/family. The CSWI is completed online by key informants in the community and results in a quantified assessment of community supports for Wraparound across multiple domains, so that researchers can determine the impact of these conditions on fidelity and outcomes. Results are structured so local groups can assess system supports for Wraparound, respond to areas of strength and weakness, and monitor improvements over time. WERT manages the administration and data collection, and Janet Walker produces a comprehensive report which is presented to local administrators and other relevant stakeholders.

To see a detailed description of CSWI methods, psychometrics, and findings, please click here.

Successful implementation of Wraparound requires a focus on effective organizational practices and supportive systems-level structures. WERT works with our partners at NWICto aid states, regions, jurisdictions, and/or communities to use research-based measures of organizational and system context to identify strengths, needs, and opportunities to improve programs and systems toward Wraparound implementation and long-term sustainability. 

 All of our measures of Wraparound system and program context are based on the Wraparound implementation and practice standards, developed by WERT and the National Wraparound Initiative (NWI). This standards framework can be used as the basis for self-assessment by a Wraparound provider organization, community, or state to identify strengths, needs, and opportunities to improve programs and systems. 

 WERT and NWIC have also developed two instruments based on the NWI standards: The Wraparound Implementation Standards – System (WISS) and the Wraparound Implementation Standards – Program (WISP) measures. Using these tools facilitates comprehensive organizational and system assessment and provision of technical assistance from the field’s leading experts on system building, leadership, and innovative, integrated financing mechanisms. The technical assistance provided by NWIC includes options for virtual and onsite consultation from the nation's leading experts on Wraparound organizational development, financing, and systems design. 

 For more information on using the Standards, WISS, and/or WISP to assess your initiative’s system and program conditions, contact UW WERT at wrapeval@uw.edu. Or reach out to NWIC via https://www.nwic.org/contact-us.

System and organizational level TA from NWIC includes:

• Building more effective system structures, including but not limited to: governance, management, quality assurance, and practice level, including care management entities and health homes; 

• Sustaining funding of high quality, high fidelity Wraparound and other needed services, including the leveraging of Medicaid and other federal, state, and local financing mechanisms; 

• Developing and incubating “centers of excellence” for ongoing implementation, quality assurance, policy, financing, and evaluation support; 

• Building, enhancing, and/or implementing workforce development initiatives outside of the Wraparound practice model, including shifting providers from residential services to quality home- and-community-based services; and 

• Implementation of Wraparound in the context of other systems of care efforts, including developing and implementing other evidence-based and promising practices, extending care coordination to child welfare and education, and many other innovations.