Outcomes

Roca is in an ideal position to achieve its growth goals, through the formalization and documentation of many of its staffing processes as well as the organizational capacity associated with the development of intensive case management and transitional employment. Much of this work has already begun, and is based on the program’s evaluation and data collection software, Efforts to Outcomes (ETO), which assesses all youth work and staff development.

While managing Roca’s strategic and organizational growth, the most important effort over the next few years will be to demonstrate an evidence-based model for working with very high-risk youth and young adults that is created by the community as opposed to a mandating authority, i.e. criminal justice or child welfare. Having worked hard over the past twenty years to identify what works, Roca is ready to work with the Crime and Justice Institute (CJI) to demonstrate an evidence-based model that addresses: (a) re-engaging young people, and (b), examining what it takes for young people to move towards self-sufficiency; and, (c) to live out of harm’s way.

Roca’s goal is to establish the efficacy of the Transformational Relationship (TR) model with a degree of scientific rigor that allows Roca to inform its own work and allows other agencies to consider the TR model for replication. However, the rigor of the evaluation must be balanced with the reality of the population with which Roca works. As such, the evaluation methodology ultimately selected will be chosen with consideration for the target population and Roca’s desire not to exclude young people from services as part of the evaluation process.

The work to demonstrate an evidence-based model will include an implementation (process), outcome and an impact evaluation. The process component of the evaluation will measure the implementation of Roca’s TR model, and the extent to which actual agency practice aligns with the model. This requires documentation of the model itself and documentation of daily practice. The process evaluation will also encompass client perspectives (e.g., opinions about and satisfaction with the Roca process). The process evaluation will include quantitative and qualitative data, and the results will be used to provide context to the outcome and impact components of the evaluation and explain the results. The process evaluation will employ multiple measurement and sampling techniques, including surveys, focus groups, key stakeholder interviews, and activity counts from ETO.

The outcome component of the evaluation will measure whether participants in the TR achieve the objectives of the model, such as maintaining employment or refraining from substance use. Outcome data can be qualitative or quantitative, but quantitative data is predominantly used in order to apply statistical tests and determine the significance of the results. The qualitative information will help to provide context to the outcomes. The way that outcomes are measured is critically important because many factors outside of Roca’s intervention can affect outcomes for young people. For example, research shows that many youth “age out” of crime and gang involvement as they approach adulthood. If a young person who spends two years involved with Roca decreases their gang activity over that time, it could be due to Roca or simply to aging. The evaluation design must make every attempt to account for other explanations for outcomes.

The evaluation will address other issues of impact, demonstrating the model’s impact on the broader community. This component of the evaluation will enable Roca to influence policy both locally and nationally, potentially influencing legislation and funding to serve high risk young people.

Roca’s evaluation agenda encompasses outcomes for participating youth, as well as a process evaluation of the Roca model, a range of outcomes data for transitional employment/job placement/retention, and related benchmarks for organizational capacity and growth. The evaluation work will assess program results and costs; increase our understanding of factors related to positive outcomes; help guide performance improvement; enhance Roca’s ongoing evaluation and performance measurement capacity; and draw lessons for the field about how to serve this population of high-risk youth. In order to be the most helpful and of value to young people, our outcomes systems will be instrumental to our success.