CEO’s Random Assignment Evaluation: Background

In 2004, the federal Department of Health and Human Services launched a multi-site national study to evaluate programs and organizations that serve “hard-to-employ” populations. The study, conducted by the independent evaluator MDRC (for more on MDRC see text box below), would use a rigorous random assignment research design and follow study participants for a three year period. CEO was selected, after a competitive process, as the study site serving people with criminal convictions. CEO entered into this evaluation to determine the impacts of its employment model exclusively for people returning from prison onto parole. To date, CEO remains one of the few organizations in the country to have subjected themselves to such a rigorous outcomes evaluation. As a learning organization, CEO welcomed the opportunity to improve the organization’s internal practices and contribute to the employment reentry field more broadly.

Study Design and Participant Characteristics

The reliability of random assignment studies makes them the “gold standard” of outcome evaluations. In the case of CEO, the process began by recruiting more people than the organization could serve, or over-enrolling. Upon walking through the front door a computer “randomly assigned” an individual to either a program group where they received the full complement of CEO services or a control group where they were provided with a pre-employment training class and then supported through a staffed employment resource room. Unlike a comparison group analysis, this type of study begins with a single collection of subjects thereby enabling the evaluator to form control and program groups that are truly random. This ensures that the impacts measured – recidivism and employment — are attributable to the intervention – in this case CEO.

Over the course of 18 months, nearly 1,000 people participated in the two study groups (568 in program; 409 in control; total 997). The outcomes of all subjects were then tracked for three years – paralleling the standard follow-up period the federal government uses to monitor recidivism. Important characteristics of those involved in the study include

• Average of 7 prior convictions
• Average 5 years in prison
• 52% had at least one conviction for a violent crime
• 2/3 African-American; 1/3 Latino
• 43% had no High School diploma or GED
• Over half of the participants were fathers; few living with their children.

Slide 1

What is MDRC?

In their own words, for more visit: www.mrdc.org

Too often, public policies that profoundly affect the lives of low-income families are shaped by hunches, anecdotes, and untested assumptions. Ineffective policies waste precious resources and feed public cynicism about government. Most important, such policies may hinder the very people they are designed to help. MDRC was created to learn what works in social policy — and to make sure that the evidence we produce informs the design and implementation of policies and programs.

Created in 1974 by the Ford Foundation and a group of federal agencies, MDRC is best known for mounting large-scale evaluations of real-world policies and programs targeted to low-income people. We helped pioneer the use of random assignment — the same highly reliable methodology that is used to test new medicines — in the evaluation of such policies and programs. In some cases, we work with others to design pathbreaking initiatives and then subject those initiatives to rigorous testing. In other cases, we conduct careful evaluations of programs designed and operated by government agencies or others.

The five main policy areas in which we work, illustrate the breadth and diversity of our current projects. Those areas include:

• Promoting Family Well-Being and Child Development
• Improving Public Education
• Promoting Successful Transitions to Adulthood
• Supporting Low-Wage Workers and Communities
• Overcoming Barriers to Employment

Working in fields where emotion and ideology often dominate public debates, MDRC is seen as a source of objective, unbiased evidence. From welfare policy to high school reform, MDRC’s research has frequently helped to shape legislation, program design, and operational practices across the country.

 

 

Evaluation Findings: The CEO Model Significantly Reduces Recidivism

MDRC’s final 3-year report have not been published by HHS, but final data has proven that CEO has a statistically significant impact on several measures of recidivism. The most pronounced effect identified was on recently released people who enroll in CEO. The study found that this subgroup – a population that CEO’s Theory of Change [link] specifically targets and defines as coming to CEO within three months of release — has significantly lower rates at all measures of recidivism, including arrests, convictions, convictions of a violent crime and returns to jail or prison, fully three years after enrolling in the program.  The results for this program and control subgroup are included in the chart below:

3-YEAR RESULTS FOR RECENTLY RELEASED SUBGROUP:

Outcome (%)	Program group	Control group	Difference	Percent Decrease
Arrested	49.1	59.1	-10.0*	[17%]
Convicted of crime	44.0	56.7	-12.7**	[22%]
Convicted of a violent crime	5.4	14.3	-8.9***	[62%]
Incarcerated in jail or prison	60.2	71.3	-11.2**	[16%]

MDRC found that the decreases in recidivism for recently released people were statistically significant, meaning that it was CEO’s intervention that accounted for the difference in outcomes. Following standard research practice, the Significance is ranked on a scale of 1to 3 stars with  more stars indicating the greater significance. The Difference column of the chart above shows the raw percentage point spread between the program and control group’s outcomes (e.g. there is a -10.0 percentage point difference in arrests between these two groups). When translated into Percent Decrease, CEO’s impact becomes even clearer: the -10.0 percentage point difference translates to a -17% percent decrease in arrests.

As random assignment evaluation is considered the most demanding and precise research method, these outcomes are all the more compelling. In fact, the evaluators described CEO’s recidivism outcomes as “rare” for a rigorous study of this kind. CEO’s recidivism results are important in several ways: they have a positive impact on public safety and the health of individuals, their families and the communities in which they live, and affect economic conditions.

Urban Institute/MRDC Study: CEO is Most Effective with High Risk Individuals Using data collected by MDRC in the first two years of their evaluation, MDRC and the Urban Institute sorted study participants by risk of reoffending (low-, medium-, and high­-risk) using factors including participants’ gender, age and number of previous arrests. The study found that CEO is most effective with individuals at the highest risk of recidivism. Overall,

“The high-risk offenders who participated in the CEO program were less likely to be rearrested, had fewer rearrests, and were less likely to be reconvicted of crimes than high-risk offenders who did not have a chance to participate in the program”.

Updated recidivism outcomes sorted by risk will be featured in the final evaluation report when it is released in late fall 2011.

Evaluation Findings: Employment Outcomes

The evaluation also showed that CEO dramatically increased employment compared to individuals in the control group for the first 12 months of the study, though the statistical significance of this impact faded in the out years of this study.  This means that CEO is having the intended effect of increasing employment during the critical period of prisoner reentry for an exceedingly hard to serve population. The recently released subgroup also had several quarters in years 2 and 3 where their unsubsidized employment was statistically significant, though these outcomes were not consistent quarter to quarter.

Overall, the findings highlight the difficulty formerly incarcerated people face in maintaining a sustained connection to the labor market, especially once they were no longer directly connected with the program.  In considering these findings CEO determined that more needed to be done if the program was to have a lasting impact on employment.  During the 2004-2005 period under study, CEO had few resources to help participants hold on to their jobs after placement or find new jobs after job loss.  In response to early findings, CEO began experimenting with job retention strategies to increase these numbers. Over the last several years, CEO has become a pioneer in this field employing full-time retention specialists who are dedicated to helping participants keep jobs they have and replace ones they lost. CEO also developed “Rapid Rewards” – a system of monthly incentives participants receive when showing paystubs that helps CEO both motivate and track participant progress. These practices have helped CEO increase its 180-day retention rate from 24% to 58% between 2004-2011 and 365 day retention from 24% to 46%. CEO hopes to test the impacts of these new practices in a follow-up random assignment evaluation (see below).

What’s to Come

Final 3-Year Study Report: The final report on CEO’s 3-year evaluation will be released in late  fall of 2011 by the Federal Department of Health and Human Services.   The report will show that CEO makes a statistically significant impact on all measures of recidivism for recently released people– arrests, conviction and reeincarceration – fully three years after enrolling in the program.

Benefit Cost Analysis: As part of the final report, MDRC worked with the Vera Institute of Justice’s new Benefit-Cost Analysis Unit to monetize the impact CEO’s program made in the study. In this collaboration, the parties worked together to quantify the benefit-cost of CEO’s model vis-à-vis public investment, cost to taxpayers, and other measures. This analysis will be included in the final 3-year report when it is released by HHS.

Future Random Assignment Evaluations

Participating in a random assignment evaluation underscored CEO’s dedication to program learning and deepened the agency’s commitment to the strictest form of independent assessment. As part of the Social Innovation Fund CEO is currently working with MDRC to design a follow-up random assignment evaluation, this time of CEO’s offices in Upstate New York. This evaluation will test whether CEO can replicate the criminal justice outcomes demonstrated in the first evaluation and the efficacy of job retention innovations on long-term employment outcomes. In preparation, in the spring of 2011, MDRC performed a fidelity study to determine whether the CEO model has been faithfully replicated in Buffalo and Albany, New York while a fidelity study of Rochester, NY was performed in December, 2011. These fidelity studies are necessary precursors to an evaluation as they determine whether all the essential elements of the model have been replicated and  whether the sites are sturdy enough to withstand the rigors of evaluation. By the second half of 2012, CEO and MDRC hope to begin enrollment in the study across multiple upstate locations.