external_evaluation_header.jpg

External Evaluation

CEO 3-YEAR RANDOM ASSIGNMENT PROGRAM EVALUATION HIGHLIGHTS

About the Study

In 2004, the federal Department of Health and Human Services launched a multi-site national study to evaluate programs and organizations that serve “hard-to-employ” populations. The study, conducted by the independent evaluator MDRC, would use a rigorous random assignment research design and follow study participants for a three year period. CEO was selected, after a competitive process, as the study site serving people with criminal convictions. CEO entered into this evaluation to determine the impacts of its employment model exclusively for people returning from prison onto parole. To date, CEO remains one of the few organizations in the country to have subjected themselves to such a rigorous outcomes evaluation. As a learning organization, CEO welcomed the opportunity to improve the organization’s internal practices and contribute to the employment reentry field more broadly.

About the Population

The study group of 977 participants are mostly young Black or Hispanic men with limited work and education histories. On average, sample members were 34 years old when they enrolled in the study; 43 percent were age 30 or younger. Just over half the sample had completed a high school diploma or a General Educational Development (GED) certificate (most of these had a GED).

The sample members had extensive histories with the criminal justice system, with an average of seven prior convictions and a total of five years in state prison. Nearly 70 percent of the sample members had prior arrests for violent offenses, and 51 percent had been convicted of a violent offense.

Study Outcomes

CEO significantly reduced recidivism with the largest impacts for the group of participants recently released from prison. This group was significantly less likely than control group members to be arrested, convicted of a crime, or incarcerated.  These impacts represent a reduction in recidivism of 16 percent to 22 percent across the three outcomes.  Reductions in recidivism are difficult to achieve and have rarely been seen in rigorous evaluations such as this one.

CEO significantly increased employment in the first year of the study.  Large employment gains faded after the first year, though employment improvements continued in years 2 and 3 for recently released people.

CEO’s financial benefits far outweigh its costs. A benefit cost analysis of the evaluation results calculated total benefits of up to $3.85 for every $1.00 spent on the program.  The majority of these benefits came in the form of reduced criminal justice expenditures and the value of services that CEO participants provided to government agencies in the transitional job work sites. Under a wide range of assumptions, the program costs less than it saves.

In addition to those recently released from prison, CEO’s impacts on employment and recidivism were stronger for those who were more disadvantaged or at higher risk of recidivism. The subgroups with the largest impacts on employment and recidivism include those with four or more prior convictions, those without a high school diploma or GED, and those with a high risk of recidivism (based on a risk index determined by age, number of prior convictions, and other static factors) at the time of random assignment. Among the subgroup with four or more prior convictions at the time of study entry, CEO reduced convictions for new crimes by 12.8 percentage points. For those at high risk of recidivism, post-program employment outcomes (Years 2-3) increased average quarterly unsubsidized employment by 11.0 percentage points (27 percent of program group members compared to 16 percent of the control group).

RANDOM ASSIGNMENT EVALUATION

The reliability of random assignment studies makes them the “gold standard” of outcome evaluations. In the case of CEO, the process began by recruiting more people than the organization could serve, or over-enrolling. Upon walking through the front door a computer “randomly assigned” an individual to either a program group where they received the full complement of CEO services or a control group where they were provided with a pre-employment training class and then supported through a staffed employment resource room. Unlike a comparison group analysis, this type of study begins with a single collection of subjects thereby enabling the evaluator to form control and program groups that are truly random. This ensures that the impacts measured – recidivism and employment — are attributable to the intervention – in this case CEO.

By participating in a random assignment evaluation CEO has proven the effectiveness of the program model scientifically, through experimental research.  While many other programs have been able to demonstrate their effectiveness by comparing their participants to similar people no other reentry employment provider in the nation has a more rigorously established evidence base proving that they reduce recidivism.

A NOTE ON THE EVALUATOR: MDRC

MDRC was created to learn what works in social policy — and to make sure that the evidence we produce informs the design and implementation of policies and programs. Created in 1974 by the Ford Foundation and a group of federal agencies, MDRC is best known for mounting large-scale evaluations of real-world policies and programs targeted to low-income people. We helped pioneer the use of random assignment — the same highly reliable methodology that is used to test new medicines — in the evaluation of such policies and programs. In some cases, we work with others to design pathbreaking initiatives and then subject those initiatives to rigorous testing. In other cases, we conduct careful evaluations of programs designed and operated by government agencies or others. [www.mdrc.org]