internal_evaluation_header.jpg

Internal Evaluation

CEO’s program model depends on extensive data tracking across a variety of measures, using an in-house adaptation of Salesforce.com’s online customer relationship management (CRM) database. This program data allows us to determine whether our Theory of Change is working as expected and whether we are having an impact in the communities which we serve. We collect data on participants and individual events (e.g. transitional work site attendance, job start readiness assessments, interview feedback, etc.) as they occur, allowing us to chart our near-, mid- and long-term goals in real time. These tools are used by management and staff every day and they are essential to achieving organizational excellence.

SALESFORCE.COM

CEO has adapted the web-based customer relationship management database, Salesforce.com, to track and retain program data. Salesforce.com allows CEO to track every participant activity from recruitment and intake to placement and retention. All case notes are documented in Salesforce.com, limiting the hard data required for case files and allowing staff and management easy access to participant information. It also has standardized the process of how a participant moves from one phase of the program to the next. This supports the integrity of the CEO model and provides clarity on the status of each participant. CEO has customized user profiles for direct program job titles and every program employee is required to document their work in the system in real time. CEO’s development of Salesforce.com also includes a robust suite of dashboards and reports to assess the progress of participants and the performance of the organization and individual employees. CEO’s adaptation of Salesforce.com has contributed significantly to the ability to share information and track outcomes at our offices throughout New York State.

Salesforce.com was implemented at CEO in 2006 and is now used by all vocational program staff and managers (about 90 users out of a total staff of 165). CEO’s data management and performance tracking efforts are a hallmark of the program model and in November 2008, CEO was awarded the “Power of Us” APPY Award from Salesforce.com, an award that recognizes innovation in the nonprofit sector.

Salesforce.com is an essential tool in tracking CEO's goals and progress. The above snapshot shows how annual, enterprise-wide goals are tracked and managed

Tracking Participant Outcomes

Upon entering CEO a detailed profile of each participant is created that includes demographic and work-related indicators that  allow our staff to meet individual needs and chart participant progress as they take part in the program. Once participants have begun transitional employment on CEO work crews, their daily attendance as well as their progress in developing appropriate on-the-job behaviors and basic skills is noted daily in their Passport to Success booklets. Collected Passport data is then entered into the Salesforce.com database and analyzed to determine when participants have achieved baseline preparation for full-time employment in the competitive workforce. Once this milestone is achieved, participants are considered “job start ready” and begin interviewing for full-time positions developed by Job Developers in CEO’s vocational services teams.

Vocational staff maintain contact with participants over the course of the following year, during which their rates of job retention are tracked to evaluate overall program success. The attendance and content of participants’ meetings with Job Coaches and Job Developers are also tracked in Salesforce.com as are youth job start readiness assessments. Job search activities are documented including interviews and interview feedback. Once placed on a job, Job Developers record the position, company, wage, and other details in Salesforce.com. In the post-placement period, Retention Specialists track their contacts with participants by documenting phone calls and/or site visits. Participants are also offered incentives to bring paystubs to CEO that verify employment at key milestones. Job placements and retention milestones are independently verified by CEO’s Follow-Up Unit by calling or emailing employers.

CEO holds itself to the highest standard of outcome verification and reporting. Only those participants that are independently verified as employed—via paystub, written confirmation by a parole/probation officer, or direct contact with an employer by someone outside the vocational staff—are counted as working. CEO does not accept self-reported information.

Participant profiles allow CEO staff to track and note the progress of a participant throughout the program. The above snapshot shows a portion of the Events Timeline of a participant.

 

Managing Organizational Outcomes

On an organization-wide level, critical outcomes such as life skills educationjob placement and post-placement services are charted on organizational “dashboards” which are reviewed by senior leadership weekly. This regular feedback on our operations allows us to discuss, investigate, assess and act quickly if performance gaps occur.

The same is true at the department level in New York City and in Upstate New York. CEO’s three workforce development teams in New York City, as well as County Directors around New York State, generate most of the organization’s key outcomes. At any given moment —minimum weekly and often daily—performance data such as total enrollments, number of interviews and job opportunities is being collected. This data is reviewed in staff meetings or in individual case conferences. This process—CEO’s performance management system—is critical to the success of individual CEO participants and achieving departmental and organizational outcomes.

CEO’s Theory of Change is also dependent on the outcome of recidivism. We seek to keep participants free from incarceration for up to a year directly and three years indirectly. As stated in the External Evaluation Section, CEO has proven through a random assignment study that recently released people who enroll at CEO have significantly lower rates at all measures of recidivism a full three years after enrolling at CEO. CEO has continued to rely on these outcomes as it is very difficult to collect and accurately interpret recidivism data in real time; however, CEO is committed to engaging new independent studies that examine this critical outcome.

Continue to Stakeholder Feedback>>