Case Study: Toujours ensemble

August 6, 2019

Refocusing and improving a program to ensure its sustainability.

“Thanks to the coaching we received from Innoweave, we were able to identify the strong points of the program as well as other areas that we did not have enough data, particularly in terms of social skills. The knowledge sharing proved to be very useful…. We would never have suspected at the outset of the project that the theory of change would be so important for the project”

– Toujours ensemble

Project overview

Sac à dos is a pilot program of Toujours ensemble, which has been providing school support in a low-income neighborhood in Montreal since 1986. The program provides after school support for primary school children in Verdun, including reading workshops, educational play workshops, and support for parents. The goal of the program is to help children, who have classroom, personal or social difficulties be successful at school by reducing two risk factors of dropout, which are lack of motivation at school and lack of commitment from parents.

Relevance and niche position for Developmental Evaluation

The Developmental Evaluation of the Sac à dos program focused on strengthening and consolidating the program to ensure that it could become sustainable. Given that it began through a trial and error process, the Developmental Evaluation had to be integrated into the “emergence” niche among various other niche positions that justify the use of this type of evaluation. The Developmental Evaluation extended over an 8-month period, under the guidance of evaluators Stephanie Garrow and Lara Evoy.

Evaluation questions

The evaluation questions aimed at clarifying the intended impact of the program, and identifying changes in program delivery to implement as it unfolded.

The questions included, among others:

  • What aspects of the project have the greatest impact on expected results?
  • What aspects have not provided the expected results and should be abandoned or replaced?
  • How can we better identify the needs of users and partners, and respond to them more efficiently?

Recommended methods and interventions

Several different evaluation methods were chosen and developed alongside the program. These included:

  1. A workshop to clarify the theory of change: activities focused on working with the team to clarify the program objectives.
  2. Analysis of available data: youth action plans, reading questionnaires, attendance, logbooks, tools and other materials.
  3. Strategic interviews with key players: teachers, parents, and employees. A questionnaire on parent satisfaction was created, distributed, and the results were compiled.
  4. Meeting with stakeholders to share knowledge.

Throughout the process, various exchanges occurred involving the data collection, analysis, and project development. This real-time feedback allowed program staff to make adjustments as the project unfolded.

Impact

The major impact of the Developmental Evaluation process was that it made it possible for the Sac à dos program to become a sustainable on-going program with an integrated evaluation method.

The Developmental Evaluation has made the following positive contributions to the development of the project, and to the organization:

  • Program improvements. DE was used to refocus program objectives concerning student motivation. The data showed that the action plan is used as a mobilization tool and its reading workshops have had a positive impact helping children to discover new books and gain an appreciation for reading. Furthermore, instead of working simultaneously on a wide range of risks, the program is now focusing on two: (a) low motivation at school and (b) lack of parental commitment.
  • Clarification of program goals. Data showed that the children’s groups represent one of the strengths of the program, but this does not necessarily meet the needs of students with special needs. The program is not sufficiently adapted to meet the needs of children with learning difficulties. The parent component also needs to be strengthened.
  • Identification of success factors: During the process, the evaluator helped illustrate the importance of the relationship between the organization and local schools, and the importance of having access to program space within the schools.
  • Integration of evaluation tools within other programs of the organization. A review and rethinking of other programs offered by the organization was an unexpected benefit of the process. Through Developmental Evaluation, Toujours ensemble has adopted an evaluation culture. In addition to being able to identify an “increase in the number of young people enrolled in regular high school classes” as an intended impact, the evaluation also led to greater linkages between the objectives and the activities of various programs. These include the Parcours and Passeport programs which focus mainly on adolescents.

Conditions for success

Readiness of the organization. One of the key success factors was the organization’s preparation, as it had taken the time to define and clarify the value and role of Developmental Evaluation. Toujours ensemble prepared its application for financial support eight months after attending the Innoweave workshop on Developmental Evaluation. During this period, the team was able to continually clarify its objectives and processes.

Task sharing between the organization and the evaluators. The evaluators provided coaching by preparing a data analysis grid, a follow-up meeting after data analysis, and helped clarify results. Toujours ensemble was responsible for data collection, compiling results and analyzing the data. It also organized the calendar of meetings. By appointing a single person from the organization to liaise between the local environment and the evaluators, it became much easier to thoroughly integrate the results of the Developmental Evaluation for the best development of the program.

Strong and active leadership. The head of the Sac à dos team attached great importance to the reflections produced by the Developmental Evaluation, which was one of the key success factors. Seeing the usefulness of the method to help his team test and reflect on the program and the organization, he found innovative ways to invite other agency programs, the Board of Directors, and key partners from the community to participate in discussions on the impact and direction of Sac à dos.