An impact evaluation focuses on the effects and outcomes of a program to see if it is working as planned and if it is efficient and effective. Since the program in this example is a pilot program and has never before been run, it will need such an impact evaluation to make sure that it is fulfilling its goals and actually helping the people it is intending to help. Impact evaluations can also identify problems with a program that need to be solved to improve efficiency.
A program like this one can combine several different methods of evaluation. Enrollment and attendance numbers for the program (in each community and as a whole) will reveal the both the levels of initial interest in the target audience as well as continuation in the program. Surveys and interviews of instructors and current students (again in each community) will fill in more personal details. Surveys and interviews with students who have dropped out of the program may also help to reveal possible problems with the program. The curriculum and its implementation should also be evaluated for each community, with special notation of different curriculum choices and instruction styles. These numbers, personal details, and curriculum/teaching evidence should be compared and contrasted in each of the twenty communities to reveal patterns. The evaluator should be on the look out for curriculum and teaching styles that correspond with higher enrollment and retention numbers as well as greater student satisfaction because these may then be implemented in other communities. Finally, the evaluator needs to closely examine the program's budget in each community to determine if it is being used efficiently and is sufficient to meet the program's needs.
An impact evaluation can be a challenging task. Sometimes numbers may be incomplete or unavailable in particular communities. Getting instructors, students, and former students to turn in surveys or agree to interviews may also be challenging. Depending upon how uniform the curriculum and instructional styles are, evaluators may run into difficulties in comparison with the enrollment and attendance numbers. Patterns may not be clear. Finally, budgets are always challenging, and the evaluator may have to make some unpopular recommendations about how money is and is not being spent.