Summary of “Program Evaluation” ch 14

Chapter 14 of Fitzpatrick, Sanders, and Worthen’s Program Evaluation text, “Collecting Evaluative Information: Design, Sampling, and Cost Choices” leads one through significant decisions one must make when choosing methods and procedures. Based on the evaluation questions developed in the evaluation plan the evaluator must develop or select a design by which information will be collected to address those questions. Sampling strategies may be considered to ensure the proper information is brought to bear on the questions from the appropriate sources. One should also consider how the information may be analyzed. Statistical methods or qualitative organization may be preferable, or mixed. Mixed methods are used to improve the validity of measurements or to procure a more accurate picture of the project and its implications. Mixed methods should, according to Greene and Caracellie, consider 3 levels: political, philosophical, technical. They also identify three “stances” to mixed methods: purist, pragmatic, and dialectical. Mixed methods appear to have largely replaced either solely causal designs (i.e. experimental designs which use random assignment and control/comparison groups to gather post-only or pre-post data to draw inferences) or case study designs (which heavily rely on responsive design and qualitative methods to explore how and why “place the best brains available in the thick of what is going on” [Stake 1994]). Quasi-experimental designs do not involve random assignments, and though they can not suggest causation yet they can counter some explanations for change other than the program. Quasi-experimental designs may include interrupted time-series, which collects data often before and after the program, focusing on trends. These are appropriate when random assignment is not feasible. A nonequivalent comparison group study seeks to study a group similar to the group that will receive the program or treatment. Regression-discontinuty selects individuals based on qualifying criteria. Descriptive designs (such as cross-sectional and time-series) are similar to but more quantitative than case studies, but do not provide similarly in-depth descriptions. Any evaluation’s design may be based on one or more of these designs, but should only be applied after careful consideration of purpose and questions.

Methods may be applied to random samplings for generalizeability, but in the case of most evaluations data will be gathred from all or most of the target population.

Finally, the authors consider the role of cost studies to determine the resource-worthiness of programs or projects using methods such as cost-benefit analysis.

Upon reflection this chapter covers a very broad range of research topics, methods, and designs. Though much of this was new to me, the purposes, justifications, and applications made sense, and I could identify evaluations I’ve been involved in or might consider conducting that would utilizes these methods or designs. Still, because I recognize fluency with this content to be critical for future work in this field, I am bookmarking this chapter for a second reading.