learn.5tein.com Jared Stein's grad-school-community blog on teaching and learning.

Summary of “Program Evaluation” ch 14

Chapter 14 of Fitzpatrick, Sanders, and Worthen's Program Evaluation text, "Collecting Evaluative Information: Design, Sampling, and Cost Choices" leads one through significant decisions one must make when choosing methods and procedures. Based on the evaluation questions developed in the evaluation plan the evaluator must develop or select a design by which information will be collected to address those questions. Sampling strategies may be considered to ensure the proper information is brought to bear on the questions from the appropriate sources. One should also consider how the information may be analyzed. Statistical methods or qualitative organization may be preferable, or mixed. Mixed methods are used to improve the validity of measurements or to procure a more accurate picture of the project and its implications. Mixed methods should, according to Greene and Caracellie, consider 3 levels: political, philosophical, technical. They also identify three "stances" to mixed methods: purist, pragmatic, and dialectical. Mixed methods appear to have largely replaced either solely causal designs (i.e. experimental designs which use random assignment and control/comparison groups to gather post-only or pre-post data to draw inferences) or case study designs (which heavily rely on responsive design and qualitative methods to explore how and why "place the best brains available in the thick of what is going on" [Stake 1994]). Quasi-experimental designs do not involve random assignments, and though they can not suggest causation yet they can counter some explanations for change other than the program. Quasi-experimental designs may include interrupted time-series, which collects data often before and after the program, focusing on trends. These are appropriate when random assignment is not feasible. A nonequivalent comparison group study seeks to study a group similar to the group that will receive the program or treatment. Regression-discontinuty selects individuals based on qualifying criteria. Descriptive designs (such as cross-sectional and time-series) are similar to but more quantitative than case studies, but do not provide similarly in-depth descriptions. Any evaluation's design may be based on one or more of these designs, but should only be applied after careful consideration of purpose and questions.

Methods may be applied to random samplings for generalizeability, but in the case of most evaluations data will be gathred from all or most of the target population.

Finally, the authors consider the role of cost studies to determine the resource-worthiness of programs or projects using methods such as cost-benefit analysis.

Upon reflection this chapter covers a very broad range of research topics, methods, and designs. Though much of this was new to me, the purposes, justifications, and applications made sense, and I could identify evaluations I've been involved in or might consider conducting that would utilizes these methods or designs. Still, because I recognize fluency with this content to be critical for future work in this field, I am bookmarking this chapter for a second reading.

Summary of “Program Evaluation” ch 8, Participant-Oriented

Chapter 8 of Fitzpatrick, Sanders, and Worthen's text Program Evaluation covers participant-oriented approaches to evaluation. Such approaches seek to focus on "the human element" and include diverse voices not typically expressed in more standard evaluation approaches or models. Stake's countenance framework describes the evaluatee and attempts to judge its value or worth. The illuminative model also focuses on description, but favors interpretation over value judgment. Naturalistic evaluations also aim at fully describing the program, but as it occurs with the evaluator in the role of the end-user or learner. Some approaches, such as Fetterman's empowerment evaluation and Merten's emancipatory evaluation attempt to address and liberate marginalized or suppressed populations.

These potentially highly-charged approaches may be valuable in certain institutions or programs seeking holistic evaluation, but I suspect the applicability or priority of such approaches are not as favorable to clients and even stakeholders as more standard evaluative approaches such as those I've summarized earlier. Case studies--and metaevaluations--using these approaches would probably help me wrap my head around their specific applicability and potential benefits for organizations and projects.