learn.5tein.com » reviews

Summary of “Program Evaluation” ch 14

Jared Stein — Mon, 17 May 2010 05:44:56 +0000

Chapter 14 of Fitzpatrick, Sanders, and Worthen’s Program Evaluation text, “Collecting Evaluative Information: Design, Sampling, and Cost Choices” leads one through significant decisions one must make when choosing methods and procedures. Based on the evaluation questions developed in the evaluation plan the evaluator must develop or select a design by which information will be collected to address those questions. Sampling strategies may be considered to ensure the proper information is brought to bear on the questions from the appropriate sources. One should also consider how the information may be analyzed. Statistical methods or qualitative organization may be preferable, or mixed. Mixed methods are used to improve the validity of measurements or to procure a more accurate picture of the project and its implications. Mixed methods should, according to Greene and Caracellie, consider 3 levels: political, philosophical, technical. They also identify three “stances” to mixed methods: purist, pragmatic, and dialectical. Mixed methods appear to have largely replaced either solely causal designs (i.e. experimental designs which use random assignment and control/comparison groups to gather post-only or pre-post data to draw inferences) or case study designs (which heavily rely on responsive design and qualitative methods to explore how and why “place the best brains available in the thick of what is going on” [Stake 1994]). Quasi-experimental designs do not involve random assignments, and though they can not suggest causation yet they can counter some explanations for change other than the program. Quasi-experimental designs may include interrupted time-series, which collects data often before and after the program, focusing on trends. These are appropriate when random assignment is not feasible. A nonequivalent comparison group study seeks to study a group similar to the group that will receive the program or treatment. Regression-discontinuty selects individuals based on qualifying criteria. Descriptive designs (such as cross-sectional and time-series) are similar to but more quantitative than case studies, but do not provide similarly in-depth descriptions. Any evaluation’s design may be based on one or more of these designs, but should only be applied after careful consideration of purpose and questions.

Methods may be applied to random samplings for generalizeability, but in the case of most evaluations data will be gathred from all or most of the target population.

Finally, the authors consider the role of cost studies to determine the resource-worthiness of programs or projects using methods such as cost-benefit analysis.

Upon reflection this chapter covers a very broad range of research topics, methods, and designs. Though much of this was new to me, the purposes, justifications, and applications made sense, and I could identify evaluations I’ve been involved in or might consider conducting that would utilizes these methods or designs. Still, because I recognize fluency with this content to be critical for future work in this field, I am bookmarking this chapter for a second reading.

Summary of “Program Evaluation” ch 8, Participant-Oriented

Jared Stein — Sat, 08 May 2010 19:45:44 +0000

Chapter 8 of Fitzpatrick, Sanders, and Worthen’s text Program Evaluation covers participant-oriented approaches to evaluation. Such approaches seek to focus on “the human element” and include diverse voices not typically expressed in more standard evaluation approaches or models. Stake’s countenance framework describes the evaluatee and attempts to judge its value or worth. The illuminative model also focuses on description, but favors interpretation over value judgment. Naturalistic evaluations also aim at fully describing the program, but as it occurs with the evaluator in the role of the end-user or learner. Some approaches, such as Fetterman’s empowerment evaluation and Merten’s emancipatory evaluation attempt to address and liberate marginalized or suppressed populations.

These potentially highly-charged approaches may be valuable in certain institutions or programs seeking holistic evaluation, but I suspect the applicability or priority of such approaches are not as favorable to clients and even stakeholders as more standard evaluative approaches such as those I’ve summarized earlier. Case studies–and metaevaluations–using these approaches would probably help me wrap my head around their specific applicability and potential benefits for organizations and projects.

Summary of “Program Evaluation” chs 4 – 7

Jared Stein — Fri, 07 May 2010 02:40:18 +0000

Last week I read four core chapters from Fitzpatrick, Sanders, and Worthen’s text Program Evaluation that dealt with major theoretical models and approaches to evaluation, including Objectives-Oriented (71 – 87), Management-Oriented (88 – 99), Consumer-Oriented (100 – 111), and Expertise-Oriented (112 – 128). Each of these chapters outlined major thinkers and practitioners who shaped the particular approaches and models, each highlighted the features of the particular model(s) that set it apart from other approaches, or made it uniquely suited for particular situations, and each listed perceived strengths and weaknesses of the approach.

The objectives-oriented approach was the most accessible to me, since it is based on logic and employs what I think of as common practices evident in even daily evaluative activities. To summarize from the chapter, Tyler designed this approach to judge the effectiveness or value of programs, but recognized that programs’ goals and objectives must first be evaluated. An emphasis on emprical evaluation seems evident in objectives-oriented approaches. There are 5 stages proposed by Provus: 1) definition, 2) installation, 3) process, 4) product, 5) cost-benefit analysis. Objectives-oriented approaches seem to have encouraged counter-approaches such as goal-free evaluations which attempt to identify and gauge unanticipated outcomes or effects of programs unanticipated by objectives-oriented evaluations.

The management-oriented approaches are aimed at providing decision-makers with information that illuminates the problem and provides new or alternate ways forward. Stufflebeam’s CIPP model addresses four kinds of decisions: Context (planning), input (resources/structuring), process (implementation), and product (summative/recycling). Stufflebeam systematically outlined evaluation design steps, and the text codifies how these four types of evaluation may be applied and categorized as formative or summative. The chapter includes Alkin’s UCLA model, which maps fairly well onto CIPP with some minor modification.

The consumer-oriented approaches, as the label suggests, deals primarily with programs aimed at satisfying end-user needs, and were born of the purported need to evaluate educational products. This approach seems also squarely aimed at commercial product or service providers as well. Scriven is one of several major contributors to such approaches and listed 13 neccessita–criteria which a given product must meet–as well as the Key Evaluation Checklist, which provides consumers with defensible results.

Expertise-oriented approaches is suggested to be the oldest and most widely used approach, and may be formal or informal. Examples include ad hoc panel reviews, blue-ribbon panels, funding agency reviews, etc. The reliance on professional judgment indicates a faith in a kind of connoisseurship, referred to as connoisseurship-criticism. Such evaluations are common in institutional and program accreditation.

All of the approaches included in these chapters are useful as background information, though with respect to the anticipated evaluation project targeting Timez Attack I expect that consumer-oriented approaches will be emphasized, though aspects of expertise-oriented evaluation and management-oriented evaluation may come into play. The expertise-orientation may be more important to the client, as they hope to have this evaluation impress potential clients with the weight of our class’s collective educational background. I’m inclined to focus on consumer-oriented and objectives-oriented evaluation in this project, as I care less about the success of the product as I do in judging it’s appropriateness and effectiveness. These are not, of course, mutually exclusive.

Reconsidering dotProject

Jared Stein — Thu, 19 Nov 2009 03:07:54 +0000

(This is a copy of a post made earlier at http://jaredstein.org)
We’ve used dotProject in my Instructional Design Services unit for almost two years. dP is an open source project management and task tracking tool that has benefited our unit organizationally, and has helped us follow-through on projects in a more efficient manner. It has also helped me manage staff resources and understand time-to-delivery of common project types.

However, dP is not without its flaws; indeed, I became so frustrated that late last week I asked my lead developer Ken Woodward to send me some examples of competitive systems that did not include BaseCamp. While BaseCamp is a great system, and probably amongst the top few project management tools out there, it’s not cheap, and we have a lot of projects running simultaneously. Plus, I am a constant advocate of OSS solutions, and in this situation, where many of my staff are student developers, having a locally-owned OSS platform that can be modified by my staff as they learn their trade is very attractive.

Let me note that I always feel guilty critiquing an OSS project, especially when I myself haven’t made any significant contributions to the code. At the same time, with increasing competition in OSS solutions, one has more liberty to leave behind that which doesn’t suit one’s needs. Further, reasonable explication of the features and failures of any system is valuable to developers who are committed to building and maintaining a quality system. So here’s a quick review of dotProject with some particular attention to the broken bits. My next post will introduce the alternatives that we are considering.

Here’s a list of dP’s strengths or notable features:

user-based and password protected
email integrated
distinction between projects and tasks (you’d be surprised how many “project management” tools omit this!)
robust data at the project level, with good hierarchical layout of tasks
project tasks may be imported into other projects, thus “templates”
calendaring
task dependency
good, flexible task logging and time tracking
decent gant charts
written in PHP
themes

Now, a list of complaints:

No RSS. For my unit, RSS or Atom feeds are taken for granted. If I can’t put my task list on my Google home page, I’m not happy. RSS is easy to script, however, so we could add this feature to our To Do list.

Inconsistent task/to do list. I haven’t quite figured this out, but one’s “tasks” list is different from the “to do” list that dP presents. This appears to just be a bug, as the To Do list–which is actually harder to get to–is more accurate.

Some weird PHP scripting. For instance, project categories: one can create categories for projects, like “in planning” or “in progress” or “pilot” or “complete”. The problem seems to be that dP attaches a simple numeric id to each of these category names. It attaches the same numeric id to the projects in the category. Makes sense, right? Wrong. The category ids correspond to the order in which they appear as tabs in the dP interface; if you change the order, the numeric ids of the categories change, and thus no longer match the project categories. I’ve actually found several examples of similarly breakable scripting in the system (including a stubborn “masking” of certain projects related to this) but this was the most frustrating one.

Calendar/date entry is inadequate. In order to set a task due date to, say, December 2010, you can’t type 12/01/2010, you have to open the calendar GUI, then click, click, click, click through the months until you get to December 2010.

Little data available outside of gant charts. And the gant charts fail to react intelligently to clearly “inactive” projects, instead extending them perpetually as if someone were still working on it (even though no hours were logged). But really the main complaint here is that there is no simple, easy way to access data on staff or projects. For example, I can’t click a user’s name and determine how many hours they logged in dP this week. I can’t look at all projects of a type and get an average of hours spent to completion. I can get clear, individual project data from individual projects

Not embeddable, widgetizeable. Several project management systems have scripts, or embeddable widgets, or even desktop apps that help you monitor your tasks. dP does not yet. If they only had RSS, we’d be able to get somewhere. XML RPC? That’d be cool.

Average GUI. The graphical user interface in dP is average at best. It’s not horrible; it’s not elegant. It could be adjusted with a new CSS, but my experimentation with theming in dP is that too much can go wrong, and I’ve tried several themes that actually break basic features of the system! OK, so maybe I just need to spend more time on this, but compared to theming in Drupal or WordPress, dP theming is no fun.

Small community support. Though the quality of people involved in dP is great, the number of people developing, contributing, and using the system is a lot smaller than I’d hoped for. The last date of a major upgrade or bug fix was July 2008. I really want to add some of my staff to that community, but right now may not be the best time in the history of our organization.

So, we’re looking at other options–and I’ll review some of those next time–but I recognize that in the end we may stick with dP. If we do, it will have to be with a greater commitment to participate in the dP community, because several of these complaints I’ve listed simply can’t be allowed to stand, and its true we haven’t given much back.