evaluation asks you to apply your skill set to a number of situations
Goals
* Understand eval foundations, issues, and approaches
* Acquire some practical strategies
* Complete an actual evaluation successfully
In 2 months we will have an eval done. Cool, but would prefer to try one on my own for the full effect rather than work as a cog in a group.
Role of evaluation in society?
determine effectiveness, refine, revise, progress, wise judgement, maximize effectiveness of choices –> save money
research is to explore
evaluation /has/ to be applied
evaluators are change agents, must convince people to adopt recommendations based on eval findings
–recommendations and solutions
interpersonal – eval as judgement vs as learning opportunity
different evaluation methods, e.g. participant evaluation,
summative evaluation after the fact (e.g. after a problem is exposed, disaster occurs)
evaluation must be self-reflective, meta-evaluation
Research v Evaluation
Who sets agenda?
Are results context-specific or generalizable/transferable?
What are the criteria being used?
What are the skill sets? researcher crunches numbers, interviews, writes; evaluators need to be interpersonal, teach
Research
“Does this work?” positivistic, quantitative
“What does it mean?” qualitative
Evaluators
“Does it work for this group A in a way that is valuable?”
*Program improvement (e.g. grants)
*Oversight and compliance (e.g. grants)
*Assessment of worth
*Knowledge development
*Social betterment
formative v summative
cook tastes the soup v. guest tastes the soup
Relative emphasis to program life, formative to summative
Purpose? f: to improve; s: judge
Use? f: encourages immediate application; s: promotes future directions
Audience? f: practioners; s: decision-makers
By whom? f: practitioners; s: external, objective evaluator
Data collection? f: qualitative; s:
Sample?
Questions? f: what can be improved?; s: is this valuable? what is worth? should it persist?
Internal v external – which is better and why?
internal is subject to social pressure; external less so
internal may be “cheaper”
what evalutions? to stop by for a bagel? did I go to bed early enough? should I be in this course? will I be able to apply this course to my work?
Big Brainz
Education Engine
mastery learning
testing with Wasatch Elementary
what do they want?
prove it’s awesome
attitude, pre-assessment post-assessment
not a comparable program, product? could be
vs. traditional teaching
mixing unrelated skill with targeted skill
distracting
Paradigms
Objectives-oriented Did you meet your goals?
Management-oriented Is it profitable? What decision should you make?
Goal-free What can we deduce from your process?
Consumer-oriented Which product should you use?
Interpretivist-oriented What does it mean from their perspective?
Participant-focused How can we make a difference by involving the participants?
Expertise-oriented What would an expert think about this?
Empowerment How can we teach you to fish?
Management
30-45 minutes
Why is paradigm exploration important?
Evaluation Type
Philosophy
Model
Questions
Methods
/narrowing the focus/
Ideas for exploration
evaluate online course development process and practices (objectives, management)
evaluate lms features (product)
evaluate effectivness of
learning
teaching
practices
strategies
that
reflection
evaluate the impact of Grand Poobah on TTIX participants
evaluate the accuracy, effectiveness, appropriateness of UVU DE’s ITR measure
IPT 661 Notes: 4-27-2010
http://www.byuipt.net/wests/rick/teaching/program-evaluation/
evaluation asks you to apply your skill set to a number of situations
Goals
* Understand eval foundations, issues, and approaches
* Acquire some practical strategies
* Complete an actual evaluation successfully
In 2 months we will have an eval done. Cool, but would prefer to try one on my own for the full effect rather than work as a cog in a group.
Role of evaluation in society?
determine effectiveness, refine, revise, progress, wise judgement, maximize effectiveness of choices –> save money
research is to explore
evaluation /has/ to be applied
evaluators are change agents, must convince people to adopt recommendations based on eval findings
–recommendations and solutions
interpersonal – eval as judgement vs as learning opportunity
different evaluation methods, e.g. participant evaluation,
summative evaluation after the fact (e.g. after a problem is exposed, disaster occurs)
evaluation must be self-reflective, meta-evaluation
Research v Evaluation
Who sets agenda?
Are results context-specific or generalizable/transferable?
What are the criteria being used?
What are the skill sets? researcher crunches numbers, interviews, writes; evaluators need to be interpersonal, teach
Research
“Does this work?” positivistic, quantitative
“What does it mean?” qualitative
Evaluators
“Does it work for this group A in a way that is valuable?”
method (research) ————- intuition (informal evaluation)
formal evaluation
Goals and Roles
*Program improvement (e.g. grants)
*Oversight and compliance (e.g. grants)
*Assessment of worth
*Knowledge development
*Social betterment
formative v summative
cook tastes the soup v. guest tastes the soup
Relative emphasis to program life, formative to summative
Purpose? f: to improve; s: judge
Use? f: encourages immediate application; s: promotes future directions
Audience? f: practioners; s: decision-makers
By whom? f: practitioners; s: external, objective evaluator
Data collection? f: qualitative; s:
Sample?
Questions? f: what can be improved?; s: is this valuable? what is worth? should it persist?
Internal v external – which is better and why?
internal is subject to social pressure; external less so
internal may be “cheaper”
what evalutions? to stop by for a bagel? did I go to bed early enough? should I be in this course? will I be able to apply this course to my work?
Big Brainz
Education Engine
mastery learning
testing with Wasatch Elementary
what do they want?
prove it’s awesome
attitude, pre-assessment post-assessment
not a comparable program, product? could be
vs. traditional teaching
mixing unrelated skill with targeted skill
distracting
Paradigms
Objectives-oriented Did you meet your goals?
Management-oriented Is it profitable? What decision should you make?
Goal-free What can we deduce from your process?
Consumer-oriented Which product should you use?
Interpretivist-oriented What does it mean from their perspective?
Participant-focused How can we make a difference by involving the participants?
Expertise-oriented What would an expert think about this?
Empowerment How can we teach you to fish?
Management
30-45 minutes
Why is paradigm exploration important?
Evaluation Type
Philosophy
Model
Questions
Methods
/narrowing the focus/
Ideas for exploration
evaluate online course development process and practices (objectives, management)
evaluate lms features (product)
evaluate effectivness of
learning
teaching
practices
strategies
that
reflection
evaluate the impact of Grand Poobah on TTIX participants
evaluate the accuracy, effectiveness, appropriateness of UVU DE’s ITR measure