IPT 564 Notes 07-29-2010
Alignment
| proxy | ||||
| Outcomes | --> | Assessment | --> | Content & activities |
| (know, be able to) | (indirect, noise, err) | (too much, too little) |
This is to justify training/educational experiences.
JMS: Sometimes this works this way, sometimes it doesn't. Sometimes the outcomes are implicit, invisible. Power of alignment: Cohen 1987, Wishnick 1989
Jacob: Because I've never seen a perfect assessment (or even a really great one) it seems basing content on assessment rather than outcomes risks cutting the actual outcomes short, by failing to provide the appropriate content and activities. The idea is that even a good assessment won't do justice to necessary instruction.
Wiley: Faculty adopts a textbook (JMS: but based on outcomes, right?) which may come with assessment
JMS: In k-12 but in higher ed I think faculty don't commonly base outcomes and assessments on the content (textbook); rather they implicitly know the outcomes and choose the content based on that, presuming that the textbook publisher did the hard work of aligning the assessment with the content.
Mager
ABCD
From Mager's Tips on Instructional Objectives
- Audience
- The who. Your objectives had better say, "The student will be able to…"
- Behavior
- An objective always says what a learner is expected to be able to do. The objective sometimes describes the product or result of the doing.
Ask yourself, what is the learner doing when demonstrating achievement of the objective?
- Condition
- An objective always describes the important conditions (if any) under which the performance is to occur.
- Degree
- Wherever possible, an objective describes the criterion of acceptable performance by describing how well the learner must perform in order to be considered acceptable.
Work Model Synthesis
systematically combining and recombining tasks and objectives that through task analysis procedures have been fragmented at a low level"
Learning Analytics
What impact might Google Analytics-style anayltics have for teaching and learning?
Students hate it (accountability vs. Big Brother)
Technology in education now lets us diagnose learners and provide laser-like feedback.
IPT 564 Notes 07-27-2010
Notes 07-27-2010
Business
Tyler built a spreadsheet that allows rubric items, score, and points
David cancelled reverse engineering
Analysis
Gibbons: If design is goal oriented choices made/process under constraints,
analysis is understanding those constraints
Analysis is an investment; it pays dividends later
Cost of change increases as time goes on (JMS: analogous to a toppling dominoes set-up; complex and elaborate set ups become more difficult to change as the set-up proceeds)
four levels of Donald Kirkpatrick's four levels of learning evaluation model:
Reaction of learner --> Learning --> Behavior --> Results
Each tends to be exponentially less reliable/dependable
Backwards Planning:
Don Clark's illustration of backwards planning model
In corporate analysis costs money --> user time == $; ROI becomes a key argument: does training (including analysis) result in savings/increased sales etc
JMS: to be fair, though, students have a time budget as well, measured often based on credit hours, contact hours, margin power load -- a class session
Wiley: classroom instruction lets us make a lot of assumptions, whereas corporate requires more concrete measures of need, roi, etc.
Front-End Analysis
http://classweb.gmu.edu/ndabbagh/Resources/Resources2/FrontEnd.htm
- Performance
- Environmental
- Learner
- Needs
Performance analysis quadrant
http://www.nwlink.com/~donclark/hrd/isd/analyze_system.html
| Motivation | Resources or Environment |
| Selection | Training |
Outcome mapping
- Outcome mapping: building learning and reflection into development programs
By Sarah Earl, Fred Carden, Michael Quinn Patton, Terry Smut - The Graphic Syllabus and the Outcomes Map: Communicating Your Course by Linda B. Nilson
gap analysis
where learners are, where we want them to be (difference)
//the constraint is the learner's prior knowledge,
e.g. pre-assessment of objectives
audience analysis
prior knowledge, past experience, sociocultural, demography
content analysis
domain, knowledge, learning, skills
task analysis
performance, behavior, actions, responsibilities
pragmatic
contract and negotiations
e.g. resources (access to a SME, SME's responsiveness, writer), budget, timelines (this deadline will slip without penalty to me IF...), etc.
TED Talk
Students and Gorilla
perceptual blindness attention blindness
Chabris and Simons 1999
What are we primed to pay attention to? What is distracting?
What do we anticipate, expect?
Advanced organizers: tell students what to expect, outlines material to activate knowledge
Does this encourage students to ignore possibly important info, or to filter out distraction
Bobby McFerrin TED Talk
Shows how fast learners can detect pattern and anticipate
A clearly pleasurable experience for participants, requires almost no prior knowledge, universal, motivating
Also hints at sheep mentality of people; we all follow along without thinking about it, without even being asked. We are easily led and controlled. we are suckers for pleasure, participation, inclusiveness, simple rewards. True, this is a harmless participatory event, requiring little judgement.
Doug: willing participants
JMS: But no more than instrument is to a musician; McFerrin is the artist, the musician. He's improvising, the audience is following.
Wiley: Same as a conducter and an orchestra or chorus. Members are controlled, etc.
Doug: I like the reptilian nature of (the pointing to self, follow me without words)
Development
Analysis paralysis = analyzing too much (ref. DoD analysis which often results in inches thick.
Do ID models follow software dev models by about 10 years?
Agile development is adaptive (as opposed to predictive, prescriptive), features direct and frequent interaction with client, allows for change, encourages f2f co-location
Doug & Wiley: Agile and OSS development models are a rejection of or reaction to ADDIE type models.
(JMS: Unless ADDIE Is not a prescriptive model, but a flexible schema, e.g. applied not on a macro (project) level, but a micro (task) level.)
IPT 564 Notes 07-15-2010
Learner certitude/confidence (motivation)
Capacity of decisions across layers (deisgn)
Matching feedback types to situation
1. Progress feedback, goal orientation (feedback)
2. Take advantage of dual-coding when relevant (cognition)
3. Manage cognitive overload, avoid split attention (cognition)
4. Chunking 7+/-2 (cognition)
5. Analysis drives design choices; alignment (design)
6. Usability of design (design); Contiguity effect
7. Exploit openess to maximize access to content, minimize user effort on generation, share and compare results (design, learner control)
8. Utilize spacing effect, lag (practice)
9. Maximize ALT - academic learning time (practice)
11. Expanded reshearsal schedule (Leitner) (practice)
12. Timing of feedback (feedback)
13. Structure +/- low level students; (13 & 11) prior knowledge; Tracking (see 1, 13, 11) (design)
14. ARCS - Attention, relevance, confidence, satisfaction (motivation) (design)
15. Learner control, allow for, Learner as agent
Increase channel capacity (cognition)
==Design==
Prototyping (more than 1) (design)
Bloom's taxononmy (design)
Reflection of environment (design)
Tutoring
2-sigma, more knowledgeable other, vygotsky's ZPD
The List
"a list of principles for effectively and efficiently teaching facts and a supporting rubric for "
5. (things from paper, with page number)
Analysis drives design choice
prior knowledge
openness
The Program...
#* Determines learner's prior knowledge. (summative)
#** Pre-assesses learner's prior knowledge to determine topics and items (summative)
#**Pre-assessment may be the first iteration of Leitner system. Prior knowledge can inform second iteration of Leitner system repetitions. (Wikipedia, Leitner system)
#* Consider prior knowledge to determine type of feedback.
#** (Mason & Bruning 15-16, 21) (high = topic contingent; low = response contingent; requires categorizing of learners according to pre-assessment, or granular correlation of practice items with pre-assessment items)
#* Conduct audience analysis to ensure objectives meet learner needs, content complements learner prior knowledge, and design meets learner abilities/preferences (design)
#**(ADDIE)
#* Determines learner's motivation profile, learning goals, modifiable influences, etc. (summative)
#**(Keller 3-6 ARCS)
#* Facilitates open content
#** Allows for user-generated content.
#** Allows user sharing of content.
#** Allows user sharing of results.
#** Allows user comparions of results.
===Notes===
Mason & Bruning. Providing Feedback in Computer-Based Instruction: What the Research Tells Us
Considering the students' level of prior knowledge allows the program to effectively implement verification or elaboration (15)
Feedback must go beyond "simple judgment of the correctness of their response" or "answer-until-correct".
"If students have relatively little prior knowledge, knowledge-of-correct-response with response-contingent feedback will help them identify the correct answer and will assist them in determining why the selected answer was incorrect. ... [it] allows them to concentrate on the specific erorr as opposed to more general misunderstandings. On the other hand if students have a solid knowledge base, knowledge-of-correct-respons with topic-contingent feedback not only will identify the correct answer but allow them to review relevant material so that they may determine where their error was made." (15)
"To the extent that students possess relevant prior knowledge, they are more likely to make effective use of the instructional material since they have a base of information from which to relate and apply the information." (15)
"Higher achieving students possessing larger knowledge bases may benefit more from feedback providing general information and allowing them to reevaluate their own answers." (16)
Degree of prior knowledge may lead to different types of feedback or levels of evaluation: Low prior knowledge should be followed by knowledge-of-correct-response with response-contingent; high prior knowledge should be followed by knowledge-of-correct-response with topic-contingent. (21)
==Feedback==
Tie feedback and relate responses to learner's goals.
Periodic feedback based on tracked data correlated to learner goals.
System allows
IPT 564 Notes 07-13-2010
Lessig presentation - possible rule-breaking in multimedia
Delivery media
Presentation modes
sensory modality
Generative theory
select (relevant info), organize, integrate
Long history of media comparison studies. To pinpoint effectiveness as a quality of a broad delivery media category is inaccurate (e.g. online vs traditional; pen vs paper). (JS: We can, I think, identify traits of new delivery media that support learning, and certain delivery media may have traits that are absent in other media.)
JS: Aren't there traits of media that facilitate different modes of learning? Aren't traits of media qualifiable, that is, of different value? Doesn't that give us something like justification for media comparison for effectiveness.
What about novelty, accustomed?
multimedia aside
dual coding, parellel processing, cognitive load, and multimedia
note to self: discover if dual coding theory (that verbal + nonverbal facilitates learning better than one) is negatively effected by multimedia combinations that increase cognitive load.
On dual coding theory: "The visual-spatial system uses mental images as the primary representational code, while the verbal system uses speech as the primary code. ... every object and concept has a verbal label in verbal memory, whereas not every object or concept has an imaginal label in visual-spatial memory" Gregory Shaw - Knowledge Representation
Also (from Wikipedia): "The nonverbal system is hypothesized to have developed earlier in evolution. Both systems rely on different areas of the brain. Paivio has reported evidence that nonverbal, visual images are processed more efficiently and are approximately twice as memorable. Additionally, the verbal and nonverbal systems are additive, so one can improve memory by using both types of information during learning."
A split-attention effect in multimedia learning
Marja: 314, is it a problem of contigency or cognitive load? working memory vs. "upload".
Is this a question about "funnels" or "capacity"? I suppose it's still capacity, because though we can see a lot of things at the same time, we can't attend to it all at the same time.
Would the same study be better labeled an efficiency study? Perhaps, and so an alternate study would be to keep time as a variable and allow repetition.
IPT 564 Notes: 06-22-2010
What is instruction?
(general class brainstorm - fun, but a bit frustrating)
event, resource, communicate, information,
presentation
acquisition of skills
kind of teaching
process
group of steps
goal
JS: the act of distributing formative experiences; instruction must be formalized; learning may not need instruction
What is learning?
development, reflexive, experimentation
JS: formative experiences, mental change - biological/behaviorial
skill: automaticity, fluency - degree of learning
JS: *individual? new? previously unavailable, inaccessible (relearning and learning)
forgetting
transfer: move learning from one context to another
Tyler: let's exclude instinct, mapped into genes, developmental (JS: But what if biological mental capabilities are merely recycled for new approaches)
memorization of facts, processes, procedures
kinds of learning vs. degrees of learning
Reigluth: instruction intends to make learning more 1 efficient, 2 effective, 3 appealing (enjoyable)
Learning & forgetting
Learner -- Environment : if this feedback loop breaks learning stops
probe & response
is attention probing? can probes be as simple and passive as interest?
JS: Is an aesthetic experience learning? related to learning? similar context as described by Wiley. In brief, I'd venture to say learning can be an aesthetic experience and vice versa, but there's more there there. Then bring in learning as a path towards truth, Poe's heresy of the didactive, the soul of poetry is in seeking for truth? or may we have poetry for the sake of poetry? poetry without objectives. Plato's theory of forms, imitation of the original, applicable to crafts, arts, but what about design?
Design is choices made under contraints (Wiley), design is the pre-product
JS: the goal of art is beauty; the goal of instruction is change (in behavior, in biology)
"Help: Toward a New Ethics-Centered Paradigm for Instructional Design and Technology" - Inouye
The purpose of ID is to help others; we must focus on this ethical objective in our practice. Also, ID is not episteme as much as poeisis, a thing whos existence is contingent rather than necessary.
Bloom's "2-Sigma Problem"
Students could learn at a much higher rate if effective instruction is employed.
Also, Bloom's Taxonomy
Marja: Levels of learning, types of learning
Doug: It implies effectiveness
David: Remembering gets a bad rap
Tyler: Levels build upon each other
JS: This page's descriptions are examples, not definitions
Notes on Eval Environment Set Up – Timez Attack at Elementary School
Rob and I set up the lab environment for the evaluation of Timez Attack this a.m. A few notes:
Lethargy of computers inhibited set up more than we expected, as did a few problematic computers. We were glad to have a small margin (about 10%) of extra computers available for each class group.
Having James the Big Brainz tech was invaluable, as he was able to not only show us how to set up swiftly, but answered questions about level selection and programming student lists.
The Teacher account can set up a class list. The Student account just plays.
Net connection needed to track data remotely.
In others' lab environments, ask if remote access or networked installation of software is available.
Students may benefit from a cheat sheet/quick ref card.
Because time to load from name selection can be as much as 3 min, Rob and I decided to enter student names ahead of time, then preload the game after selecting the student just prior to the evaluation.
Quick ref card could then include a tutorial.
Time spent to set up was approximately 2 hours (I had to leave after 1:15).
Anticipate that we will need 30 minutes to set up each class (start software, choose name, establish functionality) immediately prior to eval.
IPT 661 Notes: 5-13-2010
norm referenced testing : vs others
criterion referenced testing : vs standards
domain referenced
alternate approach
potentially disruptive
==Client discussion==
looking for signficant differences
interested in numbers -- more efficient
==What to evaluate?==
signficant differences (but we expect it will be hard to get)
look also at
satisfaction
confidence
"We have cured times tables." -- what's the malady?
Wasatch 9th E
8am at school library
May 18
9am mtg in class
==Computer lab from 2:30 - 3:15==
==Library lab from 1:30 - 3:15==
split Marie's class
===Preparation====
Instruments, scripts, student lists, nametags(?), software installation, pretests
===Procedure: Data Collection & Observation===
Pretest (to be administered by teacher?)
* Nametags
Lab startup
Physical observations
Sketch environment, details on people, work stations
Interview questions (as students leave)
Posttest (to be administered by teacher?)
IPT 661 Notes: 5-11-2010
Evaluation Planning and Questions
accurate (ethical) representation of findings
example of school district that failed to use fed funds appropriately, yet gleaned part of the eval that /was/ positive.
Darin suggests there may be whistleblower protection by the fed funding provider
Questions to Ask the Client
objective of the evaluation?
what do you hope to get out of this evaluation? what do you hope to do with the results?
objective of the software?
how is it suppsed to work?
software features:
logs and analytics
versioning
(don't have to go to IRB to collect data because we won't publish it), however should warn client. We should always try to get IRB, because you never know when you'll get something interesting.
"No eval should be conducted without a detailed description of the program being evaluated." e.g. "team teaching" but we don't know if it's team teaching
"[understand] why the program is supposed to achieve its desired impacts"
1. meet
2. describe program
3. share with description of program
3 ways to gather info
review documentation
talk to stakeholders
observe object/program
evaluand may change as you go, e.g. formativeness of feedback
(Rob = point of contact, projet mgr?)
Describe Timez Attack in a Logic Model
, user interface
IPT 661 Notes: 4-27-2010
http://www.byuipt.net/wests/rick/teaching/program-evaluation/
evaluation asks you to apply your skill set to a number of situations
Goals
* Understand eval foundations, issues, and approaches
* Acquire some practical strategies
* Complete an actual evaluation successfully
In 2 months we will have an eval done. Cool, but would prefer to try one on my own for the full effect rather than work as a cog in a group.
Role of evaluation in society?
determine effectiveness, refine, revise, progress, wise judgement, maximize effectiveness of choices --> save money
research is to explore
evaluation /has/ to be applied
evaluators are change agents, must convince people to adopt recommendations based on eval findings
--recommendations and solutions
interpersonal - eval as judgement vs as learning opportunity
different evaluation methods, e.g. participant evaluation,
summative evaluation after the fact (e.g. after a problem is exposed, disaster occurs)
evaluation must be self-reflective, meta-evaluation
Research v Evaluation
Who sets agenda?
Are results context-specific or generalizable/transferable?
What are the criteria being used?
What are the skill sets? researcher crunches numbers, interviews, writes; evaluators need to be interpersonal, teach
Research
"Does this work?" positivistic, quantitative
"What does it mean?" qualitative
Evaluators
"Does it work for this group A in a way that is valuable?"
method (research) ------------- intuition (informal evaluation)
formal evaluation
Goals and Roles
*Program improvement (e.g. grants)
*Oversight and compliance (e.g. grants)
*Assessment of worth
*Knowledge development
*Social betterment
formative v summative
cook tastes the soup v. guest tastes the soup
Relative emphasis to program life, formative to summative
Purpose? f: to improve; s: judge
Use? f: encourages immediate application; s: promotes future directions
Audience? f: practioners; s: decision-makers
By whom? f: practitioners; s: external, objective evaluator
Data collection? f: qualitative; s:
Sample?
Questions? f: what can be improved?; s: is this valuable? what is worth? should it persist?
Internal v external - which is better and why?
internal is subject to social pressure; external less so
internal may be "cheaper"
what evalutions? to stop by for a bagel? did I go to bed early enough? should I be in this course? will I be able to apply this course to my work?
Big Brainz
Education Engine
mastery learning
testing with Wasatch Elementary
what do they want?
prove it's awesome
attitude, pre-assessment post-assessment
not a comparable program, product? could be
vs. traditional teaching
mixing unrelated skill with targeted skill
distracting
Paradigms
Objectives-oriented Did you meet your goals?
Management-oriented Is it profitable? What decision should you make?
Goal-free What can we deduce from your process?
Consumer-oriented Which product should you use?
Interpretivist-oriented What does it mean from their perspective?
Participant-focused How can we make a difference by involving the participants?
Expertise-oriented What would an expert think about this?
Empowerment How can we teach you to fish?
Management
30-45 minutes
Why is paradigm exploration important?
Evaluation Type
Philosophy
Model
Questions
Methods
/narrowing the focus/
Ideas for exploration
evaluate online course development process and practices (objectives, management)
evaluate lms features (product)
evaluate effectivness of
learning
teaching
practices
strategies
that
reflection
evaluate the impact of Grand Poobah on TTIX participants
evaluate the accuracy, effectiveness, appropriateness of UVU DE's ITR measure
Recent Posts
- Design: FightCard Factual Knowledge Practice System
- IPT 564 Notes 07-29-2010
- Charles Leadbeater’s TED Talk on Innovation
- IPT 564 Notes 07-27-2010
Archives
Categories
Tags
jaredstein.org
- Weekly Notes for 2010-09-05 September 6, 2010
- Weekly Notes for 2010-08-29 September 6, 2010
- Weekly Notes for 2010-08-22 September 6, 2010
- USA Today College Looks at Creepy Treehouse (Sort Of) September 6, 2010
- Review: Nixty.com September 6, 2010
Skate > Reanimate
- Skatelog Sunday, 09-05-2010 September 6, 2010
- Skatelog Saturday, 09-04-2010 September 6, 2010
- Skatelog Friday, 09-03-2010 September 6, 2010
- Skatelog Thursday, 09-02-2010 September 6, 2010
- Skatelog Wednesday, 09-01-2010 September 6, 2010