ecological validity: not reading in the natural way(?) because of hardware, hawthorne effect, [in a distracting environment]

has anyone written about the problem of eye tracking set ups on users?

Writing
more sign posting?

literature review needs to call for the technology. why is this technology better than screen capture?

Outline

* set of understood reading behaviors – what is deep reading? how is it different from scanning skimming?
* we need to be able to measure these because (???) reading behaviors for learning are important to study– classifying and importance;
* these have primarily been studied in web design and usability context ? need to study for learning and comprehension contexts ? these haven’t been studied rigorously and extensively in online context ? need to study when user goal is learning vs info finding and task completion —
* here are the available tools & methods
* here are their strengths and limitations (people who have talked about limitations)
** has lack of access to tracking equipment inhibited studies?
** do current methods of tracking allow for ecologically valid studies (e.g. hardware, hawthorne, open web)
* let’s explore an alternative: mouse tracking – cost is lower, ecological validity is higher,
** State assumption/have to argue that eye tracking is accurate indicator of reading, if interface-tracking correlates to eye tracking, it may indicate certain aspects of reading behavior
what about verbal reporting/think alouds?
what about comprehension tests?

methods section –> details of eye tracking and how it will be done!!!

Lit review alternative

? what strategies do people use in online context to get at reading behaviors [skimming, scanning, reading]? (lit review) what are limitations and strengths?

reading behaviors as identified in literature (reading science, usability, L2)
reading for comprehension we call deep reading

To Do

Read literature on reading behaviors/strategies from reading science, L2, web usability
Read literature on tracking methods
Rework as lit review on tracking reading strategies

This is to justify training/educational experiences.

JMS: Sometimes this works this way, sometimes it doesn’t. Sometimes the outcomes are implicit, invisible. Power of alignment: Cohen 1987, Wishnick 1989

Jacob: Because I’ve never seen a perfect assessment (or even a really great one) it seems basing content on assessment rather than outcomes risks cutting the actual outcomes short, by failing to provide the appropriate content and activities. The idea is that even a good assessment won’t do justice to necessary instruction.

Wiley: Faculty adopts a textbook (JMS: but based on outcomes, right?) which may come with assessment

JMS: In k-12 but in higher ed I think faculty don’t commonly base outcomes and assessments on the content (textbook); rather they implicitly know the outcomes and choose the content based on that, presuming that the textbook publisher did the hard work of aligning the assessment with the content.

Mager

ABCD

From Mager’s Tips on Instructional Objectives

Audience

The who. Your objectives had better say, “The student will be able to…”

Behavior

An objective always says what a learner is expected to be able to do. The objective sometimes describes the product or result of the doing.

Ask yourself, what is the learner doing when demonstrating achievement of the objective?

Condition

An objective always describes the important conditions (if any) under which the performance is to occur.

Degree

Wherever possible, an objective describes the criterion of acceptable performance by describing how well the learner must perform in order to be considered acceptable.

Work Model Synthesis
systematically combining and recombining tasks and objectives that through task analysis procedures have been fragmented at a low level”

Learning Analytics

What impact might Google Analytics-style anayltics have for teaching and learning?
Students hate it (accountability vs. Big Brother)
Technology in education now lets us diagnose learners and provide laser-like feedback.

Business

Tyler built a spreadsheet that allows rubric items, score, and points
David cancelled reverse engineering

Analysis

Gibbons: If design is goal oriented choices made/process under constraints,
analysis is understanding those constraints

Analysis is an investment; it pays dividends later

Cost of change increases as time goes on (JMS: analogous to a toppling dominoes set-up; complex and elaborate set ups become more difficult to change as the set-up proceeds)

four levels of Donald Kirkpatrick’s four levels of learning evaluation model:
Reaction of learner --> Learning --> Behavior --> Results
Each tends to be exponentially less reliable/dependable

Backwards Planning:

Don Clark’s illustration of backwards planning model

In corporate analysis costs money --> user time == $; ROI becomes a key argument: does training (including analysis) result in savings/increased sales etc
JMS: to be fair, though, students have a time budget as well, measured often based on credit hours, contact hours, margin power load — a class session
Wiley: classroom instruction lets us make a lot of assumptions, whereas corporate requires more concrete measures of need, roi, etc.

Front-End Analysis

http://classweb.gmu.edu/ndabbagh/Resources/Resources2/FrontEnd.htm

• Performance
• Environmental
• Learner
• Needs

Performance analysis quadrant

http://www.nwlink.com/~donclark/hrd/isd/analyze_system.html

Outcome mapping

• Outcome mapping: building learning and reflection into development programs
  By Sarah Earl, Fred Carden, Michael Quinn Patton, Terry Smut
• The Graphic Syllabus and the Outcomes Map: Communicating Your Course by Linda B. Nilson

gap analysis

where learners are, where we want them to be (difference)
//the constraint is the learner’s prior knowledge,
e.g. pre-assessment of objectives

audience analysis

prior knowledge, past experience, sociocultural, demography

content analysis

domain, knowledge, learning, skills

task analysis

performance, behavior, actions, responsibilities

pragmatic

contract and negotiations
e.g. resources (access to a SME, SME’s responsiveness, writer), budget, timelines (this deadline will slip without penalty to me IF…), etc.

TED Talk

Students and Gorilla
perceptual blindness attention blindness
Chabris and Simons 1999

www.youtube.com/watch?v=vJG698U2Mvo

Advanced organizers: tell students what to expect, outlines material to activate knowledge
Does this encourage students to ignore possibly important info, or to filter out distraction

Bobby McFerrin TED Talk

Shows how fast learners can detect pattern and anticipate
A clearly pleasurable experience for participants, requires almost no prior knowledge, universal, motivating
Also hints at sheep mentality of people; we all follow along without thinking about it, without even being asked. We are easily led and controlled. we are suckers for pleasure, participation, inclusiveness, simple rewards. True, this is a harmless participatory event, requiring little judgement.
Doug: willing participants
JMS: But no more than instrument is to a musician; McFerrin is the artist, the musician. He’s improvising, the audience is following.
Wiley: Same as a conducter and an orchestra or chorus. Members are controlled, etc.
Doug: I like the reptilian nature of (the pointing to self, follow me without words)

Development

Analysis paralysis = analyzing too much (ref. DoD analysis which often results in inches thick.

Do ID models follow software dev models by about 10 years?
Agile development is adaptive (as opposed to predictive, prescriptive), features direct and frequent interaction with client, allows for change, encourages f2f co-location

Doug & Wiley: Agile and OSS development models are a rejection of or reaction to ADDIE type models.
(JMS: Unless ADDIE Is not a prescriptive model, but a flexible schema, e.g. applied not on a macro (project) level, but a micro (task) level.)

Matching feedback types to situation

1. Progress feedback, goal orientation (feedback)
2. Take advantage of dual-coding when relevant (cognition)
3. Manage cognitive overload, avoid split attention (cognition)
4. Chunking 7+/-2 (cognition)
5. Analysis drives design choices; alignment (design)
6. Usability of design (design); Contiguity effect
7. Exploit openess to maximize access to content, minimize user effort on generation, share and compare results (design, learner control)
8. Utilize spacing effect, lag (practice)
9. Maximize ALT – academic learning time (practice)
11. Expanded reshearsal schedule (Leitner) (practice)
12. Timing of feedback (feedback)
13. Structure +/- low level students; (13 & 11) prior knowledge; Tracking (see 1, 13, 11) (design)
14. ARCS – Attention, relevance, confidence, satisfaction (motivation) (design)
15. Learner control, allow for, Learner as agent

Increase channel capacity (cognition)

==Design==
Prototyping (more than 1) (design)
Bloom’s taxononmy (design)
Reflection of environment (design)

Tutoring
2-sigma, more knowledgeable other, vygotsky’s ZPD

The List
“a list of principles for effectively and efficiently teaching facts and a supporting rubric for ”

5. (things from paper, with page number)
Analysis drives design choice
prior knowledge
openness

The Program…
#* Determines learner’s prior knowledge. (summative)
#** Pre-assesses learner’s prior knowledge to determine topics and items (summative)
#**Pre-assessment may be the first iteration of Leitner system. Prior knowledge can inform second iteration of Leitner system repetitions. (Wikipedia, Leitner system)

#* Consider prior knowledge to determine type of feedback.
#** (Mason & Bruning 15-16, 21) (high = topic contingent; low = response contingent; requires categorizing of learners according to pre-assessment, or granular correlation of practice items with pre-assessment items)

#* Conduct audience analysis to ensure objectives meet learner needs, content complements learner prior knowledge, and design meets learner abilities/preferences (design)
#**(ADDIE)

#* Determines learner’s motivation profile, learning goals, modifiable influences, etc. (summative)
#**(Keller 3-6 ARCS)

#* Facilitates open content
#** Allows for user-generated content.
#** Allows user sharing of content.
#** Allows user sharing of results.
#** Allows user comparions of results.

===Notes===

Mason & Bruning. Providing Feedback in Computer-Based Instruction: What the Research Tells Us

Considering the students’ level of prior knowledge allows the program to effectively implement verification or elaboration (15)

Feedback must go beyond “simple judgment of the correctness of their response” or “answer-until-correct”.

“If students have relatively little prior knowledge, knowledge-of-correct-response with response-contingent feedback will help them identify the correct answer and will assist them in determining why the selected answer was incorrect. … [it] allows them to concentrate on the specific erorr as opposed to more general misunderstandings. On the other hand if students have a solid knowledge base, knowledge-of-correct-respons with topic-contingent feedback not only will identify the correct answer but allow them to review relevant material so that they may determine where their error was made.” (15)

“To the extent that students possess relevant prior knowledge, they are more likely to make effective use of the instructional material since they have a base of information from which to relate and apply the information.” (15)

“Higher achieving students possessing larger knowledge bases may benefit more from feedback providing general information and allowing them to reevaluate their own answers.” (16)

Degree of prior knowledge may lead to different types of feedback or levels of evaluation: Low prior knowledge should be followed by knowledge-of-correct-response with response-contingent; high prior knowledge should be followed by knowledge-of-correct-response with topic-contingent. (21)

==Feedback==
Tie feedback and relate responses to learner’s goals.

Periodic feedback based on tracked data correlated to learner goals.

System allows

Delivery media
Presentation modes
sensory modality

Generative theory
select (relevant info), organize, integrate

Long history of media comparison studies. To pinpoint effectiveness as a quality of a broad delivery media category is inaccurate (e.g. online vs traditional; pen vs paper). (JS: We can, I think, identify traits of new delivery media that support learning, and certain delivery media may have traits that are absent in other media.)

JS: Aren’t there traits of media that facilitate different modes of learning? Aren’t traits of media qualifiable, that is, of different value? Doesn’t that give us something like justification for media comparison for effectiveness.

What about novelty, accustomed?

multimedia aside

dual coding, parellel processing, cognitive load, and multimedia
note to self: discover if dual coding theory (that verbal + nonverbal facilitates learning better than one) is negatively effected by multimedia combinations that increase cognitive load.

On dual coding theory: “The visual-spatial system uses mental images as the primary representational code, while the verbal system uses speech as the primary code. … every object and concept has a verbal label in verbal memory, whereas not every object or concept has an imaginal label in visual-spatial memory” Gregory Shaw – Knowledge Representation

Also (from Wikipedia): “The nonverbal system is hypothesized to have developed earlier in evolution. Both systems rely on different areas of the brain. Paivio has reported evidence that nonverbal, visual images are processed more efficiently and are approximately twice as memorable. Additionally, the verbal and nonverbal systems are additive, so one can improve memory by using both types of information during learning.”

A split-attention effect in multimedia learning
Marja: 314, is it a problem of contigency or cognitive load? working memory vs. “upload”.

Is this a question about “funnels” or “capacity”? I suppose it’s still capacity, because though we can see a lot of things at the same time, we can’t attend to it all at the same time.

Would the same study be better labeled an efficiency study? Perhaps, and so an alternate study would be to keep time as a variable and allow repetition.

What is learning?
development, reflexive, experimentation
JS: formative experiences, mental change – biological/behaviorial
skill: automaticity, fluency – degree of learning
JS: *individual? new? previously unavailable, inaccessible (relearning and learning)
forgetting
transfer: move learning from one context to another

Tyler: let’s exclude instinct, mapped into genes, developmental (JS: But what if biological mental capabilities are merely recycled for new approaches)
memorization of facts, processes, procedures
kinds of learning vs. degrees of learning
Reigluth: instruction intends to make learning more 1 efficient, 2 effective, 3 appealing (enjoyable)
Learning & forgetting
Learner — Environment : if this feedback loop breaks learning stops
probe & response
is attention probing? can probes be as simple and passive as interest?

JS: Is an aesthetic experience learning? related to learning? similar context as described by Wiley. In brief, I’d venture to say learning can be an aesthetic experience and vice versa, but there’s more there there. Then bring in learning as a path towards truth, Poe’s heresy of the didactive, the soul of poetry is in seeking for truth? or may we have poetry for the sake of poetry? poetry without objectives. Plato’s theory of forms, imitation of the original, applicable to crafts, arts, but what about design?

Design is choices made under contraints (Wiley), design is the pre-product
JS: the goal of art is beauty; the goal of instruction is change (in behavior, in biology)

“Help: Toward a New Ethics-Centered Paradigm for Instructional Design and Technology” – Inouye
The purpose of ID is to help others; we must focus on this ethical objective in our practice. Also, ID is not episteme as much as poeisis, a thing whos existence is contingent rather than necessary.

Bloom’s “2-Sigma Problem”
Students could learn at a much higher rate if effective instruction is employed.
Also, Bloom’s Taxonomy
Marja: Levels of learning, types of learning
Doug: It implies effectiveness
David: Remembering gets a bad rap
Tyler: Levels build upon each other
JS: This page’s descriptions are examples, not definitions

This post is password protected. To view it please enter your password below:

Password:

Lethargy of computers inhibited set up more than we expected, as did a few problematic computers. We were glad to have a small margin (about 10%) of extra computers available for each class group.

Having James the Big Brainz tech was invaluable, as he was able to not only show us how to set up swiftly, but answered questions about level selection and programming student lists.

The Teacher account can set up a class list. The Student account just plays.

Net connection needed to track data remotely.

In others’ lab environments, ask if remote access or networked installation of software is available.

Students may benefit from a cheat sheet/quick ref card.

Because time to load from name selection can be as much as 3 min, Rob and I decided to enter student names ahead of time, then preload the game after selecting the student just prior to the evaluation.

Quick ref card could then include a tutorial.

Time spent to set up was approximately 2 hours (I had to leave after 1:15).

Anticipate that we will need 30 minutes to set up each class (start software, choose name, establish functionality) immediately prior to eval.

alternate approach
potentially disruptive

==Client discussion==
looking for signficant differences
interested in numbers — more efficient

==What to evaluate?==
signficant differences (but we expect it will be hard to get)
look also at
satisfaction
confidence
“We have cured times tables.” — what’s the malady?

Wasatch 9th E
8am at school library

May 18
9am mtg in class
==Computer lab from 2:30 – 3:15==
==Library lab from 1:30 – 3:15==
split Marie’s class

===Preparation====
Instruments, scripts, student lists, nametags(?), software installation, pretests

===Procedure: Data Collection & Observation===
Pretest (to be administered by teacher?)
* Nametags
Lab startup
Physical observations
Sketch environment, details on people, work stations
Interview questions (as students leave)
Posttest (to be administered by teacher?)

accurate (ethical) representation of findings

example of school district that failed to use fed funds appropriately, yet gleaned part of the eval that /was/ positive.

Darin suggests there may be whistleblower protection by the fed funding provider

Questions to Ask the Client

objective of the evaluation?

what do you hope to get out of this evaluation? what do you hope to do with the results?

objective of the software?

how is it suppsed to work?

software features:

logs and analytics

versioning

(don’t have to go to IRB to collect data because we won’t publish it), however should warn client. We should always try to get IRB, because you never know when you’ll get something interesting.

“No eval should be conducted without a detailed description of the program being evaluated.” e.g. “team teaching” but we don’t know if it’s team teaching

“[understand] why the program is supposed to achieve its desired impacts”

1. meet

2. describe program

3. share with description of program

3 ways to gather info

review documentation

talk to stakeholders

observe object/program

evaluand may change as you go, e.g. formativeness of feedback

(Rob = point of contact, projet mgr?)

Describe Timez Attack in a Logic Model
, user interface