learn.5tein.com Jared Stein's grad-school-community blog on teaching and learning.

3 Articles Orbiting Bloom’s 2 Sigma Problem

I've posted these annotations to the class's Google Doc for Jon Mott's IPT 692R course, but wanted to archive them here as well. These 3 article annotations seemed relevant in the discussion of Bloom's 2 sigma problem:

Cohen, A. (1987). Instructional Alignment: Searching for a Magic Bullet. Educational Researcher, 16:8, 16-20.

Cohen reviews and expands on investigations into the effect on learning outcomes of instructional alignment. Cohen explains the history of instructional alignment, going back to the 60s, and notes that though "teaching what we assess, or assessing what we teaching seems embarrassingly obvious"(19) the fact that precise instructional alignment results in better learning outcomes has often been ignored or disdained or misunderstood. Testing whether the alignment effect is as large as it looks ("approximately four times the norm"), Cohen reviews several new studies. The Koczor Study (1984) showed that instructional alignment vs misalignment provided "effect sizes ... for the lower and average aptitude students were as high as 1.10 and 2.74 sigma". The Tallarico Study (1984) showed that lower achievers average score exceeded the 85th percentile of a placebo group, equating to a 1.3 sigma effect. The Fahey study (1986) found that alignment effect increased as students moved from easy to difficult tasks; also, higher aptitude students performed better than lower aptitude students on misaligned items; finally, lower aptitude students performed higher on aligned items than did the higher aptitude students on the misaligned items, with an effect size of 1.2 sigma ("For low achievers, a little alignment goes a long way."). The Elia Study (1986) reported, overall, an alignment effect of 0.91sigma, though in the "phrase condition" it reached 1.76 sigma.

Comments:
Instructional alignment appears to be absent from Bloom's initial consideration in the 2 sigma problem. Here, Cohen shows it's importance by reviewing contemporary research studies--especially for low achievers. That the research studies often showed disparate effects for different conditions and learners implies the complexity of the 2 sigma problem, and perhaps indicts Bloom's willingness to generalize results.

Aleven, V, Koedinger, K. (2002). An effective metacognitive strategy: learning by doing and explaining with a computer-based Cognitive Tutor. Cognitive Science 26, 147-179.

Using a computer software called Cognitive Tutor for instruction and assessment of high school geometry, the researchers compared pre-test and post-test performance of two groups of students; the experimental group was required to provide an explanation for their answers--otherwise conditions were the same. Experiment 1 found that the explanation students spent more time on task, and improved more on their post-test scores than the control. Experiment 2 controlled for time on task, but the results still suggested that the explanation group performed better on the post-test, and "learned better to explain their steps" (162). The researchers investigated issues od deep learning, and found that the explanation group performed better on "harder-to-guess" items, and "more likely to reflect on the sufficiency of their knowledge, and may have achieved better transfer of skills. Researchers' conclusion: by engaging in the metacognitive strategy of explanation "students acquired better-integrated visual and verbal declarative knowledge and acquired less shallow procedural knowledge".

Comments:
first, it was amazing to discover the specificity with which these researchers considered their experiment and executed it. Their description outweighs most others I have read on similar subjects. I believe this comes from their backgrounds in cognitivism, as they seem to be seeking to pinpoint domains as well as models/structures in order to be more accurate in their experiment and results. This made me wonder about other empirical research which, at least in reporting, includes less description and specificity. Second, though the researchers' discussion of their results made sense to me, I was not familiar enough with their statistical methods to be able to fully comprehend the numbers reported for each of the 2 experiments or relate them to a "sigma" effect. Finally, this article, which targets a metacognitive strategy used by learners, also testifies to the importance of instructional design, and what is essentially an advance in programmed instruction that provides dynamic feedback and resources to the students, suggesting that many of the variables Bloom cites are too entangled or intertwined to isolate and recombine. These researchers' own reference to Bloom is of a "potential" effect conditioned by "highly effective" one-on-one tutoring (they reference another study which had lesser effects from tutoring).

Oestmann, E. & Oestmann, J. (2006). Significant difference in learning outcomes and online class size. Journal of Online Educators, 2(1), 1-8.

This study examines the outcomes of 5 large (20>) and small (<10) online masters level courses to determine if there are significant difference in interactivity and final grades. Contrary to some expectations they found that the average final grade in the large class size was 5% higher than the smaller class size. Also, the quality of discussion forum posts was judged to be greater--more substantial--in the larger class. The researchers interpret this as reflective of Vygotsky's socio-cultural learning theory "in which more opportunities for social interaction resulted in higher measures of learning outcomes"

Comments:
Though this is not directly tied to Bloom's 2 sigma problem, it is related to aim to achieve that 1-1 ideal. This research suggests that in the new online environment large groups matter. This makes sense to me, and reinforces a suspicion I had about the 2 sigma problem's relevance in the face of our changing culture and communication media practices. I have reviewed other investigations of class size in online environments, but this is among the few instances that show a positive correlation to larger class sizes. I suspect this is due to the androgogical implications of studying adult, masters-level students.