Tag Archives: Memory

Less is More: Low-Stakes Assessments and Student Success

Steve Volk, April 16, 2018
Contact at: svolk@oberlin.edu

There is a point in every semester when, it being too late to make significant changes in our current courses, we instead begin to think ahead to next semester. So, overwhelmed as you are by finishing up this term, getting in book orders, and planning research and writing for the summer, maybe you can keep this article in mind as you plan for the fall 2018 semester!

In this posting, I provide a few resources and summarize some of the reasoning supporting the argument that student learning is more enhanced by frequent, low-stakes assessments (“retrieval practices” in the technical lingo), with the opportunities they provide for continual feedback, than from infrequent, high-stakes assessments (e.g., a midterm and a final). While the research has suggested that this holds across all disciplines, the impact is noted especially in STEM fields and quantitatively based courses.

Some Research

First, a bit of the research. Where in the classroom cycle (class-studying-examination) does learning most take place? We generally think that students learn when they are studying (reading assigned texts or reviewing their notes), and that testing is the mechanism that tells us what has been learned. The same assumptions also buttress what many instructors think about classroom practice: students are assumed to learn in lectures (learning which is solidified when they study their notes), and tests help us know how much has been learned. The test, that that sense, is considered a “neutral event,” it measures without impacting learning. And yet, a considerable body of research has demonstrated that such an understanding is flawed; many cognitive psychologists now suggest that assessment practices themselves can produce large gains in long-term retention of information and concepts in comparison with studying for an exam. In other words, write  Peter C. Brown, Henry L. Roediger III, and Mark A. McDaniel in Make It Stick: The Science of Successful Learning (Harvard, 2014), using “retrieval practices” (recalling facts, concepts, data, or events from memory) is “a more effective learning strategy than review by rereading.”

To summarize his (and Andrew C. Butler) conclusions from one of Roediger’s many studies,

  1. Assessments (“retrieval practices”) often produce superior long-term retention relative to studying for the same amount of time;
  2. Repeated assessments are better than one-time testing;
  3. Providing feedback produces better results than assessment practices without feedback, although even without feedback, the learning results are better for multiple assessments;
  4. Some lag between study and test support retention and learning;
  5. The benefits of “retrieval practices” can be transferred from one context (domain) to another.

AFP; sitting for the “bac,” Paris, June 18, 2015

Cynthia J. Brame and Rachel Biel, both at Vanderbilt, offer a useful chart in CBE Life Sciences Education that summarizes the research on the relationship between testing and learning in undergraduate science courses. Among the findings of previous studies that they consider are the following:

  • Testing improved retention significantly more than restudy in delayed tests.
  • Multiple tests provided greater benefit than a single test.
  • Students in the retrieval-practice condition had greater gains in meaningful learning compared with those who used elaborative concept mapping as a learning tool.
  • Both initially correct and incorrect answers were benefited by feedback, but low-confidence answers were most benefited by feedback.
  • Testing improved retention and increased transfer of information from one domain to another through test questions that required factual or conceptual recall and inferential questions that required transfer.
  • Interim testing improved recall on a final test for information taught before and after the interim test.

But first, a note to the doubters – myself included. One of the central concerns I have always had about testing as a method of assessment, particularly high-stakes mid-terms or finals, or even post-reading quizzes, is whether they only promote rote memory learning (“When did Cortez first arrive in Mexico?” “Describe the wing structure of a bat,” etc.). I questioned whether the memorization of facts that (now) can be looked up instantaneously on a smart phone was necessary, or, in any case, whether what students learned would stick with them, or stick with them in a contextualized fashion, other than the memorization of random facts.

We all know that learning is more than the memorization and the repetition of facts, that higher education learning above all must shift students to higher cognitive levels and outcomes. So, while we know, through a variety of studies, particularly the work carried out by Andrew C. Butler, that “repeated testing produce[s] superior retention and transfer on the final test relative to repeated studying,” what do we know about the impact of frequent assessment such as tests and quizzes on helping to move students to higher cognitive levels? Can what was learned via testing be “transferred,” extended to other domains and contexts? And can attention to “retrieval practices” help students better cope with some of the negative consequences that accompany high-stakes testing? Stick around for some answers.

What to Keep in Mind

  1. Frequent, low-stakes testing is better than infrequent, high-stakes testing.

The research cited above suggests the many reasons why this is the case. To put it in practical terms:

Jenna Carter,”Test,” Flickr cc

a. We know enough about high-stakes testing to know that such assessment practices can have serious negative effects on identity-threatened groups in specific contexts: Blacks and Latinx in the sciences; women in math; etc. Further, low-stakes testing reduces the anxiety associated with high-stakes testing since one’s future isn’t riding on one or two tests. And we know that student anxiety in general has reached frightening levels. In 1985,18% of incoming first-year students agreed that they “felt overwhelmed by all I had to do.” In 2010, that figure increased to 29%. Last year, it surged to 41 percent.

b. Frequent testing allows students to practice remembering, a useful capacity building skill. But, to help students not just retain information but to use retrieval practices to move to higher order thinking, the testing should be “effortful” (e.g., short answer rather than multiple choice), spaced (allowing enough time to elapse for some forgetting to occur) and “interleaved” (one should introduce different topics and problems rather than having students master one topic at a time and then move on to the next). (These are points that Brown, Roediger, and McDaniel stress in Make it Stick.)

c. Frequent testing helps students know where they stand in a course, making it more likely that they will seek help when needed, i.e., when it can actually make a difference. Particularly in STEM courses, if students only realize that they are in a very deep pit at the mid-term point, there is little that they can do to climb out.

  1. Feedback is better than no feedback.

While the research suggests that low-stakes testing without feedback is actually better than relying on studying alone in terms of student learning, it also indicates the key role that instructor feedback plays in boosting student learning.

a. Most obviously, feedback from the instructor will let students know where they stand in a course. As faculty, we often have our own ways of intuiting how students are doing in our classes without resorting to formal testing mechanisms. Are they coming to class? Do they look engaged? Are they falling asleep in class? Have they come to office hours when asked? Some of these indicators are fairly accurate; others can be not only wrong but seriously problematic, particularly if they feed off of implicit biases. But the point is that students don’t share these “insights” as to how they are performing in a course. Many will feel that they are doing just fine, when they are actually standing at the edge of a precipice. (A recent study reported on in the New York Times suggests that men, more than women, overestimate their abilities in science classes. Who would have thought!) Concrete feedback from instructors can help students obtain a more objective sense of how they are doing. They may not always “hear” it or respond appropriately to such information, but offering feedback most often is quite welcome and useful.

b. Providing feedback to correct responses as well as incorrect answers is highly useful. As Brame and Biel discuss in the above cited article, feedback on both low-confidence correct answers and incorrect answers may further enhance the testing effect, allowing students to solidify their understanding of concepts about which they are unclear.

c. Feedback can come from instructors on exams or other assessments, or from in-class peers. One research study found that when students answer an in-class conceptual question individually using clickers, discuss it with their neighbors, and then re-vote on the same question, the percentage of correct answers typically increases.

Gary Larson, The Far Side

  1. Low-stakes, frequent testing with feedback helps students think more intentionally about their own learning.

I have written before about ways to support student metacognition. One way to do this is to encourage students to reflect on their own process of learning by having them respond to feedback on exams, quizzes, or other assignments.

a. Students can “self-test” as part of their studying process, and then measure the outcome versus their expectation, but few will actually do this, and generally those who do are not always the ones who could use additional help. Providing concrete evidence of accomplishments can help students more accurately assess their level of comprehension and (hopefully) seek help to address those areas that need attention.

b. Frequent assignments tied to reflection exercises can help students think about what they did (to prepare for an assignment, for exaple), what the outcome was, and what they will do differently the next time.

c. No-stakes formative assessment (a short quiz at the end of the class or a simple “one-minute paper”) can help students think in a more focused fashion about what they know and they need to focus on.

  1. Design questions that can move students to higher order thinking.

While not every exam must be “effortful,” with some thought, one can design multiple choice questions to lead to higher order thinking.

a. Cynthia Brame, from Vanderbilt’s Center for Teaching, provides the following advice: Presenting a problem that requires application of course principles, analysis of a problem, or evaluation of alternatives tests students’ ability to engage in higher-order thinking. Instructors can design problems that require multilogical thinking (“thinking that requires knowledge of more than one fact to logically and systematically apply concepts to a …problem”).

Here are two examples she provides:

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

b. In Dynamic Testing: The Nature and Measurement of Learning Potential , Robert Sternberg and Elena Grigorenko suggest, as the title indicates, a more dynamic approach to testing than what is typically provided in static, standardized tests. Essentially, this calls for determining the state of one’s knowledge; refocusing learning on areas that need more attention; and retesting to measure improvement. It is testing that helps the learner focus on new knowledge (i.e., on “learning”), rather than on what has already been accomplished. In this sense, well-designed testing can move the student to higher cognitive levels.

  1. Break large assignments/exams into smaller pieces

Designing, for example, eight assessments tools that are effortful, spaced, and interleaved, takes some work, particularly when you are accustomed to giving a midterm and a final. True enough, but think about breaking the high-stake assignments you already give into smaller pieces, as suggested by Sara Jones at Michigan State University’s “Inside Teaching.”

a. Replace high-stakes, large tests into several quizzes.

b. Scaffold large projects (independent research projects, term papers, etc.) into a variety of related steps: topic, preliminary bibliography, outline, final bibliography, draft, etc.

In the end, if there’s one “take-away” from this posting, it is to become more intentional when thinking about the assessments we rely on, particularly in terms of exams or quizzes. The research is quite conclusive that, when well prepared and followed by feedback, “retrieval practices” are more effective in reinforcing student learning than studying in and of itself. Frequent, low-stakes assessments are an important way to promote student learning.

Endings and Beginnings: Thoughts on Finishing the Semester

Steven Volk, May 3, 2015

Hundreds of silhouettes gradually light up over 90 seconds. Alfredo Jaar, "Geometry of Conscience," Museum of Historical Memory and Human Rights, Santiago, Chile. Image at: http://www.designboom.com/architecture/the-geometry-of-conscience-memorial-by-alfredo-jaar/

Hundreds of silhouettes gradually light up over 90 seconds. Alfredo Jaar, “Geometry of Conscience,” Museum of Historical Memory and Human Rights, Santiago, Chile. Image at: http://www.designboom.com/architecture/the-geometry-of-conscience-memorial-by-alfredo-jaar/

A few weeks ago, the Chilean-born, New York-based artist, architect and filmmaker, Alfredo Jaar, was on campus to give a lecture which he titled, “It Is Difficult.” The title comes from William Carlos Williams: “It is difficult/ to get the news from poems/ yet men die miserably every day/ for lack/of what is found there” [Asphodel, That Greeny Flower and Other Love Poems: That Greeny Flower].

 

Jaar has often taken on the difficult task of turning news into poetry, and his own poetry into news. He is well known for memorializing victims of the “dirty wars” in Chile and Argentina. He designed a deeply moving installation at Chile’s Museum of Historical Memory and Human Rights called “Geometry of Conscience.” His contribution to the Parque de la Paz (Peace Park) in Buenos Aires, “Punto Ciego (Blind Spot),” commemorating the thousands of victims of the Argentine military juntas, is a landmark work among those who labor to construct an architecture of memory that goes beyond history and into conscience.

Alfredo Jaar, "Music (Everything I know I learned the day my son was born)," Nasher Sculpture Center, Dallas, TX

Alfredo Jaar, “Music (Everything I know I learned the day my son was born),” Nasher Sculpture Center, Dallas, TX

While these are themes Jaar has explored extensively, it was his last project, at the Nasher Sculpture Center in Dallas, that came to mind as I sat down to write this semester’s final “Article of the Week.” The Nasher invited Jaar to install a work in its garden to mark the museum’s 10th anniversary. The artist contemplated a number of related themes while puzzling over the project: anniversaries and the passage of time, the nature of the museum and its particular (often exclusive and exclusionary) audience, the city of Dallas and its changing populations, endings and beginnings.

With these in mind, he examined a map of Dallas with the Nasher highlighted at its center. He then located all the hospitals with maternity wards within a certain distance from the museum, and underscored those that served predominately African American, Latino/a, and undocumented populations. He ended up visiting three wards where, with the permission of the families involved, he audio recorded the sounds of the babies at their moment of birth – their very first cries.

Alfredo Jaar, "Music (Everything I learned)," Nasher Sculpture Center, Dallas, TX: http://www.nashersculpturecenter.org/art/exhibitions/exhibition?id=37

Alfredo Jaar, “Music (Everything I learned),” Nasher Sculpture Center, Dallas, TX: http://www.nashersculpturecenter.org/art/exhibitions/exhibition?id=37

Back in the Nasher’s Sculpture Garden, he constructed a pavilion of pine and plastic in shades of green, four translucent walls reaching up perhaps 20 feet. Inside the structure, captains’ chairs are placed around the perimeter, and one sits in the quiet, listening to the murmur of the visitors outside until, at the same time every day, there is a sound, recorded and amplified: the sound of a child being born. It is played at the exact time that a specific baby was born. As part of the project, “Music (Everything I know I learned the day my son was born),” all the families that participated in the project were given a year-long membership to the museum. All the new-born babies were given a lifetime membership, Jaar’s way of addressing the fact that contemporary museums rarely serve all the populations that are closest to them.

Alfredo Jaar, "Music (Everything I learned)," Nasher Sculpture Center, Dallas, TX: http://www.nashersculpturecenter.org/art/exhibitions/exhibition?id=37

Alfredo Jaar, “Music (Everything I learned),” Nasher Sculpture Center, Dallas, TX: http://www.nashersculpturecenter.org/art/exhibitions/exhibition?id=37

For someone whose work has so often dealt with the memories of crimes against humanity, Jaar’s recent (2013-14) project reminds us of that there are no endings without beginnings, all of which is an (admittedly) round-about as well as extravagant way of signaling that as we come to the end of the semester, we also begin to think about new beginnings and where we go from here.

It was a difficult semester in many ways, often demanding that we address issues that were occurring far from campus (as well as some that are very much at home), and it is important not to ignore the toll that these events have taken on us as well as many of our students. So how do we end the semester in a way that also signals a new beginning?

 

 

A previous “Article of the Week” (Closing Time, Managing the End of the Semester, April 28, 2014) offered a number of suggestions for closing out the school year:

  • Revisit the course goals in your syllabus with your students, helping students think about why the course was structured as it was and the ways in which they can examine their own learning from that perspective.
  • Encourage students to think about how they have worked to achieve their own goals set at the start of the semester: have your students write a short (anonymous) self-evaluation of their learning, reflecting on their participation in the course.
  • Have students create a summary concept map of the course that visually traces main themes and subsidiary branches.
  • One instructor has her students present a short lesson for the class on the issue, topic, or theme they found most difficult or challenging during the semester. It is an excellent way for students to prepare for exams, since teaching a subject is often the best way to learn about it.
  • Students in small groups can discuss how their understanding has changed over the course of the semester, focusing perhaps on critical moments in their learning.

David Gooblar, at PedagogyUnbound.com, suggests having students write letters to those who will take the course in the future. They can reflect on its high and low points, offering advice to those who will sit in their seats the next semester, addressing in particular what they wish they had known going in to the class, what they would have done differently, and what future students might want to know about the course and the instructor. As Gooblar adds, “Try including specific questions in your prompt. The idea is to turn the letter-writing exercise into a kind of course review that could be useful in helping students prepare for the final exam. Ask: What are the most important aspects of the course subject? What were the most insightful readings, and why? What remains unclear at the end of the semester? Having students answer such questions is a great way to get them to review the material. Writing the letter naturally encourages students to think back to where they were at the beginning of the semester. It puts into their head the distance they’ve traveled between then and now, asking them to take stock of exactly what they’ve learned.”

uneduex-along the lines of a legacy of goodbyes. CC-Flickr

uneduex-along the lines of a legacy of goodbyes. CC-Flickr

Finally, some advice for you, the teachers:

End of semester reflection from your point of view: this is a good time to write down (while you still remember it!) what worked and what didn’t; what you should reword or redesign in your assignments, what parts of the class produced just the results you had hoped for and what was confusing or disorganized?

Terri Givens, writing in Inside Higher Education, offers 10 points to help new faculty cope with the end-of-semester stress (with some of my own comments added), but they seem just as appropriate for old-timers as well:

 

1: Clearly communicate to others that it is crunch time – some things will have to wait;
2: Relax your standards in non-essential areas of life – let the laundry pile up; let local restaurants take care of your dinners;
3: Say no to every service request from now until the end of the semester – if administrators or chairs haven’t figured out all you have to get done, they should!
4: Every day needs a plan – lists are always great, even better now: checking off tasks you have done can bring a sense of accomplishment – you are making progress;
5: Write for 30-60 minutes each day – don’t drop your writing completely, even if it tails off;
6: Only check e-mail one time per day (max) – OK, you’ll need to pay attention to some student requests, but limit your time answering email.
7: Eliminate unnecessary electronic distractions – unplug.
8: Take care of your body: walk, exercise, eat well, sleep!
9: End every day with gratitude and a treat.
10: Did I say: take care of your body? Walk, exercise, eat well, sleep!

Send me your own methods for closing out the semester.