Tag Archives: Metacognition

Less is More: Low-Stakes Assessments and Student Success

Steve Volk, April 16, 2018
Contact at: svolk@oberlin.edu

There is a point in every semester when, it being too late to make significant changes in our current courses, we instead begin to think ahead to next semester. So, overwhelmed as you are by finishing up this term, getting in book orders, and planning research and writing for the summer, maybe you can keep this article in mind as you plan for the fall 2018 semester!

In this posting, I provide a few resources and summarize some of the reasoning supporting the argument that student learning is more enhanced by frequent, low-stakes assessments (“retrieval practices” in the technical lingo), with the opportunities they provide for continual feedback, than from infrequent, high-stakes assessments (e.g., a midterm and a final). While the research has suggested that this holds across all disciplines, the impact is noted especially in STEM fields and quantitatively based courses.

Some Research

First, a bit of the research. Where in the classroom cycle (class-studying-examination) does learning most take place? We generally think that students learn when they are studying (reading assigned texts or reviewing their notes), and that testing is the mechanism that tells us what has been learned. The same assumptions also buttress what many instructors think about classroom practice: students are assumed to learn in lectures (learning which is solidified when they study their notes), and tests help us know how much has been learned. The test, that that sense, is considered a “neutral event,” it measures without impacting learning. And yet, a considerable body of research has demonstrated that such an understanding is flawed; many cognitive psychologists now suggest that assessment practices themselves can produce large gains in long-term retention of information and concepts in comparison with studying for an exam. In other words, write  Peter C. Brown, Henry L. Roediger III, and Mark A. McDaniel in Make It Stick: The Science of Successful Learning (Harvard, 2014), using “retrieval practices” (recalling facts, concepts, data, or events from memory) is “a more effective learning strategy than review by rereading.”

To summarize his (and Andrew C. Butler) conclusions from one of Roediger’s many studies,

  1. Assessments (“retrieval practices”) often produce superior long-term retention relative to studying for the same amount of time;
  2. Repeated assessments are better than one-time testing;
  3. Providing feedback produces better results than assessment practices without feedback, although even without feedback, the learning results are better for multiple assessments;
  4. Some lag between study and test support retention and learning;
  5. The benefits of “retrieval practices” can be transferred from one context (domain) to another.

AFP; sitting for the “bac,” Paris, June 18, 2015

Cynthia J. Brame and Rachel Biel, both at Vanderbilt, offer a useful chart in CBE Life Sciences Education that summarizes the research on the relationship between testing and learning in undergraduate science courses. Among the findings of previous studies that they consider are the following:

  • Testing improved retention significantly more than restudy in delayed tests.
  • Multiple tests provided greater benefit than a single test.
  • Students in the retrieval-practice condition had greater gains in meaningful learning compared with those who used elaborative concept mapping as a learning tool.
  • Both initially correct and incorrect answers were benefited by feedback, but low-confidence answers were most benefited by feedback.
  • Testing improved retention and increased transfer of information from one domain to another through test questions that required factual or conceptual recall and inferential questions that required transfer.
  • Interim testing improved recall on a final test for information taught before and after the interim test.

But first, a note to the doubters – myself included. One of the central concerns I have always had about testing as a method of assessment, particularly high-stakes mid-terms or finals, or even post-reading quizzes, is whether they only promote rote memory learning (“When did Cortez first arrive in Mexico?” “Describe the wing structure of a bat,” etc.). I questioned whether the memorization of facts that (now) can be looked up instantaneously on a smart phone was necessary, or, in any case, whether what students learned would stick with them, or stick with them in a contextualized fashion, other than the memorization of random facts.

We all know that learning is more than the memorization and the repetition of facts, that higher education learning above all must shift students to higher cognitive levels and outcomes. So, while we know, through a variety of studies, particularly the work carried out by Andrew C. Butler, that “repeated testing produce[s] superior retention and transfer on the final test relative to repeated studying,” what do we know about the impact of frequent assessment such as tests and quizzes on helping to move students to higher cognitive levels? Can what was learned via testing be “transferred,” extended to other domains and contexts? And can attention to “retrieval practices” help students better cope with some of the negative consequences that accompany high-stakes testing? Stick around for some answers.

What to Keep in Mind

  1. Frequent, low-stakes testing is better than infrequent, high-stakes testing.

The research cited above suggests the many reasons why this is the case. To put it in practical terms:

Jenna Carter,”Test,” Flickr cc

a. We know enough about high-stakes testing to know that such assessment practices can have serious negative effects on identity-threatened groups in specific contexts: Blacks and Latinx in the sciences; women in math; etc. Further, low-stakes testing reduces the anxiety associated with high-stakes testing since one’s future isn’t riding on one or two tests. And we know that student anxiety in general has reached frightening levels. In 1985,18% of incoming first-year students agreed that they “felt overwhelmed by all I had to do.” In 2010, that figure increased to 29%. Last year, it surged to 41 percent.

b. Frequent testing allows students to practice remembering, a useful capacity building skill. But, to help students not just retain information but to use retrieval practices to move to higher order thinking, the testing should be “effortful” (e.g., short answer rather than multiple choice), spaced (allowing enough time to elapse for some forgetting to occur) and “interleaved” (one should introduce different topics and problems rather than having students master one topic at a time and then move on to the next). (These are points that Brown, Roediger, and McDaniel stress in Make it Stick.)

c. Frequent testing helps students know where they stand in a course, making it more likely that they will seek help when needed, i.e., when it can actually make a difference. Particularly in STEM courses, if students only realize that they are in a very deep pit at the mid-term point, there is little that they can do to climb out.

  1. Feedback is better than no feedback.

While the research suggests that low-stakes testing without feedback is actually better than relying on studying alone in terms of student learning, it also indicates the key role that instructor feedback plays in boosting student learning.

a. Most obviously, feedback from the instructor will let students know where they stand in a course. As faculty, we often have our own ways of intuiting how students are doing in our classes without resorting to formal testing mechanisms. Are they coming to class? Do they look engaged? Are they falling asleep in class? Have they come to office hours when asked? Some of these indicators are fairly accurate; others can be not only wrong but seriously problematic, particularly if they feed off of implicit biases. But the point is that students don’t share these “insights” as to how they are performing in a course. Many will feel that they are doing just fine, when they are actually standing at the edge of a precipice. (A recent study reported on in the New York Times suggests that men, more than women, overestimate their abilities in science classes. Who would have thought!) Concrete feedback from instructors can help students obtain a more objective sense of how they are doing. They may not always “hear” it or respond appropriately to such information, but offering feedback most often is quite welcome and useful.

b. Providing feedback to correct responses as well as incorrect answers is highly useful. As Brame and Biel discuss in the above cited article, feedback on both low-confidence correct answers and incorrect answers may further enhance the testing effect, allowing students to solidify their understanding of concepts about which they are unclear.

c. Feedback can come from instructors on exams or other assessments, or from in-class peers. One research study found that when students answer an in-class conceptual question individually using clickers, discuss it with their neighbors, and then re-vote on the same question, the percentage of correct answers typically increases.

Gary Larson, The Far Side

  1. Low-stakes, frequent testing with feedback helps students think more intentionally about their own learning.

I have written before about ways to support student metacognition. One way to do this is to encourage students to reflect on their own process of learning by having them respond to feedback on exams, quizzes, or other assignments.

a. Students can “self-test” as part of their studying process, and then measure the outcome versus their expectation, but few will actually do this, and generally those who do are not always the ones who could use additional help. Providing concrete evidence of accomplishments can help students more accurately assess their level of comprehension and (hopefully) seek help to address those areas that need attention.

b. Frequent assignments tied to reflection exercises can help students think about what they did (to prepare for an assignment, for exaple), what the outcome was, and what they will do differently the next time.

c. No-stakes formative assessment (a short quiz at the end of the class or a simple “one-minute paper”) can help students think in a more focused fashion about what they know and they need to focus on.

  1. Design questions that can move students to higher order thinking.

While not every exam must be “effortful,” with some thought, one can design multiple choice questions to lead to higher order thinking.

a. Cynthia Brame, from Vanderbilt’s Center for Teaching, provides the following advice: Presenting a problem that requires application of course principles, analysis of a problem, or evaluation of alternatives tests students’ ability to engage in higher-order thinking. Instructors can design problems that require multilogical thinking (“thinking that requires knowledge of more than one fact to logically and systematically apply concepts to a …problem”).

Here are two examples she provides:

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

b. In Dynamic Testing: The Nature and Measurement of Learning Potential , Robert Sternberg and Elena Grigorenko suggest, as the title indicates, a more dynamic approach to testing than what is typically provided in static, standardized tests. Essentially, this calls for determining the state of one’s knowledge; refocusing learning on areas that need more attention; and retesting to measure improvement. It is testing that helps the learner focus on new knowledge (i.e., on “learning”), rather than on what has already been accomplished. In this sense, well-designed testing can move the student to higher cognitive levels.

  1. Break large assignments/exams into smaller pieces

Designing, for example, eight assessments tools that are effortful, spaced, and interleaved, takes some work, particularly when you are accustomed to giving a midterm and a final. True enough, but think about breaking the high-stake assignments you already give into smaller pieces, as suggested by Sara Jones at Michigan State University’s “Inside Teaching.”

a. Replace high-stakes, large tests into several quizzes.

b. Scaffold large projects (independent research projects, term papers, etc.) into a variety of related steps: topic, preliminary bibliography, outline, final bibliography, draft, etc.

In the end, if there’s one “take-away” from this posting, it is to become more intentional when thinking about the assessments we rely on, particularly in terms of exams or quizzes. The research is quite conclusive that, when well prepared and followed by feedback, “retrieval practices” are more effective in reinforcing student learning than studying in and of itself. Frequent, low-stakes assessments are an important way to promote student learning.

Metacognition II: Six Ways for Faculty to Reflect on Teaching

Steve Volk, November 13, 2017
Contact at svolk@oberlin.edu

Last week, the “Article of the Week” focused on ways to help students be more aware of how they think – to engage in metacognitive practices – in order to develop self-aware approaches that help them transfer what they learn from one course to the next, from one discipline to another, and from school to life. This week I’ll focus on six ways that we, as teachers, can reflect on our own practice so as to improve our teaching and student learning outcomes.


All images from “The Comical Hotch Potch, or The Alphabet turn’d Posture-Master” (1782)

t the start of the semester I surveyed Oberlin’s faculty on a variety of teaching issues, asking questions such as what aspects they derived the most pleasure from or what gave them the greatest heartburn. Among the questions I asked was one concerning what faculty considered “the best way/s to get help or feedback that could address the issues you face in the classroom?” Of the many possibilities, ranging from attending workshops to talking to deans or department chairs, the winner was “on-the-fly” conversations, those quickie chats squeezed in after you’ve discussed the plot lines that will emerge in Season 3 of “Stranger Things.” These most often unfold in the hallway, parking lot, around the copier, or when walking to or from a faculty meeting. “On-the-fly conversations” was almost always listed among respondents’ top three preferences.

As someone who organizes teaching and learning workshops and “brown-bag” discussions, I would have preferred a different answer, but I get it. Most of us want, or even crave, time to talk about what just went down in our classes, but we don’t have time for the 2-hour workshop or even a 45-minute chat over lunch or tots in the Feve. So “on-the-fly,” Keurig-centered conversations (often much to the annoyance of our AA’s) fill a real need.

Granted that some of these fall into the “can you believe what a student just said” mode; but many more arise from our desire to talk about something that just happened in class so that we can figure out what just happened and learn from it. These incidents could be troubling – a moment when the class seemed on the verge of spiraling out of control – or wonderful, when the semi-magical happens and the class digs down to a deeper level of understanding, a more cohesive way of interacting. The point is that as teachers, we reflect continually on what we’re doing in class, most often in the internal conversations we carry on inside our heads. But these reflections less often take place in ways that can be captured, considered, and fed back into our practice. It’s often not until the train derails at precisely the same place the next semester that we remember and wonder why we didn’t do something about it?

Reflection and Change

Lynne McAlpine and Cynthia Weston, from McGill University’s teaching and learning center, suggested that there are at least five different ways to conceptualize the role of reflection as it pertain to our teaching:

An academic orientation focuses on the organization of subject matter, a social efficiency orientation on how well practice matches what research says, a developmental orientation places priority on understanding students’ thinking, a social reconstructionist orientation sees reflection as a political act, and finally the generic orientation is one in which any reflection is good because teachers can then be more intentional and deliberate in their thinking about teaching.

The lest-costly, generic model is good for me, seeing the point of reflection as relating more to praxis than to philosophy; reflection as a way of thinking about what is off-kilter and what can be done to fix it.

As I was thinking about this kind of reflection, I remembered an essay by Atul Gawande that appeared in the New Yorker a few years ago. In “Personal Best,” Gawande considered his own practice as a surgeon. He describes how, after many years, thousands of surgeries, and results that always improved (measured by a decreasing rate of complications following operations), he leveled off. His performance rate was quite good, but it didn’t get any better. He began to wonder what professionals in other fields did to get off the plateau and keep improving. What about those who everyone would consider at the top of their game, top-ranked tennis players or singers, for example. Are they still coached or mentored in order to continually improve? Surgeons weren’t; teachers aren’t.

ondering if a violinist of the caliber of Itzhak Perlman was still getting coaching, he called him up. (“So I called Itzhak Perlman to find out what he thought.” Of course! Why didn’t I think of that? Pick up the phone and call Itzhak!) Anyway, it turns out the answer is yes. Perlman’s coach for the past 40 years has been his wife, Toby, herself a concert-level violinist whom he met at Juilliard.

“The great challenge in performing is listening to yourself,” he said. “Your physicality, the sensation that you have as you play the violin, interferes with your accuracy of listening.” What violinists perceive is often quite different from what audiences perceive. “My wife always says that I don’t really know how I play,” he told me. “She is an extra ear.” […] Her ear provided external judgment.”

Now, none of us, I’m fairly certain, is a teacher with Perlmanesque talents, but his statement about “listening to yourself” sounded so familiar that I could change “performing” to “teaching” without doing damage to his argument. One of the great challenges in teaching is that we have a hard time judging our own performance. Carrying out internal conversations about how that class just went is not likely to get us where we need to be. We need a deeper mode of reflection.

Which takes us back to Dewey. John Dewey argued that real learning comes from reflection on experience more than from the experience itself. And reflection, as Carol Rogers usefully summarized, is a “meaning-making process” that can move the learner – us, in this case – from one experience to the next “with deeper understandings of its relationships with and connections to other experiences and ideas.” Serious reflection relies on a systematic, rigorous, and disciplined way of thinking. It requires attitudes that value the personal and intellectual growth of oneself and of others. And, above all, it needs to happen “in community, in interaction with others.”

So, here are some suggestions for helping us think more reflectively about our teaching, ways to take the conversations out of our heads by putting them on paper or, better still, by reflecting in community  with others.

  1. Quick Notes:

When you return from a class, quickly jot down some notes on those aspects of the course that you think went well or poorly. Be brief, or you’ll find that it takes up too much time and you’ll soon stop doing it. Note the issue and, if a solution is easily available, add that: The students couldn’t figure a way into the reading: give them better prep questions next time. The energy drained from the room halfway through: break up the session with some activity. Amazing discussion, all initiated by Sam’s statement […]: try prompting the discussion next time with Sam’s observation.

I usually write notes on a copy of the syllabus I keep on my computer for just this purpose. It makes it much easier to locate them when I’m preparing the next iteration of the class. I can add comments the next time through to see if something made a difference.

  1. Self-questions to promote faculty metacognition about teaching.

ancy Joseph, an English professor at Oakland University, over 15 years ago began helping students think metacognitively about their writing projects, urging them to note their thoughts at every step of the writing process: when they wrote a first draft, received peer review comments, and read her comments.  But, she observed, “most responses were void of meaningful reflection, and… I detected no changes in their writing behaviors.” She thought about on what was happening and decided to teach by example, stressing the importance of reflecting on the pre-writing (planning) stage as well as the writing process:

I distributed pages of my own writing from a professional article that had proceeded in fits and starts over the previous half year…I candidly shared my thoughts as the author of this work in progress, indicating what I had been thinking when I added explanations, reorganized paragraphs, and rewrote passages. I wanted to expose my students to the decision-making strategies that writers use to address the needs of their readers. This method enabled me to understand that helping students develop metacognitive awareness requires direct instruction and demonstration, a step-by-step journey into the cognitive process of writing.

Kimberly D. Tanner, author of the article on “Promoting Student Metacognition” that I drew from last week, suggests the following ways to think about your own classes in a form that can increase their potential for metacognitive approaches to teaching.

Activity Planning Monitoring Evaluating
Class session *What are my goals for this class session?

*How did I arrive at these goals?

*What do I think students already know about this topic?

*How could I make this material personally relevant for my student? Why do I think this?

*What mistakes did I make last time I taught this and how can I not repeat these?

*What do I notice about how students are behaving during this class session? Why do I think this is happening?

*What language or active-learning strategies am I using that appear to be facilitating learning? Impeding learning?

*How is the pace of the class going? What could I do right now to improve the class session?

*How do I think today’s class session went? What evidence do I have for thinking this?

*How did the ideas of today’s class session relate to previous class sessions? To what extent do I think students saw those connections?

*How will what I think about how today’s class session went influence my preparations for next time?

Overall course *Why do I think it’s important for students pursuing a variety of careers to learn the ideas in my course? What are my assumptions?

* How does success in this course relate to my students’ career goals?

*What do I want students to be able to do by the end of this course? 5 years later?

*In what ways am I effectively reaching my goals for students through my teaching? How could I expand on these successful strategies?

*In what ways is my approach to teaching in this course not helping students learn? How could I change my teaching strategies to address this?

*How is my approach to teaching this course different from the last time I taught it? Why?

What evidence do I have that students in my course learned what I think they learned?

*What advice would I give to students next year about how to learn the most in this course?

*If I were to teach this course again, how would I change it? Why? What might keep me from making these changes?

*How is my thinking about teaching changing?

 

  1. Seeking Feedback: Once and done

Arrange for a formative observation of your course. Ask a colleague or, better yet, someone from CTIE, to visit a class, take notes, and talk to you about what went on. CTIE has developed a specific protocol (pre-observation, observation, post-observation conversation) to guide the process. As you would expect, the process works best if you have a few things that you would like the observer to pay attention to.

on’t mistake a “formative” observation – one where you invite someone to observe a class and give you feedback – with a summative observation, where you are being evaluated by a member of your department, the chair, or someone else whose job it is to make a formal assessment of your teaching. Formative observations stay with the instructor and aren’t reported to anyone else (unless you choose to include the observation in a personnel file, but that’s your choice alone). Their whole purpose is to help the instructor reflect on what is happening and, if needed, address those issues. It’s all about the fact that we can’t “hear ourselves” when we teach.

  1. Seeking Feedback: A few times is better

Teaching pairs, triangles, squares. No, not an exercise for geometry teachers. These are arrangements in which you invite a colleague (from your department or any other) into your class for a few observations and return the favor by observing their classes. It can be done, as the names imply, with 2, 3, or 4 colleagues. A central aspect of the process is that these are always formative observations that are entered into voluntarily and eagerly. They usually take place between 2-4 times a semester, and feedback is generally offered in a social setting: over lunch, coffee, a drink, or dinner. They groupings can be of “equals” (e.g., all junior faculty); mixed (pairing that bring together junior and senior faculty); colleagues from the same or different disciplines, etc. People in the arrangements can agree on how they would like to see the process develop: an initial meeting; note-taking or not; written observations or not. In short, whatever makes the process more likely to happen and more enjoyable for everyone.

How to get started? Talk to the people you want in the group now in order to begin in the spring semester. If you want some help forming a group, talk to CTIE.

  1. Seeking Feedback: All semester is best

Yes! This gives me another opportunity to talk about the Faculty-Student Partnership.  (I sent around a “recruiting” note earlier this week for those interested in joining the program in the spring; here it is again in this new context.)

One of the faculty participants in the program recently said that, “Reflection is the biggest piece [of the FSP program]. The conversations make you stop and think about what went well and what didn’t.” The research on this is utterly convincing. McAlpine & Weston stress:

“[…] Multiple, repeated observations and interactions … may be necessary [but it is] the analysis of these multiple experiences through reflection which enables one to detect patterns that then lead to knowledge.”

We don’t get that kind of feedback, or the opportunity to reflect on our practice, from the student evaluations collected at the end of the semester. But this kind of extended process of interaction is available through the Faculty-Student Partnership program. The FSP program pairs a student with a faculty partner over the course of an entire semester. The student participants cannot be enrolled in any courses taught by their faculty partners (although they may have taken courses with them in previous semesters). The students attend their partners’ designated classes once a week over the course of the semester, taking detailed observation notes of the class sessions, and meeting weekly with their faculty partners to discuss the class, often focusing on specific issues issues suggested by the faculty partner. Students also meet biweekly with the program’s coordinators to encourage the students to discuss their experiences collectively, and as an opportunity for more training and reflection. Students receive training in ethnographic note taking at the start of the semester, and discuss ways to make discussions with their faculty partners most productive. Program coordinators also meet monthly with the faculty partners to get their feedback on the progress of the partnerships. (Faculty and students engaged in the program last fall discussed it in an article in the Oberlin Review.)

The emphasis of the FSP program is on dialog, stressing the concept that teachers and students mutually benefit from seeking out one another’s perspectives and discussing how these might inform teaching and learning contexts. The goal of partnership work is not change for change’s sake but precisely to open the kind of reflection that is central to a metacognitive approach to teaching.

(My pitch, one more time: If you are interested in applying to the program in the spring 2018 semester, please contact CTIE by filling out this form.

  1. Faculty Learning Communities

aculty Learning Communities (or Faculty-Staff Learning Communities) are another way to develop a collaborative reflective practice. While there are different models for building a learning communities, the most straightforward is for 6-8 faculty (or faculty and staff) to get together over the course of a semester or year to discuss a topic of interest and concern to the group. Some are funded by the dean’s office to provide reading materials, food, or other things that can support the group. Some require the group to produce some work at the conclusion of the process that can be shared with the larger community. As Martha C. Petrone and Leslie Ortquist-Ahrens write,

FLCs provide a collaborative arena in which colleagues have the time and opportunity to reflect on their teaching, their discipline, their institution, and themselves. By creating a safe environment for the honest engagement of ideas and feelings, the FLC facilitator helps to move the faculty outside of their disciplinary comfort zones and into the realm of intellectual and interpersonal connections. Through this process, teaching and learning are meaningfully enhanced and often transformed. [Martha C. Petrone, Leslie Ortquist-Ahrens, “Facilitating Faculty Learning Communities: A Compact Guide to Creating Change and Inspiring Community,” in Milton D. Cox and Laurie Richlin, eds., New Directions for Teaching and Learning: Building Faculty Learning Communities (Spring 2004), Volume 2004, Issue 97, Pages 1–157.]

The best way to initiate a Faculty (or Faculty-Staff) Learning Community around teaching and pedagogy is the most straightforward: talk to a colleague, define a theme, find others who would be interested in joining and talk to them. CTIE would be delighted to help either identify faculty or staff who would be interested or to provide a bibliography that can inform and orient your inquiry.

Conclusion

It has been said many times, but is probably worth saying it again: College and university faculty are among the very few professionals who aren’t actually trained in what we spend most of our time doing: teaching. We are experts in our particular domains and subfields, but few of us have read extensively in the literature on pedagogy, theories of learning, or child and adolescent development. That’s how it has always been. And while there are many developments that can help instructors think about their classroom practices, often coming from teaching and learning centers such as CTIE, we learn most from our own experiences. We can get the most out of that by reflecting on our experiences, by ourselves, but especially in community with others.

 

“What Am I Doing? Is It Getting Me Anywhere?” Scaffolding Student Metacognition

Steve Volk
Contact at: svolk@oberlin.edu

Illustration from Edwin D. Babbitt’s The Principles of Light and Color (1878)

Hands up those of you who have had students come to your office hours anguishing over the poor grades they received on an exam. I’m not talking about the student who thinks his grade should be higher, rather the student who can’t figure out why she got such a low grade since she worked really hard preparing for the exam:

I studied all Saturday night and Sunday; I re-read all the assigned textbook readings and went over my notes. I even took the time to memorize all the words bolded in the text. I worked really hard, but still got a C-. I don’t know what to do!

So, what’s your advice, dear reader? Tell the student: “It looks like you’re not studying hard enough. You need to work harder; you need to figure out how to apply yourself better”?

Or what about the student who has done well on all the quizzes and exams you have given but seems to struggle when asked in class to explain the reasoning behind her proofs or the concept from which the equation is drawn?

Or what about the student who took the entry-level physics class, and did very well, but doesn’t appear to have carried over what he learned in that class to the next level?

It’s not surprising that the great majority of our students have learned to be good at school – they know how to take tests, draw between the lines, memorize what they are told to memorize, and give us what we seem to be asking for. They probably wouldn’t have been admitted if they weren’t good at school. But that doesn’t mean that they have learned how to be good at learning, that they know how to make learning their own, or have gained some insight on how they learn, not just what they learn, and can transfer what they have learned in one domain (say the close attention to detail that helps them excel in art history) to other domains (perhaps psychology, or a close and critical reading of texts). And, ultimately that would be a shame because the latter are the students who will better apply what they have learned in school to help them succeed in many fields after they graduate.

The historian Brook Adams, writing in the Atlantic in 1879, observed: “Knowing that you cannot teach a child everything, it is best to teach a child how to learn.” Adams charged that the Boston school system, on whose board he served, was failing miserably at this. Francis W. Parker, the superintendent of the Quincy, Massachusetts system around the same time, wanted all students to, as he put it, make their learning their own, to gain from their schooling a resource they could rely on throughout their lives. Le plus ça change?

If the tension between being helping students be good at school and helping them be good at learning is an old one, it’s probably also one that each of us continues to confront in our classes. We are trained as scholars and creative artists in specific fields, analytic philosophy or printmaking, hydrogeology or contemporary Hebrew poetry. We probably wouldn’t have made it as far as we have if we hadn’t become experts not just in the content of our subjects of study, but also in understanding – explicitly or implicitly, consciously or unconsciously –how we best could approach our own learning. And yet, probably for most of us – and I’d certainly include myself here –we couldn’t, or can’t, articulate just what it was that we did to achieve this. Nor did we have many (any?) grad school professors who stopped us along the way to ask: do you know what you did to get over that roadblock, to make learning your own?

Phrenological chart, public domain.

As instructors, we are hired to teach students the content that we have become expert in. And, except for some in psychology or those in the field of education, we don’t necessarily think about whether we should make our students aware of how they can also master learning, and not just history or music theory. And even if we did, would we know how to go about this?

We’re now in the realm of metacognition and helping our students think about their own metacognitive strategies. In this week’s article, the first of two on the topic, I’ll discuss some approaches to how we can help our students become better learners by focusing on metacognitive strategies in class sessions, on exams and papers, and after they have finished a course. The research literature on metacognition is rich and quite deep, and I’ll include a short bibliography at the end for those interested in  pursuing it further (and I am sure that our colleagues in Psychology can be of much greater help than I in this regard). This essay draws from a number of research publications, but it is primarily indebted to Kimberly D. Tanner, “Promoting Student Metacognition,” which appeared in Life Sciences Education (American Society of Cell Biology) in 2012.

Next week I’ll focus on how we, as teachers, can use megacognitive strategies to improve as instructors.

Metacognition

Without getting too deep in the definitional weeds, we could use some working concept of metacognition. In its simplest sense, metacognition is “thinking about thinking.” But we can also push it a bit further. Any process that helps us think about how we got to an answer, rather than the answer itself, can probably be thought of as a metacognitive process. The Socratic method would qualify as a metacognitive one in the sense that it focuses on a guided set of questions that can produce an answer. Math problems that require the student to record step-by-step reasoning rather than just writing the answer are in the metacognitive mode, at least in a broad sense. My go-to guy, John Dewey, who argued that students would learn more from reflecting on their experiences than from the actual experiences themselves, introduces a metacognitive approach – and we’ll return to him shortly.

Metacognition, according to John Flavell, the psychologist credited with the developing the term, “refers to one’s knowledge concerning one’s own cognitive processes…the learning-relevant properties of information or data.” Flavell divided metacognitive processes into three categories, summarized here.  The divisions are useful as we think of methods that can support our students’ metacognitive understandings.

  • “Knowledge of person variables” concern general knowledge about how we process information and our own awareness of this, i.e. “learning about learning.” The first example I gave, that of the frustrated student who can’t figure out why she’s doing poorly on exams, probably has a fairly low awareness of how she learns the best. Perhaps she’s studying while listening to music, checking her text messages frequently, or trying to study when she’s just too tired.
  • “Knowledge of task variables” refer to our deeper understanding about the task that we are given. Students who leave as much time to read a short story as they do for a 40-page article by Judith Butler or a chapter in a physics text could easily be in trouble because they haven’t properly assessed the task at hand.
  • “Knowledge about strategy variables” include understanding cognitive and metacognitive strategies as well as conditional knowledge about when and where it is appropriate to employ such strategies. “Knowledge,” Jennifer Livingston writes, “is considered to be metacognitive if it is actively used in a strategic manner to ensure that a goal is met. For example, a student may use knowledge in planning how to approach a math exam: ‘I know that I (person variable) have difficulty with word problems (task variable), so I will answer the computational problems first and save the word problems for last (strategy variable).’”

 Supporting Student Metacognition: Student Self-Questioning

While there has been more research on the impact of metacognitive thinking on K-12 students than on college-age students, one clear conclusion from the research is that students with low metacognitive abilities generally can’t evaluate their own strengths and weaknesses appropriately. (Remind you of anyone?) They quite often overestimate their abilities and therefore are less likely to prepare well for assignments. James Lang, who writes frequently on teaching and learning for the Chronicle of Higher Education, some years ago referenced the cringe-worthy opening episodes of each season of “American Idol” calling them a mashup of lousy singing and dreadful metacognition: Don’t they know that they have no talent as singers? Returning to higher education: students who over-estimate their knowledge, skills, or level of preparation are unlikely to do well for the assigned task.

“Convolutions,” : Dr Alesha Sivartha, “The Book of Life: The Spiritual and Physical Constitution of Man,” 1898.

Paul Pintrich, on the other hand, has argued that “students who know about the different kinds of strategies for learning, thinking, and problem solving will be more likely to use them.” So how do we teach metacognitive strategies to our students and what can we do to help them?

Tanner’s article provides a wonderful place to begin, starting with prompting questions that students should ask themselves as they prepare for class, assignments, exams, or other tasks. For each type of activity (class session, active-learning task and/or homework assignment, quiz or exam, overall course), Tanner provides a set of questions-to-self that students can explore when planning an activity, monitoring their progress, or evaluating the outcome. (Here’s a great idea for an app to all you programmers out there: A metacognition app that prompts students, as in: “Siri: What do I already know about this topic?” “Alexa: What strategies should I use to study?”)

I’ll include a few of Tanner’s questions here. You can find the whole chart at the linked article.

Activity Planning Monitoring Evaluating
Class session *What are the goals of the class session going to be?

*What do I already know about this topic?

*Where should I sit and what should I do (or not do) to best support my learning during class?

*What insights am I having during this class session?

*What am I confused about?

*Can I distinguish important information from details? How will I figure this out?

*What was today’s class about?

*How can I get my remaining questions answered?

*How did the ideas of today’s class relate to previous classes?

Active-learning task and/or homework assignment *What is the instructor’s goal in having me do this task?

*What are all the things I need to do to be successful in this task?

*How much time will this take me?

*What strategies am I using that are working well or not working well to help me learn?

*What is most challenging for me about this task? Most confusing?

*To what extent did I successfully accomplish the goals of the task?

*When I do an assignment or task like this again, what do I want to remember to do differently? What worked well for me that I should use next time?

Quiz or exam *What strategies will I use to study? Study groups, problem sets, etc.

*How much time should I be studying? Over what period of time?

*What should I be spending more or less time on based on my current understanding?

*Am I taking advantage of all the learning supports available to me?

*Am I struggling with my motivation to study? Do I remember why I am taking this course?

*Which of my confusions do I have to get clarified? How will I do this?

*What about my exam preparation worked well that I should remember to do next time?

*What didn’t work so well so that I should not do next time or should change?

*What questions did I get wrong and what confusions do I have that still need to be clarified?

Overall course *Why is it important to learn the material in this course?

*How does success in this course relate to my larger goals?

*How am I going to actively monitor my learning in this course?

*In what ways is the teaching in this course supportive of my learning? How could I maximize this?

*How interested am I in this course? How confident am I in my learning? What could I do to increase my interest and confidence?

*What will I still remember 5 years from now that I learned in this course?

*What advice would I give a friend about how to learn the most in this course?

*What have I learned about how I learn in this course that I could use in my future courses?

Supporting Student Metacognition: Other Approaches

Providing significant formative feedback to students can also help them more accurately assess their knowledge about a topic. Many of these ideas will be familiar to loyal “Article of the Week” readers: minute papers, muddy point writing, clicker questions, etc. Here are a few others:

ConcepTests: Developed by the legendary Eric Mazur, a physics professor at Harvard, ConcepTests focus on a single concept; can’t be solved using equations; have good multiple-choice answers; are clearly worded; and are of intermediate difficulty. Stephen Chew, chair and professor of psychology at Stamford University (Birmingham, AL) describes how he uses them: After putting up a multiple-choice question, “Students select their answers individually, and I poll the class. They can then discuss their answer with other students, after which I poll the class again. Finally, we discuss the answers as a class. This gives me a sense of how well students understand the material. I can identify and address problem areas.” (Chew also offers a 6 short videos on metacognition and “how to get the most out of studying,” including a final one: “I Blew the Exam, Now What?”

Retrospective Post-assessments as a means of pushing students to be aware of conceptual change: Cognitive psychologists often define “learning” as “a student-centered activity in which students change their ideas about a topic” (Posner et al, 1982). Approached in this way, learning doesn’t happen until students examine how their understanding of a concept or idea has changed. This insight is reminiscent of Dewey’s argument that reflection on experience is more important than the experience itself. The learning happens in the reflection. Again, the question is how to help our students be reflective. One way is to engage students in a post-assessment exercise in which they are asked to discuss how they thought about the topic or concept before the assignment or other activity and to compare that with how they think about it after having completed the assignment.

Memo to Self: What did I learn? A relatively simple way to help students reflect on their learning is to have them write a letter or memo to their “future selves” after they have completed and been graded on an assignment. Encourage them to note what about their preparation for the exam or paper or other assignment worked well enough that it should be repeated the next time they take on a similar task. Did they work with a study group? Go to a peer instruction session? Attend office hours? Get two hours extra sleep the night before? Have a good breakfast? Similarly, they need to think about what didn’t work so well and shouldn’t be repeated?

Prompts to use in Class: Finally, again borrowing from Tanner’s excellent article, here are a list of questions you can ask students after standard classroom activities as a way of supporting their metacognitive thinking.

For “pairing-up” types of discussion after a ConcepTest, clicker question, or other multiple choice question posed to the class:

  • What did you think the question was asking?
  • What process did you use to arrive at your answer? Why did you choose “b,” for example?
  • What were the main reasons you didn’t chose “a” or “c”?
  • Compare your ideas with your discussion partner.
  • What most confused you about this question?
  • How confident are you in your answer? Why? What more would you need to increase your confidence?

For homework or active-learning tasks:

  • Pose three questions you had about the concepts you explored in this assignment that you still can’t answer.
  • Describe at least two ideas related to this assignment that you found confusing.
  • Do you agree that you “learned a lot” from the assignment? Why? Why not?
  • Think about how you approached completing this assignment and compare it with the way that you completed the last assignment.
  • What advice would you have for yourself based on what you know if you were just beginning the assignment?

Preparation for Quizzes and Exams:

  • How do you plan on preparing for the upcoming exam? Why?
  • What resources are available to support you? How will you make sure to use these?
  • Compare your exam preparation strategy with three others enrolled in the class – ask them how they are preparing.
  • What concepts have you found most confusing so far? What has been clearest? Given this, how could you best spend your time preparing for the exam?
  • Based on your performance on the first exam, write a memo to yourself with advice about preparing for the next exam.

Conclusion: It’s OK to be Confused

The final point, again from Tanner, is that we need to create an environment in the class where students know that they are both allowed and encourage to share their confusions as well as their brilliant insights. We can do this by recounting for students what happened when you taught the course before: “Students in previous semesters really seem to hit a speed bump when they got to this concept, so please share your questions and confusions; that way the whole class can benefit from the discussion.” In a previous article I suggested that, when addressing students who answer questions in class, it is as important to spell out why a particular answer was correct as why it was wrong. Pulling out the conceptual or procedural reasoning behind a question helps students as they think about their own thinking. Help your student not just to be good at school, but to be good at learning.

Next week: Modeling metacognition for students.


Some beginning resources on metacognition and teaching and learning:

“Odors and Flavors,” Dr Alesha Sivartha, “The Book of Life: The Spiritual and Physical Constitution of Man”

Ambrose, S.A., Bridges, M.W., Lovett, M.C., DiPietro, M and Norman, M.K. (2010). How Learning Works: Seven Research-based Principles for Smart Teaching. San Francisco, CA: Jossey-Bass. (Short summary here.)

Angelo, T. A. & Cross, K. P. (1993). Classroom Assessment Techniques: A Handbook for College Teachers. San Francisco, CA: Jossey-Bass.

Chew, S.  Five-part video series on learning.

Coutinho S.A. (2007). The relationship between goals, metacognition, and academic success. Educate 7, 39–47.

Kruger J, Dunning D (1999). Unskilled and unaware of it: how differences in recognizing one’s own incompetence lead to inflated self-assessments. J Personality Soc Psychol 77, 1121–1134.

National Research Council (2000). How People Learn: Brain, Mind, Experience, and School, Washington, DC: National Academies Press.

Pintrich P. (2002). The role of metacognitive knowledge in learning, teaching, and assessing. Theory Pract 41, 219–226.

Smith, M. K., Wood, W. B., Adams, W. K., Wieman, C., Knight, J. K., Guild N., Su T. T. (2009). Why Peer Discussion Improves Student Performance on In-Class Concept Questions, Science, 323 (5910), 122-124.

“Choose your Own Adventure”: New Approaches to Assignments

Wendy Hyman, Associate Professor of English (Oberlin College), April 17, 2017

"Macbeth," Holinshed Chronicles Folio. Public domain

“Macbeth,” Holinshed Chronicles Folio. Public domain

Like most teaching faculty, I’ve experimented with an array of strategies for augmenting (and evaluating) student learning over the years. Some have been fairly conventional: response papers, short quizzes, class reports, blackboard posts, final research papers, cumulative exams, and the like. Others have been more creative and comparative: writing “biographies” of books in Special Collections (trying to discern something like the “life story” of a 400- or 500-year old object), curating an exhibit in the Allen Memorial Art Museum  (although a literature professor, my scholarship sometimes delves into visual studies), writing imagined dialogues between literary characters, selecting among textual variants in order to create mini “editions,” or creating “liner notes” to speculate whether Petrarch or Dante is the mysterious “Italian poet from the thirteenth century” in Dylan’s “Tangled Up in Blue.”

But until recently, one thing I had never experimented with was giving students a choice about which assignments to complete or when to complete them. After all, standardizing the “what” and “when” of student work not only enables one to anticipate the grading tsunamis, but also ameliorates the grading process itself (read a few essay-based exams in a row, and you quickly develop a rubric for what “A” or a “B” answers looks like). Plus, as a female—and, for a while there, young—faculty member, I worried that I needed to be stringent in order to be taken seriously. Like developing a cherished reputation as a hard grader, I thought of inflexible deadlines as my friend. My reasons for keeping things simple and keeping them on a set schedule were not entirely self-interested, however. Like many professors, I found myself persuaded by the oft-repeated truism that it is part of our job to teach our students respect for deadlines. Surely it is the rare academic who has never turned in an article, book review, or assessment report after a promised deadline, and I admittedly felt a bit hypocritical playing bad cop. But we hear again and again that the work world, unlike higher education, will not tolerate the foibles of the disorganized and dilatory; in order to best prepare our students to flourish after graduation, it thereby seems like it must be our mandate to keep them on schedule.

A Midsummer Night's Dream act IV, scene I. Engraving from a painting by Henry Fuseli, published 1796. Public domain

A Midsummer Night’s Dream act IV, scene I. Engraving from a painting by Henry Fuseli, published 1796. Public domain

But over the years I began to resent occupying the role of the disciplinarian who insisted on calibrating demerits for each day of lateness. In terms of contact with students, deadline management also places inevitable emotional friction into the least edifying part of the student-teacher relationship. I’d so much rather talk about how my student’s thinking is evolving than adjudicate whether they really needed another 24 hours to turn in the essay reflecting said thinking. More important, the rigidity seemed inimical to the kind of creative, inventive, higher-order thinking I wanted to foster in both my students and myself, and the impression I wanted to leave of their encounter with the material. Do I want them to remember that I was the person who taught them to turn in an essay on time even if they have the flu? Or the person that enabled then to do their best work because I had not discouraged self-care?

Stepping Gradually into the New Assignment Waters

A few years ago, a new class gave me a chance to test out doing a few things differently. It was a new lecture-style course, a 100-level Introduction to Shakespeare directed at non-majors. I therefore anticipated many first-time visitors to college-level humanities courses: opera singers and oboe players, geologists and mathematicians, pre-med students taking a required English class, or first-years who had promised a grandparent they would try out Shakespeare. I wanted to signal my understanding that each of these students would bring different interests and strengths to the table, and I also wanted the “opportunity costs” of trying out something new to be low. At the same time, with 50 students in the class (and half that many in another that was writing intensive), there was a limit to how much I could really personalize things. So I considered a compromise. What if, in addition to short quizzes and a final exam, I gave students a menu of 4 choices for the remaining 10% of their grade? And what if I allowed them to turn in or perform their student-choice assignment whenever they chose?

Christian de Köhler, Othello with Desdemona, 1859. Public domain

Christian de Köhler, Othello with Desdemona, 1859. Public domain

To my delight, the experiment went very well. Most of the students chose to do a recitation (option: to perform a soliloquy or group scene either in class or in my office), while others cheered them on. Several chose to analyze a film adaptation, while another analyzed a performance of the opera Otello. A few discussed a scholarly article about one of the plays, and a couple others even examined primary sources reproduced in our textbook. While a few did treat the assignment as peripheral, most really relished the opportunity to take agency in their choice, and to round-out the course’s approach with something more personal. Certainly, the recitations animated the classroom, and created a sense of mutual respect and group bonding that made the lecture hall feel more intimate. But best of all, I was enabling a kind of multi-modal learning even in a lecture course, and in a way that I think increased the student sense of buy-in.

All In: Choose Your Own Adventure

I was pleased enough with how this went that, last semester, I decided to radically increase the student-choice component of an upper-level course, giving students what I described as “choose your own adventure” approach to the majority of assignments. It took a good amount of planning to set up, a significant amount of flexibility to administer, and a willingness to take a real leap of faith in my students. But if it enabled me to focus less on my role as Keeper of Deadlines, and more on my role as enabler of scrupulous analysis, elegant expression, and metacognition, how could that be bad? It certainly seemed like a class called Shakespeare and Metamorphosis, which read Shakespeare in conversation with classical Ovidian myths about transformation, was an ideal place to ask for more self-direction from students—and to try out something new myself.

Ovid, Metamorphoseon libri XV.... Title-page. Collection of Hayden White and Margaret Brose, 1556. Public domain

Ovid, Metamorphoseon libri XV…. Title-page. Collection of Hayden White and Margaret Brose, 1556. Public domain

Of course, the details of any such experiment would need to be adapted to the learning goals of the course, the skill level of the students, and the exigencies of the professor’s schedule (full disclosure: I am not doing anything like this now, during my hectic 3-course semester). But for me, it looked like this: all students were required to write one analytical essay (6-8 pages) worth 25% of the grade; and all students would receive 15% of their grade for their attendance and participation in discussion. Beyond this, it was up to each student to determine the means by which they would find it most beneficial to be evaluated according to a menu of options that I provided. Admittedly, this took me some time to work out:

 

 

 

5%:    Start off class discussion with a substantive response to something we’ve read; pose         questions.

10%:  Oral presentation (5-10 min) of an article—summarize, analyze, point to ideas it raises;

          or Visual analysis (3pp) of an object in the AMAM in relation to Ovid/Shakespeare;

          or Response paper (3pp.) analyzing any primary or secondary reading for the day.

15%:  Annotated bibliography of 3 articles not on the syllabus (one paragraph per source);

          or Analysis (3-4pp) of a translation of one of our primary texts.

20%:  Attempt to create your own myth (!);

          or Continue the story of any Ovidian character.

25%:  6-7 pp. analytical essay on any primary source from the second half of term;

          or an Ovidian Shakespeare text (e.g. Venus and Adonis) not on the syllabus;

          or one or more myth(s) from another culture/linguistic tradition, etc.;

          or a Final exam comprising discussion of terms, passage analysis, and one essay.

50%:  Final original research paper, 16-20pp.; requires 6-8 secondary sources and a consultation with a librarian.

?%:    Online virtual exhibition/web page: let’s discuss what you envision, and agree on scope.

          or? I am open to other unconventional approaches.

This menu of options was followed by five samples of how a student might get to 100 points, and some ground rules: All students had to submit a personal plan by week 2 for how they intended to meet the course requirements (they were allowed to modify the plan after this date if their interests changed, as long as they submitted a revised plan and explanation). I stipulated that no more than three of any element could be submitted, urging that in most cases two would be better to expand the skill set developed. All students had to earn at least 20 “student choice” points by the end of week 10, unless writing the research paper, and all were required to turn in a sheet showing their “accounting” at the end of the semester.

Ovid, Metamorphoses, "Minerva at the House of Envy," 1664 - 1700

Ovid, Metamorphoses, “Minerva at the House of Envy,” 1664 – 1700. Public domain

There were scheduling issues I had to think through. I needed to allow only two discussion starters per day so as to not take too much time away from class discussion; that part was therefore handled by sign-up sheet. I had to take into account the registrar’s rules for the submission of final projects. And I had to help the students, primarily through one-on-one conversations, think through how to figure out their rubric. But here is where the somewhat dizzying logistics made way for intentionality and responsibility: because each student had to give serious thought to what their aims and intentions for the course were. Is this a semester, I asked them, in which you really want to work on your writing by tackling a series of short essays with ongoing feedback? Is this the semester in which you want to try bringing together your English and Creative Writing majors by writing your own myth? Are you interested in working on your research skills, and therefore want to develop an annotated bibliography? Do you need more practice giving in-class presentations, and can we set up a schedule for those? It almost goes without saying that by asking them to take responsibility for what they most wanted to work on, they were more likely to achieve it. And far from thinking that flexibility was a synonym for slack, the class truly set itself wonderfully high expectations: from a scholarly 20-page research papers for a graduate-school bound student, to a site-specific installation by a TIMARA major in Tappan Square, replete with an original score, which reimagined the town green as an allegorical landscape.

And what about those deadlines? As you might imagine, students were initially a bit overwhelmed by this array of options. They did their earnest best to come up with plans they could stick to. But as they kept learning, their interests changed, and so did their intentions. I accepted every revision. Likewise, as deadlines piled up, many students had to revise their own due dates. I accepted every new date. Each time, the students seemed apologetic that they had fallen short in some way, but I told them that, instead, they could use this as a wonderful opportunity to self-reflect about how they worked best, what they might need to prioritize differently in the future, and how much time they required for various tasks. Because in reality, the world of work very rarely dictates singular deadlines that can be tackled one by one. Instead, we all must learn to multitask in increasingly complex and demanding environments that reward us more for nimbleness than for rigidity. And that, too, is something that my Shakespeare and Metamorphosis students so deftly taught me.

Evaluation Time!

Steven Volk, December 6, 2015

The debate over the value of Student Evaluations of Teaching (SETs) is a long one, which I have reported on a number of times (see here and here, among others). As we move into the last week of the semester, I’d like to suggest two additional approaches to end-of-semester evaluations that can help both you and your students think about the learning that occurred in your classes. I’ll also include a “guide” I wrote in 2010 for reading your SETs when they are returned to you after grades are in.

What Helped Your Learning?

Students enter the "Natio Germanica Bononiae,“ University of Bologna (15th century). Public domain. Wikimedia

Students enter the “Natio Germanica Bononiae,“ University of Bologna (15th century). Public domain. Wikimedia

SETs are largely about how students experienced your course, and so the questions focus on issues of organization, pacing, clarity, grading, etc. As numerous articles have pointed out, Student Evaluations of Teaching don’t tell you about student learning, and they provide very little information to suggest what it is you are doing to support (or hinder) the leaning that goes on in the class. Linda Shadiow and Maryellen Weimer, writing in Faculty Focus on Nov. 23, suggest a series of questions that can help foreground student learning issues. They offer a series of fairly simple sentence stems for students to complete. For example,

  • It most helped my learning of the content when…because…
  • It would have helped my learning of the content if…because…
  • The assignment that contributed most to my learning was…because…
  • The reading that contributed the most to my learning was…because…
  • The kinds of homework problems that contributed most to my learning were…because…
  • The approach I took to my own learning that contributed the most for me was…because…
  • The biggest obstacle for me in my learning the material was…because…
  • A resource I know about that you might consider using is…because…
  • I was most willing to take risks with learning new material when…because…
  • During the first day, I remember thinking…because…
  • What I think I will remember five years from now is…because…

Shadiow and Weimer recommend that faculty also complete the same sentences. Having (almost) completed the semester, we probably have a good idea which assignments worked from our point of view and which didn’t; what readings brought out the most in discussion, and what didn’t; what homework assignments stretched student learning and what brought basically “meh” responses. Comparing our answers to the students can be revealing (or, perhaps, horrifying!).

These questions can be added to the bottom of the current SETs that you will be handing out. You can simply add an additional sheet with these questions which, still anonymously, can be returned directly to you rather than being tabulated by the department AA’s or becoming a part of your official file. Once you have you have let some time pass (see “SETs for Beginners,” below), you can look at them, particularly before preparing classes for next semester.

Student Self-Assessment

Cours de philosophie à Paris Grandes chroniques de France; Castres, bibliothèque municipale – late 14th century. Public domain. Wikimedia

Cours de philosophie à Paris Grandes chroniques de France; Castres, bibliothèque municipale – late 14th century. Public domain. Wikimedia

David Gooblar, in his “Pedagogy Unbound” blog for the Chronicle of Higher Education, recently wrote a very useful post on student self-assessment. [Note: For a good introduction to student self-assessment, see Heidi Andrade & Anna Valtcheva, “Promoting Learning and Achievement Through Self-Assessment,” Theory Into Practice 48:1 (2009): 12-19.] Ask your students to reflect upon the learning strategies they used over the course of the semester, and to consider their own habits of thinking. “Explain that the act of reflection is itself a valuable learning strategy,” he writes. Ask them how they studied for tests or what they did to prepare for their assignments; what worked for them and what didn’t? The answers you get may be quite basic, but the more often we ask students to reflect on their own learning, the more practiced at it they will become.

Gooblar calls attention to the work on metacognition undertaken by Kimberly D. Tanner, a professor of biology at San Francisco State University, particularly the “retrospective postassessments” she uses. Tanner asks students to think about what they now know about the subject (in general terms) compared to what they knew at the start of the semester. (For example: “Before this course, I thought neoliberalism was _____. Now, I think neoliberalism is _____.”) You can ask students to write about the specific ways they have changed their thinking about the topics you covered. Or you can have them write a letter to a future student of the course, to reflect on its high and low points, and tell incoming students what they wished they would have had known going in to the class and what they wish they would have done differently over the course of the semester?

Since I have my students set out their learning goals in a short paper at the start of the class – which I collect and keep – I return these to them at the end of the semester and have them reflect, one final time, on which of these goals they feel they have achieved, and which they didn’t, what they did to reach their goals and what they will do differently in their next classes. Whatever method you chose, the end of the semester is a good time to encourage students to reflect on the journey they have undertaken with you and how they are different at the end of the trip.

“SETs for Beginners” (first written Feb. 7, 2010 and slightly updated here)

800px-Meeting_of_doctors_at_the_university_of_Paris

A meeting of doctors at the university of Paris. From the “Chants royaux” manuscript, Bibliothèque Nationale, Paris.

By now, as you well know, there is a very large literature on student evaluations of teaching (SETs). A lot of the research points to the validity and reliability of these instruments in terms of measuring very specific areas of a student’s experience in a completed class. Some writers continue to argue that they are a worthless exercise, citing evidence that evaluations handed out after the first day of a class will yield strikingly similar results to surveys conducted at the end of the semester, that they are a measure of the entertainment-value of a class, not any value added in terms of student learning, or that they can be easily influenced (just hand out doughnuts with the questionnaires). I have come to accept three basic realities about the use of SETs:

(1) on a broad level, they help to identify outliers – a class which seems to have been extremely successful or highly troubled; (2) they should not be used as the only evaluation of teaching (informed peer evaluation following a standard observation protocol and an examination of course syllabi by experts in the field are strongly recommended as well); and (3) SETs are not a substitute for an assessment of student learning in a course. But, when read carefully, they can tell you something about your teaching on a very specific level. The question is how to read them to get that specific information, and on that score, there is very little literature.

So here’s a first attempt at this question, a kind of “SETs for Beginners.”

Because of college rules, which are likely similar everywhere, we receive our teaching evaluations back only after grades are turned in. (You should consult with your department chairs for information as to how and when to hand out SETs, whether your students can complete them online or only in class, and how they are to be collected if you do them in-class.) At some point by mid-January or mid-June, after our hard-working administrative assistants have tabulated and organized the data, we are informed that our SETs are ready to be picked up!

First decision: do you rush in to get them, play it cool, like a cat walking around a particularly lovely kibble before pouncing, or pretend that they aren’t there until, sure enough, you have forgotten all about them? I usually take the middle route on this, but, in any case, I certainly won’t pick them up on a day when the auto shop called to tell me that the problem’s in the drive train or the best journal just rejected my article. Another hit that day, I just don’t need.

When I do finally make the move, I take them to my office, put them on my desk, pretend that they aren’t there while I take a look at my email which I already checked 90 seconds earlier. Enough, already. I open the folders and read, rapidly, the overall numbers: not what I hoped for, better than it could have been, whatever… Then I put them away for at least a day or two. I don’t think I’m ready to take them on-board just yet, whether the numbers are good, bad, or indifferent. I go back to the email, the article, the gym, until I’ve absorbed the larger quantitative landscape and feel mentally prepared to explore the terrain a bit more carefully.

When I do return to the SETs, I give myself the time (and space) to read them carefully (and privately). I don’t pay much attention to any individual numbers – those have been summarized for me, but I read the comments with care… and a mixture of interest, confusion, skepticism, and wonder. How is it that the student who wrote “Volk is probably the most disorganized professor I’ve ever encountered” attended the same class as the one who commented, “This was a marvel of organization and precision”? What is one to make of such clearly cancelling comments? So here are my tricks for trying to give my SETs the kind of close reading that I think they merit:

  • Do not dwell on the angry outliers. That’s advice more easily given than taken. I have read enough teaching evaluations, my own as well as others, to know that there are some students who just didn’t like our classes and have not figured out any helpful or gracious way to say that. The fact that these are (hopefully) a tiny minority and are directly contradicted by the great majority of other comments doesn’t seem to decrease their impact, or the fact that we continue to obsess about them. (I can still quote, verbatim, comments that were written in 1987!) These bitter communiqués probably serve a purpose for the student, but they really don’t help you think at all usefully about your teaching. Let ’em go.
  • Evaluate the “cancellers”. What do you do when three students thought you were able to organize and facilitate discussions with a high degree of skill and three thought you couldn’t organize a discussion to save your life? These are harder to deal with and can add to the cynicism of those who think that the whole SET adventure is a waste of time. For the “cancellers,” I try to figure out a bit more about them to see if they represent some legitimate (i.e., widespread) concern about the class or not. Is one side of the debate generally supported by the numbers? Do I score lower in the discussion-oriented questions than in other areas, lower than in previous iterations of the course, or lower than I would have really wanted? Does the demographic information that I know about the student evaluator add context that is useful and that I should take on board? I am more likely to trust comments from seniors than from first-years. Does it appear that there is a striking gender or racial difference in terms of how students respond to specific questions? That is extremely important information to lean from and it is why we collect demographic information from our student respondents. A careful reading of this information can help us understand what is going on in our classes on a more granular level.
Brian Carson, Backyard Flowers in Black and White, No. 2. Flickr Creative Commons

Brian Carson, Backyard Flowers in Black and White, No. 2. Flickr Creative Commons

And, if none of the above helps me think about why something I have done works for some and not others, I make a note to myself to ask students explicitly about it the next time I offer the class: Please, tell me if you don’t think these (discussions, paper assignments, readings, etc.) are working so that I can consider other approaches.

  • Focus on those areas that seem to be generating the most amount of concern from students. Are they having a hard time trying to figure out how the assignments relate to the reading? Do a considerable number worry that they aren’t getting timely or useful feedback from you? Is there a widespread upset that classes run too long and students don’t have enough time to get to their next class? For each of the areas where I find a concern that has reached a “critical mass” level and is not just an angry-outlier grievance, I consider what I think about their criticism and whether, given my own goals in the course, I find it legitimate. For example, I will pay no attention to students who complain about the early hour of my class. Getting work back on time depends on the size of the class and what I have promised: in a 50-person class if I say I’ll return work within two weeks, and do so, then I won’t think much about students who complain that I only returned their work two weeks after they turned it in.

Other issues force me to think more about how I teach and what impact that has on student learning. What of students who protest that “there’s too much work for a 100-level class”? I get a lot of those comments, and it makes me think: why do students think a 100-level class should involve less work than a 300-level class? Do we, the faculty, think that a 100-level class should be less work than a senior seminar? Certainly, upper-level classes will be more “difficult” than 100-level classes (i.e., they demand that the students have acquired significant prior knowledge and skills needed to engage at a higher level), but should there be any less work involved in the entry-level class? I, for one, don’t think so – and so I won’t change that aspect of the course.

But, ultimately, when student comments suggest what is a real area of concern, when they point to something I am doing in the class that negatively impacts how students are able to learn, than I need to regard that issue with the seriousness it deserves. I will think about how I might correct the problem, and, often, the best way to do that is to talk to my colleagues and find someone in my department or outside who can read my SETs with me. That has served me well every time, and it does point to the ultimate utility of SETs for the individual faculty member on a formative level: they can help us to design our teaching to more effectively promote student learning.

  • Finally, since Oberlin really does attract faculty who care about their students and the challenges of student learning, then my guess is that your evaluations are generally good, and you need to take great satisfaction in that (see: “Don’t dwell on the angry outliers”). I have never failed to find some comments on my own evaluations that remind me yet again about how perceptive our students can be and how fortunate I am to be here.