Tag Archives: self-assessment

Evaluation Time!

Steven Volk, December 6, 2015

The debate over the value of Student Evaluations of Teaching (SETs) is a long one, which I have reported on a number of times (see here and here, among others). As we move into the last week of the semester, I’d like to suggest two additional approaches to end-of-semester evaluations that can help both you and your students think about the learning that occurred in your classes. I’ll also include a “guide” I wrote in 2010 for reading your SETs when they are returned to you after grades are in.

What Helped Your Learning?

Students enter the "Natio Germanica Bononiae,“ University of Bologna (15th century). Public domain. Wikimedia

Students enter the “Natio Germanica Bononiae,“ University of Bologna (15th century). Public domain. Wikimedia

SETs are largely about how students experienced your course, and so the questions focus on issues of organization, pacing, clarity, grading, etc. As numerous articles have pointed out, Student Evaluations of Teaching don’t tell you about student learning, and they provide very little information to suggest what it is you are doing to support (or hinder) the leaning that goes on in the class. Linda Shadiow and Maryellen Weimer, writing in Faculty Focus on Nov. 23, suggest a series of questions that can help foreground student learning issues. They offer a series of fairly simple sentence stems for students to complete. For example,

  • It most helped my learning of the content when…because…
  • It would have helped my learning of the content if…because…
  • The assignment that contributed most to my learning was…because…
  • The reading that contributed the most to my learning was…because…
  • The kinds of homework problems that contributed most to my learning were…because…
  • The approach I took to my own learning that contributed the most for me was…because…
  • The biggest obstacle for me in my learning the material was…because…
  • A resource I know about that you might consider using is…because…
  • I was most willing to take risks with learning new material when…because…
  • During the first day, I remember thinking…because…
  • What I think I will remember five years from now is…because…

Shadiow and Weimer recommend that faculty also complete the same sentences. Having (almost) completed the semester, we probably have a good idea which assignments worked from our point of view and which didn’t; what readings brought out the most in discussion, and what didn’t; what homework assignments stretched student learning and what brought basically “meh” responses. Comparing our answers to the students can be revealing (or, perhaps, horrifying!).

These questions can be added to the bottom of the current SETs that you will be handing out. You can simply add an additional sheet with these questions which, still anonymously, can be returned directly to you rather than being tabulated by the department AA’s or becoming a part of your official file. Once you have you have let some time pass (see “SETs for Beginners,” below), you can look at them, particularly before preparing classes for next semester.

Student Self-Assessment

Cours de philosophie à Paris Grandes chroniques de France; Castres, bibliothèque municipale – late 14th century. Public domain. Wikimedia

Cours de philosophie à Paris Grandes chroniques de France; Castres, bibliothèque municipale – late 14th century. Public domain. Wikimedia

David Gooblar, in his “Pedagogy Unbound” blog for the Chronicle of Higher Education, recently wrote a very useful post on student self-assessment. [Note: For a good introduction to student self-assessment, see Heidi Andrade & Anna Valtcheva, “Promoting Learning and Achievement Through Self-Assessment,” Theory Into Practice 48:1 (2009): 12-19.] Ask your students to reflect upon the learning strategies they used over the course of the semester, and to consider their own habits of thinking. “Explain that the act of reflection is itself a valuable learning strategy,” he writes. Ask them how they studied for tests or what they did to prepare for their assignments; what worked for them and what didn’t? The answers you get may be quite basic, but the more often we ask students to reflect on their own learning, the more practiced at it they will become.

Gooblar calls attention to the work on metacognition undertaken by Kimberly D. Tanner, a professor of biology at San Francisco State University, particularly the “retrospective postassessments” she uses. Tanner asks students to think about what they now know about the subject (in general terms) compared to what they knew at the start of the semester. (For example: “Before this course, I thought neoliberalism was _____. Now, I think neoliberalism is _____.”) You can ask students to write about the specific ways they have changed their thinking about the topics you covered. Or you can have them write a letter to a future student of the course, to reflect on its high and low points, and tell incoming students what they wished they would have had known going in to the class and what they wish they would have done differently over the course of the semester?

Since I have my students set out their learning goals in a short paper at the start of the class – which I collect and keep – I return these to them at the end of the semester and have them reflect, one final time, on which of these goals they feel they have achieved, and which they didn’t, what they did to reach their goals and what they will do differently in their next classes. Whatever method you chose, the end of the semester is a good time to encourage students to reflect on the journey they have undertaken with you and how they are different at the end of the trip.

“SETs for Beginners” (first written Feb. 7, 2010 and slightly updated here)

800px-Meeting_of_doctors_at_the_university_of_Paris

A meeting of doctors at the university of Paris. From the “Chants royaux” manuscript, Bibliothèque Nationale, Paris.

By now, as you well know, there is a very large literature on student evaluations of teaching (SETs). A lot of the research points to the validity and reliability of these instruments in terms of measuring very specific areas of a student’s experience in a completed class. Some writers continue to argue that they are a worthless exercise, citing evidence that evaluations handed out after the first day of a class will yield strikingly similar results to surveys conducted at the end of the semester, that they are a measure of the entertainment-value of a class, not any value added in terms of student learning, or that they can be easily influenced (just hand out doughnuts with the questionnaires). I have come to accept three basic realities about the use of SETs:

(1) on a broad level, they help to identify outliers – a class which seems to have been extremely successful or highly troubled; (2) they should not be used as the only evaluation of teaching (informed peer evaluation following a standard observation protocol and an examination of course syllabi by experts in the field are strongly recommended as well); and (3) SETs are not a substitute for an assessment of student learning in a course. But, when read carefully, they can tell you something about your teaching on a very specific level. The question is how to read them to get that specific information, and on that score, there is very little literature.

So here’s a first attempt at this question, a kind of “SETs for Beginners.”

Because of college rules, which are likely similar everywhere, we receive our teaching evaluations back only after grades are turned in. (You should consult with your department chairs for information as to how and when to hand out SETs, whether your students can complete them online or only in class, and how they are to be collected if you do them in-class.) At some point by mid-January or mid-June, after our hard-working administrative assistants have tabulated and organized the data, we are informed that our SETs are ready to be picked up!

First decision: do you rush in to get them, play it cool, like a cat walking around a particularly lovely kibble before pouncing, or pretend that they aren’t there until, sure enough, you have forgotten all about them? I usually take the middle route on this, but, in any case, I certainly won’t pick them up on a day when the auto shop called to tell me that the problem’s in the drive train or the best journal just rejected my article. Another hit that day, I just don’t need.

When I do finally make the move, I take them to my office, put them on my desk, pretend that they aren’t there while I take a look at my email which I already checked 90 seconds earlier. Enough, already. I open the folders and read, rapidly, the overall numbers: not what I hoped for, better than it could have been, whatever… Then I put them away for at least a day or two. I don’t think I’m ready to take them on-board just yet, whether the numbers are good, bad, or indifferent. I go back to the email, the article, the gym, until I’ve absorbed the larger quantitative landscape and feel mentally prepared to explore the terrain a bit more carefully.

When I do return to the SETs, I give myself the time (and space) to read them carefully (and privately). I don’t pay much attention to any individual numbers – those have been summarized for me, but I read the comments with care… and a mixture of interest, confusion, skepticism, and wonder. How is it that the student who wrote “Volk is probably the most disorganized professor I’ve ever encountered” attended the same class as the one who commented, “This was a marvel of organization and precision”? What is one to make of such clearly cancelling comments? So here are my tricks for trying to give my SETs the kind of close reading that I think they merit:

  • Do not dwell on the angry outliers. That’s advice more easily given than taken. I have read enough teaching evaluations, my own as well as others, to know that there are some students who just didn’t like our classes and have not figured out any helpful or gracious way to say that. The fact that these are (hopefully) a tiny minority and are directly contradicted by the great majority of other comments doesn’t seem to decrease their impact, or the fact that we continue to obsess about them. (I can still quote, verbatim, comments that were written in 1987!) These bitter communiqués probably serve a purpose for the student, but they really don’t help you think at all usefully about your teaching. Let ’em go.
  • Evaluate the “cancellers”. What do you do when three students thought you were able to organize and facilitate discussions with a high degree of skill and three thought you couldn’t organize a discussion to save your life? These are harder to deal with and can add to the cynicism of those who think that the whole SET adventure is a waste of time. For the “cancellers,” I try to figure out a bit more about them to see if they represent some legitimate (i.e., widespread) concern about the class or not. Is one side of the debate generally supported by the numbers? Do I score lower in the discussion-oriented questions than in other areas, lower than in previous iterations of the course, or lower than I would have really wanted? Does the demographic information that I know about the student evaluator add context that is useful and that I should take on board? I am more likely to trust comments from seniors than from first-years. Does it appear that there is a striking gender or racial difference in terms of how students respond to specific questions? That is extremely important information to lean from and it is why we collect demographic information from our student respondents. A careful reading of this information can help us understand what is going on in our classes on a more granular level.
Brian Carson, Backyard Flowers in Black and White, No. 2. Flickr Creative Commons

Brian Carson, Backyard Flowers in Black and White, No. 2. Flickr Creative Commons

And, if none of the above helps me think about why something I have done works for some and not others, I make a note to myself to ask students explicitly about it the next time I offer the class: Please, tell me if you don’t think these (discussions, paper assignments, readings, etc.) are working so that I can consider other approaches.

  • Focus on those areas that seem to be generating the most amount of concern from students. Are they having a hard time trying to figure out how the assignments relate to the reading? Do a considerable number worry that they aren’t getting timely or useful feedback from you? Is there a widespread upset that classes run too long and students don’t have enough time to get to their next class? For each of the areas where I find a concern that has reached a “critical mass” level and is not just an angry-outlier grievance, I consider what I think about their criticism and whether, given my own goals in the course, I find it legitimate. For example, I will pay no attention to students who complain about the early hour of my class. Getting work back on time depends on the size of the class and what I have promised: in a 50-person class if I say I’ll return work within two weeks, and do so, then I won’t think much about students who complain that I only returned their work two weeks after they turned it in.

Other issues force me to think more about how I teach and what impact that has on student learning. What of students who protest that “there’s too much work for a 100-level class”? I get a lot of those comments, and it makes me think: why do students think a 100-level class should involve less work than a 300-level class? Do we, the faculty, think that a 100-level class should be less work than a senior seminar? Certainly, upper-level classes will be more “difficult” than 100-level classes (i.e., they demand that the students have acquired significant prior knowledge and skills needed to engage at a higher level), but should there be any less work involved in the entry-level class? I, for one, don’t think so – and so I won’t change that aspect of the course.

But, ultimately, when student comments suggest what is a real area of concern, when they point to something I am doing in the class that negatively impacts how students are able to learn, than I need to regard that issue with the seriousness it deserves. I will think about how I might correct the problem, and, often, the best way to do that is to talk to my colleagues and find someone in my department or outside who can read my SETs with me. That has served me well every time, and it does point to the ultimate utility of SETs for the individual faculty member on a formative level: they can help us to design our teaching to more effectively promote student learning.

  • Finally, since Oberlin really does attract faculty who care about their students and the challenges of student learning, then my guess is that your evaluations are generally good, and you need to take great satisfaction in that (see: “Don’t dwell on the angry outliers”). I have never failed to find some comments on my own evaluations that remind me yet again about how perceptive our students can be and how fortunate I am to be here.

Learning from the Semester

As the semester races (or crawls) to a close, it’s a good time to think about capturing what you (as opposed to your students) learned from the semester. I’ve posted some material on this in earlier “Article of the Week” columns (see, for example December 5, 2011: “End of Semester Self-Evaluation,” and November 28, 2011: “Assessing your own Teaching”). Here are three different ways to track your teaching, each slightly more intensive. For the Shostakovich fans out there, I’ve labeled each so you can plan your time accordingly.

Shostakovich, Gadfly Suite (Amazon.com)

(1) End of Semester Reflections: Short, think of it as the Gadfly Suite, Op. 97a

While you can reflect on your teaching at any point of the semester (see nos. 2 and 3 below), there are two times that I have found to be particularly productive: Some 2-3 weeks before the semester ends (when you already have a very good sense about how the semester has gone), and about 2-3 weeks after the semester ends (once you have had a chance to read the student evaluations). Granted that everyone is unbelievably busy right now, try to set aside 30 minutes to begin to answer these questions (and return to them when you can). It is useful to engage in this process before you get student evaluations of your teaching. You want to think from your own perspective as to why the semester worked out as it did.

  • What do you feel was the strongest part of your teaching (and student learning) this semester?

What did you accomplish (think of it concretely: the assignment I designed which was intended to help me evaluate whether students were reading the text closely worked really well; the discussions worked much better than previously; their recall of basic material as shown in their exams was better than I had expected; I was able to establish a dynamic in class where student could begin to talk about extremely difficult topics; etc.).

  • Why do you think that happened? Link outcomes to your teaching methods.

What did you do differently? Was it a matter of the composition of the class or of your methods? If outcomes were different than in previous years, try to think why that was the case.

  • Do you think you achieved your learning goals for the course? This, of course, should lead you back to your learning objectives, help you think about them again, and consider whether you can actually answer this question.

What methods of assessment – papers, tests, projects, etc. – do you use that can help you answer this question concretely?

  • What do you think basically didn’t work in the course? What do you feel least pleased, or most uneasy, about?  What left you thinking: next time, I just won’t do that?

Think about this in terms of your teaching approach (pedagogy); circumstances (make sure I’m never teaching in that classroom again; avoid 2:30 PM classes; this is a class that works for 20 students but is a disaster for 8; I let one student assert too much control over the other students, etc.); materials and assignments (too much, improperly paced, directions unclear, etc.)

  • As above: Why did you (or didn’t you) reach your learning objectives? Link outcomes to your teaching approach.
  • Getting concrete: what do you want to at least think about doing differently next time?

Readings; the way discussions are structured; adding more active lecturing techniques, dropping a particular content area, etc.

  • Very briefly: If you are not sure what to do to change the results, who are the people and what are the resources that can help?

Talk to colleagues, mentors here or elsewhere, consult with CTIE, find materials that address the topic, at CTIE, or elsewhere, develop an action plan, etc.

Try to go through the same exercise after you have read and digested the student evaluations of your teaching. (For advice on how/when to read your students’ evaluations, see the “Article of the Week” from Feb. 8, 2010: Reading Student Evaluation of Teaching).  Get a sense of whether your self-evaluation finds resonance in the students’ comments, or whether you come to different conclusions. Think about – or talk to a colleague about – any disparities: just because the students liked your class (i.e., gave you favorable ratings), it doesn’t mean that you met your learning objectives. Just because some students didn’t like certain aspects of the course, it doesn’t mean that they should be jettisoned.

Shostakovich, Jazz Suite No. 2 (Allmusic.com)

(2) Annotated Syllabus: Medium, maybe the Suite for Jazz Orchestra No. 1

Create a “dummy” syllabus for your class. If your regular syllabus doesn’t include information on what you are planning to do on a class-by-class basis, make sure that this dummy syllabus does. So, for example:

Wednesday, November 27: Classify polysaccharides based on function in plants and animals and describe how monomers join to form them.

Each day, after that class has finished, enter some notes on the syllabus as to how the class went, paying particular attention to whether you think that the class helped the students get to the objectives you have set out (in this case classifying polysaccharides…). Also think about what evidence you have to answer this question (do you ask for “muddy points” responses at the end of class? Do you use clickers or other audience response systems that let you know whether the students are “getting” it?).

Jot down notes of what, to you, worked and what didn’t: was it the way you broke them up into discussion groups? The amount or nature of the reading assigned? The presence or absence of contextualizing material? The literal day you presented the material (Duh! It was the day before Thanksgiving; what was I thinking!).

Finally, enter some notes as to what you would do differently the next time around: Less/more reading; start with a quiz to see where they are at; have them work in groups; make the goals of the class more transparent; work to create an atmosphere where students can talk more easily about controversial issues; etc.

Don’t be hard on yourself if you miss annotating classes now and again. This is a jazz suite, after all, improvise. Maybe your best bet is to try to open a syllabus template that you can get to whenever you can. If you set impossible goals, you won’t accomplish them, and the purpose is not to find another reason to feel guilty (now, where did I leave my hairshirt?) but to begin a practice that can empower you.

(Flickr, Creative Commons – exquisitur)

(3) The Teaching Portfolio: We’re talking Symphony No. 5

When you’re up to Shostakovich’s Symphony No. 5, you’re willing to spend some quality time thinking about your teaching. At some level, the teaching portfolio is an ongoing conversation between #2 (the daily syllabus annotations) and #1 (the end of semester reflections). The syllabus annotation is at the heart of a teaching portfolio, but the portfolio allows you greater space for reflection on your teaching philosophy, pedagogical approaches, readings on – and thoughts about – learning theory, longer blog posts (either public or private), articles that have influenced your thinking, etc.

You can set up a portfolio quite easily using Google sites or any one of a number of (free) commercial products (WordPress, IMCreator, etc.). The main issue is not to get hung up on the technology. Perhaps all you want is a set of folders (on your computer or actual folders) into which you can place these materials.

The main goal of the teaching portfolio, as far as I’m concerned, is to complete the feedback loop that ties together action, reflection, and reformulation. For example: I tried a very directed set of primary source readings in my philosophy class to get the students to understand John Stuart Mill’s concept of liberalism and the individual. I don’t think it worked given that their answers to a short reflection piece at the end of the class and the papers on the topic which they turned in two weeks later were imprecise and often factually incorrect. I’ve thought about what I tried to do in that class, talked about it with a colleague in the department, and read more about what other philosophy teachers do when teaching Mill. Here’s what I’ll do the next time and why I think it might work better (and so on and so forth…).

Again, you may want to simply expand the “dummy syllabus” idea presented in #2 or construct a more traditional teaching portfolio (for more on this, see “Teaching Portfolios” at CTIE’s “Resources” site). But reflection leading to redesign is the key to this approach. It takes some time, but it’s worth it.

The Rehearsal (Flickr-Creative Commons - Steve Bowbrick)

Conclusion:

The main point, and let me stress this again, is to begin a reflective process about your teaching that you feel comfortable with and, most importantly, works for you. Starting with the Symphony No. 5 will make sense for some, but not many. Maybe you work best with études. But in all cases do as Shakespeare suggested and play on.

Steve Volk

Director, Center for Teaching Innovation and Excellence (CTIE)

November 25, 2013