Tag Archives: Evaluation

Learning from the Semester: 2.0

Steven Volk (April 25, 2016)

[The following is an edited and updated version of a post from 2013.]

From Guy Newell Boothby, "Doctor Nikola" (London: Ward, Lock, & Co,  1986), p. 335. British Library.

From Guy Newell Boothby, “Doctor Nikola” (London: Ward, Lock, & Co, 1986), p. 335. British Library.

As the semester moves to it close (insert fist pump), it’s a good time to reflect on what you learned from the semester as well as considering what you think your students are taking away from your classes. To begin, here are three ways to track your teaching, from the quick and simple to the more time consuming.

End of Semester Snapshop

While you can, and probably should, reflect on your teaching at many points during the semester (see nos. 2 and 3 below), two moments can be particularly productive: Some 2-3 weeks before the semester ends (when you already have a very good sense about how the semester has gone), and about 2-3 weeks after the semester ends (or once you have had a chance to read student evaluations). You are all unbelievably busy right now, but try to set aside 30 minutes to begin to answer these questions – and then return to them when you can. It is useful to engage in this process before you read the students’ evaluations, as you want to be able to consider from your own perspective why the semester turned out as it did.

(1) What do you feel was the strongest part of your teaching (and student learning) this semester?

What did you accomplish? Try to answer this question concretely. Was it the assignment you designed to help you evaluate whether students were reading the text closely and which worked exactly as planned? The discussions, which were a lot livelier than other times you taught the class? The students’ ability to recall basic materials, as demonstrated by better exam results than in previous years? The fact that you were able to establish a dynamic in class that allowed students to talk about extremely difficult topics? In short: What worked well in the class?

(2) Why do you think that happened? Can you link these outcomes to your teaching methods.

What did you do differently? Was it a matter of the composition of the class or of your methods? If outcomes were different than in previous years, reflect on why that was the case.

"Lilliput Lyrics," Edited by R. Brimley Johnson; Illustrated by Chas. Robinson (London: J. Lane, 1899), p. 253. British Library

“Lilliput Lyrics,” Edited by R. Brimley Johnson; Illustrated by Chas. Robinson (London: J. Lane, 1899), p. 253. British Library

(3) Did you achieve your learning goals for the course?

This, of course, should lead you back a consideration of your learning objectives, help you think about them again, and consider whether you can actually answer this question.

Did you use assessment methods – papers, tests, projects, etc. – that can help you answer this question reasonably? If you find that you have learning goals that aren’t being assessed, you should make a note to change that next semester.

(4) What were you dissatisfied with in terms of how the course is turning out?

What didn’t work as you would have liked it in your classes? What do you feel least pleased, or most uneasy, about?  What left you thinking, “Next time, I probably shouldn’t do that”?

You can think about this in a variety of ways. For example:

(a) The pedagogy you employed. The mix of discussion and lecture, more active learning techniques, preparation for discussions, group work, student presentations, etc.

(b) Structural factors: Maybe you have found that teaching after lunch is not the best time; that the classroom you were assigned did not help your teaching and should be changed, that the class size did not lend itself to the particular pedagogy you employed.

(c) Classroom management issues. Did you allow one student to assert too much sway over the other students? Did you not step in where you should have? Did you not address management issues early enough? Should laptops be banned in your class as students are not using them appropriately? Should you have a “bathroom” policy to prevent a continual in-and-out of students from the class? How have you responded to challenges to your authority? How have you dealt with tensions that have come up in the class?

(d) Course Materials: Were students doing the readings? If not, why? Was the reading too basic? Too theoretical? Did mechanical issues (not being able to upload files, etc.) get in the way of their being able to complete assigned readings? Were the readings improperly paced (too much right during midterms) or unengaging (even for you!).

(e) Assignments: Too many? Too few to give students proper feedback? Should you be assigning multiple drafts of papers? Would smaller quizzes work better than one or two high-stakes exams? Did you assign collaborative work without preparing for it?

    "Lilliput Lyrics," Edited by R. Brimley Johnson; Illustrated by Chas. Robinson (London: J. Lane, 1899), p. 328. British Library.

“Lilliput Lyrics,” Edited by R. Brimley Johnson; Illustrated by Chas. Robinson (London: J. Lane, 1899), p. 328. British Library.

(5) As with your successes, think about why things didn’t work and what you can do the next time to change those aspects that you can change.

If time doesn’t permit you to plan out a concrete strategy for doing things differently next semester, jot down a note to remind you about the things that you should consider addressing.

(6) Who can help?

If you are not sure what to do to change those aspects of your course that you agree should be changed, jot down the name of the person/people you can talk to or the resource you can use.  Who are the colleagues and mentors, on campus or elsewhere, who you should be emailing to set up a coffee date? Where can you find materials that address the topics of your concern?

After the SETs Come In

Try to go through the same exercise after you have read and digested the student evaluations of teaching (SETs) for your courses. (For advice on how and when to read your students’ evaluations, see the “Article of the Week” from Feb. 7, 2010: Reading Student Evaluation of Teaching).  Get a sense of whether your self-evaluation finds any resonance in the students’ comments, or whether you come to different conclusions – and you need to think about why that’s the case. Reflect on – or talk to a colleague about – any disparities. Just because the students liked your class (i.e., gave you favorable ratings), it doesn’t mean that you met your learning objectives. Just because some students didn’t like certain aspects of the course, it doesn’t mean that those aspects should be jettisoned.

Longer-term Reflection: Annotated Syllabus

    "Lilliput Lyrics," Edited by R. Brimley Johnson; Illustrated by Chas. Robinson (London: J. Lane, 1899), p. 241. British Library.

“Lilliput Lyrics,” Edited by R. Brimley Johnson; Illustrated by Chas. Robinson (London: J. Lane, 1899), p. 241. British Library.

While it is useful to reflect back on your class at the end of the semester, you can gain more insight by reflecting on your classes in real time. This is particularly useful for people like me whose memory, to quote Billy Collins, has “decided to retire to the southern hemisphere of the brain, to a little fishing village where there are no phones.” Create a “dummy” syllabus for your class. If your regular syllabus doesn’t include information on what you are planning to do on a class-by-class basis, make sure that this dummy syllabus does. So, for example:

Wednesday, November 27: Make goal of class: Help students classify polysaccharides based on function in plants and animals and describe how monomers join to form them.

Each day, after that class has finished, enter some notes on the syllabus as to how the class went, paying particular attention to whether you think that the class helped the students reach the objectives you have set out (in this case classifying polysaccharides). Also think about what evidence you have to answer this question (do you ask for “muddy points” responses at the end of class? Do you use clickers or other audience response systems that let you know whether the students are “getting” it?).

Jot down notes of in your opinion worked and what didn’t: was it the way you broke them up into discussion groups? The amount or nature of the reading assigned? The presence or absence of contextualizing material? The day you chose to examine the topic (The day before Thanksgiving? What was I thinking!).

Finally, enter some notes as to what you would do differently the next time around: Less/more reading; start with a quiz to see where they are at; have them work in groups; make the goals of the class more transparent; work to create an atmosphere where students can talk more easily about controversial issues; etc.

Don’t be hard on yourself if you miss annotating classes now and again. The last thing you need is to be hard on yourself. Maybe your best bet is to try to open a syllabus template that you can get to whenever you can. If you set impossible goals, you won’t accomplish them, and the purpose is not to find another reason to feel guilty (and we all have many of those) but to begin a practice that can be empowering.

In For a Penny, In for a Pound: The Teaching Portfolio

    "Lilliput Lyrics," Edited by R. Brimley Johnson; Illustrated by Chas. Robinson (London: J. Lane, 1899), p. 226. British Library.

‘Lilliput Lyrics … Edited by R. Brimley Johnson. Illustrated by Chas. Robinson’ 226

To contemplate creating a teaching portfolio is to accept that you’re willing to spend some quality time reflecting on your teaching. At some level, the teaching portfolio is an ongoing conversation between #2 (the daily syllabus annotations) and #1 (the end of semester reflections). The syllabus annotation is at the heart of a teaching portfolio, but the portfolio allows you greater space for reflection on your teaching philosophy, pedagogical approaches, readings on – and thoughts about – learning theory, longer blog posts (either public or private), articles that have influenced your thinking, etc.

You can set up a portfolio quite easily using Google sites or any one of a number of (free) commercial products (WordPress, IMCreator, etc.). The main issue is not to get hung up on the technology. Perhaps all you want is a set of folders (either on your computer or actual folders) into which you can place these materials: standard syllabus, annotated syllabus, reflections on particular classes or on the course in general, emerging “philosophy” of teaching, notes on pedagogy, classroom management style, essays on finding your own teaching style, articles that have proven particularly important in your teaching, comments from people who have observed your teaching, student reflections, student work in response to particular prompts, comments from mentors and colleagues, etc., etc.

The main goal of the teaching portfolio, as far as I’m concerned, is to complete the feedback loop that ties together action, reflection, and reformulation. For example: Tried a very directed set of primary source readings in philosophy class to get students to understand John Stuart Mill’s concept of liberalism and the individual. Don’t think it worked given that their answers to a short reflection piece at the end of the class; papers on topic turned in two weeks later were imprecise and often factually incorrect. Thought about goals for that class, talked about it with a colleague in the department, and read more about what other philosophy teachers do when teaching Mill. Here’s a plan for the next time…

For more on teaching portfolios, consult the excellent handbook written by Hannelore B Rodriguez-Farrar (The Teaching Portfolio: A Handbook for Faculty, Teaching Assistants, and Teaching Fellows) at Brown University, the materials prepared by the Center for Teaching at Vanderbilt University, or the paper (“The Teaching Portfolio”) by Matthew Kaplan at the University of Michigan’s Center for Research on Teaching and Learning.

Final Reflections: What Have Your Students Carried Away?

    "Lilliput Lyrics," Edited by R. Brimley Johnson; Illustrated by Chas. Robinson (London: J. Lane, 1899), p. 227. British Library.

“Lilliput Lyrics,” Edited by R. Brimley Johnson; Illustrated by Chas. Robinson (London: J. Lane, 1899), p. 227. British Library.

The end of the semester is a time, all too often, of exhaustion and, at some level and speaking for myself, disappointment. In light of this, reflecting on what we think our students have actually absorbed from our classes is a useful exercise.

One of the most complicated issues we face in teaching is understanding in a comprehensive fashion what our students have taken away from the course. I think of this as somewhat different from what they have “learned.” We can get a good sense of that through our students’ written work or quizzes and examinations. What I’m talking about is more speculative: what do we think they will carry with them into the future, what will shape the way they think about the subject of our classes or more broadly? What will they remember 10 or 20 years in the future?

This is, of course, one of the devilishly hard questions of assessment. In the humanities, in particular, we know that more often than not, many students will “get it” only after the course is over. Synapses will be closed that remained wide-open during the class; light bulbs will finally turn on. And, more often than not, when this happens, it won’t be tied back to a particular class or even a particular course.

Of course, there is no way to know what the group of students just completing your class will take away from it. But thinking into the future is actually the starting point of “backward planning” and, as such, the first step for planning your next course syllabus. So, what do we think they will put in their backpacks and carry away with them?

I’ll use my own teaching this semester as an example. One of my classes is on museum studies (“Museum Narratives”). I am quite sure that only a few – OK, no one – will remember anything about exhibition morphology, how depth, ring factor, and entropy work in exhibition design. But I think that most, when they walk into a museum in the future, will think about how exhibition layout relates to content and audience, will search for the museum’s narrative rather than only focusing on its artifacts, and will continue to consider what Stephen Greenblatt meant when he divided museum exhibitions between those that worked through resonance versus those that work by wonderment.

And maybe that’s good enough.

Contract Improv – Three Approaches to Contract Grading

Steve Volk, March 27, 2016

Benin Plaque, c. 16- 17th century, brass “lost-wax” technique, British Museum Af1898,0115.38

Benin Plaque, c. 16- 17th century, brass “lost-wax” technique, British Museum Af1898,0115.38

Students in museums studies, more so than casual visitors, are frequently confronted with the question of how specific artifacts made their way from their point of origin into the museum where they are displayed for our enjoyment and edification. For some objects, the answer is relatively straightforward: the painting originally in, say, the French royal collection, was purchased by a dealer who sold it to a collector who donated it to the museum. For other artifacts, particularly if the museum in question is the British Museum (the end point of a vast collection of imperial booty), the origins of the artifact is more troubled. The catalog entry for this “Benin Plaque” (left), dating from the 16th-17th centuries, calmly notes that “following the British occupation of Benin City (Edo) in 1897 objects made of brass, ivory and wood were seized by British force from the royal quarters and various storerooms.”

But as this information doesn’t appear on the object’s label in the gallery, the viewer has little sense of the violent history — the imperial relations — that underwrote the trajectory of the plaque from Benin City to its current abode on Great Russell Street in London. Museologically and culturally speaking, that’s a problem. If museums are to represent (and not simply appropriate) objects from their colonial empires, the history of that displacement must be kept in sight.

This may seem an unusual way to begin an essay on grading, but I thought of the Benin Plaques and their absent labels as I prepared another set of grades for my students. Grading (as I’ve written many times before here and hardly need to remind you) is about as eagerly anticipated by teachers as a colonoscopy (and at least those are served up with propofol.) There are any number of reasons why this is the case, and at least some of the problems of grading do come with relatively straight-forward solutions. If you can’t bear reading the 27th paper on the role of the cottage in Frankenstein, then open your assignments to allow for a greater variety of responses. If the assignment essentially requires that students feed back to you what you’ve given to them, don’t expect to have an enjoyable experience reading them. Try completing your own assignments and if you find them boring or not conducive of learning, change them so that students can use the assignment to demonstrate both mastery and application.

Full Disclosure

Other issues involved in grading are more difficult to resolve, which brings us back to the Benin Plaques. What everyone knows, yet no label discloses, is that grades represent the ultimate power that faculty hold over our students. As much as our professional code of conduct requires – demands – that we grade fairly, objectively, and without regard to extraneous factors, there is no denying that we are humans and that, when it comes to grading, we are both shaped by, and must contend with, a variety of factors that make that difficult, if not impossible. These range from simple tiredness to complex issues of prejudice including racism and sexism. [See, for example, here (the impact of the teacher’s emotional state) and here (the impact of the stereotype threat) as examples.). Perhaps, just as the Benin Plaques should include on their label an indication of the nature of the power that brought them to the British Museum, so too should we include a label on all of our tests and assignments:

Warning: As much as I will try to grade your assignments objectively, fairly, and without prejudice, and as much as I will attempt to forget how annoyed I was with you when you [fill in appropriately]: didn’t do the reading/watched a Beyoncé video on your laptop instead of listening to what I was saying/left the class three times to go to the bathroom, I am only human, so caveat emptor!

When Life Is Young, British Library HMNTS 10413.dd.1

When Life Is Young, British Library HMNTS 10413.dd.1

Grading has a way of reversing the intent of teaching, not only closing off a formative process of dialog and reflection, but often contradicting what we have been insisting all semester: “It’s not about the grade.” Well, what if it is? And how do we tell students with a straight face not to worry so much about their grades when they know (as do we) that when all is said and done, the grade we give them can/will influence whether they get the fellowship they need to pursue their studies. I would venture that, for most of us, the problem is not that we feel pressured to give “C” work an “A” (although grade inflation, particularly at elite institutions, might suggest otherwise), but rather how we maintain a straight face when we suggest there is a clear and obvious difference between a “B” and a “B+,” between a “B+ and an “A-.” Particularly in the humanities and social studies, but likely in the sciences as well, we know full well that extraneous considerations (those extra trips to the bathroom!) can influence our decisions. There’s no way around the fact that a serious evaluation of our students’ work is so much more complex than can be expressed in that single letter, and giving a student a “B+/A-” really doesn’t resolve the problem.

What else is wrong with grades? Let me count the ways! As  Steven Mintz, Executive Director of the University of Texas System’s Institute for Transformational Learning and Professor of History at the University of Texas at Austin, pointed out,

  • A single, over-all grade conflates elements that need to be disentangled.
  • They tend to overly reward lower-order thinking skills (such as memorization and recall) rather than higher order skills (involving analysis, application, and synthesis).
  • Grades too often fail to accurately reflect student learning or mastery.
  • They are frequently de-motivating and discouraging.

Nor is this a recent finding: studies as early as 1912 questioned the utility (and validity) of grades, and research has fairly consistently underlined some of the main problems in grading practices.

Typical-student, British Library HMNTS 10413.dd.1

“Typical figure, showing tendency of student life,” British Library HMNTS 10413.dd.1

Does that mean that we should stop giving grades? Very few colleges or universities have abandoned the grading system altogether. Hampshire College, where faculty write narratives of their students’ learning rather than assigning grades, remains the exception. But even there, complex narratives probably won’t work in a class of 80 students, nor is Pass/Fail an option without draw-backs in a world in which grades are the norm. A “Pass” in Organic Chemistry might not help a student when she applies to med school.

Valen E. Johnson, professor and head of the department of statistics at Texas A&M University at College Station argues further that if we didn’t grade, “students probably wouldn’t work as much and wouldn’t do homework and wouldn’t study for exams”? While this is not universally the case, we can (and probably should) admit that we share at least some of the blame for not making learning more intrinsically motivating. But such an observation can only get us so far. Ultimately, we need to think about new ways of grading that can address some of the shortcomings of the current system. That’s where contract grading comes in.

CONTRACT GRADING: THREE OPTIONS

Contract grading, which actually was first used some decades ago, more recently has been gaining traction in higher ed. Briefly, contract grading attempts to reduce the subjectivity of the grading process for faculty and the induced passivity of students within the evaluation system in an attempt to arrive at a more integrative and meaningful process of assessment.

There are a variety of approaches to contract grading, each designed to meet an instructor’s intended outcomes, but all share the fundamental goal of clarifying the grading process for students so that they can make more informed decisions about their actions. While there are a number of different types of contract grading options, I’ve  summarized three different contract-grading approaches here. At the same time, I’d encourage you to talk to your colleagues about how they grade; raise the issue at department meetings: you’d be surprised how many have adopted this method of grading.

Contract grading as a means of negotiating authority

Songs of a Savoyard, British Library HMNTS 11651.k.42

Songs of a Savoyard, British Library HMNTS 11651.k.42

The essential factor in determining a grading approach, at least as I see it, is deciding what you hope the process of grading can achieve in the broadest terms. For some, revealing and addressing the nature of power relations within a classroom environment is the central element that a grading system can address. Ira Shore, for example, has written much about the importance of creating a democratic classroom in which power is both fully disclosed and openly negotiated with students. Similarly, Isabel Moreno Lopez argues that teachers should create a critical classroom in which “authority and responsibilities are shared between teacher and students, empowering all course members to become active, responsible participants of the learning process, not merely passive consumers.” For both, grading is a means by which power can be shared through a collectively negotiated contract co-constructed at the beginning of the semester.

Here, in a condensed version, is how Moreno Lopez describes her contract grading system:

The negotiation process starts at the beginning of the semester when the teacher presents the elements of a contract grading system to the students. In general terms, the grading system is based on the quality and quantity of work students are willing and capable of doing. That is, if a student signs a contract for an “A,” s/he will do more work in the course than the student who contracts for a “C.” The quality of work will also reflect the contracted grade. Students are permitted to rewrite the written assignments as many times as necessary to attain the contracted grade.

At the start of the semester, then, the teacher opens up class-time to discuss both the syllabus and the grading system. Then, s/he asks for questions, amendments, and comments on the original proposal. A debate follows, after which the students sign the contract, as amended by themselves, and keep a copy for their records. During the semester, the negotiation process continues, both in class discussions and in comments in the students’ journals. At the end of the semester, based on the contracts and their performance, students discuss with the teacher their final grades. This grade might be the same they contracted or might have varied depending on their performance and progress.

Moreno Lopez suggests that this negotiated grading system is valuable in two ways: it helps students see learning as a process and not an end, and it “encourages students to be active participants in their own learning process by allowing them to cooperate in what is usually considered the ultimate prerogative of the teacher: the assessment process.”

Shor, Moreno Lopez and others who engage in this form of critical pedagogy identify the classroom as a political arena where differences of power are necessarily, and properly, brought into the center of teaching where they are negotiated. In such a context, struggle and conflict is both inevitable and appropriate insofar as it is a reflection of the larger society, not a “bubble” separate from it.

Non-negotiated contract grading to improve learning

Spectroscope_British Library HMNTS 10027.ee

Spectroscope_British Library HMNTS 10027.ee

The grading contracts used by Jane Danielewicz and Peter Elbow in their composition courses are similar in some respects, but they are less concerned about using the classroom to negotiate authority than Shor or Moreno Lopez. Instead, they see their goal as creating “a classroom where both teachers and students get to give as much time and attention as possible to writing—not politics and culture. Of course political and cultural issues turn up in student writing, but our tendency is to discuss the effectiveness of the writing more than political and cultural issues themselves (not that one can ever completely separate the two).”

Danielewicz and Elbow present the grading contracts to students at the beginning of the semester rather than co-constructing them with student input. By using contracts, they seek “not only to help students learn more and function better as learners; we also want a grading system that encourages them to be the kind of persons our world needs; furthermore, we want to make our own teaching easier and more satisfying.” And they add, “That’s all.” Indeed, that would be plenty.

Here is a summary of the main elements of the Danielewicz-Elbow grading contract:

  1. Attend class regularly—not missing more than a week’s worth of classes.
  2. Meet due dates and writing criteria for all major assignments.
  3. Participate in all in-class exercises and activities.
  4. Complete all informal, low stakes writing assignments (e.g. journal writing or discussion-board writing).
  5. Give thoughtful peer feedback during class workshops and work faithfully with their group on other collaborative tasks (e.g., sharing papers, commenting on drafts, peer editing, on-line discussion boards, answering peer questions).
  6. Sustain effort and investment on each draft of all papers.
  7. Make substantive revisions when the assignment is to revise—extending or changing the thinking or organization—not just editing or touching up.
  8. Copy-edit all final revisions of main assignments until they conform to the conventions of edited, revised English.
  9. Attend conferences with the teacher to discuss drafts.
  10. Submit their mid term and final portfolio.

In other words, students get a “B” based solely on what they do, not on any evaluation of their work by the professor. Grades higher than a “B,” however, depend on the teacher’s evaluation of the quality of their writing. They will discuss in class what “exceptionally high quality” writing means, making the criteria as public and concrete as possible, but they don’t give students power over “high-grade” decisions.

Although they don’t evaluate the quality of their students’ writing up to a “B” grade, they also don’t withhold evaluation as they continue to provide students with feedback on the strengths and weaknesses in their work, both drafts and final version. But the evaluation (up to a “B”) is decoupled from grades. “As a result,” they write, “students don’t have to heed any of our judgments or advice when they revise their papers (though they must revise).” They want their students to feel that the evaluations they conduct are “from individual persons: yes, experts about writing, but individuals, nevertheless, who cannot pretend to be wholly impersonal or fair.”

Their article (“A Unilateral Grading Contract to Improve Learning and Teaching”) offers a fascinating discussion of how they came to the various elements of the contract, why, for example, they picked a “B” grade as the base-line mark for their contract (“Our original reasoning was merely timid—crassly negative and pragmatic: we were scared to ‘go all the way.’”), or whether their contract is actually less “fuzzy” than a standard grading system. “How can we defend ambiguous and arguable criteria like ‘conscientious effort,’ ‘thoughtful feedback,’ and ‘conscientious participation?’” they ask. “First, we don’t accuse someone of failing to meet one of these fuzzy criteria (‘no-effort,’ for example), unless the violation is grossly flagrant (e.g. drafts far short of the required length). Secondly, we’ll always take the students word for it.” In terms of a final, summative, grade they are persuaded that their decisions were relatively easy to make. Students who didn’t fulfill the contract (including some who were excellent writers), were disqualified. They then focused more closely on the remaining final portfolios that they found to be particularly strong.

Contract grading to encouraging active learning and community building

"The Life of George Barnwell; or, the London apprentice of the last century," British Library HMNTS 12621.dd.5.

“The Life of George Barnwell; or, the London apprentice of the last century,” British Library HMNTS 12621.dd.5.

The final example of contract-style grading is Asao B. Inoue’s community-based assessment approach. Similar to all contract models, Inoue, a writing instructor at Washington State University, moves away from teacher-centered assessment and evaluation while encouraging students to take more initiative. But, more than in the previous models, Inoue seeks to create a classroom in which “students take control of all writing assignments, their instructions, assessment criteria, and the practices and reflective activities that go along with their writing.” Such an approach, he maintains, “encourages a community of writers that are implicated in each others’ writing and assessment practices, and gets them to critically engage with these practices.”

Inoue’s model underscores the fact that assessment is a vital component in the act of writing. He spends considerable time discussing with students what they want out of their papers and how they should be read and assessed. It is a complex and recursive process that begins when the class collectively creates its first assessment rubric, a set of guidelines that everyone agrees to, and that they will use both as writers and assessors. This first rubric will be revised continually as the class moves from the early stages of writing (paragraph writing) to position papers and final essays.

Any student can suggest a rubric revision or raise a question about the rubric at any time. To test and revise the iterated rubric, class members write two separate paragraphs, each receiving three peer assessments that use the in-process rubric. The class — instructor and students alike — uses what it has learned from the paragraph assessments to revise the rubric, which becomes the new starting point for on-going assignments, and so on. Over a month, each student writes a position paper, receives responses and assessments from the entire class (both on paper and through class discussions), posts a revision of the position paper based on those discussions and input, gets a more formal peer-assessment of the revision by a few colleagues, writes an essay (often based on the position paper), and finally receives a formal peer-evaluation of the essay. The same process is repeated for a second paper. (The process is schematized in the illustration below.)

ChartWhen the students assess each others’ writing, they are not looking to identify an “A” paragraph or an “exemplary,” or “outstanding” one. Rather they use the rubrics to help them identify proficient paragraphs, ones that reach the proficiency markers they set out at the start of the process. If a paragraph hits these markers, then it has done its job.

Here, for example, is what the class came up as a “proficient” paragraph with after their discussions: A proficient and adequate paragraph will . . .

  • Contain a consistent claim
  • Support claim with appropriate evidence (when needed)
  • Elicit thought on the part of the audience
  • Adapt to or consider its audience
  • Use clear and concise language
  • Use appropriate language and grammar
  • Contain three or more sentences

They continue to refine this set of criteria over the course of the semester.

As Inoue explains,

I try simply to provide the structures for my students to create a rubric, re-think it, write from it, use it to assess each other, and, of course, reflect continually upon all these practices. I distribute guidelines, provide due dates, post weekly reflection prompts, and pose additional questions in class that facilitate assessment discussions on student writing. In short, I try to coach them toward sound assessment practices and active learning stances by making them do the hard work of assessment. I encourage them to voice disagreement, show agreement, and elaborate and qualify ideas. I act as a facilitator, questioner, and listener when we talk about each other’s writing. I try to keep us focused on our rubric in our assessment discussions, yet not be a guard to ivory towers… Our class writing isn’t about what I want — it’s about what the class can agree on they want and can justify in some way so that agreements can be made… My students must debate and decide on all the important decisions regarding their writing in the course from start to finish. The class is about them learning not me teaching.

The key to making assessment work pedagogically, according to Inoue, is periodic reflection on the assessment activities. He does it once a week based on open-ended prompts to point the students to the areas he wants them to reflect on. Community-based assessment pedagogy also offers ways to build a pragmatic sense of community that is active and purposeful.

If our purpose, as teachers, in assessing and evaluating student writing is to help students learn — if assessment is inherently a learning practice (which I think it is) — then the teacher shouldn’t control all of the process. And Inoue concludes:

Community-based assessment pedagogy, as described here, boils down to three classroom imperatives: (1) encourage active learning stances by allowing students to assess and evaluate their own and their colleagues’ writing practices, and make these assessments meaningful and purposeful, (2) situate assessment practices within a community of knowledge makers who construct assessment rubrics and define and justify assessment practices, i.e., encourage the class to work for one another as mutual agents working with and for each other’s benefit, writing for each other, and negotiating hard agreements together, and (3) give lots of opportunities to reflect on assessment that speaks to the larger class community, in order to theorize about writing, rhetorical conventions, assessment, and the judging of writing from specific criteria, i.e., what we say about what we are doing (or did) can help us do it better in the future. In my versions of this pedagogy, these imperatives rest on a framework of recursive, repeated writing and assessment activities.


 

As you will have noticed, none of these models makes grading “easy.” Contract grading is not the contemporary equivalent of throwing the papers down the steps and handing out marks depending on where they land. But, by bringing students into the assessment process, contract grading can help make assessment criteria clearer, remove some subjective aspects of grading, bolster student learning, and build community. And, by foregrounding the grading process as reflective of the inherent power of faculty (i.e., we may be their friends, but ultimately we will give them grades), contract grading provides a needed “label” for students and an invitation to faculty to re-imagine classroom practices.


Some additional bibliography:

Elbow, Peter and Jane Danielwicz. “A Unilateral Grading Contract to Improve Learning and Teaching.” English Department Faculty Publication Series. Paper 3.

Huot, B. “Toward a new discourse of assessment for the college writing classroom.” College English 65 (2002): 163–180.

Inoue, Asao B. “Community-based Assessment Pedagogy.” Assessing Writing 9 (2005) 208–238.

Kohn, Alfie. Punished by Rewards: The Trouble with Gold Stars, Incentive Plans, A’s, Praise, and Other Bribes. Boston, New York: Houghton Mifflin, 1993.

Moreno-Lopez, Isabel. “Sharing Power with Students: The Critical Language Classroom.” Radical Pedagogy 7:2 (2005).

Radican, Lynda S. “Contract Grades: An Agreement between Students and Their Teachers.” In Stephen Tchudi, ed. Alternatives to Grading Student Writing. Urbana, IL: NCTE, 1997, 285-290.

Shor, Ira. When Students Have Power: Negotiating Authority in a Critical Pedagogy. Chicago: University of Chicago Press, 1996.

Tchudi, Stephen, ed. Alternatives to Grading Student Writing. Urbana, IL: NCTE, 1997.

Evaluation Time!

Steven Volk, December 6, 2015

The debate over the value of Student Evaluations of Teaching (SETs) is a long one, which I have reported on a number of times (see here and here, among others). As we move into the last week of the semester, I’d like to suggest two additional approaches to end-of-semester evaluations that can help both you and your students think about the learning that occurred in your classes. I’ll also include a “guide” I wrote in 2010 for reading your SETs when they are returned to you after grades are in.

What Helped Your Learning?

Students enter the "Natio Germanica Bononiae,“ University of Bologna (15th century). Public domain. Wikimedia

Students enter the “Natio Germanica Bononiae,“ University of Bologna (15th century). Public domain. Wikimedia

SETs are largely about how students experienced your course, and so the questions focus on issues of organization, pacing, clarity, grading, etc. As numerous articles have pointed out, Student Evaluations of Teaching don’t tell you about student learning, and they provide very little information to suggest what it is you are doing to support (or hinder) the leaning that goes on in the class. Linda Shadiow and Maryellen Weimer, writing in Faculty Focus on Nov. 23, suggest a series of questions that can help foreground student learning issues. They offer a series of fairly simple sentence stems for students to complete. For example,

  • It most helped my learning of the content when…because…
  • It would have helped my learning of the content if…because…
  • The assignment that contributed most to my learning was…because…
  • The reading that contributed the most to my learning was…because…
  • The kinds of homework problems that contributed most to my learning were…because…
  • The approach I took to my own learning that contributed the most for me was…because…
  • The biggest obstacle for me in my learning the material was…because…
  • A resource I know about that you might consider using is…because…
  • I was most willing to take risks with learning new material when…because…
  • During the first day, I remember thinking…because…
  • What I think I will remember five years from now is…because…

Shadiow and Weimer recommend that faculty also complete the same sentences. Having (almost) completed the semester, we probably have a good idea which assignments worked from our point of view and which didn’t; what readings brought out the most in discussion, and what didn’t; what homework assignments stretched student learning and what brought basically “meh” responses. Comparing our answers to the students can be revealing (or, perhaps, horrifying!).

These questions can be added to the bottom of the current SETs that you will be handing out. You can simply add an additional sheet with these questions which, still anonymously, can be returned directly to you rather than being tabulated by the department AA’s or becoming a part of your official file. Once you have you have let some time pass (see “SETs for Beginners,” below), you can look at them, particularly before preparing classes for next semester.

Student Self-Assessment

Cours de philosophie à Paris Grandes chroniques de France; Castres, bibliothèque municipale – late 14th century. Public domain. Wikimedia

Cours de philosophie à Paris Grandes chroniques de France; Castres, bibliothèque municipale – late 14th century. Public domain. Wikimedia

David Gooblar, in his “Pedagogy Unbound” blog for the Chronicle of Higher Education, recently wrote a very useful post on student self-assessment. [Note: For a good introduction to student self-assessment, see Heidi Andrade & Anna Valtcheva, “Promoting Learning and Achievement Through Self-Assessment,” Theory Into Practice 48:1 (2009): 12-19.] Ask your students to reflect upon the learning strategies they used over the course of the semester, and to consider their own habits of thinking. “Explain that the act of reflection is itself a valuable learning strategy,” he writes. Ask them how they studied for tests or what they did to prepare for their assignments; what worked for them and what didn’t? The answers you get may be quite basic, but the more often we ask students to reflect on their own learning, the more practiced at it they will become.

Gooblar calls attention to the work on metacognition undertaken by Kimberly D. Tanner, a professor of biology at San Francisco State University, particularly the “retrospective postassessments” she uses. Tanner asks students to think about what they now know about the subject (in general terms) compared to what they knew at the start of the semester. (For example: “Before this course, I thought neoliberalism was _____. Now, I think neoliberalism is _____.”) You can ask students to write about the specific ways they have changed their thinking about the topics you covered. Or you can have them write a letter to a future student of the course, to reflect on its high and low points, and tell incoming students what they wished they would have had known going in to the class and what they wish they would have done differently over the course of the semester?

Since I have my students set out their learning goals in a short paper at the start of the class – which I collect and keep – I return these to them at the end of the semester and have them reflect, one final time, on which of these goals they feel they have achieved, and which they didn’t, what they did to reach their goals and what they will do differently in their next classes. Whatever method you chose, the end of the semester is a good time to encourage students to reflect on the journey they have undertaken with you and how they are different at the end of the trip.

“SETs for Beginners” (first written Feb. 7, 2010 and slightly updated here)

800px-Meeting_of_doctors_at_the_university_of_Paris

A meeting of doctors at the university of Paris. From the “Chants royaux” manuscript, Bibliothèque Nationale, Paris.

By now, as you well know, there is a very large literature on student evaluations of teaching (SETs). A lot of the research points to the validity and reliability of these instruments in terms of measuring very specific areas of a student’s experience in a completed class. Some writers continue to argue that they are a worthless exercise, citing evidence that evaluations handed out after the first day of a class will yield strikingly similar results to surveys conducted at the end of the semester, that they are a measure of the entertainment-value of a class, not any value added in terms of student learning, or that they can be easily influenced (just hand out doughnuts with the questionnaires). I have come to accept three basic realities about the use of SETs:

(1) on a broad level, they help to identify outliers – a class which seems to have been extremely successful or highly troubled; (2) they should not be used as the only evaluation of teaching (informed peer evaluation following a standard observation protocol and an examination of course syllabi by experts in the field are strongly recommended as well); and (3) SETs are not a substitute for an assessment of student learning in a course. But, when read carefully, they can tell you something about your teaching on a very specific level. The question is how to read them to get that specific information, and on that score, there is very little literature.

So here’s a first attempt at this question, a kind of “SETs for Beginners.”

Because of college rules, which are likely similar everywhere, we receive our teaching evaluations back only after grades are turned in. (You should consult with your department chairs for information as to how and when to hand out SETs, whether your students can complete them online or only in class, and how they are to be collected if you do them in-class.) At some point by mid-January or mid-June, after our hard-working administrative assistants have tabulated and organized the data, we are informed that our SETs are ready to be picked up!

First decision: do you rush in to get them, play it cool, like a cat walking around a particularly lovely kibble before pouncing, or pretend that they aren’t there until, sure enough, you have forgotten all about them? I usually take the middle route on this, but, in any case, I certainly won’t pick them up on a day when the auto shop called to tell me that the problem’s in the drive train or the best journal just rejected my article. Another hit that day, I just don’t need.

When I do finally make the move, I take them to my office, put them on my desk, pretend that they aren’t there while I take a look at my email which I already checked 90 seconds earlier. Enough, already. I open the folders and read, rapidly, the overall numbers: not what I hoped for, better than it could have been, whatever… Then I put them away for at least a day or two. I don’t think I’m ready to take them on-board just yet, whether the numbers are good, bad, or indifferent. I go back to the email, the article, the gym, until I’ve absorbed the larger quantitative landscape and feel mentally prepared to explore the terrain a bit more carefully.

When I do return to the SETs, I give myself the time (and space) to read them carefully (and privately). I don’t pay much attention to any individual numbers – those have been summarized for me, but I read the comments with care… and a mixture of interest, confusion, skepticism, and wonder. How is it that the student who wrote “Volk is probably the most disorganized professor I’ve ever encountered” attended the same class as the one who commented, “This was a marvel of organization and precision”? What is one to make of such clearly cancelling comments? So here are my tricks for trying to give my SETs the kind of close reading that I think they merit:

  • Do not dwell on the angry outliers. That’s advice more easily given than taken. I have read enough teaching evaluations, my own as well as others, to know that there are some students who just didn’t like our classes and have not figured out any helpful or gracious way to say that. The fact that these are (hopefully) a tiny minority and are directly contradicted by the great majority of other comments doesn’t seem to decrease their impact, or the fact that we continue to obsess about them. (I can still quote, verbatim, comments that were written in 1987!) These bitter communiqués probably serve a purpose for the student, but they really don’t help you think at all usefully about your teaching. Let ’em go.
  • Evaluate the “cancellers”. What do you do when three students thought you were able to organize and facilitate discussions with a high degree of skill and three thought you couldn’t organize a discussion to save your life? These are harder to deal with and can add to the cynicism of those who think that the whole SET adventure is a waste of time. For the “cancellers,” I try to figure out a bit more about them to see if they represent some legitimate (i.e., widespread) concern about the class or not. Is one side of the debate generally supported by the numbers? Do I score lower in the discussion-oriented questions than in other areas, lower than in previous iterations of the course, or lower than I would have really wanted? Does the demographic information that I know about the student evaluator add context that is useful and that I should take on board? I am more likely to trust comments from seniors than from first-years. Does it appear that there is a striking gender or racial difference in terms of how students respond to specific questions? That is extremely important information to lean from and it is why we collect demographic information from our student respondents. A careful reading of this information can help us understand what is going on in our classes on a more granular level.
Brian Carson, Backyard Flowers in Black and White, No. 2. Flickr Creative Commons

Brian Carson, Backyard Flowers in Black and White, No. 2. Flickr Creative Commons

And, if none of the above helps me think about why something I have done works for some and not others, I make a note to myself to ask students explicitly about it the next time I offer the class: Please, tell me if you don’t think these (discussions, paper assignments, readings, etc.) are working so that I can consider other approaches.

  • Focus on those areas that seem to be generating the most amount of concern from students. Are they having a hard time trying to figure out how the assignments relate to the reading? Do a considerable number worry that they aren’t getting timely or useful feedback from you? Is there a widespread upset that classes run too long and students don’t have enough time to get to their next class? For each of the areas where I find a concern that has reached a “critical mass” level and is not just an angry-outlier grievance, I consider what I think about their criticism and whether, given my own goals in the course, I find it legitimate. For example, I will pay no attention to students who complain about the early hour of my class. Getting work back on time depends on the size of the class and what I have promised: in a 50-person class if I say I’ll return work within two weeks, and do so, then I won’t think much about students who complain that I only returned their work two weeks after they turned it in.

Other issues force me to think more about how I teach and what impact that has on student learning. What of students who protest that “there’s too much work for a 100-level class”? I get a lot of those comments, and it makes me think: why do students think a 100-level class should involve less work than a 300-level class? Do we, the faculty, think that a 100-level class should be less work than a senior seminar? Certainly, upper-level classes will be more “difficult” than 100-level classes (i.e., they demand that the students have acquired significant prior knowledge and skills needed to engage at a higher level), but should there be any less work involved in the entry-level class? I, for one, don’t think so – and so I won’t change that aspect of the course.

But, ultimately, when student comments suggest what is a real area of concern, when they point to something I am doing in the class that negatively impacts how students are able to learn, than I need to regard that issue with the seriousness it deserves. I will think about how I might correct the problem, and, often, the best way to do that is to talk to my colleagues and find someone in my department or outside who can read my SETs with me. That has served me well every time, and it does point to the ultimate utility of SETs for the individual faculty member on a formative level: they can help us to design our teaching to more effectively promote student learning.

  • Finally, since Oberlin really does attract faculty who care about their students and the challenges of student learning, then my guess is that your evaluations are generally good, and you need to take great satisfaction in that (see: “Don’t dwell on the angry outliers”). I have never failed to find some comments on my own evaluations that remind me yet again about how perceptive our students can be and how fortunate I am to be here.

Students Evaluating Teaching – The Unending Conversation

The New York Times Magazine for September 21, 2008 is the “College Issue,” with a cover title, “It’s All About Teaching.” Among the articles is one on “Judgment Day” by Mark Oppenheimer. While making the point that there have been over 2,000 studies on the value of student teaching evaluations (i.e., those evaluations which all students are required to fill out at the end of our classes), the research on their utility is still mixed. The article calls attention to the way that the evaluations are subject to particular gender/race biases, how they can reward “entertainment value” over good teaching, how it is difficult to rate the sciences/math (i.e. very vertical curricula) vs. the humanities and social sciences, etc. Oppenheimer closes his article (spoiler alert!) with the following: “When students in the 1960s demanded more say in academic governance, they could not have predicted that their children would play so outsize a role in deciding which professors were fit to teach them. Once there was a student revolution, which then begat a consumer revolution, and along with more variety in the food court and dorm rooms wired for cable, it brought the curious phenomenon of students grading their graders. Whether students are learning more, it’s hard to say. But whatever they believe, they’re asked to say it.”

What do you think? SET’s are required at Oberlin and we have put a fair amount of time trying to make them more reliable and uniform. Are there better ways to evaluate teaching? What would you like to see (other than superlative comments from students on ALL your classes)?