27 Aug 2014

Writing versus winning

I've used peer review/grading in several of my classes because I think it gives students a different perspective (from mine) on their work.

I think the process leads students to...
  • write differently -- and perhaps more carefully
  • get more written feedback than I can provide
  • learn from their peers' writing and perspectives
  • improve their skills when critiquing others' work
These benefits bring some additional costs. The most obvious is the extra paperwork and sorting that I need to do, to move the process along. The most dangerous is students' perception that their peers are being unfair to them.

This is how the system used to work:
  1. Each author's essay is given to three peers
  2. Each peer ranks the three different essays they get as A, B or C
  3. Peers then give written feedback to accompany their ranks
  4. Authors then rank the quality of the peers' feedback as A, B or C
This system ensures that the "average" essay gets a B at the same time as it removes the problem of subjective awarding of points (one student would give A, A and B; another B, C and C).

Although I worried about students who complained to me that their peers were being unfair (or even sabotaging them, to help themselves in a roundabout way), I told them that three grades from peers should help reduce bias, on average.

But I also underestimated two problems. The first was a propensity for authors to give bad ranks to peers in exchange for bad ranks. We reduced this problem by separating ranks from the written critiques. The second was the potential for a peer to write a review that justified their rank.* Such an action would lead to flowery reports for the A essay and brutal reports for the C essay.** Indeed, I had seen examples of terrible comments given to essays that didn't deserve them.

Luckily for my students, LUC's policy forbids peer grading, which forced me to rethink and reform the process into a better structure. What's interesting is that it's nearly the same as the system peer-reviewed journals use: A paper goes to three peers, who write anonymous critiques that the editor uses to decide whether to accept or reject the paper.***

I will therefore use a peer-review process that's modified in three ways. First, I'll ask peers to give constructive criticism (e.g., "what did they miss, how can they improve, what did you learn?"). Second, I will grade peers on the quality of their critiques (journals do not do this, formally). Third, I will end the process with the Author's grade, rather than another revision.

Bottom Line: Students will help each other more if they are graded on the quality of the help rather than their rationalization for a grade they may not even want to give.

* Cornelia pointed this out, and it's obvious in hindsight. I assumed students would read and critique and THEN rank, but the need to give A, B or C meant that students would rank first, justify later.
** Psychologists have shown that people are clever in rationalizing pretty much any involuntary situation as the outcome of free choice.
*** The most common move is in-between, i.e., "revise and resubmit" for potential acceptance or rejection.

No comments:

Post a Comment

Note: only a member of this blog may post a comment.