Can the outcomes of CJ be used for grading purposes? We’ve found that people often think they cannot, or that grading based on CJ has to be norm-referenced rather than criterion-referenced.
This is not the case at all!
The outcome of marking is scores that are used to rank students, and grade boundaries can be applied using whatever method is preferred. Substitute “marking” for “CJ” and exactly the same principle applies.
One difference is that marking typically results in some students getting the same score, whereas CJ results in each student getting a unique score. However this if anything is a strength, allowing grade boundaries to be set between any two students.
Norm-referencing involves allocating grades to predetermined percentages of students. For example, the top 10% might be allocated grade A. This is possible using the outcomes of marking or CJ, although given that CJ allocates unique scores to students we can be more precise.
Criterion-referencing involves grading students in terms of their understanding of the material that has been taught. For example, a script might be selected that is deemed to represent the boundary between grade A and grade B. Students who scored above the selected script are allocated grade A, and the procedure can be repeated to identify the boundary between grades B and C, and so on. No assumption need be made regarding whether the scores came from marking or CJ.
An example of this was provided by Jones & Alcock (2013). Experts judged a conceptual calculus test sat by 193 undergraduate mathematicians. The lecturer wished to allocate a grade of 1, 2 or 3 to each student dependent on their CJ score. She began by studying a graph of the students’ scores to identify potential boundaries, and then scrutinised scripts either side. Through an iterative process of studying the graph and scrutinising scripts final boundaries were decided. The grades contributed to the students’ overall score for the module. The lecturer also provided detailed grade descriptors to the students, along with advice about how students at each grade might improve their understanding.
But CJ can go further than this. Exemplar scripts can be sourced beforehand and included in the scripts to be judged. These may be real scripts selected from a previous year, contrived scripts designed to represent the boundaries, or not a script at all but an explicit statement of achievement such as “typical Grade A student”. The exemplars will be allocated a score and can be used as markers to inform criterion-referenced grading.
So not only can CJ make the grade, we believe it can do so better than marking!
Jones, I. & Alcock, L. (2013). Peer assessment without assessment criteria. Studies in Higher Education. doi: 10.1080/03075079.2013.821974 http://tinyurl.com/kghro4u