CJ Beyond Educational and Psychometric testing

Comparative judgement studies are most frequently applied in educational or psychometric testing. 
However, as we show in this post, they can also be of great benefit to other academic disciplines as well as commercial businesses. 

In co-operation with the developers of NoMoreMarking, my colleagues and I from the University of Cambridge used NoMoreMarking to gain insights about personality-specific image preferences. 

Which image do you like more?

Before using NoMoreMarking we did what most psychologists do and measured people’s preferences for certain image characteristics with the help of a 7-point Likert scale (e.g. “How much do you like this picture?”). However, from our own experience with rating the 20 pictures in our study as well as from the feedback we got from participants we had to realise that it is incredibly difficult to distinguish between subtle preferences and to respond consistently across the whole set of images.

Several participants, for example, reported that they would have liked to change previous ratings once they had seen the later images. This was supported by the finding that the range and variance of ratings were more restricted than we would have hoped for. Most participants rated the images somewhere between 4 (= average) and 6, indicating that they found most images reasonably likable. This restriction of range made it almost impossible to filter out the subtle preferences for certain characteristics across images.

When I discussed the problem with one of my colleagues, he mentioned the NoMoreMarking platform. Although I had heard about CJ studies and their benefits for psychometric testing before, it would have never occurred to me to use them in one of my studies. The reason for that is simple: most psychologists like myself are not familiar with how to set up and score CJ studies. Hence, we often take the easier route and stick to methods that we feel comfortable with.

Once I had a look at the NoMoreMarking website, I knew that this was the ideal platform to conduct my study. Not only is the set-up of tests extremely easy and flexible when it comes to the integration of pictures, but it also offers different ways of scoring the data automatically. Even if I had not been familiar with the concepts of CJs or IRT (which is probably the case for many non-academics or even psychologists from other areas of expertise), it would have been possible for me to use the platform. This is particularly true as the platform developers provided me with excellent support whenever I needed it!

In order to compare the usefulness of Likert scale vs. CJ items for our research hypothesis, we investigated the consistency of image ratings in both samples (e.g. if image A is preferred over B and B is preferred over C then A should be preferred over C). As expected, we found the consistency of CJ items to be much higher than that of Likert scale items. The advantage of using CJ items was further supported by a comparison of results. While the expected effects of personality on image ratings were small to non-significant when using Likert scale items, they were much more pronounced and clear when using the results of the CJ.

I hope that our example has shown how easy and valuable it can be to use NoMoreMarking in research that is not directly related to the traditional use of Comparative Judgement. I am extremely grateful for the support that the developers have provided me with and can only encourage everybody to explore the benefits of the platform themselves!

Sandra Matz
PhD Candidate
University of Cambridge Psychometrics Centre


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s