Analysing Comparative Judgement data in R

For many years researchers working in Comparative Judgement have been grateful for the R package BradleyTerry2. While the package is fine for smaller datasets, for the size of datasets we are now producing the package is slow and the model fitting unreliable.

To fill the gap we produced an implementation of the Rasch algorithms detailed in Pollitt’s (2012) paper. The algorithms are open-source, but do require some knowledge of node – which is unfamiliar to many.

Following a conversation with Alexander Robitzsch, the author of the popular package TAM, we were delighted to receive this email:

Hi Chris,
I implemented (a rather simple version) of an extended Bradley-Terry model for pairwise comparisons in the recent sirt package (1.6-5). The function is ‘btm’. The speed should be sufficient also for large datasets. See
https://sites.google.com/site/alexanderrobitzsch/software


Sufficient?

With a dataset of 15,354 decisions the btm function in sirt fitted the model in half a second. Our own node estimation took 4 seconds, while BradleyTerry2 was still running 25 minutes later.

Comparing the results of btm in sirt with the Rasch model in No More Marking the impact of Alexander’s implementation of bias reduction techniques is clear. Standard errors at the extremes are reduced. 



We were so impressed we have now implemented Alexander’s model with bias correction in No More Marking. From now on you will be able to get exactly the same results whether you use sirt or whether you simply download results from No More Marking.

If you are interested in trying out btm in sirt you’ll need the latest development version of sirt (at least version 1.6-9).

Here is the script we used:

library(sirt)

decisions <- read.csv('decisions.csv', stringsAsFactors=F)

#Create a unique factor level for each candidate
players <- unique(c(decisions$Candidate.Chosen,decisions$Candidate.Not.Chosen))

#### Fit B-T ####
df <- data.frame(id1=decisions$Candidate.Chosen, id2=decisions$Candidate.Not.Chosen, result=1)

df$id1 <- factor(df$id1,levels=players)
df$id2 <- factor(df$id2,levels=players)

mod1 <- sirt::btm(df , maxit=400 , fix.eta=0 , ignore.ties=TRUE )

summary(mod1)

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s