Open Sourcing Our Comparative Judgement Algorithms

This week we released our Comparative Judgement Algorithms under the GNU GENERAL PUBLIC LICENSE Version 3. That means that you are free to use them, modify them, but most importantly we hope you will collaborate with us to improve them!

We took the decision to open source our algorithms following a meeting with the team who are developing the Digital Platform for the Assessment of Competences (D-PAC). The D-PAC project is a consortium of Antwerp University, iMinds and Ghent University funded by the Flemish government. They, like us, had become enamoured with the concept of Comparative Judgement, and, like us, had been disappointed with the lack of products to support researchers and assessment practitioners in this area. The D-PAC project aims to produce an open-source solution for Comparative Judgement that will support their research program, and will support the growth of research in this field more generally.

As we were both working in the same programming language, and we had already developed certain key algorithms, we immediately saw an opportunity. By working together we were more likely to spread the reach of Comparative Judgment and more likely to produce robust products.

Six months later, the D-PAC team have produced their first Comparative Judgement tool, while we have benefitted from the scrutiny, optimisation and testing of our algorithms by the D-PAC team and iMinds. Together we will work on the crucial issue of script selection, and we will soon be travelling to Antwerp to help them design their first simulation studies that will produce evidence in this nascent area.

All this with no money changing hands!

While I would love to say this is pure altruism, in fact, to us, it makes perfect business sense.

Firstly, we do not charge people to use We hope this will always be the case. The reason we can do this is because we use to support our own development of large-scale assessments. These assessments need 
a highly scalable and efficient distributed judging solution. It costs us little to make this solution available to others who wish to use it.

Secondly, we support our main business model, developing assessments, with providing psychometric support for companies who are changing their assessment models. We provide the expertise in assessment design, sampling, test-equating, predictive models and quality assurance that companies need if they are to incorporate Comparative Judgment into their processes in a robust manner.

Finally, as a small company, we have already seen the benefits of collaboration. You no longer have to take our word for it that our algorithms work, you can take the word of the D-PAC team, and if that is not good enough, you can scrutinise them for yourselves. Most importantly, you can improve on them, and adapt them to suit your particular needs.

So, if you want to use Comparative Judgement, go to

If you want to get involved in Comparative Judgement, download our package here:

If you want to further the field of Comparative Judgement, join the team here:

Chris Wheadon, 18th November 2014

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s