Evaluating the Effectiveness of Interventions

We were excited to receive this guest blog from Derek Williams:

I am a full-time faculty member at a community college in Raleigh, NC (USA). In the past, the course teams – faculty teaching the same course – have collected data on student performance on specific learning objectives. Every couple of years, we identify a learning objective that has caused students trouble and we design a common intervention plan (all members of the course team use the same intervention plan). Once everybody on the course team has used the intervention, we collect data on student performance and then we collect data again on the final exam question(s) pertaining to the same learning objective. 

However, in the past, each faculty member scores their own exams and the interventions differently, and then reports the proportion of students who earned at least 70% credit on each assignment. For an 8 point item on the final exam, students must earn 6, 7, or 8 points in order to be considered part of the proficient population. But what is the difference between me scoring a student at 5/8 and another member scoring their student at a 6/8?

Without a rubric – and some training – all of our data are subjective and have very poor inter-rater reliability. This was completely unacceptable to me; especially from the statistics course team.

Then, I attended a conference (SIGMAA-RUME) where Dr. Matthew Inglis presented on Comparative Judgement (CJ), and I was hooked! I have thought of dozens of ways in which CJ could be used in research and evaluation in [mathematics] education; one of which was fixing the problem mentioned above. Accordingly, I shared a method for evaluating the effectiveness of our common intervention with the course team, and we are trying it.
My hopes are that from CJ we can determine 1) if the intervention is effective and 2) a confidence interval for the proportion of students taking the course that are proficient with the learning objective being intervened. I am still waiting on a couple of judges to finish their duty, but I am excited to dig into the data and see what I can find.
Derek A. Williams
Instructor of Mathematics
Wake Technical Community College
PhD student
North Carolina State University

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s