Transforming University admissions with Adaptive Comparative Judgement

Discover how Adaptive Comparative Judgement is revolutionizing university admissions in Canada with Adaptive Comparative Judgement, making the process fairer, more efficient, and insightful by focusing on holistic evaluations over traditional rubric-based assessments.

Dr. David Gill is currently an Associate Professor in the Faculty of Education at Memorial University of Newfoundland. Part of his role involves leading a small team of colleagues in the annual selection process of new course participants.

Current processes followed a traditional pattern where candidates submitted a lengthy letter of application for consideration. The applications were distributed between a small team of experts for marking against an established rubric. At the end of the marking period there was a moderation / standardisation process to ensure consistency between markers.

Dr. Gill and his team were interested in finding ways that can enhance the fairness, efficiency, and depth of its admission process, leading to a more rounded and insightful assessment of potential students.

The Assessment Challenge

The process was unsatisfactory for a number of reasons:

  • Subjectivity and Inconsistency: Even with a rubric, individual marker biases and interpretations can influence the scoring. This subjectivity can lead to inconsistencies in how applications are evaluated, especially if markers have different levels of experience or understanding of the criteria
  • Time-Consuming: The process of marking each application in detail, followed by a moderation session to ensure consistency, can be very time-consuming. This can be inefficient, particularly when dealing with a large number of applications
  • Limited Feedback: Traditional marking provides quantitative scores but often lacks qualitative feedback that could be more beneficial for understanding an applicant's strengths and weaknesses beyond the rubric's scope
  • Overemphasis on Rubric Conformity: Focusing strictly on rubric criteria might overlook other valuable aspects of an applicant's profile, such as creativity, resilience, or unique experiences that do not neatly fit into predefined categories.

Dr. David Gill, April 2024"Placing applicants successfully is of the utmost importance for the University and the students themselves. Our current process carries a number of risks and is burdensome on already over-worked staff."

The project

  • Items: 28 applications were added to RM Compare. Each application was in text form. The longest ran to 8 pages, the shortest to 1.
  • Judges: 6 judges considered the applications. Each Judge was a domain expert.
  • Rounds: Each item was seen 16 times. Most judges they were asked to completed around 45 judgements.
  • Workload: The time to judgement varied by averaged out at less than 2 minutes. This meant that the Judges spent around 90 minutes completing their judgements.

The results

In the rank (below) we can see that the team achieved very high levels of reliability over 16 rounds. The Misfit data showed high levels of consensus between the judges concerning the relative quality of submissions.

The Item Misfit analysis (below) identified a couple of Items which were harder for the Judges to agree on – these are shown by the block dots close to or over the solid red line. Interestingly one of the Items consisted of more pages than average (8), while the other had less (1). Despite the 'misfitting' we can still be confident that each item found itself on the right place in the rank.

Improvements Through Adaptive Comparative Judgement

Adaptive Comparative Judgement (ACJ) offers a holistic approach that could address these weaknesses effectively:

  • Reducing Subjectivity: ACJ involves comparing pairs of applications to decide which one is better, rather than scoring them against a fixed rubric. This method can help mitigate individual marker biases as decisions are based on direct comparisons, and the process inherently involves multiple judgements, leading to a more balanced view.
  • Efficiency: ACJ can be more time-efficient compared to traditional grading. Since markers are only making comparative judgements rather than detailed rubric-based assessments, the process can move faster while still producing reliable results.
  • Enhanced Feedback: Through the process of comparative judgement, markers often gain deeper insights into what distinguishes a strong application from a weaker one. This can provide more informative and actionable feedback for applicants.
  • Holistic Evaluation: ACJ allows evaluators to consider the application as a whole rather than focusing narrowly on specific criteria. This holistic view can capture the essence of an applicant's potential more effectively, recognizing qualities that might not be explicitly listed in a rubric but are nonetheless important for success in the university environment.
  • Scalability and Adaptability: ACJ uses algorithms to adapt the process based on the results of previous rounds of comparison, which helps in quickly establishing a reliable ranking of applications. This scalability makes it suitable for handling large volumes of applications.

This blog was original published by RM Compare https://compare.rm.com/blog/transforming-university-admissions-with-adaptive-comparative-judgement/

Further blogs

Knowledge and skills are most powerful in combination cover

Knowledge and skills are most powerful in combination

A Labour government would aim to break down the divide between academic and vocational education. But what would that mean for students and schools?...

The Inequity of Exams

How our assessment system perpetuatates harmful stereotypes about disadvantaged groups

By only measuring knowledge, skills and strengths through exams, we are in danger of perpetuating harmful stereotypes about some of our most disadvantaged students especially those at the intersection of race and class…
Learner Profiles: A richer picture of learning for students cover

Learner Profiles: A richer picture of learning for students

Discover how Wapping High School is designing and implementing a Learner Profile, going beyond traditional assessments to capture the wider skills essential for life....