What is the difference between rating and ranking applications?

For the Foundation Grant competition, CIHR asks reviewers to provide a rating as well as a ranking.

Reviewers provide their rating using the letter scale (O++, O+, O, etc.) to assess each criterion, which ultimately leads to a rating for the application overall. The letter scale is broad (ranging from Poor to Outstanding++), but ratings tend to cluster in the O and E parts of the scale. Given the caliber of Foundation Grant applicants, it is not surprising that the majority of applications rate so highly. This is part of what makes the competition so competitive, and it is also one of the reasons that CIHR asks reviewers to rank the applications that they have been assigned.

Reviewers use the ranking to give CIHR the order (from highest to lowest) of applications, based on their assessments. In this sense, the rating acts as a tool to help the reviewers determine what their ranking list should be (for example, an O++ application would rank higher than an O+ application). As mentioned above, however, the ratings tend to cluster – which basically means that the rating alone could, for example, lead to a five-way tie on a reviewer’s list. The ranking forces reviewers to break those ties and decide which of these excellent applications is slightly better than the rest.

It is the ranking system that may lead to a scenario where an applicant who receives a rating of O+ on each criterion could still end up receiving a low consolidated rank. The consolidated rank shows how they fared compared to the other applications in the competition and not necessarily how the applicant fared in terms of the assessment criteria.

Date modified: