Postdoctoral Fellow Peer Review Pilots

CIHR is committed to supporting the development of the next generation of health researchers by accelerating their research independence and leadership. Given that, developing peer review skills through hands-on experience is important for the career development of emerging researchers, CIHR invited postdoctoral fellows (PDFs) to participate as reviewers for the Doctoral Research Award (DRA) competitions.

The objectives of these pilots were:

  1. To give PDFs an opportunity to develop their peer review skills;
  2. To assess the proficiency of PDFs as peer reviewers within the DRA program; and
  3. To determine how best to support the peer review experience and quality of PDFs; including the need for updated learning resources.

Two pilot studies were conducted through the 2016 and 2017 DRA competitions.

Postdoctoral Peer Review Pilots 1 and 2

Methods

Reviewer Groups: As per the DRA competition review process, each application was reviewed by two reviewers. In each pilot, there were two reviewer groups: a Regular-Regular group that only included experienced reviewer pairs with previous experience of reviewing; and a PDF-Regular group consisting of one PDF reviewer and one experienced reviewer with more than 2 years of review experience. In the PDF-Regular reviewer group, each application was assigned a maximum of one PDF reviewer.

Reviewer Participation: Overall, 1175 and 1205 applications were received in 2016 and 2017 DRA competition, respectively. In pilot 1 (2016 DRA competition), a total of 238 reviewers participated, of which 47 were Banting PDFs. In pilot 2 (2017 DRA competition), a total of 250 reviewers participated, of which 50 were PDFs (23 Banting PDFs and 27 CIHR PDFs). In both pilots, PDFs were provided with standard peer reviewer training available to all reviewers.

Table 1 provides overall competition statistics of the two pilot studies.

Pilot 1
(2016 DRA Competition)
Pilot 2
(2017 DRA Competition)
Total number of applications 1175 1205
Total number of reviewers 238 250
Number of Regular reviewers 191 200
Number of PDF reviewers (BPF, MFE) 47 50 (23, 27)
Percent of reviewers who are PDFs 20% 20%

BPF = Banting postdoctoral fellows
MFE = CIHR postdoctoral fellows
Regular reviewers = Experienced reviewers who have previously taken part in the review process.

The overall success rate for the 2016 and 2017 DRA competitions was 13%.

Reviewer Workload: In pilot 1, both reviewer types (Regular or PDF) were assigned an average of 10 applications. In pilot 2, Regular reviewers were assigned an average of 10 applications compared with an average of 9 applications assigned to PDF reviewers.

Stages of Review: An application within the DRA competition can undergo up to three review stages depending on how discrepant the scores are between the two reviewers.

Figure 1: The three review stages that a DRA application can undergo depending on the reviewer scores. The pilot studies followed the same three-staged review process.

Figure 1 – long description

Review Process of the DRA competition. An application within the DRA competition can undergo up to three review stages depending on how discrepant the scores are between the two reviewers.

  • Initial Review: Applications are reviewed and written feedback/scores returned to CIHR.
  • Re-Review: For any application with a discrepant score (i.e. ˃ 0.5 difference), both reviewers are invited to re-evaluate the application and to change the scores and/or comments if they feel it is relevant.
  • Third Review: Any application that still has a discrepant score following the re-review process is sent to an impartial third experienced Regular reviewer for assessment.

Both the Regular-Regular group and the PDF-Regular group were involved in all stages of review. In cases where applications were re-reviewed, the PDF reviewers were encouraged to reach out to their assigned mentors (who were always reviewers with two or more years of DRA review experience) and/or CIHR staff for any questions or concerns. For applications where score discrepancy could not be resolved through the re-review process, the third reviewer was always an experienced reviewer.

At the end of each pilot study, a survey was conducted where both reviewer types were invited to provide feedback on their overall experience. Recommendations from the survey results of pilot 1 informed the training materials of the pilot 2 and were incorporated accordingly.

Results

CIHR analyzed the competition data of the 2016 and 2017 DRA competitions to compare the review behavior of the PDF reviewers to that of the experienced Regular reviewers. The analysis was conducted at each of the three review stages. Table 2 outlines the types of analyses conducted. Results from the survey were analyzed separately. For pilot 1, an additional qualitative analysis was additionally conducted to compare the quality of the reviews.

Table 2: Types of analysis conducted for Pilot 1 and Pilot 2.

A. Quantitative Analysis: By Review Stage (Pilot 1 and Pilot 2)
  DRA Competition Process Description Analysis Performed
1. Initial Review Applications are reviewed and scores/written feedback returned to CIHR.
  1. Comparison of Overall Scoring Behavior
  2. Scoring Spread
2. Re-Review For any application with discrepant scores (i.e. ˃ 0.5 SD difference), both reviewers are asked to re-evaluate the application and to change the scores and/or comments if they felt it was relevant.
  1. Comparison of the proportion of applications that underwent re-review
  2. Comparison of the scoring behavior
3. Third Review Any application that still has a discrepant score following the re-review process, CIHR asks a third, impartial reviewer to review the application and assign a score.
  1. Comparison of the proportion of applications that underwent 3rd review
  2. Comparison of the scoring behavior (could not perform due to low N value)
B. Qualitative Analysis (Pilot 1 only)
4. Comparison of the quality (robustness and utility) of reviewer written assessments
C. Survey Analysis (Pilot 1 and Pilot 2)
5. Reviewer feedback analysis
Date modified: