Peer Reviewer Resource: Broadening your assessment of research contributions and impacts
February 2024
As a signatory of DORA, CIHR has aligned the Project Grant competition, Awards programs, and Priority Driven Research programs with DORA's recommendations to funders. As such, the guidance to Peer Reviewers, Chairs and Scientific Officers within these programs has been updated.
To support peer reviewers in implementing DORA in their reviews, Chairs and Scientific Officers will do the following:
- Chairs, in collaboration with CIHR staff, are asked to ensure peer review is conducted in accordance with DORA.
- Scientific Officers are asked to capture comments relevant to and in alignment with DORA.
- Chairs and Scientific Officers should intervene in deliberations when conversations are misaligned with DORA principles.
Reviewing in alignment with DORA principles means assessing productivity and progress broadly, recognizing broad research outputs, while taking the context of the applicant into consideration.
In addition, when completing their assessments, reviewers should only consider the information provided by applicants in their applications. It is important to avoid making assumptions to “fill in the blanks” in an application, which can lead to bias in peer review. The onus is on the applicant to address the funding opportunity criteria and make the case to reviewers that their application is meritorious for funding.
Applicants have the opportunity to highlight the quality and impact of their research contributions (whatever their form), for example, through the summary of progress or most significant contributions sections of their application.
Examples of comments that are misaligned with DORA principles and suggestions for improvement:
Example 1: A comment that uses journal-based metrics as a surrogate measure of quality, for example:
- The applicant has an excellent track record of publishing in gold standard journals
- The applicant has a high number of publications, however most in low-impact journals
What to do: Peer reviewers should reframe the comment to remove reference to journal-based metrics and instead focus on the scientific content of the applicant’s publications, rather than publication metrics.
The reviewer can consider commenting on:
- the quality and impact of the contribution; and/or
- whether the applicant explained their choice of venue for dissemination.
Example 1, reframed:
- The applicant has published important papers in the field of X which has significantly increased our understanding of Y. The applicant has engaged in meaningful and culturally safe research throughout their career.
- The applicant has published an excellent paper in the field of X which I found to be eloquently written and meaningfully advanced how we study this disease.
- The applicant has published a number of papers recently; however, their findings did not significantly advance the needle with regards to answering the research question.
Example 2: A reviewer refers to an applicant as “unproductive” or “productive” without added justification and/or with consideration of grant or journal-based metrics alone.
What to do: Peer reviewers should reflect on their assessment of the applicant’s productivity, keeping in mind that it is necessary to consider a broad range of research outputs (e.g., research publications, training and mentorship, knowledge mobilization activities, datasets, etc.), with attention to quality and impact. It is equally important to avoid consideration of the number of publications, or the journals they have published in, in isolation, as signs of productivity.
- The reviewer should consider:
- the context of the applicant, as disclosed in the application, and the opportunities they have had to date (for example, disciplines, environments, career stage, barriers to access the field, etc.)
- any disclosed delays to research (for example, family-related leave, pandemic impact, career interruptions due to illness, caregiving, etc.) and the impact this has had on the applicant’s productivity.
- The reviewer can consider commenting on:
- whether the applicant disclosed any delays to research, and/or
- the applicant’s track-record, with consideration of context, and highlight any notable contributions and/or gaps while providing an assessment.
Example 2, reframed:
- The applicant has a very modest publication record with ~1 publication per year. Of note, the applicant indicates that there have been medical issues in 2015-2016 which impacted productivity. The applicant has an established history of supervision (graduate, PDF, and technicians).
- Productive PhD before MD, and productive researcher even during the period of intensive clinical training.
The Peer Reviewer resource, Broadening your assessment of research contributions and impacts and Applicant resource, How to highlight your resea rch contributions and impacts provide practical tips to support DORA-related guidance.
- Date modified: