CIHR Reviewers’ Guide for New Investigator Salary Awards
October 2014
Table of Contents
- Introduction
- Peer Review At CIHR
- Summary Of The Peer Review Process
- Stage 1: Individual Review
- Stage 2: Committee Review
- Feedback
- Appendix A – CIHR New Investigator Salary Awards Adjudication Criteria
- Appendix B – CIHR New Investigator Salary Awards Reviewer Worksheet
Introduction
On behalf of CIHR, we would like to thank the reviewers for agreeing to serve as a peer review committee member. The success of the peer review process is made possible by dedicated people like you who generously give their time and expertise. Your efforts are greatly appreciated by CIHR and the scientific community.
The purpose of this document is to provide instructions on the peer review process specific to the CIHR New Investigator Salary Awards program.
Peer Review at CIHR
Information on CIHR's objectives, governance and policies; an outline of the roles and responsibilities of peer review committee members; and the policies, principles and procedures for peer review of applications can be found in the CIHR Peer Review Guide for Training and Salary Awards. It is important that reviewers become familiar with this document, as well as the present document, before starting the reviews.
Summary of the Peer Review Process
The CIHR New Investigator Salary Awards program uses multi-stage structured review process. The review process is completed in two (2) stages: an individual evaluation of an assigned set of applications using the online ResearchNet platform, followed by a face-to-face committee meeting for a discussion of a selected group of applications.
There are five (5) peer review committees for this program. Each application received will be assigned to the committee with the mandate that most closely aligns with the applicant's training, credentials and area of research. The link to the committees' mandates is listed under the section "Read the pertinent documentation".
Reviewers are asked to follow the step-by-step instructions below to successfully complete all peer review tasks:
Stage 1: Individual review
Step 1: Read the pertinent documentation
Step 2: Determine ability to review and identify conflicts
Step 3: Conduct preliminary review
Step 4: Submit initial reviews and ratings on ResearchNet
Stage 2: The committee review
Step 1: Determine ability to review and identify conflicts
Step 2: Conduct overall assessment of your assigned applications
Step 3: Attend the committee meeting
Stage 1: Individual Review
Step 1: Read the pertinent documentation
The peer review process for this program is described in details in this document. It is essential to read the document and be familiar with it. It is also important to read the following:
- the CIHR Peer Review Guide for Training and Salary Awards;
- the description of the program's committee mandate;
- the Funding Opportunity.
Step 2: Determine ability to review and identify conflicts
To determine the ability to review and identify conflicts of interests, reviewers are to follow these steps:
- Log into ResearchNet.
- On the home page, click on the link of their assigned committee to open the main task list.
- Complete the task "Review Confidentiality and Conflict of Interest Guidelines" (once completed, it will "open" the other tasks).
- Open the "Manage Conflicts/Ability to Review" task.
- For each of the assigned application, use the information provided to indicate their ability to review (high-medium-low) or if they have a conflict of interest including conflict of language. If there is a conflict, CIHR will assign the application to another reviewer.
It is important to note that many candidates will likely be conducting research outside of the reviewer's research specialty. Therefore, they should review the application with a generalist's perspective and assess the overall quality of the research proposed by the candidate, using the appropriate adjudication criteria. However, if a reviewer feels that their level of comfort reviewing an application is unacceptably low, they should choose the expertise option "Not Enough Expertise".
Assignment of Applications
Based on the ability to review/conflict information, CIHR, in collaboration with the committee's co-chairs and Scientific Officer, will proceed with assigning the applications to reviewers. Efforts will be made to ensure a balanced workload, taking into consideration potential conflicts, language capabilities and areas of expertise. The final authority for the assignment of applications rests with CIHR.
Each application will be assigned to three (3) reviewers to ensure the reliability of the rankings.
Step 3: Conduct preliminary review
Once the individual assignments are released, reviewers will have access to the full content of their assigned applications under the task "Conduct Reviews". They should then follow the steps below.
3.1 Review the adjudication criteria
Reviewers should first become familiar with the adjudication criteria for this funding opportunity. They can be found at the end of this document in Appendix A. This appendix provides an interpretation of each criterion and identifies which elements of the application to review for that criterion.
3.2 Read the assigned applications
Reviewers should read all of their assigned applications in details before rating any of them; and jot down notes to capture their impressions. The CIHR New Investigator Salary Awards Reviewer Worksheet in Appendix B provides a template that they may wish to use. This worksheet is strictly for the reviewers' own personal use and will not be filed with CIHR.
It is important to note that, for this program, expectations should differ based on the research area/discipline of the candidate. For example, publication productivity can vary when comparing a biomedical researcher, a clinician or a social scientist; where a health professional with a minimal amount of publications may have a publication record of superior quality than a biomedical researcher with numerous publications.
To ensure that all applications are treated equally, reviewers are asked to base their evaluation only on the content of the application and not to complete any additional research (e.g. publications via PubMed, etc.) on the candidate or the proposed research institution. They are however free to consult published lists of journal impact factors when assessing the candidate's research accomplishments. It is important to note however that journal impact factors vary from one discipline to another and that they do not necessarily indicate the quality of individual articles.
Reviewers should also be alert to unconscious bias related to gender, discipline or geographic location as detailed in the CIHR Peer Review Guide for Training and Salary Awards.
3.3 Rate the assigned applications
Reviewers are then asked to rate their assigned applications against each of the adjudication criteria described in Appendix A. This funding opportunity uses a binning system for the rating of applications. This method is meant to help reviewers differentiate between highly promising applications and ensure that the full range of the scale is used. For this system to work effectively, it is essential that the full range within each bin be used.
Reviewers are not bound by this initial rating and can change it at the peer review committee meeting.
Binning Descriptor | Scoring Range |
---|---|
Excellent | 4.2 – 4.9 |
Very Good | 3.5 – 4.1 |
Not Competitive | 0.0 – 3.4 |
Reviewers are to first assign each application to a bin; then provide a numerical score from the appropriate range. They are required to assign a minimum of 25% of their assigned applications to each bin (ResearchNet will not allow to submit if that minimum distribution is not respected). The remaining 25% can be distributed between the three bins as they deem appropriate.
3.4 Provide a written assessment for each assigned application
Reviewers are asked to provide a short written assessment for each assigned applications that supports their ratings. The objectives of the written review are to:
- provide constructive advice to applicants to assist them in improving the quality and efficiency of the proposed training;
- provide valuable information for the reviewers of Stage 2 in regards to the strength and the weaknesses of the application that justify their evaluation;
- provide points of discussion for the reviewers of Stage 2.
Comments should focus on the strengths and weaknesses of each adjudication criterion:
- Keep it simple;
- Use familiar descriptors that align with your rating;
- Include justification, context and an explanation of the comments, if applicable, for each topic introduced (e.g. refer to specific sections);
- Be clear and concise;
- While brevity is acceptable (e.g. using bullets), express complete thoughts and ensure the length is sufficient enough to inform the reader;
- Use objective and non-inflammatory language;
- Carefully avoid language that might be construed as sarcastic, flippant, arrogant or inappropriate in any way.
The applicant will receive the review as it is submitted by the reviewer. For this reason, reviewers are to refrain from inserting scores in the comments and should not identify themselves in order to ensure the confidentiality of the review process.
3.5 Flag issues for CIHR's attention
Any concerns regarding eligibility, ethics, human stem cells, etc. should be reported to CIHR staff immediately for follow-up and should not be noted in the written comments. For the full list of potential issues, please refer to the CIHR Peer Review Guide for Training and Salary Awards. These issues should not be considered as criteria for evaluation, except as they may impact on the scientific quality of the application. Concerns may be expressed by email at New.Investigators@cihr-irsc.gc.ca.
Step 4: Submit initial reviews and ratings on ResearchNet
Once the reviewers have submitted their scores and written comments, ResearchNet will open the task "View Score Distribution/Submit". If the score distribution is not balanced appropriately within each bin, they will not be able to submit. In this case, they will have to go back to their applications and review the scores they entered.
Once they are finished and the scores are balanced, they will have to "submit twice". Once under the "View Score Distribution/Submit" task (by clicking on "Submit"); then under the "Conduct Reviews" task (which will open automatically after the first submission) (by clicking on "Submit Review").
Once the reviewers have clicked the final "Submit Review" button, they won't be able to access their reviews anymore and will no longer be able to make changes.
It is important for reviewers to respect the deadline provided by submitting their reviews and scores in ResearchNet by the date specified via correspondence with CIHR staff responsible for this program. Delays in the peer review process will jeopardize CIHR's ability to release decisions to applicants by the published date. If, at any point in the process, a reviewer determines that he/she may not be able to submit his/her reviews on or before the deadline, he/she must contact CIHR staff as soon as possible.
Once the preliminary reviews and ratings are submitted, CIHR will rank the application. The rank list of applications will be used to identify applications that will be recommended for funding as well as those that will be submitted for further review at the Stage 2 face-to-face meeting.
Stage 2: Committee Review
Stage 2 consists of the face-to-face meeting of a multidisciplinary peer review committee and the objective of this meeting is to validate the rank list of applications.
Because of the rigorous streamlining that is completed following the first stage of the peer review process, reviewers of the second stage will have more time to judge and discriminate between potentially successful applications and ultimately, identify the most deserving applications to receive funding.
The committee's co-Chairs from the Stage 1 peer review process will be invited to participate as reviewers in the second stage of the review process; and a Chair and a Scientific Officer (SO) will be assigned to the committee.
Step 1: Determine ability to review and identify conflicts
To determine the ability to review and identify conflicts of interests, reviewers are to follow these steps:
- Log into ResearchNet.
- On the home page, click on the new link for the Stage 2 committee called "New Investigators". This will open the main task list.
- Just like Stage 1, complete the task "Review Confidentiality and Conflict of Interest Guidelines" (once completed, it will "open" the other tasks).
- Open the "Manage Conflicts/Ability to Review" task.
- For each of the assigned application, use the information provided to indicate their ability to review (high-medium-low) or if they have a conflict of interest including conflict of language. If there is a conflict, CIHR will assign the application to another reviewer.
At the meeting, any committee member who has a conflict of interest with an application cannot take part in the discussion of that application and will asked to leave the room for the duration of the discussion. The Chair and CIHR staff will be responsible for monitoring conflicts and for resolving areas of uncertainty.
Assignment of Applications
Based on the ability to review/conflict information, CIHR, in collaboration with the Chair and SO, will proceed with assigning the applications to reviewers. The final authority for the assignment of applications rests with CIHR.
Each application will be assigned to two (2) reviewers and one (1) reader.
The reviewers will have access to the written reviews submitted in the first stage of the review process.
Step 2: Conduct overall assessment of the assigned applications
In preparation for the committee meeting, reviewers are encouraged to:
- read their assigned applications in details;
- read the Stage 1 reviews submitted for those applications;
- conduct an overall assessment.
Written reviews are not required for this stage of the process. However, just like in Stage 1, reviewers that wish to take notes to capture their impressions can use the CIHR New Investigator Salary Awards Reviewer Worksheet in Appendix B. This worksheet is strictly for their own personal use and will not be filed with CIHR.
Step 3: Attend the committee meeting
During the meeting, any committee member who has a conflict of interest with an application cannot take part in the discussion of that application and will be asked to leave the room for the duration of the discussion. The Chair and CIHR staff will be responsible for monitoring conflicts and for resolving areas of uncertainty.
The discussion of each application will proceed as follows:
- The two reviewers assigned to the application announce their initial ratings to one decimal place.
- The application is streamlined (not discussed) if it meets the following conditions:
- The average of the internal reviewer's initial rating is below 3.5;
- There is no objection from the other reviewers that the application not be discussed1.
If the application is not streamlined, the discussion proceeds as follows:
- The primary reviewer assigned to the application will summarize the reviews from the stage 1 review process and provide their own assessment of the application, highlighting the strengths and weaknesses of the application.
- The second reviewer follows, concentrating on points of agreement or disagreement, and elaborate on points not addressed by the first reviewer.
- The reader will then concentrate on points of agreement or disagreement, and elaborate on points not addressed by the first and second reviewers.
- The Chair leads the discussion of the application by all committee members.
- The Scientific Officer reads the SO notes, capturing the key elements of the discussion to be considered when rating the application.
- The Chair seeks a consensus rating from the two reviewers assigned to the application. The internal reviewers may revise their initial ratings as they see fit. If a consensus cannot be reached, the mean value of the ratings of the two internal reviewers is used (round up, if necessary, to obtain a single decimal point).
- All committee members, including the two reviewers and the reader but excluding the Chair and Scientific Officer, cast individual confidential votes within +/- 0.5 of the consensus rating (they are not bound by the consensus rating). The final rating to be assigned to the application will be the average of these confidential votes. A vote will be taken even if the consensus rating is below 3.5 (i.e. not in the fundable range).
At the end of the meeting, if the peer review committee feels that any application(s) has been treated inconsistently, re-review of one or a small number of applications is permitted. Any committee member with a conflict of interest must again leave the room. Following the discussion, a consensus rating is determined by the two reviewers and voting proceeds as before. The committee may review the overall ranking of all applications at the end of the meeting.
Finalizing the rank-order list
Once all the scores from the Stage 2 face-to-face committee meeting have been received, a final rank-order priority list will be generated to make funding decisions.
Feedback
An important component of the peer review process is the review of the committee's effectiveness and functioning, and feedback on policy issues that may have arisen in the course of the process. This feedback provides an opportunity for CIHR staff to address any concerns of the committee members and for staff to record comments on the peer review process as part of CIHR's ongoing efforts to maintain an effective and high quality peer review system.
This discussion will occur at the end of meeting. Feedback can also be provided to the committee coordinator by email at New.Investigators@cihr-irsc.gc.ca.
Appendix A – CIHR New Investigator Salary Awards Adjudication Criteria
The evaluation of the New Investigator Salary Award applications is based on the following three criteria. These criteria should be weighted equally.
It should be noted that within their CCV, candidates were requested to only provide their contribution (e.g. publications) for the last 5 years.
Additional factors to be considered under each criterion may also be described in the funding opportunity details. Please contact the Program Delivery Coordinator if you need further guidance on how to apply the individual criteria. All aspects of the applications listed above should be reviewed taking into consideration the career stage and discipline of the candidate.
Criterion | Information Source | Notes and advice for reviewers on how to review |
---|---|---|
1. Applicant Track Record |
Common CV 5-year Research Plan 3 sponsor letters |
The candidate should demonstrate that they have already established a track-record of significant achievement that shows their commitment to research and the originality and impact of their research relative to their career stage. It should also demonstrate that the candidate is forging an international reputation for excellence in their fields and exceptional promise for the future.
|
2. Research Plan |
5-year Research Plan (references may be added to the Research proposal appendix) Summary of Progress Summary of Research Proposal (which is now in reference to the research plan) Letters of collaboration (if applicable) Common CV to evaluate the applicant's supervisory experience Common CV and Contribution Details (if applicable) to evaluate dissemination and outreach activities |
The candidate's research plan should cover the full duration of the salary award. A well prepared research plan should include details on the expected goals and their rationale, an explanation of how the goals are likely to be achieved (methodology), a rationalization as to where the research will be carried out (environment and resources), in collaboration with whom (team and collaborators) and finally details on the timeframe. Note: the Research Proposal and Research Proposal Appendix should be viewed as a reference to the 5-year Research Plan.
|
3. Environment and Support |
Dean of Faculty/Research Director letter Appendix 2A (parts 1, 2 and 3) completed by the Head of Department |
It is imperative that the salary award application demonstrates a strong institutional/organizational commitment to the continued scientific development and productivity of the candidate. The application should reveal a clear commitment from the institution/organization to ensure that the majority of the candidate's efforts will be devoted directly to the research plan, detailed in the application, with the remaining efforts being devoted to an appropriate balance of teaching, administrative, and clinical responsibilities.
|
Appendix B – CIHR New Investigator Salary Awards Reviewer Worksheet
The following table is meant to guide reviewers in the evaluation of the application. It is strictly for their working notes and will not be filed with CIHR.
Applicant Name:
Application #:
Criterion | Reviewer's Comments |
---|---|
1. Applicant Track Record | |
|
Strengths: Weaknesses: |
|
|
|
|
|
|
|
|
|
|
2. Research Plan | |
|
Strengths: Weaknesses: |
|
|
|
|
|
|
|
|
|
|
3. Environment and Support | |
|
Strengths: Weaknesses: |
|
|
|
|
|
|
|
|
Score: |
Footnotes
- Footnote 1
-
If an application is not discussed, the applicant will receive a copy of all internal reviewers' reports and the SO notes will only carry notification of the decision to streamline. Committee members do not vote on the rating; it is calculated as the mean of the initial ratings of the two internal reviewers.
- Date modified: