CIHR Reviewers’ Guide for New Investigator Salary Awards

October 2014

Table of Contents

Introduction

On behalf of CIHR, we would like to thank the reviewers for agreeing to serve as a peer review committee member. The success of the peer review process is made possible by dedicated people like you who generously give their time and expertise. Your efforts are greatly appreciated by CIHR and the scientific community.

The purpose of this document is to provide instructions on the peer review process specific to the CIHR New Investigator Salary Awards program.

Peer Review at CIHR

Information on CIHR's objectives, governance and policies; an outline of the roles and responsibilities of peer review committee members; and the policies, principles and procedures for peer review of applications can be found in the CIHR Peer Review Guide for Training and Salary Awards. It is important that reviewers become familiar with this document, as well as the present document, before starting the reviews.

Summary of the Peer Review Process

The CIHR New Investigator Salary Awards program uses multi-stage structured review process. The review process is completed in two (2) stages: an individual evaluation of an assigned set of applications using the online ResearchNet platform, followed by a face-to-face committee meeting for a discussion of a selected group of applications.

There are five (5) peer review committees for this program. Each application received will be assigned to the committee with the mandate that most closely aligns with the applicant's training, credentials and area of research. The link to the committees' mandates is listed under the section "Read the pertinent documentation".

Reviewers are asked to follow the step-by-step instructions below to successfully complete all peer review tasks:

Stage 1: Individual review
Step 1: Read the pertinent documentation
Step 2: Determine ability to review and identify conflicts
Step 3: Conduct preliminary review
Step 4: Submit initial reviews and ratings on ResearchNet

Stage 2: The committee review
Step 1: Determine ability to review and identify conflicts
Step 2: Conduct overall assessment of your assigned applications
Step 3: Attend the committee meeting

Stage 1: Individual Review

Step 1: Read the pertinent documentation

The peer review process for this program is described in details in this document. It is essential to read the document and be familiar with it. It is also important to read the following:

Step 2: Determine ability to review and identify conflicts

To determine the ability to review and identify conflicts of interests, reviewers are to follow these steps:

  • Log into ResearchNet.
  • On the home page, click on the link of their assigned committee to open the main task list.
  • Complete the task "Review Confidentiality and Conflict of Interest Guidelines" (once completed, it will "open" the other tasks).
  • Open the "Manage Conflicts/Ability to Review" task.
  • For each of the assigned application, use the information provided to indicate their ability to review (high-medium-low) or if they have a conflict of interest including conflict of language. If there is a conflict, CIHR will assign the application to another reviewer.

It is important to note that many candidates will likely be conducting research outside of the reviewer's research specialty. Therefore, they should review the application with a generalist's perspective and assess the overall quality of the research proposed by the candidate, using the appropriate adjudication criteria. However, if a reviewer feels that their level of comfort reviewing an application is unacceptably low, they should choose the expertise option "Not Enough Expertise".

Assignment of Applications

Based on the ability to review/conflict information, CIHR, in collaboration with the committee's co-chairs and Scientific Officer, will proceed with assigning the applications to reviewers. Efforts will be made to ensure a balanced workload, taking into consideration potential conflicts, language capabilities and areas of expertise. The final authority for the assignment of applications rests with CIHR.

Each application will be assigned to three (3) reviewers to ensure the reliability of the rankings.

Step 3: Conduct preliminary review

Once the individual assignments are released, reviewers will have access to the full content of their assigned applications under the task "Conduct Reviews". They should then follow the steps below.

3.1 Review the adjudication criteria

Reviewers should first become familiar with the adjudication criteria for this funding opportunity. They can be found at the end of this document in Appendix A. This appendix provides an interpretation of each criterion and identifies which elements of the application to review for that criterion.

3.2 Read the assigned applications

Reviewers should read all of their assigned applications in details before rating any of them; and jot down notes to capture their impressions. The CIHR New Investigator Salary Awards Reviewer Worksheet in Appendix B provides a template that they may wish to use. This worksheet is strictly for the reviewers' own personal use and will not be filed with CIHR.

It is important to note that, for this program, expectations should differ based on the research area/discipline of the candidate. For example, publication productivity can vary when comparing a biomedical researcher, a clinician or a social scientist; where a health professional with a minimal amount of publications may have a publication record of superior quality than a biomedical researcher with numerous publications.

To ensure that all applications are treated equally, reviewers are asked to base their evaluation only on the content of the application and not to complete any additional research (e.g. publications via PubMed, etc.) on the candidate or the proposed research institution. They are however free to consult published lists of journal impact factors when assessing the candidate's research accomplishments. It is important to note however that journal impact factors vary from one discipline to another and that they do not necessarily indicate the quality of individual articles.

Reviewers should also be alert to unconscious bias related to gender, discipline or geographic location as detailed in the CIHR Peer Review Guide for Training and Salary Awards.

3.3 Rate the assigned applications

Reviewers are then asked to rate their assigned applications against each of the adjudication criteria described in Appendix A. This funding opportunity uses a binning system for the rating of applications. This method is meant to help reviewers differentiate between highly promising applications and ensure that the full range of the scale is used. For this system to work effectively, it is essential that the full range within each bin be used.

Reviewers are not bound by this initial rating and can change it at the peer review committee meeting.

Binning Descriptor Scoring Range
Excellent 4.2 – 4.9
Very Good 3.5 – 4.1
Not Competitive 0.0 – 3.4

Reviewers are to first assign each application to a bin; then provide a numerical score from the appropriate range. They are required to assign a minimum of 25% of their assigned applications to each bin (ResearchNet will not allow to submit if that minimum distribution is not respected). The remaining 25% can be distributed between the three bins as they deem appropriate.

3.4 Provide a written assessment for each assigned application

Reviewers are asked to provide a short written assessment for each assigned applications that supports their ratings. The objectives of the written review are to:

  • provide constructive advice to applicants to assist them in improving the quality and efficiency of the proposed training;
  • provide valuable information for the reviewers of Stage 2 in regards to the strength and the weaknesses of the application that justify their evaluation;
  • provide points of discussion for the reviewers of Stage 2.

Comments should focus on the strengths and weaknesses of each adjudication criterion:

  • Keep it simple;
  • Use familiar descriptors that align with your rating;
  • Include justification, context and an explanation of the comments, if applicable, for each topic introduced (e.g. refer to specific sections);
  • Be clear and concise;
  • While brevity is acceptable (e.g. using bullets), express complete thoughts and ensure the length is sufficient enough to inform the reader;
  • Use objective and non-inflammatory language;
  • Carefully avoid language that might be construed as sarcastic, flippant, arrogant or inappropriate in any way.

The applicant will receive the review as it is submitted by the reviewer. For this reason, reviewers are to refrain from inserting scores in the comments and should not identify themselves in order to ensure the confidentiality of the review process.

3.5 Flag issues for CIHR's attention

Any concerns regarding eligibility, ethics, human stem cells, etc. should be reported to CIHR staff immediately for follow-up and should not be noted in the written comments. For the full list of potential issues, please refer to the CIHR Peer Review Guide for Training and Salary Awards. These issues should not be considered as criteria for evaluation, except as they may impact on the scientific quality of the application. Concerns may be expressed by email at New.Investigators@cihr-irsc.gc.ca.

Step 4: Submit initial reviews and ratings on ResearchNet

Once the reviewers have submitted their scores and written comments, ResearchNet will open the task "View Score Distribution/Submit". If the score distribution is not balanced appropriately within each bin, they will not be able to submit. In this case, they will have to go back to their applications and review the scores they entered.

Once they are finished and the scores are balanced, they will have to "submit twice". Once under the "View Score Distribution/Submit" task (by clicking on "Submit"); then under the "Conduct Reviews" task (which will open automatically after the first submission) (by clicking on "Submit Review").

Once the reviewers have clicked the final "Submit Review" button, they won't be able to access their reviews anymore and will no longer be able to make changes.

It is important for reviewers to respect the deadline provided by submitting their reviews and scores in ResearchNet by the date specified via correspondence with CIHR staff responsible for this program. Delays in the peer review process will jeopardize CIHR's ability to release decisions to applicants by the published date. If, at any point in the process, a reviewer determines that he/she may not be able to submit his/her reviews on or before the deadline, he/she must contact CIHR staff as soon as possible.

Once the preliminary reviews and ratings are submitted, CIHR will rank the application. The rank list of applications will be used to identify applications that will be recommended for funding as well as those that will be submitted for further review at the Stage 2 face-to-face meeting.

Stage 2: Committee Review

Stage 2 consists of the face-to-face meeting of a multidisciplinary peer review committee and the objective of this meeting is to validate the rank list of applications.

Because of the rigorous streamlining that is completed following the first stage of the peer review process, reviewers of the second stage will have more time to judge and discriminate between potentially successful applications and ultimately, identify the most deserving applications to receive funding.

The committee's co-Chairs from the Stage 1 peer review process will be invited to participate as reviewers in the second stage of the review process; and a Chair and a Scientific Officer (SO) will be assigned to the committee.

Step 1: Determine ability to review and identify conflicts

To determine the ability to review and identify conflicts of interests, reviewers are to follow these steps:

  • Log into ResearchNet.
  • On the home page, click on the new link for the Stage 2 committee called "New Investigators". This will open the main task list.
  • Just like Stage 1, complete the task "Review Confidentiality and Conflict of Interest Guidelines" (once completed, it will "open" the other tasks).
  • Open the "Manage Conflicts/Ability to Review" task.
  • For each of the assigned application, use the information provided to indicate their ability to review (high-medium-low) or if they have a conflict of interest including conflict of language. If there is a conflict, CIHR will assign the application to another reviewer.

At the meeting, any committee member who has a conflict of interest with an application cannot take part in the discussion of that application and will asked to leave the room for the duration of the discussion. The Chair and CIHR staff will be responsible for monitoring conflicts and for resolving areas of uncertainty.

Assignment of Applications

Based on the ability to review/conflict information, CIHR, in collaboration with the Chair and SO, will proceed with assigning the applications to reviewers. The final authority for the assignment of applications rests with CIHR.

Each application will be assigned to two (2) reviewers and one (1) reader.

The reviewers will have access to the written reviews submitted in the first stage of the review process.

Step 2: Conduct overall assessment of the assigned applications

In preparation for the committee meeting, reviewers are encouraged to:

  • read their assigned applications in details;
  • read the Stage 1 reviews submitted for those applications;
  • conduct an overall assessment.

Written reviews are not required for this stage of the process. However, just like in Stage 1, reviewers that wish to take notes to capture their impressions can use the CIHR New Investigator Salary Awards Reviewer Worksheet in Appendix B. This worksheet is strictly for their own personal use and will not be filed with CIHR.

Step 3: Attend the committee meeting

During the meeting, any committee member who has a conflict of interest with an application cannot take part in the discussion of that application and will be asked to leave the room for the duration of the discussion. The Chair and CIHR staff will be responsible for monitoring conflicts and for resolving areas of uncertainty.

The discussion of each application will proceed as follows:

  1. The two reviewers assigned to the application announce their initial ratings to one decimal place.
  2. The application is streamlined (not discussed) if it meets the following conditions:
    • The average of the internal reviewer's initial rating is below 3.5;
    • There is no objection from the other reviewers that the application not be discussed1.

If the application is not streamlined, the discussion proceeds as follows:

  1. The primary reviewer assigned to the application will summarize the reviews from the stage 1 review process and provide their own assessment of the application, highlighting the strengths and weaknesses of the application.
  2. The second reviewer follows, concentrating on points of agreement or disagreement, and elaborate on points not addressed by the first reviewer.
  3. The reader will then concentrate on points of agreement or disagreement, and elaborate on points not addressed by the first and second reviewers.
  4. The Chair leads the discussion of the application by all committee members.
  5. The Scientific Officer reads the SO notes, capturing the key elements of the discussion to be considered when rating the application.
  6. The Chair seeks a consensus rating from the two reviewers assigned to the application. The internal reviewers may revise their initial ratings as they see fit. If a consensus cannot be reached, the mean value of the ratings of the two internal reviewers is used (round up, if necessary, to obtain a single decimal point).
  7. All committee members, including the two reviewers and the reader but excluding the Chair and Scientific Officer, cast individual confidential votes within +/- 0.5 of the consensus rating (they are not bound by the consensus rating). The final rating to be assigned to the application will be the average of these confidential votes. A vote will be taken even if the consensus rating is below 3.5 (i.e. not in the fundable range).

At the end of the meeting, if the peer review committee feels that any application(s) has been treated inconsistently, re-review of one or a small number of applications is permitted. Any committee member with a conflict of interest must again leave the room. Following the discussion, a consensus rating is determined by the two reviewers and voting proceeds as before. The committee may review the overall ranking of all applications at the end of the meeting.

Finalizing the rank-order list

Once all the scores from the Stage 2 face-to-face committee meeting have been received, a final rank-order priority list will be generated to make funding decisions.

Feedback

An important component of the peer review process is the review of the committee's effectiveness and functioning, and feedback on policy issues that may have arisen in the course of the process. This feedback provides an opportunity for CIHR staff to address any concerns of the committee members and for staff to record comments on the peer review process as part of CIHR's ongoing efforts to maintain an effective and high quality peer review system.

This discussion will occur at the end of meeting. Feedback can also be provided to the committee coordinator by email at New.Investigators@cihr-irsc.gc.ca.

Appendix A – CIHR New Investigator Salary Awards Adjudication Criteria

The evaluation of the New Investigator Salary Award applications is based on the following three criteria. These criteria should be weighted equally.

It should be noted that within their CCV, candidates were requested to only provide their contribution (e.g. publications) for the last 5 years.

Additional factors to be considered under each criterion may also be described in the funding opportunity details. Please contact the Program Delivery Coordinator if you need further guidance on how to apply the individual criteria. All aspects of the applications listed above should be reviewed taking into consideration the career stage and discipline of the candidate.

Criterion Information Source Notes and advice for reviewers on how to review
1. Applicant Track Record

Common CV

5-year Research Plan

3 sponsor letters

The candidate should demonstrate that they have already established a track-record of significant achievement that shows their commitment to research and the originality and impact of their research relative to their career stage. It should also demonstrate that the candidate is forging an international reputation for excellence in their fields and exceptional promise for the future.

  • Assess the quality and diversity of the academic and research training received by the applicant. Are the candidate's prior training and research experience relevant and appropriate for this award?
  • What awards or acknowledgements of academic and research achievement has the applicant received?
  • What peer-reviewed grant funding has been obtained or applied for? Consider the applicant's role in obtaining multi-investigator grants and the numbers and categories of grants that support their research.
  • Does the applicant demonstrate leadership in the field through knowledge translation activities (e.g. commercialization, clinical activities, public health activities, invited speaker or moderator, etc.), and contributions to research training, mentorship and supervisory activities, contributions to professional activities (e.g. peer review committee, review of scientific manuscripts, review of grant/award applications, graduate committees, departmental and extra-departmental administrative, etc.)?
  • Does the applicant have a good publication record in peer-reviewed journals? To what extent does the applicant appear to have contributed to the work published? Patents and other significant contributions should be considered. Consider publications or papers accepted for publications, demonstrated capacity to publishing as principal author without mentors, the quality of journals and the continuity of production since the start of the career.
  • What research has been accomplished to date and has the applicant clearly demonstrated independence and originality? Significant contributions to team research should also be considered.
  • Are the sponsor letters from three reputable individuals in the field, and do they show evidence that the candidate has the characteristics and skills that correlate with career research achievement?
2. Research Plan

5-year Research Plan (references may be added to the Research proposal appendix)

Summary of Progress

Summary of Research Proposal (which is now in reference to the research plan)

Letters of collaboration (if applicable)

Common CV to evaluate the applicant's supervisory experience

Common CV and Contribution Details (if applicable) to evaluate dissemination and outreach activities

The candidate's research plan should cover the full duration of the salary award. A well prepared research plan should include details on the expected goals and their rationale, an explanation of how the goals are likely to be achieved (methodology), a rationalization as to where the research will be carried out (environment and resources), in collaboration with whom (team and collaborators) and finally details on the timeframe.

Note: the Research Proposal and Research Proposal Appendix should be viewed as a reference to the 5-year Research Plan.

  • Is the research plan relevant to the candidate's research career objectives?
  • Are the ideas put forward in the research plan innovative and/or original?
  • Does the research plan have the potential of significantly advancing our understanding of the area?
  • Is the proposed research feasible, given the resources and support available to the investigator?
  • Has preliminary data been accumulated to support the 5 year research plan?
  • Have strong research interactions and collaborations been established? What are the candidate's contributions to the proposed collaborations? Are the proposed collaboration of high quality and what level of impact could they have on the professional growth of the candidate?
  • Will the quality and extent of proposed dissemination and outreach activities be within and/or beyond the academic community?
  • Is the applicant's leadership role clearly outlined in his/her research plan? Does the research plan provide evidence of the applicant's leadership in the design and conduct of the proposed research? Leadership can be demonstrated through the applicant's engagement as a mentor, their ability to manage research, to contribute novel ideas to their research program, to make decisions that are crucial to the success of the research program, to lead his/her research collaboratively, have excellent working relationships with those around him/her, etc.
  • For New Investigators, has the applicant demonstrated independence or shown promise to become independent from former supervisors?
3. Environment and Support

Dean of Faculty/Research Director letter

Appendix 2A (parts 1, 2 and 3) completed by the Head of Department

It is imperative that the salary award application demonstrates a strong institutional/organizational commitment to the continued scientific development and productivity of the candidate. The application should reveal a clear commitment from the institution/organization to ensure that the majority of the candidate's efforts will be devoted directly to the research plan, detailed in the application, with the remaining efforts being devoted to an appropriate balance of teaching, administrative, and clinical responsibilities.

  • What is (or will be) the applicant's position within the institution?
  • What space, operating funds, infrastructure and/or other resources will be available to the candidate and are they adequate?
  • Has the institution demonstrated a commitment to enable the candidate to devote full time to research and related duties by releasing the applicant from teaching, administration, clinical work and/or other responsibilities?
  • Has the institution demonstrated support for the scientific development of the candidate and their independent research program?
  • Does the institution or organization demonstrate leadership in the candidate's chosen field?
  • Will the candidate receive adequate scientific and career guidance?

Appendix B – CIHR New Investigator Salary Awards Reviewer Worksheet

The following table is meant to guide reviewers in the evaluation of the application. It is strictly for their working notes and will not be filed with CIHR.

Applicant Name:
Application #:

Criterion Reviewer's Comments
1. Applicant Track Record
  • Academic and research training received by the applicant

Strengths:

Weaknesses:
  • Prestigious awards or acknowledgements of academic achievement
  • Publications and related research activity
  • Research accomplished to date - has the applicant clearly demonstrated independence and originality? Contributions to team research?
  • To what extent does the applicant demonstrate leadership in the field? Quality of the applicant's training, mentorship and supervisory activities.
  • Record of funding
2. Research Plan
  • Are the ideas put forward innovative and/or original?

Strengths:

Weaknesses:
  • Could the research plan extend significantly our understanding of the area?
  • Is the project feasible, given the resources and support available to the investigator?
  • Have strong research interactions and collaborations been established?
  • Has the applicant shown evidence of leadership?
  • Has the applicant demonstrated independence or shown promise to become independent from former supervisors? (New Investigators)
3. Environment and Support
  • What is (or will be) the applicant's position within the institution?

Strengths:

Weaknesses:
  • Institutional commitment to space, operating funds and/or infrastructure.
  • Has the institution demonstrated a commitment to protect the candidate's research time?
  • Does the institution or organization demonstrate leadership in the candidate's chosen field?
  • Letters of support
Score:

Footnotes

Footnote 1

If an application is not discussed, the applicant will receive a copy of all internal reviewers' reports and the SO notes will only carry notification of the decision to streamline. Committee members do not vote on the rating; it is calculated as the mean of the initial ratings of the two internal reviewers.

1

Date modified: