Peer Review Manual – Project
Table of Contents
- Policies Impacting Peer Review
- Principles of Peer Review
- Project Grant Application Review Process
- 4.1 Assigning Applications
- 4.2 Reviewing Applications
- 4.2.1 Conducting reviews
- 4.2.2 Adjudication Criteria and Interpretation Guidelines
- 4.2.3 Budget Assessment
- 4.2.4. Indigenous Health Research (IHR)
- 4.2.5 Randomized Controlled Trial (RCT) Evaluation Criteria and Headings
- 4.2.6 Commercialization Projects
- 4.3 The face-to-face Committee Meeting
- 4.4 After the Meeting
- CIHR Review Committees: Selection and Roles
The purpose of this manual is to provide highlights on the policies and information on procedures for peer review of applications in the Project Grant competition, as well as to outline the roles and responsibilities of peer review committee members. This manual also includes the description of the iterative peer review process for Indigenous Health Research designed in consultation with the Reference Group on Appropriate Review Practices for Indigenous Health Research.
The mandate of the Canadian Institutes of Health Research (CIHR) is to "To excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened health care system."
The Project Grant program is open to applicants in all areas of health research that are aligned with the CIHR mandate. It is designed to capture ideas with the greatest potential for important advances in fundamental or applied health-related knowledge, the health care system, and/or health outcomes, by supporting projects with a specific purpose and a defined endpoint. The best ideas may stem from new, incremental, innovative, and/or high-risk lines of inquiry or knowledge translation approaches.
Project Grant applications follow a single-stage committee-based peer review process. This process involves the evaluation of applications by a group of reviewers, who have the required experience and expertise to assess the quality and potential impact of the proposed research and/or research related activities, within the context of the program’s objectives. These reviewers are grouped into Peer Review Committees based on their expertise and the topics of applications submitted to these committees. More information about the selection process for members of Peer Review Committees and their roles can be found in section 5.
Peer Review Committees (PRCs) are responsible for:
- evaluating individual applications;
- rating each application;
- discussing applications at the face-to-face committee meeting;
- recommending a budget to support the proposed research if the application is approved; and,
- advising CIHR on the final ratings of the applications and providing recommendations on funding level and term.
Members of the Peer Review Committees should review the material in this manual as they prepare for and conduct evaluations of applications. In addition to reviewing the material in this manual, it is essential that members of the committees read and become familiar with the Funding Opportunity. Furthermore, all CIHR peer reviewers are also expected to:
- Become familiar with CIHR funding policies and guidelines as outlined in CIHR's Funding Policies.
- Formally agree to abide by CIHR's Confidentiality and Conflict of Interest Policy through ResearchNet.
- Ensure that all review materials are handled according to CIHR’s Guide on Handling Documents used in Peer Review.
- Provide feedback on the adjudication process to CIHR. For detailed regulations concerning all aspects of CIHR funding programs, please refer to the Grants and Awards Guide.
- Complete training on the Project review process through online learning modules and attendance to webinars.
2. Policies Impacting Peer Review
- Global Health Research and International Collaborations;
- Knowledge Translation;
- Tri-Agency Open Access Policy on Publications;
- Gender, Sex and Health Research;
- Official Languages.
2.1 Global Health Research and International Collaborations
As outlined in the CIHR Grants and Awards Guide Global Health Research Section and the International Collaborations Section, applicants who are eligible for CIHR funding may apply for funds to support research to be carried out in, or in direct collaboration with researchers and/or knowledge users based in other countries. CIHR supports international research projects and collaborations to address a range of areas including but not limited to established priorities in global health research and contributes to the development of health-research capacity both internationally and nationally.
2.2 Knowledge Translation
Knowledge translation (KT) is integral to CIHR's mandate and falls into two main categories, end of grant KT and integrated KT. With both categories of knowledge translation, CIHR expects researchers to disseminate their findings and facilitate their translation into improved health, more effective products or services, and/or a strengthened healthcare system. Note that the costs of dissemination are eligible expenditures in all CIHR grants.
For end of grant KT, many means of dissemination exist and the onus is on the researcher to select the most appropriate vehicle for the intended knowledge-user audience to ensure maximum impact. When the primary knowledge users are researchers, dissemination of results through the publication of articles in high quality and accessible journals is appropriate, although other strategies that increase awareness of the results and facilitate their application may also be appropriate. When knowledge-user audiences outside the research community should be informed of specific research findings, dissemination plans with more ambitious goals and comprehensive strategies are expected. With integrated KT, stakeholders or potential research knowledge users are engaged in the entire research process and the research is directed at producing solutions to issues or problems the stakeholders/knowledge users have identified.
2.3 Open Access Policy on Publications
CIHR believes that greater access to research publications and data will promote the ability of researchers and knowledge users in Canada and abroad to use and build on the knowledge needed to address significant health challenges. Open access will promote accessibility to CIHR-funded research and will serve to increase the international visibility of Canadian research. CIHR grant recipients are reminded to adhere to the responsibilities outlined in the Tri-Agency Open Access Policy on Publications. Under this policy, grant recipients must ensure that research papers and bio-molecular data generated from CIHR funding are freely accessible online.
2.4 Gender, Sex and Health Research
Applicants are encouraged to demonstrate the use of Gender and Sex-Based Analysis in their applications. Gender and sex-based analysis is an approach to research which systematically inquires about biological (sex-based) and sociocultural (gender-based) differences between women and men, boys and girls, without presuming that any such differences exist. The purpose of this line of inquiry is to promote rigorous health research which expands the understanding of health determinants in both sexes and results in improvements in health and health care. One helpful reference document for applicants and peer reviewers is the Sex, Gender and Health Research Guide: A Tool for CIHR Applicants. This document includes CIHR's definitions for gender and sex-based analysis and information on applying gender and sex-based analysis to research proposals.
2.5 Official Languages
In order to ensure that positive measures are undertaken within CIHR's mandate to contribute to the vitality of English and French minority communities in Canada, CIHR:
- encourages the appropriate inclusion of Official Language considerations and of English and French speaking Canadians, including those living in linguistic minority communities, in health research design, conduct and application to improve health outcomes, and
- facilitates equitable access to its programs and services for health research stakeholders in official language minority communities.
3. Principles of Peer Review
The integrity of the review process relies on well-established principles and policies that:
- ensure fair and effective evaluation; and,
- support CIHR’s mandate and objectives.
CIHR’s principles of review are:
Confidentiality is information about a person that will not be disclosed directly or indirectly to anyone else without that person's prior expressed consent. The information provided by applicants in their applications is protected by the Privacy Act and is made available to external assessors for reviewing purposes only. The use of this information for any other purpose is a breach of the Privacy Act and could result in a CIHR investigation and/or report to the federal Privacy Commissioner's Office.
CIHR instructs committee members to not approach or communicate in any way with applicants, or anyone outside of the committee, any information relating to the review of a specific application, or offer opinions on the applicant’s chances of success or failure. In turn, applicants are not to contact committee members, including the Chair and Scientific Officers, regarding the status of their applications (for example ratings).
By law, applicants have access to their own application files. Therefore, all written materials used in evaluating an application are made available to the applicants when they are notified of CIHR's funding decision for a competition.
The identity of the reviewers will not be revealed to the applicants. However, a list of review committee members is typically published on the CIHR website 60 days after the approval of funding of a competition.
3.2 Conflict of Interest
CIHR makes every effort to ensure that its decisions are fair and objective by identifying and addressing any conflict of interest between an applicant and a reviewer. According to the Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations (COIC), a conflict of interest means a conflict between a participant's (e.g., reviewers and observers) duties and responsibilities with regard to the review process, and a participant's private, professional, business or public interests. There may be a real, perceived or potential conflict of interest when the participant:
- would receive professional or personal benefit resulting from the funding opportunity or application being reviewed;
- has a professional or personal relationship with an applicant or the applicant's institution; or,
- has a direct or indirect financial interest in a funding opportunity or application being reviewed.
A conflict of interest may be deemed to exist or be perceived as such when committee members and observers:
- are a relative or close friend, or have a personal relationship with the applicants;
- are in a position to gain or lose financially/materially from the funding of the application;
- have had long-standing scientific or personal differences with the applicants;
- are currently affiliated with the applicants' institutions, organizations or companies—including research hospitals and research institutes*;
- are closely professionally affiliated with the applicants, as a result of having in the last six years:
- frequent and regular interactions with the applicants in the course of their duties at their department, institution, organization or company;
- been a supervisor or a trainee of the applicants;
- collaborated, published or shared funding with the applicants, or have plans to do so in the immediate future; or,
- been employed by the institution, when an institution is the applicant; and/or
- feel for any reason unable to provide an impartial review of the application.
All committee members (Chair, Scientific Officers, reviewers) and observers are subject to the same conflict of interest guidelines. CIHR staff is responsible for resolving areas of uncertainty. All committee members must read and agree to abide by the COIC policy prior to viewing any application information.
*It is important to manage both real and perceived conflicts of interest, but at the same time the review process should not suffer because of an overly strict application of the conflict of interest guidelines. For instance, a reviewer is not automatically in conflict with an application if they are from the same institution as the applicant, but do not know or work with the applicant.
Success of the review system is critically dependent upon the willingness and ability of all reviewers to:
- exercise rigorous judgement;
- be impartial and reasonable;
- understand and take into account in a balanced way the particular context of each application; and,
- provide constructive, quality review which helps the applicant by pointing out strengths and weaknesses that contributed to the application rating.
CIHR ensures transparency in the review process through a number of different mechanisms. All applications submitted to CIHR are evaluated by reviewers who provide an overall assessment of the application. This assessment is performed using the application requirements and review criteria established for each individual competition, as seen in the relevant funding opportunity. In addition, the Scientific Officer (SO) notes capture the discussion during the peer review deliberations. All reviews and SO notes, as applicable, are shared with applicants.
4. Project Grant Application Review Process
4.1 Assigning Applications
The competition process begins with the applicant submitting a registration followed by a full application. Applicants are required, within the ResearchNet registration process, to indicate their suggestion of most relevant peer review committees. Suggested committees must remain unchanged between registration and application. Applicants have the opportunity to indicate up to two potential review committees. Following submission of registration, the Chairs and Scientific Officers, together with CIHR staff, review the registrations assigned to their committees. Together, they are responsible for ensuring their committees are equipped with the appropriate expertise and, upon accepting applications for review by their committees, accept responsibility for ensuring that their committees conduct a fair review. In some cases, applications may not meet a particular committee's mandate and may need to be reassigned. CIHR will make the final decision on which peer review committee will review each application based on the summary of proposed research received at registration. The final committee selected may not necessarily be the applicant’s first or second choice.
After the final list of applications is compiled following the submission of full applications, committee members are given access to the application summaries to declare any conflicts of interest and indicate their level of expertise (i.e. High, Medium/Low/Not enough expertise).
Chairs, Scientific Officers and CIHR staff then assign each application to three committee members (three internal reviewers; a primary and two secondary reviewers) based on optimal alignment between the application content and reviewer’s declared expertise. Each reviewer will receive a maximum of 10 applications to review. External reviews can also be solicited if required. Four to six weeks before the peer review committee meeting, all committee members, excluding external referees, are given access to the full applications assigned to their committee. External referees see only those applications assigned to them. It is possible that once access to the full applications is granted to reviewers, a reviewer may decide that they do not, in fact, have the appropriate expertise to review applications that have been assigned to them. If this happens, reviewers are to contact CIHR immediately so that the application could be re-assigned appropriately.
4.2 Reviewing Applications
4.2.1 Conducting reviews
In preparation for the face-to-face committee meeting, expert reviewers will review their assigned applications by focusing on a project’s concept and feasibility, which are founded on a sound and important idea supported by a feasible plan of execution. Concept accounts for 25%, while feasibility accounts for 75% of the overall assessment.
Research proposals submitted to the Project Grant competition can be up to 10 pages (including tables and figures), should stand alone (i.e. it should contain all the information required to support the research plan) and should contain a complete description of the project. Although the applicants can structure their research proposal as they see fit, the reviewers assess the application based on the specific adjudication criteria as outlined in Section 4.2.2 below. Please note that the research proposal must adhere to the guidelines for attachments on the Acceptable Application Formats and Attachments.
If applicants are resubmitting an unsuccessful application, they have the option of including a response to previous reviews (maximum two pages). If the response to previous reviews is included by the applicant, reviewers are required to read this document. Please note that committee members are instructed to treat all applications, including resubmissions, as new applications.
In addition to their research proposal, applicants can include unlimited attachments. However, reviewers are not required to read those attachments.
Reviewers are also asked to review the applicant(s) CV(s). Through their CVs, applicants highlight their recognitions, funding history, activities and contributions that best demonstrate their leadership, significant contributions and productivity in the context of their research field(s).
For each application, reviewers will be asked to submit in ResearchNet:
- a rating per evaluation sub-criterion
- top/bottom selection
- a summary of the application
- a justification of the ratings by stating the strengths and weaknesses of the application based on the evaluation criteria
Success of the peer review system is critically dependent upon the willingness and ability of all reviewers to exercise rigorous scientific judgement, to be fair and reasonable, and to understand and take into account in a balanced way the particular context of each application. The opinions expressed in the reviews are those of the reviewers alone and are provided to the applicant without prior editing by CIHR. In circumstances where the Chair, the Scientific Officers or CIHR deems that the review expresses comments that could be construed as sarcastic, flippant, arrogant, or inappropriate in any way or contains factually incorrect information regarding CIHR’s policies, CIHR reserves the right to remove such information from the review. A constructive, quality review which helps the applicant by pointing out strengths and weaknesses that could be improved upon in a future submission will help demonstrate to an applicant that a fair assessment of the proposal was provided.
Completing the review in ResearchNet
The adjudication criteria, evaluation checkpoints and evaluation guidelines and other considerations regarding the sub-criteria are outlined in section 4.2.2 below.
To ensure consistency, reviewers must adhere to the following rating scale:
4.5 – 4.9
The application excels in most or all relevant aspects.
Any short-comings are minimal.
4.0 – 4.4
The application excels in many relevant aspects, and reasonably addresses all others.
Certain improvements are possible.
3.5 – 3.9
The application excels in some relevant aspects, and reasonably addresses all others.
Some improvements are necessary.
3.0 – 3.4
The application broadly addresses relevant aspects.
Major revisions are required.
0.0 – 2.9
The application fails to provide convincing information and/or has serious inherent flaws or gaps.
In ResearchNet, reviewers provide their initial ratings for each adjudication sub-criterion to one decimal place in advance of the peer review meeting. ResearchNet will automatically display the overall calculated rating. Note that reviewers are not bound by the initial overall calculated rating and can change it at the peer review committee meeting.
Reviewers also indicate whether the application reviewed falls into their top or bottom group. The top group should include the applications considered to be highly competitive and most deserving of being funded. This assessment is to be based on the reviewers' total experience with applications over the past five years and will be used during the streamlining phase of review (Section 4.3.1). Note that this allows reviewers to make an assessment even if they have only reviewed a few applications for that committee meeting. The proportion of applications in each group may vary depending on the overall quality of the applications reviewed.
Reviewers provide a summary of the application to demonstrate their understanding of the research work that is being proposed.
Reviewers also provide the strengths and weaknesses of the application based on the evaluation criteria. Although the reviewers are not required to provide strengths and weaknesses for each evaluation criterion, strengths and weaknesses that contributed to the application rating should be clearly articulated as they will be used to:
- provide the other reviewers assigned to the application with a justification for the rating given to the application, and
- to provide applicants with feedback.
For security reasons, ResearchNet times-out rather frequently. Unfortunately, we are unable to retrieve any lost information; therefore, we suggest to reviewers to save their work often so that it is not lost. Alternatively, reviewers can write their comments offline in a Word document and then copy and paste the text directly into the appropriate text box on ResearchNet when ready.
Reviewers will also be asked to flag applications in ResearchNet that involve any of the issues outlined below. These issues are not to be considered in the evaluation of the application unless they impact the scientific quality of the application. For detailed regulations concerning these issues, please refer to the CIHR Grants and Awards Guide.
- Ethical and other policy requirements: Responsibility for ensuring that all research meets ethical standards is delegated by CIHR to the applicant’s institution. Ethics forms are not required as part of the application. However, reviewers may flag specific issues, such as the use of human participants, animals, human tissues or hazardous material, or research that involve Indigenous peoples, if they feel they have not been adequately addressed.
- Human pluripotent stem cell research: Applications recommended for funding that involve the use of human stem cells must also be reviewed by the Stem Cell Oversight Committee (SCOC). Applicants are instructed to check the relevant box in the section entitled “Certification Requirements”. It is essential that the use of these stem cells be also assessed by reviewers.
The deadline for uploading reviews to ResearchNet, along with assessments of overall quality and initial ratings, is one week before the meeting date. Once reviewers have submitted their reviews, they will also be able to see the reviews of the other reviewers assigned to the same applications. Once reviews are submitted, reviewers will no longer be able to modify them prior to the committee meeting. Reviewers who wish to revise their reviews after the committee meeting will have one week to directly upload their changes to ResearchNet. Committee members are also able to view the external referee reports as they become available before the peer review committee meeting.
- Of note, it is the responsibility of all peer reviewers to familiarize themselves in advance of the committee meeting with all applications to be assessed by their committee as this will facilitate discussions at the face-to-face meeting.
4.2.2 Adjudication Criteria and Interpretation Guidelines
In this section, each of the sub-criteria relating to the concept and feasibility are described in more detail. A set of interpretation guidelines and considerations have been summarized for each sub-criterion. These are intended to provide guidance for the assessment of the application.
Criterion 1. Concept (25%)
- Significance and Impact of the Research (25%)
Criterion 2. Feasibility (75%)
- Approaches and Methods (50%)
- Expertise, Experience, and Resources (25%)
Reviewers should take into consideration the career stage, research field and institution setting of all applicants when assessing each criterion. The evidence should be notable compared to peers in similar fields and career stages.
- Specific considerations will need to be taken into account in review of applications in the Indigenous Health Research (IHR) committee (see IHR Committee Considerations under each Adjudication Sub-criterion below).
- Evaluation of all applications reviewed in the Randomized Controlled Trial (RCT) committee will need to consider the specific RCT evaluation criteria in their assessment (section 4.2.5).
- Evaluation of applications reviewed in the Commercialization committee will need to include the assessment of both Research and Technical plan and Commercialization plan according to specific criteria (section 4.2.6).
4.2.2.a Criterion 1: Concept (25%)
Significance and Impact of the Research
This sub-criterion is intended to assess the quality of what is being proposed. Each Evaluation Checkpoint is presented as a question and is followed by the Evaluation Guideline(s) for that Checkpoint.
- Is the project idea creative?
- The project idea is among the best formulated ideas in its field, stemming from new, incremental, innovative, and/or high-risk lines of inquiry; new or adapted research and knowledge translation/commercialization approaches/methodologies; and opportunities to apply research findings nationally and internationally.
- Is the rationale of the project idea sound?
- The project rationale is based on a logical integration of concepts.
- Are the overall goals and objectives of the project well-defined?
- The overall goal and objectives of the project are well-defined and clear.
- The goal states the purpose of the project, and what the project is ultimately expected to achieve.
- The objectives clearly define the proposed lines of inquiry and/or activities required to meet the goal.
- The proposed project outputs (i.e., the anticipated results of the project) are clearly described and aligned to the objectives.
- Are the anticipated project contributions likely to advance health-related knowledge (which includes basic science, model organisms, and other discovery research), health care, health systems and/or health outcomes?
- The context and needs (issues and/or gaps) of the project are clearly described.
- The anticipated contribution(s) (e.g. publishing in peer-reviewed journals) are clearly described, and should be substantive and relevant in relation to the context of the issues or gaps.
- The anticipated contribution(s) are realistic, i.e., directly stemming from the project outputs, as opposed to marginally related.
This sub-criterion is not intended to assess feasibility of the project, expertise of the team or the potential of success. These will be assessed under Criterion #2: Feasibility
Research should focus on addressing an issue (e.g., hypothesis or question, problem, need or gap) in any area across the spectrum of health (basic biomedical, health-related knowledge, health care, health systems, and/or health outcomes).
Depending on the nature of the project, it may have a research and/or knowledge translation / commercialization focus. Also, depending on the nature of the project, the rationale may be well-supported by evidence (e.g., literature review, previous findings, environmental scan, market analysis, stakeholder or partner input). However, this level of justification is not required for all types of projects (e.g., high-risk lines of inquiry).
In cases where projects have a primary implementation, or knowledge translation/commercialization (application and uptake of research findings) focus, the importance of the research should be validated as being substantive and relevant by stakeholders and partners, i.e., by those who could directly benefit from, or make use of, the project outputs.
IHR committee Considerations
The proposed research must be relevant to First Nations, Inuit and/or Métis priorities and have the potential to produce valued outcomes from the perspective of First Nations, Inuit and/or Métis participants and Indigenous peoples more broadly.
Global Health Research
Projects that have a global health research focus, or include international collaborations, are eligible for support through the Project Grant program. CIHR welcomes all research, from fundamental to applied, with the potential to advance health-related knowledge, and/or improve health outcomes for Canadians and the broader global community.
4.2.2.b Criterion 2: Feasibility (75%)
4.2.2.b-1 Approaches and Methods (50%)
This sub-criterion is intended to assess the quality of the Project's design and plan; including how and when the project will be completed. Each Evaluation Checkpoint is presented as a question and is followed by the Evaluation Guideline(s) for that Checkpoint.
- Are the approaches and methods appropriate to deliver the proposed output(s) and achieve the proposed contribution(s) to advancing health-related knowledge, health care, health systems, and/or health outcomes?
- The research and/or knowledge translation/commercialization approaches, methods and/or strategies are well-defined and justified in terms of being appropriate to accomplish the objectives of the project.
- Opportunities to maximize project contributions to advance health-related knowledge, health care, health systems and/or health outcomes should be proactively sought and planned for, but may also arise unexpectedly.
- Are the timelines and related deliverables of the project realistic?
- Timelines for the project should be appropriate in relation to the proposed project activities. Key milestones and deliverables should be aligned with the objectives of the project, and be feasible given the duration of the project.
- Does the proposal identify potential challenges and appropriate mitigation strategies?
- Critical scientific, technical, or organizational challenges should be identified, and a realistic plan to tackle these potential risks should be described. An exhaustive list is not expected.
Gender and Sex Considerations (if applicable)
Evidence demonstrates that biological, economic and social differences between women and men contribute to differences in health risks, health services use, health system interaction and health outcomes. All applicants to CIHR are expected to integrate gender and sex considerations into their research design, where appropriate, to maximize the relevance and applicability of health research findings to both men and women. This requirement is based on Canada's Health Portfolio Sex and Gender-Based Analysis Policy.
IHR committee Considerations
In addition to demonstrating scientific excellence (Western, Indigenous, or both), the proposed research approaches and methods must respect Indigenous values and ways of knowing and sharing, and abide by the Tri-Council Policy Statement Chapter 9: Research Involving the First Nations, Inuit and Métis Peoples of Canada, and/or Indigenous partnering community/organizational ethical guidelines, or clearly explain why other guidelines have been developed and agreed upon with the study governance body.
Applications submitted to the Project Grant competition may include an integrated knowledge translation approach or may have a knowledge translation focus, with at least one knowledge-user and one researcher. CIHR defines a knowledge user as an individual who is likely to be able to use the knowledge generated through research to make informed decisions about health policies, programs and/or practices. A knowledge user can be, but is not limited to, a practitioner, policy-maker, educator, decision-maker, health care administrator, community leader, or an individual in a health charity, patient group, private sector organization or media outlet.
CIHR defines integrated knowledge translation as a way of doing research with researchers and knowledge users working together to shape the research process – starting with collaborations on setting the research questions, deciding the methodology, being involved in data collection and tools development, interpreting the findings and helping disseminate the research results.
Designs, approaches, and methodologies will vary by project (e.g., field of research, target audience) and should include a knowledge translation approach that is appropriate to the nature of the project outputs.
4.2.2.b-2 Expertise, Experience, and Resources (25%)
This sub-criterion is intended to assess the appropriateness of the complement of expertise, experience, and resources among the applicants (Nominated Principal Applicant, Principal Applicant(s) and Co-Applicant(s)), and their institutions/organizations, as it relates to the ability to collectively deliver on the objectives of the project.
It is the responsibility of the Nominated Principal Applicant to ensure the proposed project is poised for success.
An estimate of the number of hours per week (contribution) for each applicant working on the project should be provided.
Each Evaluation Checkpoint is presented as a question and is followed by the Evaluation Guideline(s) for that Checkpoint.
- Does the applicant(s) bring the appropriate expertise and experience to lead and deliver the proposed output(s), and to achieve the proposed contribution(s)?
- The applicant(s) should demonstrate the combined expertise and experience needed to execute the project (i.e., deliver the proposed outputs as well as achieve the proposed contribution(s)).
- The roles and responsibilities of each applicant should be clearly described, and linked to the objectives of the project.
- Is there an appropriate level of engagement and/or commitment from the applicant(s)?
- The level of engagement (e.g., time and other commitments) of each applicant should be appropriate to the roles and responsibilities described.
- Is the environment (academic institution and/or other organization) appropriate to enable the conduct and success of the project?
- Project applicants should have access to the appropriate infrastructure, facilities, support personnel, equipment, and/or supplies to:
- Carry out their respective roles, and;
- As a collective, manage and deliver the proposed output(s), and achieve the proposed contribution(s).
- Project applicants should have access to the appropriate infrastructure, facilities, support personnel, equipment, and/or supplies to:
IHR Committee Considerations
Appropriateness of the team based on their overall scientific experience (Western, Indigenous, or both) and skills as well as their Indigenous community-based research experience, track record, relevance of past experience, including expertise related to Indigenous lived experience(s).
The required complement of expertise will vary by project. Applications with an integrated knowledge translation approach or knowledge translation focus must include knowledge users in defining/refining research questions, informing the research plan, conducting research, interpreting research findings, understanding the receptor community, leading dissemination activities, etc. Knowledge users may also be responsible, and accountable, for the application/uptake of the project outputs. The nature, breadth and depth of the applicant’s experiences and contributions should be assessed in the context of the applicant’s career stage.
Nominated Principal Applicants and Principal Applicants who have taken leaves of absence in the past seven years (e.g., parental, bereavement, medical, or administrative leave) may include a PDF document in their application to supplement the publication information included in their Common CV. Whatever length of time an applicant has taken off from research in the past seven years is the amount of time that they may include in the attachment. Reviewers should therefore review this document in order to ensure that they have an accurate profile of applicants’ research activities and achievements.
Project environments should be assessed according to their ability to support the proposed project activities. Institutions often function as "networked" environments or interdisciplinary networks, which means there may be multiple satellite environments contributing to the support environment. Reviewers should consider that for smaller institutions, or affiliated research facilities, resources and/or services may be obtained through networks, or may be contracted out.
4.2.3 Budget Assessment
Please note that the budget assessment must not be factored into the scientific assessment.
Applicants are asked to submit their budget using a modular template with budget increments. Reviewers are required to check the requested budget and justification although detailed scrutiny of each item is not expected. Reviewers are required to determine if the requested budget is appropriate to support the proposed project and if it is realistic and well-justified. Appropriate budget planning should be demonstrated. The requested resources, together with any existing resources, should be adequate to financially support the full scope of the project.
Partner contributions, if applicable (required for partnered applications), should be considered when reviewing support for the proposed project. Partner contributions will be outlined in letters of contribution and in the budget module through ResearchNet.
Reviewers may recommend that the budget remain as requested or recommend a reduction. If a reviewer suggests an adjustment to the budget, he/she will be required to provide comments to justify their recommendation.
The budget recommendation is intended to inform CIHR funding levels. Grant value and duration will be commensurate with the requirements of the project proposed, and will vary depending on the field, proposed approach, and scope of activities.
While budget recommendations are not factored into the overall rating of an application, CIHR values the experience and perspectives of reviewers in estimating overall budget requirements of the proposed project.
Each Evaluation Checkpoint is presented as a question and is followed by the Evaluation Guideline(s) for that Checkpoint.
- Is the requested funding appropriate to support the project? Is it realistic and well-justified, taking into account any other sources of funding?
- Appropriate budget planning should be demonstrated. The requested resources, together with any existing resources, should be adequate to financially support the full scope of the project.
- Reviewers are asked to consider the proposed budget of the project, and to provide a justified recommendation as to whether the budget should be:
- Accepted, as described; or
- Reduced by a specific dollar amount (total).
The applicants are asked to complete their budget using pre-set incremental levels and to provide a comprehensive description of what the funds will be used for, in order to justify the amount requested.
A detailed item-by-item scrutiny by the reviewer is not expected.
Peer review committees are asked to recommend budgets for grant applications irrespective of other sources of funds received or applied for, i.e., committees should not reduce recommended budgets to take into account potential overlap.
4.2.4. Indigenous Health Research (IHR)
Applications with a central focus on carrying out ethical and culturally competent research involving Indigenous peoples, with the intent to promote health through research that is in keeping with Indigenous values and traditions and follows the TCPS 2 - Chapter 9 Research Involving the First Nations, Inuit and Métis Peoples of Canada guidelines may be reviewed by the Indigenous Health Research (IHR) Committee. The IHR Committee may deem an application eligible for the Iterative Peer Review Process. A significant portion of the IHR committee will be composed of reviewers with lived-experience, knowledge, and/or expertise in Indigenous Health Research. Reviewers will provide their written reviews with a mentorship lens.
At the time of application submission, the research proposal must also explicitly describe engagement with the community in relation to the research. The IHR committee will take specific considerations into account when evaluating applications submitted to the committee (see IHR Committee Considerations under each Adjudication Sub-criterion above). Applications that do not fit with the IHR committee mandate will be reviewed by another committee.
All applications will be discussed at the face-to-face committee meeting to allow for development of an explicit mentoring approach to enable all promising Indigenous Health Research to continue to develop.
4.2.5 Randomized Controlled Trial (RCT) Evaluation Criteria and Headings
A RCT is a study in which investigators randomly assign eligible human research participants or other units of study (e.g., classrooms, clinics, playgrounds) into groups to receive or not receive one or more interventions that are being compared. The results are analyzed by comparing outcomes in the groups.
Please consult the RCT Evaluation Criteria and Headings website for detailed information on the specific criteria that need to be taken into account by the RCT committee when evaluating applications.
4.2.6 Commercialization Projects
Commercialization projects are designed to advance discoveries/inventions towards commercializable technologies/products, with a view to attract new investment, create new science-based businesses, organizations and initiatives, and ultimately improve health outcomes for Canadians.
Please consult the Commercialization Projects website for detailed information on the specific criteria that need to be taken into account by the Commercialization committee when evaluating applications.
4.3 The face-to-face Committee Meeting
The prime responsibilities of a peer review committee include discussion and evaluation of applications submitted for a particular competition by using CIHR's rating scale. When applicable, the committee will also discuss and recommend the grant term and a budget sufficient to support the proposed research. It is important that committees follow defined procedures in order to function in a consistent manner.
Of note, any committee member who has a conflict of interest with an application (as defined in Section 3.2, above) must not take part in the discussion of that application and must leave the meeting room before the application is discussed. The Chair and CIHR staff are responsible for monitoring conflicts and for resolving areas of uncertainty.
4.3.1 Streamlining Process
Only a portion of the Project Grant applications submitted to a competition will ultimately be funded. Therefore, it is important that committees focus their discussions on the most competitive applications to ensure that an accurate comparative evaluation is achieved. To help support this goal, a streamlining process is used to remove applications that will not be funded from the discussion process, thereby allowing peer reviewers more time to judge and discriminate between potentially successful applications and helping to ensure that the most deserving applications receive funding. Applicants whose proposals are streamlined still benefit from the review process, as they receive the written reviews from the assigned reviewers (but no detailed Scientific Officer notes).
Prior to the committee meeting
The average of the initial ratings and top/bottom placements from the three assigned reviewers are recorded in a spreadsheet to be shared with the committee.
At the committee meeting
The spreadsheet containing averages of the initial scores and top/bottom placements of applications is shared with the committee. The assessment of each application begins with reviewers announcing their initial ratings to one decimal place. An application is then eligible to be streamlined if it meets the following conditions:
- At least two reviewers placed the application in their bottom group;
- The average of the initial scores places the application in the bottom 25 to 35% of all applications in the committee.
An application is subsequently streamlined only if there is no objection from the other committee members regarding the fact that the application will not be discussed. Note that some of the streamlined applications may have an initial average rating above 3.5.
If an application is not discussed, the applicant will receive a copy of all reviewers' reports, but the Scientific Officer notes will only carry notification of the decision to streamline the application. Committee members do not vote on the rating; it is calculated as the average of the initial ratings of the assigned reviewers.
4.3.2 Rating of Applications
If an application is not streamlined, the committee discussion proceeds as follows:
The primary reviewer gives a brief synopsis of the proposal and presents his/her assessment, describing strengths and weaknesses of the proposal (see Section 4.2.2 for adjudication criteria);
The two secondary reviewers follow, concentrating on points of agreement or disagreement with the other reviewers, and elaborating on points not already addressed;
If external reviews have been provided, a committee member reads the reviews;
The Chair leads the discussion of the proposal by all committee members;
The Scientific Officer reads back the Scientific Officer notes, capturing the key elements of the committee discussion to be considered by the committee when rating the application;
The Chair seeks a consensus rating from the three assigned reviewers. The reviewers may revise their initial ratings in light of the committee discussion. If a consensus cannot be reached, the Chair declares the consensus score, usually by using the mean value of the reviewers’ scores after committee discussion (round up, if necessary, to obtain a single decimal point).
For the applications that are discussed, the three assigned reviewers first reach a consensus score. Then, all the committee members not in conflict (excluding the Chair and Scientific Officers) will be asked to vote (+/- 0.5 of the consensus score) for the final rating of each application.
4.3.4 Budget and term
The budget requested will not be factored into the scientific assessment of the application; however, CIHR will seek the advice of peer reviewers on the budget and the term requested. For additional information, please see section 4.2.3.
CIHR reserves the right to determine the final amount awarded to the grants.
4.3.5 Flagging of Applications
Issues related to ethics, eligibility, human pluripotent stem cell research and other concerns need to be flagged for CIHR's attention. The rating should not be influenced by such factors. For additional information, please see section 4.2, and for detailed regulations concerning these issues, please refer to the CIHR Grants and Awards Guide.
4.3.6 End of Meeting Review
Once all applications have been reviewed, if the peer review committee feels that any application(s) has been treated inconsistently, re-review of one or a small number of applications is permitted. Any committee member with a conflict of interest must again leave the room. Following discussion, a consensus rating is determined by the three internal reviewers and voting proceeds as before. The committee does not review the final ratings of all applications at the end of the meeting as individuals with conflicts of interest would be present.
An important component of any peer review committee meeting is the final review of the committee's effectiveness and functioning, and a discussion of policy issues that may have arisen in the course of its deliberations. This discussion provides an opportunity for CIHR staff to address any concerns of the committee members and for staff to record feedback on the peer review process as part of CIHR's ongoing efforts to maintain an effective and high quality peer review system.
4.4 After the Meeting
Upon completion of peer review, for all applications discussed, CIHR will collect the final ratings and recommendations on funding level and grant term from all the committee members. This information will be used for funding decision-making purposes.
Applications that receive a rating below 3.5 will not be funded.
Applicants are informed of the results of the competition as follows:
All applicants are sent a Notice of Decision, indicating whether or not their proposals were approved, and if approved, with what budget, which may or may not coincide with that recommended by the peer review committee. Applicants also receive a copy of all reviews and the Scientific Officer notes (for the applications discussed by the committees).
Applications that have been flagged for special attention and followed up by CIHR staff are withheld as "pending". The applicants will be notified if further information is required.
5. CIHR Review Committees: Selection and Roles
5.1 Selection of CIHR Review Committee Members
CIHR extends invitations to members of the health research community to join specific Project Grant Peer Review Committees (PRCs), based on their area(s) of expertise. CIHR recruits reviewers based on a set of selection criteria and in consultation with Chairs and Scientific Officers. The Chairs also have a role in the selection of Scientific Officers.
5.2 Committee Members
Project Grant Peer Review Committees consist of a Chair, one or two Scientific Officers and reviewers. Committee membership as a whole considers one or more of the following aspects:
- the need to cover the full range of research areas, relevant methodologies and experience for which the committee is responsible;
- the necessity for reviewing capability in both English and French so that applications in either official language can be evaluated by the committee; and,
- the need for national representation and representation by gender proportionate to membership in the Canadian health research community.
5.3 Committee Members Roles and Responsibilities
5.3.1 Committee Chair
Persons serving the role of a Chair in a committee are expected to follow a clear set of general instructions as defined by CIHR.
Before the committee meeting, the Chair:
- Suggests names of potential committee members (Scientific Officers and reviewers) to CIHR during both the formation of the review committee and the assignment of applications, as required. This could include identifying potential external referees to fill expertise gaps.
- Works with the Scientific Officer and CIHR staff to assign applications to specific peer review committees and to select reviewers for each application.
- Works with CIHR staff to establish a review sequence or agenda, and to determine when streamlining of applications will be employed.
- Works with CIHR staff to manage conflicts of interest relative to review committee members.
At the committee meeting and in the role of moderator, it is the Chair’s responsibility to ensure that the review committee functions smoothly, effectively and objectively, and that a positive, constructive, fair-minded environment in which research proposals are evaluated is established and maintained. To do so, the Chair, may:
- Provide opening remarks to the committee.
- Make the proper introductions and explain the meeting process to members, as appropriate.
- Ensure that all committee members who are in conflict with an application leave the meeting room before the discussion about that particular application begins.
- Appoint a delegate as Chair or Scientific Officer when either individual leaves the meeting room due to a conflict of interest with an application or any other reason. Whenever possible, the same individual should not occupy both roles in the context of the same committee meeting.
- Fulfill an oversight role – does not rate applications or vote during the committee meeting.
- Ensure the involvement of the entire committee in evaluating each application.
- Work with the Scientific Officer(s), as required, to summarize the discussion around each application, before the consensus rating is reached.
- Ensure that a consensus rating is reached.
- Ensure that both the budget and term of the grant are discussed amongst committee members and that comments are added to the Scientific Officer notes, as appropriate.
- Ensure that specific ethical concerns and other CIHR requirements are addressed, and that any related discussion is captured in the Scientific Officer notes.
At the end of the committee meeting, the Chair may:
- Propose that any application that has been identified by committee members as being inconsistently reviewed in the review process is discussed again.
- Allow time to receive feedback from the committee members regarding the effectiveness and functioning of the committee.
5.3.2 Scientific Officer (SO)
Individuals serving the role of Scientific Officer (SO) at a committee meeting are responsible for supporting the Chair in his/her role during the committee meeting and are expected to follow a clear set of general instructions as defined by CIHR.
Before the committee meeting, the Scientific Officer may:
- Suggest names of potential committee reviewers to CIHR during the formation of the review committee, as well as during the assignment of applications, as required. This could include identifying potential external reviewers to fill specific expertise gaps.
At the committee meeting, the Scientific Officer:
- Takes official notes of the committee discussions for each application (SO notes). These will be sent to applicants along with the reviewer reports, if applicable.
- Reads back the SO notes to receive validation from committee members and to receive additional input before a consensus on the application is reached by the committee and member votes are cast.
- Reads back the summary of changes to the budget and term of the grant on the application, and ensures that special considerations related to ethics and/or other issues have been flagged for the attention of CIHR and that these are recorded within the evaluation, if applicable.
At the end of the committee meeting, the Scientific Officer:
- Finalizes and posts the Scientific Officer notes directly to ResearchNet or sends them to the CIHR Program Coordinator no later than 5 working days after the meeting.
- Scientific Officers do not rate applications or vote during the committee meeting.
- The Scientific Officer notes should provide the applicant with insight into the committee discussion of his/her application. Their purpose is two-fold:
- to identify the determining factors in the evaluation of the application by the committee; and,
- to describe how the budget and term was determined by the committee (if applicable).
- The Scientific Officer notes should be clear and concise, and give objective and constructive feedback to the applicant. They should:
- include the strengths and weaknesses of the application;
- address the issues that had the greatest impact on the evaluation, as they relate to the program’s evaluation criteria;
- address aspects of the committee discussion that were not captured in the reviewer written reports;
- describe how reviewer disagreements, as seen in the individual reviewer reports, were reconciled by identifying which view was favored by the committee.
- CIHR will not edit or modify the Scientific Officer notes without the Scientific Officer’s approval.
Reviewers are committee members assigned to review one or more applications. Applications are assigned to three reviewers for assessment, who participate in the rating of applications and submit an in-depth written report that is provided to the applicant. They present their review at the committee meeting where they lead the discussion, and vote during the meeting. They also participate in the discussion and rating of all other applications before the committee for which they are not in conflict.
Before the committee meeting, reviewers must:
- Evaluate the applications assigned to them by providing a critical assessment of the applications, as well as constructive feedback based on the program’s objectives and evaluation criteria described in the funding opportunity:
- consider all factors and the strengths and weaknesses of the applications in relation to each criterion.
- focus their comments on the factors most relevant to the rating;
- provide comments on the budget requested and a formal recommendation to CIHR, including clear and detailed reasons for any recommended budget or term cuts, if applicable; and,
- provide comments on issues that they feel should be flagged, as required. These concerns should not influence the rating or budget recommendations, unless they bear on the scientific merit of the application.
- Provide their initial ratings to one decimal place on ResearchNet in advance of the committee meeting. Reviewers also divide their applications into a top and bottom group, based on their overall quality.
- Familiarize themselves in advance of the meeting with all applications to be assessed by their committee as this will facilitate discussions at the face-to-face committee meeting.
At the committee meeting, reviewers:
- Present the applications assigned to them for assessment (see Section 4.2.2 for adjudication criteria);
- Participate in the committee discussions
- Vote on applications
- Provide recommendations for budget and term cuts, if applicable
At the end of the committee meeting, reviewers:
- Have the option to revise their reviews within a week of the end of the meeting.
- Date modified: