Peer Review Manual – Project
Table of Contents
- Policies and Considerations Impacting Peer Review
- Principles of Peer Review
- Project Grant Application Review Process
- 4.1 Assigning Applications
- 4.2 Reviewing Applications
- 4.2.1 Conducting reviews
- 4.2.2 Adjudication Criteria and Interpretation Guidelines
- 4.2.3 Budget Assessment
- 4.2.4. Indigenous Health Research (IHR)
- 4.2.5 Randomized Controlled Trials (RCTs)
- 4.2.6 Commercialization Projects
- 4.3 The face-to-face Peer Review Committee Meeting
- 4.4 After the Committee Meeting
- Peer Review Committees: Selection and Roles of Members
The purpose of this manual is to provide highlights on the policies and information on procedures for the peer review of applications submitted to the Project Grant competition, as well as to outline the roles and responsibilities of peer review committee members. This manual also includes the description of the iterative peer review process for Indigenous Health Research designed in consultation with the Reference Group on Appropriate Review Practices for Indigenous Health Research.
The Project Grant program is open to applicants in all areas of health research that are aligned with the CIHR mandate. It is designed to capture ideas with the greatest potential for important advances in fundamental or applied health-related knowledge, health care, health systems, and/or health outcomes, by supporting projects or programs of research conducted by individual researchers or groups of researchers in all areas of health. The best ideas may stem from new, incremental, innovative, and/or high-risk lines of inquiry or knowledge translation approaches.
Project grant applications follow a single-stage committee-based peer review process. This process involves the evaluation of applications by a group of reviewers, who have the required experience and expertise to assess the quality and potential impact of the proposed research and research related activities, within the context of the program’s objectives. These reviewers are grouped into Peer Review Committees based on their expertise and the topics of applications submitted to these committees.
Peer Review Committees (PRCs) are responsible for:
- evaluating individual applications
- rating each application
- discussing applications at the face-to-face committee meeting
- recommending a budget and term to support the proposed research if the application is approved.
Members of the Peer Review Committees should review the material in this manual as they prepare for and conduct evaluations of applications. It is essential that members of the committees read and become familiar with the Funding Opportunity. They are also expected to:
- Become familiar with CIHR funding policies and guidelines as outlined in CIHR's Funding Policies.
- Formally agree to abide by CIHR's Confidentiality and Conflict of Interest Policy through ResearchNet.
- Ensure that all review materials are handled according to CIHR’s Guide on Handling Documents used in Peer Review.
- Refer to the CIHR Application Administration Guide for detailed regulations on CIHR funding
- Complete training on the Project review process through online learning modules and attendance to webinars.
- Support CIHR’s efforts to strengthen the integration of sex as a biological variable and gender as a social determinant of health in health research by: 1) completing the appropriate sex- and gender-based analysis (SGBA) training, as provided by CIHR; 2) becoming familiar with the SGBA tools available to support peer review activities, as provided by CIHR; and 3) providing feedback on the integration of sex and/or gender into research proposals, where appropriate.
2. Policies and Considerations Impacting Peer Review
- Global Health Research and International Collaborations
- Knowledge Translation
- Tri-Agency Open Access Policy on Publications
- Sex- and Gender-Based Analysis (SGBA) and Health Research
- Official Languages
2.1 Global Health Research and International Collaborations
As outlined in the on CIHR’s International and Global Health Collaborations webpage, applicants who are eligible for CIHR funding may apply for funds to support research to be carried out in, or in direct collaboration with researchers and/or knowledge users based in other countries. CIHR supports international research projects and collaborations to address a range of areas including but not limited to established priorities in global health research and contributes to the development of health-research capacity both internationally and nationally.
2.2 Knowledge Translation
Knowledge translation (KT) is integral to CIHR's mandate and falls into two main categories, end of grant KT and integrated KT. With both categories of knowledge translation, CIHR expects researchers to disseminate their findings and facilitate their translation into improved health, more effective products or services, and/or a strengthened healthcare system. The costs of dissemination are eligible expenditures in all CIHR grants.
For end of grant KT, many means of dissemination exist and the onus is on the researchers to select the most appropriate vehicle for the intended knowledge-user audience to ensure maximum impact. When the primary knowledge users are researchers, dissemination of results through the publication of articles in high quality and accessible journals is appropriate, although other strategies that increase awareness of the results and facilitate their application may also be appropriate. When knowledge-user audiences outside the research community should be informed of specific research findings, dissemination plans with more ambitious goals and comprehensive strategies are expected. With integrated KT, stakeholders or potential knowledge users are engaged in the entire research process and the research is directed at producing solutions to issues or problems the stakeholders/knowledge users have identified.
2.3 Open Access Policy on Publications
CIHR believes that greater access to research publications and data will promote the ability of researchers and knowledge users in Canada and abroad to use and build on the knowledge needed to address significant health challenges. Open access will promote accessibility to CIHR-funded research and will serve to increase the international visibility of Canadian research. CIHR grant recipients are reminded to adhere to the responsibilities outlined in the Tri-Agency Open Access Policy on Publications. Under this policy, grant recipients must ensure that research papers and bio-molecular data generated from CIHR funding are freely accessible online.
2.4 Sex- and Gender-Based Analysis (SGBA) and Health Research
CIHR expects that all applicants will integrate sex and gender into their research designs when appropriate. Sex- and gender-based analysis (SGBA) is an approach that systematically examines sex-based (biological) and gender-based (socio-cultural) differences between men, women, boys, girls and gender-diverse people. The purpose of SGBA is to promote rigorous science that is sensitive to sex and gender and therefore has the potential to expand our understanding of health determinants for all people. Reviewers are asked to explicitly assess whether the integration of sex (as a biological variable) and/or gender (as a socio-cultural factor) is a strength, a weakness or not applicable to the proposal. Reviewers are also asked to comment on their assessment and to provide recommendations to the applicants on how they might improve the strength of their applications with respect to the integration of sex and/or gender. The SGBA section of the CIHR website provides helpful resources for applicants and peer reviewers alike, including CIHR's definitions for sex, gender and SGBA, as well as information on applying SGBA to the development and assessment of research proposals.
2.5 Official Languages
In order to ensure that positive measures are undertaken within CIHR's mandate to contribute to the vitality of English and French minority communities in Canada, CIHR:
- encourages the appropriate inclusion of Official Language considerations of English and French speaking Canadians, including those living in linguistic minority communities, in health research design, conduct and application to improve health outcomes, and
- facilitates equitable access to its programs and services for health research stakeholders in official language minority communities.
3. Principles of Peer Review
The integrity of the peer review process relies on well-established principles and policies that:
- ensure fair and effective evaluation
- support CIHR’s mandate and objectives
CIHR’s principles of review are:
Confidentiality is information about a person that will not be disclosed directly or indirectly to anyone else without that person's prior expressed consent. The information provided by applicants in their applications is protected by the Privacy Act and is made available to external assessors for reviewing purposes only. The use of this information for any other purpose is a breach of the Privacy Act and could result in a CIHR investigation and/or report to the federal Privacy Commissioner's Office.
CIHR instructs committee members to not approach or communicate in any way with applicants, or anyone outside of the committee, any information relating to the review of a specific application, or offer opinions on the applicant’s chances of success or failure. In turn, applicants are not to contact committee members, including the Chairs and Scientific Officers, regarding the status of their applications (for example ratings).
By law, applicants have access to their own application files. Therefore, all written materials used in evaluating an application are made available to the applicants when they are notified of CIHR's funding decision for a competition.
The identity of the reviewers will not be revealed to the applicants. However, a list of review committee members is typically published on the CIHR website 60 days after the approval of funding of a competition.
3.2 Conflict of Interest
CIHR makes every effort to ensure that its decisions are fair and objective by identifying and addressing any conflicts of interest between an applicant and a peer review committee member. According to the Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations (COIC), a conflict of interest means a conflict between a participant's (e.g., committee members and observers) duties and responsibilities with regard to the review process, and a participant's private, professional, business or public interests. A conflict of interest may be deemed to exist or be perceived as such when committee members and observers:
- are a relative or close friend, or have a personal relationship with the applicants;
- are in a position to gain or lose financially/materially from the funding of the application;
- have had long-standing scientific or personal differences with the applicants;
- are currently affiliated with the applicants' institutions, organizations or companies*;
- are closely professionally affiliated with the applicants, as a result of having in the last six years:
- frequent and regular interactions with the applicants in the course of their duties at their department, institution, organization or company;
- been a supervisor or a trainee of the applicants;
- collaborated, published or shared funding with the applicants, or have plans to do so in the immediate future; or,
- been employed by the institution, when an institution is the applicant.
- feel for any reason unable to provide an impartial review of the application.
*A reviewer is not automatically in conflict with an application if they are from the same institution as the applicant, but do not know or work with the applicant.
All committee members (Chairs, Scientific Officers, reviewers) and observers are subject to the same conflict of interest guidelines. CIHR staff is responsible for resolving areas of uncertainty. All committee members must read and agree to abide by the COIC policy prior to viewing any application information.
Success of the peer review system is critically dependent upon the willingness and ability of all reviewers to:
- exercise rigorous judgement
- be impartial and reasonable
- understand and take into account in a balanced way the particular context of each application and,
- provide constructive, quality review which helps the applicant by pointing out strengths and weaknesses that contributed to the application rating.
CIHR ensures transparency in the peer review process through a number of different mechanisms. All applications submitted to CIHR are evaluated by reviewers who provide an overall assessment of the application. This assessment is performed using the application requirements and review criteria established for each individual competition, as seen in the funding opportunity details. In addition, the Scientific Officer (SO) notes capture the committee discussion during the peer review deliberations. All reviews and SO notes, as applicable, are shared with applicants.
4. Project Grant Application Review Process
4.1 Assigning Applications
The competition process begins with the applicants submitting a registration followed by a full application. Applicants are required, within the ResearchNet registration process, to indicate their suggestion of up to two most relevant peer review committees and to provide a justification for the committee(s) selected. Suggested committee(s) along with the corresponding justification(s) must remain unchanged between registration and application. Following submission of registrations, the Chairs and Scientific Officers, together with CIHR staff, review the registrations assigned to their committees. Together, they are responsible for ensuring that their committees have the appropriate expertise and, upon accepting applications for review, accept responsibility for ensuring that their committees conduct a fair review. In some cases, applications may not meet a particular committee's mandate and may need to be reassigned. CIHR will make the final decision on which peer review committee will review each application based on the summary of proposed research received at registration. The final committee selected may not necessarily be the applicant’s first or second choice. Applicants will be informed of the peer review committee that reviewed their applications when they receive the notice of decision.
Following the submission of full applications, committee members are given access to the summaries of proposed research to declare any conflicts of interest and indicate their level of expertise (i.e. High/Medium/Low/Not enough expertise).
Chairs, Scientific Officers and CIHR staff then assign each application to three committee members (a primary and two secondary reviewers) based on optimal alignment between the application content and the reviewer’s declared expertise. Each reviewer will receive a maximum of 8 to 10 applications to review. Reviews can also be solicited from external referees when additional, specific expertise is required. Four (4) to 5 weeks before the peer review committee meeting, all committee members, excluding external referees, are given access to the full applications assigned to their committee. External referees see only those applications assigned to them. It is possible that once access to the full applications is granted to reviewers, a reviewer may determine that he/she does not have the appropriate expertise to review one or more of the applications assigned to him/her. In such cases, reviewers are to contact CIHR immediately so that the application(s) can be re-assigned appropriately. Under certain circumstances, committee reviewers may participate via teleconference for the review of one or more applications.
4.2 Reviewing Applications
4.2.1 Conducting reviews
In preparation for the face-to-face committee meeting, reviewers will review their assigned applications by focusing on the project’s concept and feasibility, which should be founded on a sound and important idea supported by a feasible plan of execution. Concept accounts for 25%, while feasibility accounts for 75% of the overall assessment.
Research proposals submitted to the Project Grant competition can be up to 10 pages for English applications and 12 pages for French applications* (including figures, tables, charts and photographs), should stand alone (i.e., should contain all the information required to support the research proposal) and should contain a complete description of the project.
*Applications submitted in French are now allowed two additional pages of research proposal. This provision will ensure an equitable amount of space for applications written in either official language, as evidence demonstrates that documents written in French require approximately 20% more space than similar documents in English.
Although the applicants can present their research proposals in a free form format, the reviewers assess the application based on the specific adjudication criteria outlined in Section 4.2.2 below. In order to ensure that all applicants have exactly the same amount of space to write their research proposals, applicants must adhere to the following formatting requirements:
- Font: 12 point or larger. Do not use condensed/narrow font sizes or type density. Smaller text in tables, charts, figures, and graphs is acceptable, as long as it is legible when the page is viewed at 100%.
- Line spacing: A minimum of single line spacing
- Text colour: Black type
- Margins: Not be less than 2 cm (3/4 inch) on all sides
- Observe page limitations
- Use only letter size (21.25 X 27.5 cm / 8.5" X 11") white paper/background for all attachments.
- Photo-reduce the supporting documents if the originals are larger than 21.25 X 27.5 cm / 8.5" X 11".
- Attachments must be uploaded in PDF format (unprotected).
- The size of the attached document(s) cannot exceed 30 MB per document.
- For more information about converting documents to PDF, please refer to Help with Accessibility page.
Failure to comply with these requirements will negatively impact the evaluation/rating of the applications and can lead to their withdrawal from the Project Grant competition.
If applicants are resubmitting a previously unsuccessful application, they may provide a response (up to 2 pages) to previous reviewers’ comments. Applicants who upload a “response to previous reviews” must include all the reviews received in the last round of submission in the response (the reviews do not count toward the 2 page response limit). You do not have to respond to all the comments in the reviews, only those that are relevant to your revised application. The response and the previous reviews must be uploaded as a single PDF document.
Reviewers are not required to read the applicants’ responses to previous reviewers’ comments if the applicants chose not to include all the previous reviews being addressed.
If applicants have provided their previous ratings, reviewers are not to consider them in the evaluation/rating of the applications. Applications should be evaluated/rated based on their content and not on their previous ratings.
For applicants who have held a Foundation grant as an ECR, an additional half page is permitted in the Response to Previous Reviews attachment to contextualize their Foundation grant as deemed appropriate by the applicant.
If the response to previous reviews exceeds the 2-page limit (2.5 page limit for ECR Foundation Grantees), reviewers will not be required to read the additional pages.
Additional attachments including letters of support, questionnaires, surveys, and up to five publications from the last five years, may be uploaded under Other Application Material task in ResearchNet as outlined in the Application instructions. However, reviewers are under no obligation to read Other Application Materials.
Reviewers are also asked to review the applicant(s) CV(s). The Nominated Principal Applicant, all Principal Applicants and all Co-Applicants are required to submit a CIHR Biosketch CV, using the Canadian Common CV (CCV) interface. Through their CVs, applicants highlight their recognitions, funding history, activities and contributions that best demonstrate their leadership, significant contributions and productivity in the context of their research field(s).
For each application, reviewers will be asked to submit in ResearchNet:
- a summary of the research proposal (in the reviewer’s own words)
- a rating for each evaluation sub-criterion
- a justification of the ratings by stating the strengths and weaknesses of the project based on the evaluation criteria
- comments on the integration of sex and/or gender in the research proposal
- top (competitive) or bottom (non-competitive) group selection
Success of the peer review system is critically dependent upon the willingness and ability of all reviewers to exercise rigorous scientific judgement, to be fair and reasonable, and to understand and take into account in a balanced way the particular context of each application. Reviews must be written in the reviewer’s own words and must not be directly copied from elsewhere (including comments from other reviewers or their own previous reviews). The opinions expressed in the reviews are those of the reviewers alone and are provided to the applicants without prior editing by CIHR. In circumstances where the Chairs, the Scientific Officers or CIHR deem that a review expresses comments that could be construed as sarcastic, flippant, arrogant, or inappropriate in any way or contains factually incorrect information regarding CIHR’s policies, CIHR reserves the right to remove such information from the review. A constructive, quality review that helps the applicant by pointing out the strengths and weaknesses to be improved upon in a future submission will help demonstrate to an applicant that a fair assessment of the proposal was provided.
Completing the review in ResearchNet
The adjudication criteria, evaluation checkpoints, evaluation guidelines and other considerations regarding the sub-criteria are outlined in section 4.2.2 below.
To ensure consistency, reviewers must adhere to the following rating scale:
4.5 – 4.9
The application excels in most or all relevant aspects. Any short-comings are minimal.
4.0 – 4.4
The application excels in many relevant aspects, and reasonably addresses all others. Certain improvements are possible.
3.5 – 3.9
The application excels in some relevant aspects, and reasonably addresses all others. Some improvements are necessary.
3.0 – 3.4
The application broadly addresses relevant aspects. Major revisions are required.
0.0 – 2.9
The application fails to provide convincing information and/or has serious inherent flaws or gaps.
Summary of the Research Proposal
Reviewers provide a summary of the project to demonstrate their understanding of the research work that is being proposed. This summary must be written in the reviewer’s own words and must not be directly copied from the application, from previous reviews (including their own, if applicable), or from comments of other reviewers.
Ratings per Evaluation Sub-Criterion
In ResearchNet, reviewers provide their initial ratings for each adjudication sub-criterion to one decimal place in advance of the peer review committee meeting. ResearchNet will automatically display the overall calculated rating. Note that reviewers are not bound by the initial overall calculated rating and can change it during the peer review committee meeting.
Strength and Weaknesses of the Project
Reviewers also highlight the strengths and weaknesses of the project based on the evaluation criteria. Reviewers are encouraged to provide strengths and weaknesses for each evaluation criterion; strengths and weaknesses that contributed to the application ratings must be clearly articulated, as they will be used to:
- provide the other reviewers assigned to the application with a justification for the ratings given to the application
- provide applicants with feedback
Integration of Sex and/or Gender in the Research Proposal
Reviewers comment on whether the integration of sex (as a biological variable) and/or gender (as a socio-cultural determinant of health) is a strength, a weakness or not applicable to the proposal. Reviewers will also be asked to provide recommendations to the applicants on how they might improve the strength of their applications with respect to the integration of sex and/or gender. Resources to support this assessment can be found on the CIHR website.
Top (competitive)/Bottom (non-competitive) Selection
Reviewers also indicate if each of their assigned applications is competitive (top) or non-competitive (bottom) in its current form. The proportion of applications deemed competitive or non-competitive (top or bottom), may vary depending on the overall quality of the applications being reviewed. Initial scores and top/bottom are recorded by CIHR staff and communicated to the committee Chair and Scientific Officers. This information will be used for the streamlining phase of the review (section 4.3.1 below).
For security reasons, ResearchNet times-out rather frequently, and CIHR is unable to retrieve any lost information. Therefore, reviewers should save their work often so that it is not lost. Alternatively, reviewers can write their comments offline in a Word document and then copy and paste the text directly into the appropriate text box on ResearchNet when ready.
Reviewers will also be asked to flag applications in ResearchNet that involve any of the issues outlined below. These issues are not to be considered in the rating of the applications unless they impact the scientific quality of the application, as detailed below:
- Ethical and other policy requirements: Responsibility for ensuring that all research meets ethical standards is delegated by CIHR to the applicants’ institutions. Ethics forms are not required as part of the application. However, reviewers may flag specific issues, such as the use of human participants, animals, human tissues or hazardous material, or research that involve Indigenous Peoples, if they feel they have not been adequately addressed.
- Human pluripotent stem cell research: Applications recommended for funding that involve the use of human stem cells must also be reviewed by the Stem Cell Oversight Committee (SCOC). Applicants are instructed to check the relevant box in the section entitled “Certification Requirements”. It is essential that the use of these stem cells be also assessed by reviewers.
The deadline for uploading the reviews to ResearchNet, along with the assessment of overall quality (top/bottom groups) and the initial ratings, is five business days before the committee meeting date. Once reviewers have submitted their reviews, they are able to see the reviews of the other reviewers assigned to the same applications. Once reviews are submitted, reviewers will no longer be able to modify them prior to the committee meeting. Reviewers who wish to revise their reviews after the committee meeting will have 5 business days to directly upload their changes (ratings and comments) to ResearchNet. Committee members are also able to view the external referee reports as they become available before the committee meeting.
- Of note, it is the responsibility of all peer reviewers to familiarize themselves in advance of the committee meeting with the applications to be assessed by their committee, as this will facilitate discussions at the face-to-face meeting.
4.2.2 Adjudication Criteria and Interpretation Guidelines
In this section, each of the sub-criteria related to the concept and feasibility are described in more detail. A set of interpretation guidelines and considerations have been summarized for each sub-criterion. These are intended to provide guidance for the assessment of the application.
Of note, in the interpretation of the adjudication criteria, it is important to keep in mind that the research proposal may exert only a basic/mechanistic impact, which is as important as the translational impact. The impact does not only mean near-future clinical relevance. Reviewers should evaluate whether the work proposed will significantly advance the proposed area of research.
Criterion 1. Concept (25%)
- Significance and Impact of the Research (25%)
Criterion 2. Feasibility (75%)
- Approaches and Methods (50%)
- Expertise, Experience and Resources (25%)
Reviewers should take into consideration the career stage, research field and institution setting of the applicants when assessing each criterion. The evidence should be notable compared to peers in similar fields and career stages.
- Specific considerations should be taken into account in the review of applications submitted to the Indigenous Health Research (IHR) committee (see IHR Committee Considerations under each Adjudication Sub-criterion below).
- Evaluation of all applications containing a Randomized Controlled Trial (RCT) as a major component will need to consider the specific RCT evaluation criteria in their assessment, irrespective of the assigned peer review committee (section 4.2.5 below).
- Evaluation of applications reviewed in the Commercialization committee will need to include the assessment of both the Research and Technical plan and the Commercialization plan according to specific review criteria (section 4.2.6 below).
4.2.2.a Criterion 1: Concept (25%)
Significance and Impact of the Research
This sub-criterion is intended to assess the quality of what is being proposed. Each Evaluation Checkpoint is presented as a question and is followed by the Evaluation Guideline(s) for that Checkpoint.
- Is the project idea creative?
- The project idea is among the best formulated ideas in its field, stemming from new, incremental, innovative, or high-risk lines of inquiry; new or adapted research in basic science, or health care, or health systems or health outcomes. When applicable, knowledge translation/commercialization approaches/methodologies should be considered, as well as opportunities to apply research findings nationally and internationally.
- Is the rationale of the project idea sound?
- The project rationale is based on a logical integration of concepts.
- Are the overall goals and objectives of the project well-defined?
- The overall goal and objectives of the project are well-defined and clear.
- The goal states the purpose of the project, and what the project is ultimately expected to achieve.
- The objectives clearly define the proposed lines of inquiry and/or activities required to meet the goal.
- The proposed project outputs (i.e., the anticipated results of the project) are clearly described and aligned to the objectives.
- Are the anticipated project contributions likely to advance basic health-related knowledge, or health care, or health systems or health outcomes?
- The context and needs (issues and/or gaps) of the project are clearly described.
- The anticipated contribution(s) (e.g. publishing in peer-reviewed journals) are clearly described, and should be substantive and relevant in relation to the context of the issues or gaps.
- The anticipated contribution(s) are realistic, i.e., directly stemming from the project outputs, as opposed to marginally related.
This sub-criterion is not intended to assess feasibility of the project, expertise of the team or the potential of success. These will be assessed under Criterion #2: Feasibility
Research should focus on addressing an issue (e.g., hypothesis or question, problem, need or gap) in any area across the spectrum of health (basic biomedical, health-related knowledge, health care, health systems, and/or health outcomes).
Depending on the nature of the project, it may have a research and/or knowledge translation / commercialization focus. Also, depending on the nature of the project, the rationale may be well-supported by evidence (e.g., literature review, previous findings, environmental scan, market analysis, stakeholder or partner input). However, this level of justification is not required for all types of projects (e.g., high-risk lines of inquiry).
In cases where projects have a primary implementation, or knowledge translation / commercialization (application and uptake of research findings) focus, the importance of the research should be validated as being substantive and relevant by stakeholders and partners, i.e., by those who could directly benefit from, or make use of, the project outputs.
Indigenous Health Research (IHR) committee Considerations
The proposed research must be relevant to First Nations, Inuit and/or Métis priorities and have the potential to produce valued outcomes from the perspective of First Nations, Inuit and/or Métis participants and Indigenous Peoples more broadly.
Global Health Research
Projects that have a global health research focus, or include international collaborations, are eligible for support through the Project Grant program. CIHR welcomes all research, from fundamental to applied, with the potential to advance health-related knowledge, and/or improve health outcomes for Canadians and the broader global community.
4.2.2.b Criterion 2: Feasibility (75%)
4.2.2.b-1 Approaches and Methods (50%)
This sub-criterion is intended to assess the quality of the project's design and plan; including how and when the project will be completed. Each Evaluation Checkpoint is presented as a question and is followed by the Evaluation Guideline(s) for that Checkpoint.
- Are the approaches and methods appropriate to deliver the proposed output(s) and achieve the proposed contribution(s) to advancing health-related knowledge, health care, health systems, and/or health outcomes?
- The research and/or knowledge translation/commercialization approaches, methods and/or strategies are well-defined and justified in terms of being appropriate to accomplish the objectives of the project.
- Opportunities to maximize project contributions to advance health-related knowledge, health care, health systems and/or health outcomes should be proactively sought and planned for, but may also arise unexpectedly.
- Are the timelines and related deliverables of the project realistic?
- Timelines for the project should be appropriate in relation to the proposed project activities. Key milestones and deliverables should be aligned with the objectives of the project, and be feasible given the duration of the project.
- Does the proposal identify potential challenges and appropriate mitigation strategies?
- Critical scientific, technical, or organizational challenges should be identified, and a realistic plan to tackle these potential risks should be described. An exhaustive list is not expected.
Sex and Gender Considerations (if applicable)
Evidence demonstrates that biological and social differences between women and men contribute to differences in health risks, health services use, health system interaction and health outcomes. Accounting for sex and gender in health research has the potential to make health research more rigorous, more reproducible, and more widely applicable. CIHR expects that all research applicants will integrate sex and gender into their research designs when appropriate, as indicated on the Sex, Gender and Health Research webpage. Resources to assist reviewers in their assessment of the integration of sex and gender in the research design are available on CIHR’s website.
Indigenous Health Research (IHR) committee Considerations
In addition to demonstrating scientific excellence (Western, Indigenous, or both), the proposed research approaches and methods must respect Indigenous values and ways of knowing and sharing, and abide by the Tri-Council Policy Statement Chapter 9: Research Involving the First Nations, Inuit and Métis Peoples of Canada, and/or Indigenous partnering community / organizational ethical guidelines, or clearly explain why other guidelines have been developed and agreed upon with the study governance body.
Applications submitted to the Project Grant competition may include an integrated knowledge translation approach or may have a knowledge translation focus, with at least one knowledge-user and one researcher. CIHR defines a knowledge user as an individual who is likely to be able to use the knowledge generated through research to make informed decisions about health policies, programs and/or practices. A knowledge user can be, but is not limited to, a practitioner, policy-maker, educator, decision-maker, health care administrator, community leader, or an individual in a health charity, patient group, private sector organization or media outlet.
CIHR defines integrated knowledge translation as a way of doing research with researchers and knowledge users working together to shape the research process – starting with collaborations on setting the research questions, deciding the methodology, being involved in data collection and tools development, interpreting the findings and helping disseminate the research results.
Designs, approaches, and methodologies will vary by project (e.g., field of research, target audience) and should include a knowledge translation approach, when applicable, that is appropriate to the nature of the project outputs.
4.2.2.b-2 Expertise, Experience and Resources (25%)
This sub-criterion is intended to assess the appropriateness of the complement of expertise, experience and resources among the applicants (Nominated Principal Applicant, Principal Applicant(s) and Co-Applicant(s)), and their institutions/organizations, as it relates to the ability to collectively deliver on the objectives of the project.
It is the responsibility of the Nominated Principal Applicant to ensure the proposed project is poised for success.
An estimate of the number of hours per week (contribution) for each applicant working on the project should be provided.
Each Evaluation Checkpoint is presented as a question and is followed by the Evaluation Guideline(s) for that Checkpoint.
- Does the applicant(s) bring the appropriate expertise and experience to lead and deliver the proposed output(s), and to achieve the proposed contribution(s)?
- The applicant(s) should demonstrate the combined expertise and experience needed to execute the project (i.e., deliver the proposed outputs as well as achieve the proposed contribution(s)).
- The roles and responsibilities of each applicant should be clearly described, and linked to the objectives of the project.
- Is there an appropriate level of engagement and/or commitment from the applicant(s)?
- The level of engagement (e.g., time and other commitments) of each applicant should be appropriate to the roles and responsibilities described.
- Is the environment (academic institution and/or other organization) appropriate to enable the conduct and success of the project?
- Project applicants should have access to the appropriate infrastructure, facilities, support personnel, equipment, and/or supplies to:
- Carry out their respective roles, and;
- As a collective, manage and deliver the proposed output(s), and achieve the proposed contribution(s).
- Project applicants should have access to the appropriate infrastructure, facilities, support personnel, equipment, and/or supplies to:
Indigenous Health Research (IHR) Committee Considerations
Appropriateness of the team based on their overall scientific experience (Western, Indigenous, or both) and skills as well as their Indigenous community-based research experience, track record, relevance of past experience, including expertise related to Indigenous Health Research.
The required complement of expertise will vary by project. Applications with an integrated knowledge translation approach or knowledge translation focus must include knowledge users in defining/refining research questions, informing the research plan, conducting research, interpreting research findings, understanding the receptor community, leading dissemination activities, etc. Knowledge users may also be responsible, and accountable for the application/uptake of the project outputs. The nature, breadth and depth of the applicants’ experiences and contributions should be assessed in the context of the applicants’ career stages.
Applicants that have taken leaves of absence in the past seven years (e.g., parental, bereavement, medical, or administrative leave) may include a PDF document (no page limits) to supplement the publication information for that equivalent period of time. Whatever length of time an applicant has taken off from research in the past seven years is the amount of time that they may include in the attachment. Note that leaves of absence should also have been included in the appropriate section of the CV. Reviewers should therefore review this document in order to ensure that they have an accurate profile of applicants’ research activities and achievements.
Project environments should be assessed according to their ability to support the proposed project activities. Institutions often function as "networked" environments or interdisciplinary networks, which means there may be multiple satellite environments contributing to the support environment. Reviewers should consider that for smaller institutions, or affiliated research facilities where resources and/or services may be obtained through networks, or may be contracted out.
4.2.3 Budget Assessment
The budget assessment must not be factored into the scientific assessment and must not influence the rating of applications.
Applicants are asked to submit their budget using a modular template with pre-set incremental levels, and to provide a comprehensive description of what the funds will be used for, in order to justify the amount requested.
Reviewers are required to check the requested budget and justification although detailed scrutiny of each item is not expected. Reviewers are required to determine if the requested budget is appropriate to support the proposed project and if it is realistic and well-justified. Appropriate budget planning should be demonstrated. The requested resources, together with any existing resources, should be adequate to financially support the full scope of the project.
Partner contributions, if applicable (required for partnered applications), should be considered when reviewing support for the proposed project. Partner contributions will be outlined in letters of contribution and in the budget module.
Reviewers may recommend that the budget remains as requested or recommend a reduction. If a reviewer suggests an adjustment to the budget, he/she will be required to provide comments in the “Budget” section of their reviews, to justify their recommendation.
The budget recommendation is intended to inform CIHR funding levels. Grant value and duration will be commensurate with the requirements of the project proposed, and will vary depending on the field, proposed approach, and scope of activities.
While budget recommendations are not factored into the overall rating of an application, CIHR values the experience and perspectives of reviewers in estimating overall budget requirements of the proposed project.
Each Evaluation Checkpoint is presented as a question and is followed by the Evaluation Guideline(s) for that Checkpoint.
- Is the requested funding appropriate to support the project? Is it realistic and well-justified?
- Appropriate budget planning should be demonstrated. The requested resources should be adequate to financially support the full scope of the project.
- Reviewers are asked to consider the proposed budget of the project, and to provide a justified recommendation as to whether the budget should be:
- Accepted, as described; or
- Reduced by a specific dollar amount (total).
Peer review committees are asked to recommend budgets for grant applications irrespective of other sources of funds received or applied for, i.e., committees should not reduce recommended budgets to take into account potential overlap.
4.2.4. Indigenous Health Research (IHR)
Applications with a central focus on carrying out meaningful and culturally safe research involving Indigenous Peoples, with the intent to promote health through research that is in keeping with Indigenous values and traditions and follows the TCPS 2 - Chapter 9 Research Involving the First Nations, Inuit and Métis Peoples of Canada guidelines may be reviewed by the Indigenous Health Research (IHR) committee. A significant portion of the IHR committee will be composed of reviewers with living-experience, knowledge, and/or expertise in Indigenous Health Research. Reviewers will provide their written reviews with a mentorship lens.
At the time of application submission, the research proposal must also explicitly describe engagement with the community in relation to the research. The IHR committee will take specific considerations into account when evaluating applications (see IHR Committee Considerations under each Adjudication Sub-criterion above). Applications that do not fit with the IHR committee mandate will be reviewed by another discipline-based committee.
All applications will be discussed at the face-to-face committee meeting to allow for the development of an explicit mentoring approach to enable all promising Indigenous health research to continue to evolve.
The IHR committee may assess an application as being eligible for the Iterative Peer Review Process. The objective of the Iterative Peer Review Process is to allow applicants whose applications have been deemed excellent, the opportunity to provide minor clarifications that would see the application improve to become outstanding. Additional information can be found in the TCPS 2 - Chapter 9 Research Involving the First Nations, Inuit and Métis Peoples of Canada guidelines.
4.2.5 Randomized Controlled Trials (RCTs)
A randomized controlled trial (RCT) is an experiment in which investigators randomly assign eligible human subjects or other units of study (e.g., classrooms, clinics, playgrounds) into groups to receive or not receive one or more interventions that are being compared. The results are analyzed by comparing outcomes in the groups. Applications involving a RCT are accepted and encouraged where appropriate based on scientific and methodological grounds.
Irrespective of the suggested peer review committee, evaluation of all applications containing a RCT as a major component will need to consider the specific RCT evaluation criteria. Please consult the RCT Evaluation Criteria and Headings website for detailed information on the specific criteria that need to be taken into account by the committees when evaluating RCT applications.
For applications including a RCT:
- if the budget requested is less than $250,000 per year, applicants may apply to the most relevant discipline-based peer review committee OR to the RCT peer review committee;
- if the overall budget requested is greater than or equal to $250,000 per year, applicants must apply to the RCT peer review committee.
- irrespective of the suggested peer review committee and regardless of the level of funding requested, all applications containing an RCT as a major component must use the RCT headings in the research proposal. Detailed information on the RCT headings is found in the Evaluation Criteria and Headings website
4.2.6 Commercialization Projects
Commercialization projects are designed to advance discoveries/inventions towards commercializable technologies/products, with a view to attract new investment, create new science-based businesses, organizations and initiatives, and ultimately improve health outcomes for Canadians.
Please consult the Commercialization Projects website for detailed information on the specific criteria that need to be taken into account by the Commercialization committee when evaluating applications.
4.3 The face-to-face Peer Review Committee Meeting
The prime responsibility of a peer review committee is the evaluation of applications submitted to a competition by using CIHR's rating scale. The committee will also recommend the grant term and a budget sufficient to support the full proposed research. It is important that committees follow defined procedures in order to function in a consistent manner.
Of note, any committee member who has a conflict of interest with an application (as defined in the section 3.2 above) must not take part in the evaluation process of that application and must leave the meeting room before the application is discussed during the committee meeting. The Chair and CIHR staff are responsible for monitoring conflicts and for resolving areas of uncertainty.
4.3.1 Streamlining of Applications
Only a portion of the Project Grant applications submitted to a competition will ultimately be funded. Therefore, it is important that during the face-to-face meeting, committee members focus their discussions on the most competitive applications to ensure and achieve an accurate comparative evaluation. To help support this goal, a streamlining process is used to remove applications that are not competitive from the discussion process, thereby allowing committee members more time to judge and discriminate between potentially successful applications and helping to ensure that the most deserving applications receive funding. Applicants whose proposals are streamlined still benefit from the review process, as they receive the written reviews from the assigned reviewers (but no detailed Scientific Officer notes).
Applications that are streamlined will not be eligible to receive funding from the CIHR Priority Announcements (PAs). For more information on Priority Announcements please refer to the Project Grant: Fall 2020 Priority Announcement (Specific Research Areas) funding opportunity and the Frequently Asked Questions.
Prior to the committee meeting
Reviewers provide their initial ratings on ResearchNet in advance of the committee meeting. They also indicate if each of their assigned applications is competitive (top) or non-competitive (bottom) in its current form. The applications in the competitive (top) group should include those applications considered to be the most deserving of being funded. The proportion of applications deemed competitive or non-competitive (top or bottom), may vary depending on the overall quality of the applications reviewed. Averages of the initial scores and top/bottom are recorded by CIHR staff and provided to the committee Chair and Scientific Officers. This information is used to determine which applications are eligible for streamlining.
An application is eligible for streamlining if it meets the following conditions:
- The average of the reviewers’ scores places the application in the bottom 50 % of all applications in the committee;
- At least one reviewer identified the application as non-competitive (bottom).
- There is no objection from any committee member to streamline the application.
At the committee meeting
All the committee members will be informed of the applications identified prior to the committee meeting as eligible for streamlining. If there are no objections from any committee members, these applications will be streamlined. If there is an objection to an application being streamlined, it will be discussed by the committee. Note that some of the streamlined applications may have an initial average rating above 3.5.
During the course of the committee meeting, additional applications may also be identified as eligible for streamlining in accordance with the conditions described above.
If an application is not discussed, the applicant will receive all the reviewers' reports, but the Scientific Officer notes will only carry notification of the decision to streamline. Committee members do not vote on the rating of a streamlined application; the final rating is calculated as the average of the ratings from the assigned reviewers.
4.3.2 Assessment of Applications
The assessment of applications begins with the assigned reviewers announcing their initial ratings to one decimal place. The reviewers may revise their initial ratings in light of the committee discussion.
If an application is not streamlined, the committee discussion proceeds as follows:
The primary reviewer gives a brief synopsis of the proposal and presents his/her assessment, describing strengths and weaknesses of the proposal (see section 4.2.2 above for the adjudication criteria);
The two secondary reviewers follow, focusing on points of agreement or disagreement with the other reviewers, and elaborating on points not already addressed;
If external reviews have been provided by external referees, the reviewers read the reviews;
The Chair leads the discussion of the proposal by the committee members;
The Scientific Officer reads back the Scientific Officer notes to the committee, capturing the key elements of the committee discussion to be considered by the committee when rating the application;
The Chair seeks a consensus rating from the three reviewers. If a consensus cannot be reached, the Chair determines the consensus rating, usually by using the mean value of the reviewers’ ratings after committee discussion (round up, if necessary, to obtain a single decimal point).
All the committee members not in conflict (excluding the Chair and Scientific Officers) are asked to vote (+/- 0.5 of the consensus rating) for the final rating of each application.
4.3.4 Budget and Term
The budget requested will not be factored into the scientific assessment and rating of the applications. However, CIHR will seek the recommendation from the reviewers on the budgets and terms requested. For additional information, please see section 4.2.3 above.
CIHR reserves the right to determine the final amount awarded to the grants.
4.3.5 Flagging of Applications
Issues related to ethics, eligibility, human pluripotent stem cell research and other concerns need to be flagged for CIHR's attention. The rating should not be influenced by such factors. For additional information, please see section 4.2.
4.3.6 End of Meeting Review
Once all applications have been reviewed, if the peer review committee feels that any application(s) has been treated inconsistently, the re-review of one or a small number of applications is permitted. Any committee member with a conflict of interest must again leave the meeting room. Following discussion, a consensus rating is determined by the three reviewers and voting proceeds as before.
An important component of any peer review committee meeting is the assessment of the committee's effectiveness and functioning, and a discussion of policy issues that may have arisen in the course of its deliberations. This committee debrief provides an opportunity for CIHR staff to address any concerns of the committee members and to record feedback on the peer review process as part of CIHR's ongoing efforts to maintain an effective and high quality peer review system.
4.4 After the Committee Meeting
Upon completion of the committee meeting, for all applications discussed, CIHR will collect the final ratings and recommendations on funding level and grant term from all the committee members. This information will be used for funding decision-making.
All applicants will receive a letter of decision and a Notice of Decision, indicating whether or not their proposals were approved, and if approved, with information on the allocated funds. Applicants also receive a copy of all reviews and the Scientific Officer notes (for the applications discussed by the committees).
Applications that have been flagged for special attention and followed up by CIHR staff are withheld as "pending". The applicants are notified if further information is required.
5. Peer Review Committees: Selection and Roles of Members
5.1 Selection of Peer Review Committee Members
Standing peer review committees have been established for the Project Grant competition. Committee core membership will be recruited for a term of service (typically 3 years or 6 competitions). To maintain stability in membership, while providing a mechanism for membership renewal, a rotational system will be established for one third of the membership on a yearly basis. The membership may also be supplemented by additional members as required for a specific competition, based on the applications received and expertise needed for their review. These terms will also address the benefits of renewing the membership so that new perspectives are continually incorporated into the peer review process.
Invitations to members of the health research community to join specific Project Grant Peer Review Committees (PRCs) are based on their area(s) of expertise, experience in grant writing, peer review process, and in consultation with the committee Chairs and Scientific Officers. The Chairs also have a role in the selection of Scientific Officers.
5.2 Committee Memberships
Project Grant Peer Review Committees consist of a Chair, two Scientific Officers and reviewers. The committee membership as a whole considers one or more of the following aspects:
- the need to cover the full range of relevant research areas, methodologies and experience according to the committee’s defined mandate)
- the necessity for reviewing capability in English and French so that applications in either official language can be evaluated by the committee
- the need for national representation and representation by gender proportionate to the Canadian health research community
5.3 Committee Members Roles and Responsibilities
5.3.1 Committee Chair
The committee Chair is expected to follow a clear set of general instructions as defined by CIHR.
Before the committee meeting, the Chair will:
- suggests names of potential committee members (Scientific Officers and reviewers) to CIHR during both the formation of the review committee and the assignment of applications, as required. This could also include identifying potential external referees to fill expertise gaps.
- works with the Scientific Officers and CIHR staff to assign applications to specific peer review committees and to select reviewers for each application.
- works with CIHR staff to establish a review sequence or a meeting agenda, and to identify the eligible applications for streamlining.
- works with CIHR staff to manage conflicts of interest of committee members.
During the committee meeting and in the role of moderator, it is the Chair’s responsibility to ensure that the review committee functions smoothly, effectively and objectively, and that a positive, constructive, fair-minded environment in which research proposals are evaluated is established and maintained. The Chair will:
- provide opening remarks to the committee
- explain the meeting process to the committee
- ensure that all committee members who are in conflict with an application leave the meeting room before the discussion of the application.
- appoint a delegate as Chair or Scientific Officer when either individual leaves the meeting room due to a conflict of interest with an application or for any other reasons.
- fulfill an oversight role – does not rate applications nor vote during the committee meeting.
- ensure the involvement of the entire committee in evaluating each application.
- work with the Scientific Officers, as required, to summarize the discussion around each application, before the consensus rating is reached.
- ensure that a consensus rating is reached by the assigned reviewers.
- ensure that both the budget and term of the project are discussed by the committee and that comments are added to the Scientific Officer notes, as appropriate.
- ensure that specific ethical concerns and other CIHR requirements are addressed, and that any related discussion is captured in the Scientific Officer notes.
At the end of the committee meeting, the Chair may:
- propose that any application that has been identified by committee members as being inconsistently reviewed during the meeting is discussed again.
- allow time to receive feedback from the committee members regarding the effectiveness and functioning of the committee.
5.3.2 Scientific Officers
Scientific Officers (SOs) are responsible for supporting the Chair in his/her role during the committee meeting and are expected to follow a clear set of general instructions as defined by CIHR.
Before the committee meeting, the Scientific Officers will:
- suggest names of potential committee reviewers to CIHR during the formation of the review committee, as well as during the assignment of applications, as required. This could include identifying potential external reviewers to fill specific expertise gaps.
- work with the Chair and CIHR staff to assign applications to specific peer review committees and to select reviewers for each application.
- work with the Chair and CIHR staff to establish a review sequence or a meeting agenda, and to identify the eligible applications for streamlining.
- work with the Chair and CIHR staff to manage conflicts of interest of committee members.
During the committee meeting, the Scientific Officers will:
- take official notes of the committee discussions for each application (SO Notes). These will be sent to the applicants along with the reviewers’ reports.
- the SO Notes should provide the applicants with insight into the committee discussion of their applications.
- they should be clear and concise, and give objective and constructive feedback to the applicants. They should:
- include the strengths and weaknesses of the applications discussed by the committee.
- address the issues that had the greatest impact on the evaluation, as they relate to the program’s evaluation criteria.
- address aspects of the committee discussions that were not captured in the reviewers’ reports.
- describe how reviewer disagreements, as seen in the individual reviewer reports, were reconciled by identifying which view was favored by the committee.
- read back the SO Notes to the committee for validation and for additional input before a consensus rating on each application is reached by the assigned reviewers, and all the members’ votes are cast.
- read back the summary of recommendations to the budget and term of the project, and ensure that special considerations related to ethics and/or other issues are also recorded in the SO Notes, if applicable.
- Scientific Officers do not rate applications or vote during the committee meeting.
- CIHR will not edit or modify the SO Notes without the Scientific Officers’ approval
At the end of the committee meeting, the Scientific Officers will:
- finalize and post the SO Notes directly on ResearchNet or send them to the CIHR staff for posting no later than 5 business days after the committee meeting.
Reviewers are committee members assigned to review one or more applications (maximum 8-10 applications) for which they have declared expertise and are not in conflict with. Applications are assigned to three reviewers who participate in their rating and submit in-depth written reports that are provided to the applicants. They present their reviews at the committee meeting where they lead the discussion, and vote during the meeting. They also participate in the discussion and rating of all the other applications before the committee for which they are not in conflict.
Before the committee meeting, reviewers must:
- evaluate each of the applications assigned to them by providing a critical assessment of the applications, as well as constructive feedback based on the program’s objectives and adjudication criteria described in the funding opportunity by :
- considering all factors and the strengths and weaknesses of the applications in relation to each adjudication criterion.
- focusing their comments on the factors most relevant to their ratings.
- providing feedback that is original and written in their own words;
- providing comments on the integration of sex (as a biological variable) and/or gender (as a socio-cultural determinant of health) in the applications, if applicable.
- providing comments on the budgets requested and a formal recommendation to CIHR in the “Budget” section, including clear and detailed reasons for any recommended budget or term cuts, if applicable.
- providing comments on issues that they feel should be flagged, as required. These concerns should not influence the rating or budget recommendations, unless they bear on the scientific merit of the applications.
- provide their initial ratings to one decimal place on ResearchNet.
- divide their applications into a top (competitive) and bottom (non-competitive) group, based on their overall quality; and
- familiarize themselves with the applications to be assessed by their committee as this will facilitate discussions at the face-to-face committee meeting.
During the committee meeting, reviewers will:
- present to the committee the review of their assigned applications (see section 4.2.2 above, for the adjudication criteria).
- participate in the committee discussions.
- vote on all the applications discussed by the committee and for which they are not in conflict with
At the end of the committee meeting, reviewers will:
- have the option to revise their reviews within 5 business days after the end of the committee meeting.
- Date modified: