Peer Review Guide
Pilot of the Tri-Agency Interdisciplinary Peer Review Committee
- 1 Introduction
- 2 Objective
- 3 Review Process
- 4 Roles and Responsibilities
- 5 Description of Evaluation Criteria
- 6 Resources
The aim of this document is to serve as a resource for reviewers adjudicating applications submitted to the Tri-Agency Interdisciplinary Peer Review Committee. This shared peer review committee is being piloted through the SSHRC 2021-2022 Insight Grants, CIHR Fall 2021 Project Grant, and NSERC 2022 Discovery Horizons Grants funding opportunities.
On behalf of SSHRC, CIHR, and NSERC, we would like to thank the peer review committee members. These dedicated experts who generously give their time and expertise are essential to ensuring the success of the peer review process.
If you need further guidance or would like to have more information about this review process, please do not hesitate to contact:
- SSHRC: firstname.lastname@example.org
- CIHR: ProjectGrant-TIR-SubventionsProjet@cihr-irsc.gc.ca
- NSERC: email@example.com
Individuals applying for SSHRC Insight Grants, CIHR Fall Project Grant, and NSERC Discovery Horizons grants will be eligible to apply to the Tri-Agency Interdisciplinary Peer Review Committee. This committee provides an option for researchers working in interdisciplinary research to direct their application to a committee with expertise from across the social sciences, humanities, natural sciences, engineering, and health sciences.
The agencies recognize that the terms "multidisciplinary," "transdisciplinary," and "interdisciplinary" have different connotations and that there may be a lack of consistency in their usage across and between research disciplines. For the purpose of this committee, "interdisciplinary" is used as an umbrella term to refer to all three concepts inclusively.
3. Review Process
3.1 Important Dates
|CIHR Fall Project Grant Application deadline||Sept 15, 2021|
|SSHRC Insight Grant Application deadline||October 1, 2021|
|NSERC Discovery Horizons Grant Application deadline||October 18, 2021|
|Conflict of interest and ability to review deadline||Fall 2021|
|Preliminary meeting(s)||Late Fall 2021|
|Assignments ready, reviewers receive access to applications for review||Based on Agency Timelines|
|Reviewers submit reviewer report and initial scores||One Week before Meeting|
|Peer review meetings||Winter 2021/2022|
|Anticipated Notification of decision to applicants||Spring 2022, based on Agency Timeline.|
3.2 Assigning Applications
Following the submission of full applications, committee members are given access to the summaries of proposed research to declare any conflicts of interest and indicate their level of expertise (i.e. High/Medium/Low/Not enough expertise). The workload of the reviewers will be balanced across the applications from the three agencies.
Each application will be read by at least 5 committee members: 3 as Reviewers and 2 as Readers (see section 4, below, for more details on the roles):
- The application will be assigned by agency staff to three committee members as Reviewers with different but relevant perspectives based on the subject matter of the application.
- The application will be assigned to two additional committee members as Readers (who will read but not provide preliminary ratings on the application, although they will vote on the consensus scores of the applications to which they are assigned).
- The agencies will also seek to obtain at least two External Assessment Reports for every application from subject matter experts of the content of the application.
3.2.1 Committee Orientation and Calibration
Agency staff will convene preliminary meetings of the Tri-Agency Interdisciplinary Peer Review Committee in advance of the peer review meeting for the purposes of committee orientation and committee calibration.
As this committee's processes and evaluation criteria have been harmonized and will include members from across the social sciences, humanities, natural sciences, engineering, and health sciences, committee orientation provides an opportunity to explain processes and procedures that will apply to this committee, as well as address any questions from reviewers stemming from their previous peer review experiences. Agency staff will also introduce the peer review platform and other related information technologies being used to support the peer review meeting.
The committee calibration offers an opportunity for discussion of the consistent use of evaluation criteria and the scoring system, time-management strategies for the adjudication meeting, and other topics.
3.3 Review of Applications before the peer review meeting
All applications must be treated as protected information and all committee members will sign a tri-agency Conflict of Interest and Confidentiality Agreement. Committee members must not divulge application contents to others or use any information herein for any other purpose than peer review. Ensure that all Peer Review materials are handled safely and disposed of according to the document "Guide on Handling Documents used in Peer Review".
3.4 Review Process
Reviewers will be asked to submit a written review whereby they will provide strengths and weaknesses of the project in relation to the Tri-Agency Interdisciplinary Peer Review Committee evaluation criteria for the competition. Reviewers and Readers should consider, but are not bound by, the External Assessment Reports.
Note: A template will be provided to the reviewers to facilitate their written report (which can then be copied and pasted into the online system).
After the reviewers have submitted their final reviews, they will be able to access the other finalized reviewers' reports for that application. These reports can be used in preparation for the discussion at the committee meeting.
If applicable, reviewers may also comment on the budget requested and make a formal recommendation, including clear and detailed reasons for any recommended budget cuts.
3.4.1 Evaluation Criteria and rating scale
To support the strategic objectives of this funding opportunity, the following evaluation criteria will be used. Reviewers will enter scores for each of the three criteria, which will be given equal weight.
|Merit of the Proposed Research||Quality of the Proposal||The clarity, scope and originality of research questions and objectives|
|Methods and Approaches||
Appropriateness of the proposed methods and approaches
Appropriate incorporation of sex, gender and diversity in the research design, if applicable
Evidence of commitment to the development of active and meaningful partnerships with appropriate individuals and organizations, if applicable
Extent to which the budget is appropriate and justified in relation to the proposed activities
Appropriateness of the environment (academic institution and/or other organization) to enable the conduct and success of the proposed activities
|Anticipated Outcomes||Impact||Potential for the outcomes of the proposed activities to have influence and impact|
|Knowledge mobilization and dissemination||Quality and appropriateness of knowledge mobilization plans, including effective dissemination, exchange and engagement with stakeholders within and/or beyond the research community, where applicable|
Extent to which the proposed activities incorporates plans for the training and development of highly qualified personnel
Appropriate inclusion of considerations of equity, diversity and inclusion in the training plan, where applicable
|Applicant(s) Record||Expertise and Experience||
Appropriate expertise to undertake the proposed research, including complementary and interdisciplinary knowledge, expertise and experience
Experience and track record of the investigator(s) – importance, originality, quality and impact of past research
|Past Contribution to Training of HQP||Quality and extent of past contributions to the training of highly qualified personnel|
See section 5 below for a more detailed discussion of the evaluation criteria.
The following rating scale will be used:
|Outstanding||4.5 – 4.9||Fundable range||The application excels in most or all relevant aspects. Any shortcomings are minimal.|
|Excellent||4.0 – 4.4||The application excels in many relevant aspects, and reasonably addresses all others. Certain improvements are possible.|
|Very good||3.5 – 3.9||The application excels in some relevant aspects, and reasonably addresses all others. Some improvements are necessary.|
|Fair||3.0 – 3.4||Non-fundable range||The application broadly addresses relevant aspects. Major revisions are required.|
|Poor||0.0 – 2.9||The application fails to provide convincing information and/or has serious inherent flaws or gaps.|
Applications are submitted according to the guidelines, formatting, page limits, required attachments etc. of the funding opportunity to which they apply. Applicants should not be penalized as long as they are following the rules appropriate to their individual agencies. To facilitate this, applications will be reviewed in bundles by agency during the peer review meeting.
3.4.2 Peer Review Meeting
The Tri-Agency Interdisciplinary Peer Review Committee will meet in one or more session to review the eligible and relevant applications submitted via the three agency funding opportunities. As noted above, the applications will be reviewed in bundles by agency and the review may occur over a number of sessions (i.e. all CIHR applications will be reviewed together as a bundle during one session of the committee over one or more days; then the SSHRC applications will be reviewed in another session; then the NSERC applications will be reviewed in a final session). Please note that the agencies may streamline applications based on existing practices, and as a result the committee may not discuss all files.
During the peer review meetings, and before the application is discussed, the assigned reviewers will be asked to announce their initial ratings per each criterion. One reviewer will be assigned as 'primary' reviewer and will open the discussion by presenting their assessment of the strengths and weaknesses of the proposal according to their perspective and specific to each of the evaluation criteria for the funding opportunity. The remaining reviewers follow and supplement the discussion by concentrating on points of agreement or disagreement, and elaborating points not addressed by the primary reviewer, with attention given to their specific perspective. The open discussion will be structured and discussed in the order of each evaluation criteria.
Once the discussion is complete, the Chair will invite reviewers to vote on the application. Following the individual confidential voting, the panel will review the requested budget and make a formal recommendation. Any recommended budget modifications, if applicable, must be detailed and clearly outlined. Any other concerns should be discussed and, if necessary, flagged for staff to address (i.e. ethics, eligibility, human stem cells, etc.). These issues are not to be considered as criteria for evaluation, except if they impact the quality of the application. In such cases, reviewers should bring these issues to the committee earlier in the process for discussion.
Upon conclusion of the review and rating of all applications, reviewers will be asked to provide feedback to the funding agencies regarding the pilot and the effectiveness and functioning of the committee and the peer review process in its entirety.
3.5 After the peer review meeting
Reviewers who wish to make changes to their reviews will have one week to do so after the meeting. Reviewer reports may be made available (without the reviewer's name or score) to the applicants as part of the feedback package provided by their agency. As such, reviewers should refrain from divulging their identities in their reviews to ensure the confidentiality of the review process.
The review should be clear and concise, using objective and non-inflammatory language, and include justification (see, for example, the CIHR College of Reviewer's Review Quality). Constructive advice to the applicant will allow them to improve the quality and efficiency of the proposed research. Note: in circumstances where the Chair or staff deems that the review expresses comments that could be construed as sarcastic, flippant, arrogant, or inappropriate in any way or contain factually incorrect information regarding funding agency's policies, the funding agencies reserve the right to remove such information from the review.
While applications from the three funding opportunities will be reviewed by the shared Tri-Agency Interdisciplinary Peer Review Committee, funding decisions will be made, funding will be provided, and grants will be administered by the agency to which the application was submitted, in accordance with that agency's respective guidelines and decision making processes. Similarly, applicant feedback packages will follow the format of the agency to which the application was submitted.
4. Roles and Responsibilities
The Tri-Agency Interdisciplinary Peer Review Committee consists of a Chair, three Scientific Officers, and Peer Review Committee Members. In addition to the committee, External Assessors will also be engaged to support the committee through the provision of External Assessor Reports on each application. The committee is supported by program staff at each of the three agencies.
The committee Chair is responsible for ensuring that the peer review committee functions smoothly, effectively and objectively. They establish a positive, constructive, fair-minded environment in which the applications are to be evaluated. The Chair (along with the Scientific Officers and program staff) fulfills an oversight role and does not rate applications. Their responsibilities include:
- Working with program staff to manage conflicts of interest;
- Ensuring the involvement of the committee members, as appropriate, with recommendations for each application;
- Ensuring that the perspectives of all reviewers assigned to an application are heard and that they participate actively to the discussion;
- Guiding the committee discussions which will contribute to a final voting score;
- Working with the Scientific Officers and/or program staff to summarize the discussion of each application; and,
- Abide by the agencies' guidelines on conflict of interest and confidentiality.
4.2 Scientific Officers (SO)
The Scientific Officers (SO) are responsible for supporting the Chair during the peer review process as appropriate, including by acting as back-up chairs when the Chair has a conflict of interest. In addition, the SO will take notes on the committee proceedings which may form part of the applicant feedback packages. One SO will be recruited by each agency. In addition to the duties shared with the Chair, the Scientific Officer:
- Supports the Chair in their role throughout the peer review process, including by acting as a back-up chair in the cases where the chair has a conflict of interest;
- Takes notes of the discussion as it is proceeding, which may be sent to applicants as part of their feedback package, according to existing agency practice;
- Ensures that issues of ethics, budgetary adjustments and other concerns that have been flagged for the attention of the funding agencies are recorded for each application, following each agencies' existing practices; and,
- Abide by the agencies' guidelines on conflict of interest and confidentiality.
4.3 Peer Review Committee Members
Peer Review Committee Members are recruited from across the social sciences, humanities, natural sciences, engineering, and health sciences. They will be assigned as Reviewers and Readers to applications assessed by the committee, as supported by External Assessor reports. The primary responsibilities of the peer review committee members are to:
- Indicate their 'ability to review' (i.e., level of expertise and conflicts) for each application;
- Provide a reviewer report and rating for each assigned application as Reviewer. Reviewers should consider, but not be bound by, the External Assessment Reports;
- Review all application material (including External Assessment Reports) and be prepared to discuss and vote on applications to which they are assigned as Reader;
- Attend the committee meeting and participate in the discussions of the applications; and,
- Abide by the agencies' guidelines on conflict of interest and confidentiality.
4.4 External Assessors
The funding agencies will seek to obtain at least two External Assessment Reports from subject matter experts for each application. Committee members should take into consideration, but are not bound by, these reports while conducting their reviews. External Assessors are not part of the committee. The role of the External Assessors is to provide written expert reviews of the application(s) assigned to them. External Assessors will also abide by the agencies' guidelines on conflict of interest and confidentiality.
4.5 Funding Agency Staff
Funding agency staff are involved throughout the funding opportunity process, including supporting the Peer Review Committee in its work. The responsibilities of Funding Agency staff include:
- Assign applications to committee members;
- Identify and recruit External Assessors based on the content of the applications;
- Provide guidance and advice to members on agency guidelines and policies;
- Keep notes on procedural aspects of the committee's functions;
- Record recommendations and concerns raised by the committee on issues requiring later attention by staff (e.g. ethics, etc.); and,
- Record feedback about the committee as a whole and the meeting process.
5. Description of Evaluation Criteria
The Tri-Agency Interdisciplinary Peer Review Committee employs harmonized evaluation criteria. While the specific definition of terms may vary between agency-specific funding opportunities, applicants and committee members will consider the harmonized description of the evaluation criteria. Please note that not every participating funding opportunity has the same requirements in their applications and that different amounts of space may be provided between applications. Applicants and committee members are asked to communicate with their agency contacts with any questions.
Merit of the Proposed Research
Quality of the Proposal:
- The clarity, scope and originality of research questions and objectives
Applicants will clearly describe the context and needs (issues and/or gaps) of the proposed project, including its importance, originality, and anticipated contribution to knowledge. The objectives of the project will clearly define the proposed lines of inquiry and/or activities required to meet the goal.
The proposed project must be based on a logical integration of concepts including a proper framing of the research in the context of the relevant scholarly literature with appropriate reference(s) to other relevant work in the field. The theoretical approach or framework (if applicable) will be well-defined and clear.
Committee members will consider the extent to which the proposal suggests and explores novel or potentially transformative concepts and lines of inquiry and the extent to which the proposal will lead to advances in the knowledge in the relevant field(s).
Methods and Approaches:
- Appropriateness of the proposed methods and approaches
Applicants will provide a clear and detailed description of the research methodologies (approaches/procedures) that will be used to achieve the stated objectives. The methodologies must be appropriate for achieving the stated research goals.
The timelines for the project should be appropriate in relation to the proposed project activities, and the applicant will demonstrate how the project's goals and objectives will be met within the proposed timeline.
Critical challenges should be identified (such as scientific, technical, or organizational), and the applicant should present a realistic plan and mitigating measures to tackle these potential risks. An exhaustive list is not expected.
- Appropriate incorporation of sex, gender and diversity in the research design, if applicable
Applicants will describe their consideration of sex (biological), gender (socio-cultural) and/or diversity in their research design, if applicable. This may include clearly describing the rationale and methodology for including sex, gender and diversity in the research design as well as demonstrating how aspects of sex, gender and/or diversity are addressed in the research design, methods, analysis and interpretation, and/or dissemination of findings.
If appropriate, committee members will be asked to recommend to the applicants on how they might improve the strength of their applications with respect to the integration of sex, gender and/or diversity.
- Extent to which the budget is appropriate and justified in relation to the proposed activities
Applicants will demonstrate appropriate budget planning and clearly explain the suitability of the budget in relation to the proposed methodology(ies) and expected results in terms of scale and feasibility of research plans. They should also clearly describe any financial and in-kind contributions from other sources, where appropriate, including demonstrating that funds requested in the current application are not for expenses supported or submitted for support through other sources.
The committee is asked to adhere to the principle of "Minimal Essential Funding", which requires committee members to carefully examine budgets and cut expenses which are deemed to be inflated or if they judge that a particular request is inappropriate or inadequately justified, and if it is confident that savings can be achieved without jeopardizing the project objectives. Committee members will provide a justified recommendation as to whether the budget should be: Accepted, as described; or Reduced by a specific dollar amount or percentage (total), as appropriate.
- Appropriateness of the environment (academic institution and/or other organization) to enable the conduct and success of the proposed activities
Applicants should have access to the appropriate infrastructure, facilities, support personnel, equipment, resources, and/or supplies to carry out their respective roles. This might include evidence of commitment to the development of active and meaningful partnerships with appropriate individuals and organizations, if applicable.
Committee members will assess project environments, according to their ability to support the proposed project activities. Institutions often function as "networked" environments or interdisciplinary networks, which means there may be multiple satellite environments contributing to the support environment.
Committee members also should consider that for smaller institutions and/or affiliated research facilities that resources and/or services may be obtained through networks or may be contracted out.
- Potential for the outcomes of the proposed activities to have influence and impact
Applicants must clearly describe the potential for the project results to have influence and impact within and/or beyond the project's research field(s), such as the potential for policy impacts or the achievement of results with importance to a broad range of applications or the potential to advance knowledge in the field and influence the direction of thought and activity. When applicable, knowledge translation and/or commercialization approaches and/or methodologies should be described, as well as opportunities to apply research findings nationally and internationally.
Committee members will consider whether the anticipated impact(s) are realistic, i.e., directly stemming from the project outputs, as opposed to marginally related, as well as the significance of developed applications to general and/or limited end users (firms, institutions, etc.).
Knowledge mobilization and dissemination:
- Quality and appropriateness of knowledge mobilization plans, including effective dissemination, exchange and engagement with stakeholders within and/or beyond the research community, where applicable
Applicants will clearly describe the anticipated contribution(s) (e.g. publishing in peer-reviewed journals and other outputs, as appropriate) of the project through a knowledge mobilization plan that includes a description of the effective dissemination, exchange and engagement with stakeholders within and/or beyond the research community, as appropriate. These proposed project outputs should be aligned to the project's objectives.
Committee members will consider whether the anticipated contribution(s) are realistic, i.e., directly stemming from the project outputs, as opposed to marginally related. In addition, committee members will determine whether the research and/or knowledge translation and/or commercialization approaches, methods and/or strategies are well-defined and justified in terms of being appropriate to accomplish the objectives of the project.
- Extent to which the proposed activities incorporates plans for the training and development of highly qualified personnel
Applicants must provide a training plan which describes the specific roles and responsibilities of students, research assistants, and other highly qualified personnel including how their duties will complement their academic training (if applicable). The plan will describe how the applicant will interact with research personnel as well as the approach taken to train and impart knowledge to future researchers, including the skillsets to be imparted to ensure their success.
The research training and development opportunities provided for highly qualified personnel can include, but are not limited to: participation and involvement of highly qualified personnel in outreach activities; interdisciplinary research experiences; promoting equity, diversity, and inclusion in the research community; as well as collaborations, and/or interaction with the private and public sectors.
Committee members will consider the appropriateness of the level and mix of highly qualified personnel for the proposed research project (e.g. is the proposed work suitable for an undergraduate student, a master's student, PhD candidate, or postdoctoral fellow?). Committee members will also take into consideration, where appropriate, the quality of training opportunities to be provided to students, such as the level of intellectual involvement of highly qualified personnel in the proposed research project and how this training will complement the students' own academic training (i.e., the proposed research should leave room for growth and development and highly qualified personnel should be more than simply extra hands for the researcher).
- Appropriate inclusion of considerations of equity, diversity and inclusion in the training plan
As appropriate, applicants will describe how they will increase the inclusion and advancement of under-represented and disadvantaged groups in the research enterprise as one way to enhance excellence in research and training. They should include a discussion of the challenges or barriers encountered in ensuring an inclusive research and training environment as well as a qualitative description of specific actions implemented to support equity and inclusion in recruitment practices, mentorship approaches, and initiatives aimed at ensuring an inclusive research and training environment and trainee growth.
Important: trainee demographic data is not requested, nor required to assess impacts related to equity, diversity and inclusion in the research and training environment.
Expertise and Experience:
- Appropriate expertise to undertake the proposed research, including complementary and interdisciplinary knowledge, expertise and experience
Applicants will clearly demonstrate the combined expertise and experience needed to execute the project. While the principal investigator's achievements should form an important component of the evaluation of the team, as this person is responsible for the planning, co-ordination and intellectual direction of the entire research project, the team's overall capability will be assessed as a collective in light of each member's role in the project, placing more weight on the achievements of those with more central roles.
The level of engagement (e.g., time and other commitments) of the applicant(s) and/or research team should be appropriate to the roles and responsibilities described as well as their capacity to undertake the planned program given their commitments to other research endeavours, as presented in the application. The rationale for the team composition with regards to diversity and equity must be explained, as applicable.
Committee members will consider, as applicable, whether the research team contains the required expertise from complementary disciplines to address the defined objectives competently and complete the research successfully. The contributions of individuals to the research effort must be clear.
- Experience and track record of the investigator(s) – importance, originality, quality and impact of past research
Committee members will consider contributions made by the applicant(s) over the past six years and based on their stage of career. Committee members are also required to take into consideration any special circumstances or career interruptions that may have delayed or interrupted their research career during that period. Applicants with eligible leaves of absence may include contributions from their most recent active research period prior to the last six years for a period equivalent to the duration of the leave.
Assessment of this sub-criterion must be based on the quality and impact of all contributions, not only on the number of publications or conference presentations. Venues with the highest impact (as measured by readership or attendance) may not be the most appropriate for an applicant's research results and it is the responsibility of the applicant to explain the choice of venues for dissemination. As a result, committee members will assess productivity broadly (i.e., not just based on publications) following the San Francisco Declaration on Research Assessment (DORA), which recognizes that scholarly outputs are not limited to published journal articles but can include a broader range of outputs.
Possible outputs include, but are not limited to:
- refereed contributions (e.g. books, book chapters, article in scholarly refereed journals, review articles, conference proceedings, papers presented at scholarly meetings or conferences, articles in professional or trade journals, etc.);
- non-refereed contributions (e.g. book reviews, published reviews of the researcher's work, research reports, policy papers, chairing conference sessions, public lectures and other knowledge mobilization activities, etc.);
- creative outputs (e.g. exhibitions, performances, publications, presentations, film, video, audio recordings, etc);
- involvement in public outreach activities (e.g. organizing research promotional events, taking on leadership positions in research outreach, etc.);
- contributions to the research enterprise (e.g. membership on committees, editorial boards and/or advisory boards, and/or the promotion of equity, diversity and inclusion in the research enterprise); and,
- research results from previous research grants.
Past Contribution to Training of HQP:
- Quality and extent of past contributions to the training of highly qualified personnel
Applicants must demonstrate their contributions to the training and mentoring of future researchers in a research training environment that generates new knowledge or insights, including the supervision of graduate theses and/or the involvement of students in research activities in the context of their current career stage (i.e. Early Career Researchers may not have had the opportunity to develop a large training record).
Committee members will consider the depth and breadth of highly qualified personnel's participation in the applicant's research contributions (e.g. as co-authors, depending on the discipline), which can include but is not limited to conferences, presentations, publications, patents, and/or technical reports; and/or awards, scholarships and/or fellowships won by the highly qualified personnel. Other examples of training impact may include the progression of highly qualified personnel into further studies and/or careers that have impact, whether as professionals in the private, public, and/or academic sectors. While this impact can be either within or beyond the research enterprise, it needs to be clear how the skills gained in the applicant's research training environment are being used by the highly qualified personnel.
Applicants should review the funding opportunity specific guidelines and documents available on their websites:
In addition to the funding opportunity specific resources, applicants should also review the following tri-agency guidelines:
- Selecting the Appropriate Federal Granting Agency
- Research Ethics
- Responsible Conduct of Research
- Conflict of Interest and Confidentiality
- Tri-Agency Open Access Policy on Publications
- Tri-Agency Guide on Financial Administration
- Date modified: