Peer Review Expert Panel: Input from College of Reviewer Chairs
Approach To Engagement Of College Of Reviewer Chairs
On October 27th, 2016, at the first meeting of College Chairs, staff from the Performance and Accountability Branch at CIHR facilitated a semi-structured discussion about the Peer Review Expert Panel questions. Twelve of the 17 Chairs and were present for the discussion. The discussion was not audio-recorded, but detailed notes were taken by staff and those notes informed the summary below. The College Chairs subsequently provided feedback on the summary below and also added some details on discussions regarding PREP which took place at other points in the meeting.
In addition to the summary of this facilitated discussion, College Chairs recommended that the Peer Review Expert Panel be provided with a summary of the Chairs’ meeting minutes, which include an overview of the College of Reviewers core functions, as well as a summary of the planned strategic activities both in the short (definition of College Chair mandate, terms of their interaction with external international review) and long term (e.g., recruitment, learning and mentoring, incentives and recognition and review quality assurance).
Summary of Discussion Themes
The panel: General concerns and opportunity
The College Chairs had questions and concerns regarding the composition of the Panel, the Panel as selected by CIHR, and the questions that will be submitted to this panel. Specifically, the College Chairs advised CIHR to ensure that the Panel’s focus is on "how" the design and grant competitions should be conducted to ensure quality and efficiency of peer review.
The Chairs expressed strong interest in the data and other information that would be provided to the PREP for their work and would like access to these data and analyses. There, the College Chairs were also concerned that the data should be analyzed and accompanied by contextual information to allow the PREP members to understand the key outcomes embedded in the data. The Chairs indicated that it is critical for the Panel to understand the outcomes of the July 13th meeting with the research community and progress made toward implementing these outcomes.
A concern was also raised regarding the appropriateness and ability of the international panel to evaluate the iterative review process in development by the Reference Group on Appropriate Review Practices for Indigenous Health Research due to the lack of indigenous expertise and the exclusively international membership on the panel. It was clarified that the panel’s mandate will not include the iterative review process for Indigenous health research.
The Chairs also asked to be kept informed of the proceedings of the PREP panel (within reason), in particular with respect to the relationship between the PREP mandate and that of the College of Reviewers and the roles of the College Chairs in particular.
Views on program architecture and implementation challenges
The College Chairs have a plurality of views on the design of Foundation and Project Schemes including the organization of the reviews, and newly structured applications. Clearly the implementation of the peer review reform encountered serious problems including reviewer recruitment, expertise matching, quality of review, and engagement of reviewers in post-review discussions. The Chairs agreed that as a result, the Canadian research community lost confidence in this critical function of the CIHR.
Several Chairs were concerned that the Foundation Scheme budget calculations depended largely upon previously held open grants and did not include previous priority-driven funding held by the NPI or team funding. This practice could have unfairly limited budget for some type of research that had been more successful in attracting CIHR priority-driven or interdisciplinary group projects compared to others that had been previously supported by CIHR "open" funding.
Several Chairs expressed concern about the fairness of adjudication based on gender, pillar, and French language applications. Review of Indigenous and participatory research proposals was also noted as an area of concern. It was suggested that the College Chairs and CIHR could consider application formats for different fields of research, particularly the distinct structure of participatory research applications and applicants who often come from the relevant communities and not from academies.
All the College Chairs agreed that CIHR’s overall flat-lined budget had a significant impact on success rates and the morale of the Canadian research community.
Strong endorsement for the College and creation of strong support structures
The College was felt to be a necessary, but insufficient mechanism to ensure peer review quality. Although reviewers and Competition Chairs recruitment and expertise matching (to individual grants and to clusters respectively) are key immediate challenges, the incentives to engage, learning curriculum, mentorship programs and reviewer quality were all seen as important supporting structures within the College. These programs designed to support the membership base will enhance the performance of the College.
The College Chairs agreed that CIHR’s delay in assembling and operationalizing the College of Reviewers was one factor which contributed to compromising the quality of the review process of the early Foundation and Project competitions. A top priority is the implementation of the College and related activities. As implementation progresses, the College Chairs will participate with CIHR in monitoring the impact of these efforts on peer review quality.
Peer review is a human and social enterprise. Members of the College must be engaged to make peer review work. Technology to support implementation is valuable, but cannot substitute for ongoing efforts to improve reviewer engagement with the process. The College and the College Chairs need to become the link between CIHR and the research community, supporting engagement with the review process. The College also represents an important structure to engage the reviewer community in all aspects of peer review.
Indicators and Methods for Measuring Quality and Efficiency
College Chairs expressed the necessity of high-quality data on peer reviewers, the review process and outcomes at all stages of peer review. They stressed the importance of obtaining feedback from the fall 2016 Project Competition Chairs on the quality of review and reviewer performance at both review stages (stage 1 and at stage 2 face to face discussion).
Other data of interest include absolute and proportionate application and success rates by language, pillar, and gender as examples of indicators of fairness and equity. For this part of the mandate, CIHR will provide the College Chairs with metrics on these indicators so that we can return to this topic at a later meeting.
- Date modified: