Management Response to the 2014 Evaluation of the Vanier CGS Program

Context

The Vanier CGS program was created by the government of Canada in 2008 to strengthen Canada’s ability to attract and retain the world’s top doctoral students and establish Canada as a global centre of excellence in research and higher learning. The program is implemented through the three Federal Granting agencies and invests approximately $25 million annually to support 500 Canadian and international doctoral students studying at Canadian universities.

This is the first evaluation of the Vanier CGS program and was led by the Canadian Institutes of Health Research (CIHR) in partnership with the Natural Sciences and Engineering Research Council (NSERC) and the Social Sciences and Humanities Research Council (SSHRC). The evaluation covers the time period from the program’s inception in 2008-2009 to the end of fiscal year 2012-2013 and examines the relevance and performance of the program as well as its design and delivery in order to meet Treasury Board requirements for evaluation and inform the renewal of its Terms and Conditions which expire July 31, 2014.

A final report addressing the main issues of this evaluation was approved by the Tri-Agency Programs Steering Committee in June 2014. The response from Vanier Banting Program Management to the evaluation recommendations and the proposed action plan are detailed below.

Vanier Banting Management Response

Overall Comments

The Vanier CGS Executive Management regards the evaluation report as a sound and accurate summary and analysis of the information obtained through multiple sources of evidence and agrees with the conclusions of the report and its recommendations.

The evaluation recommendations and management action plan are presented in the table below.

Vanier CGS Evaluation 2014 - Management Response

Recommendation Response (Agree or Disagree) Management Action Plan Responsibility Timeline

1. To enable the Vanier CGS program to better meet its objective of attracting and recruiting world-class doctoral students to Canadian universities, the following changes to the allocation and application processes should be considered:

1. a) The three year allocation cycle should be changed to annual allocations with restrictions placed on carrying forward unused quotas.

Agreed.

The agencies will implement equal annual targets within the existing three year allocations distribution methodology.

The Vanier-Banting Secretariat.

1. b) A portion of nomination allocations should be targeted to foreign students not already enrolled in the institution in which they are seeking Vanier support, with the amount calculated, at least in part, on an institution’s international student enrollment rate. The Vanier CGS program should monitor the extent to which the target of foreign student nominees is being met, per institution and in total, after each competition and use this information as part of the calculation to determine targets and allocations for the next competition year.

Agreed.

The agencies will work with nominating institutions to identify mitigation strategies to reduce these barriers at the level of the institution and to develop approaches that would increase incentives for institutions to identify foreign students for these awards; these may be within or over and above the current allocation system.

The Vanier-Banting Secretariat.

For 2015-2016 fiscal year.

1. c) The timing of the application deadlines for the Vanier CGS program should be re-considered in light of findings from the evaluation.

Agreed.

The agencies will review the evaluation findings regarding the application deadline, conduct further analysis and change the application deadline as appropriate.

The Vanier-Banting Secretariat.

For 2016-2017 fiscal year.

2. To help improve the assessment of the leadership criterion in the University and Federal Granting Agency selection processes and increase the extent to which it is interpreted consistently, Vanier CGS program management should establish a clearer definition of what leadership is and how it should be evaluated – especially for foreign students. Suggestions for improvement obtained through the evaluation should be taken into consideration and any changes made should be communicated to universities and Federal Granting Agency selection committee members.

Agreed.

As the Agencies continue to clarify the definition of the leadership criterion and its indicators, they will consider the evaluation findings and communicate any changes made.

The Vanier-Banting Secretariat.

For 2015-2016 fiscal year.

3. To help inform the implementation of a Vanier Alumni Network, data obtained through the evaluation on the communication preferences of Vanier scholars for interaction with other award holders and program staff should be considered to ensure that Vanier CGS graduates are connected to the program and to other scholars after completion of their studies.

Agreed.

Ongoing activities such as the creation of Vanier CGS Facebook and LinkedIn pages as well as cohort distribution lists are aligned with the evaluation findings regarding the Vanier Alumni Network.

The CIHR Communication and Public Outreach branch and the Vanier-Banting Secretariat.

For 2014-2015 fiscal year.

4. Evaluation findings demonstrate that the Vanier program is meeting the educational related financial needs of almost all of its recipients and that it is viewed as a highly prestigious award in Canada. However, the evaluation was unable to assess what incremental outcomes are associated with the higher value of the scholarship in comparison to the CGS and Federal Granting Agency doctoral awards as evaluations of those programs were in progress at the time of this study and comparison data was not yet available. The surveys used in these scholarship evaluations were designed to enable a comparative analysis of data across programs.

As such, it will be important to undertake a further analysis to assess the similarities and differences in the results achieved across programs as it can provide insight into what effect features of the Vanier scholarship, most importantly its award amount, have on outcomes. Issues to be examined would include incentives to enroll in studies, financial situation of students, training (including skill acquisition), research productivity, retention rates and employment. Based on this, it is recommended that Vanier CGS program management consider the results of this analysis in the future planning and design of the program.

Agreed.

The Vanier Banting Secretariat will work with the CIHR Evaluation Unit to conduct a meta-analysis of information collected through recent evaluations of training programs and Vanier CGS program management will consider the findings in their decision-making regarding the design and implementation of the program.

The CIHR Evaluation unit and the Vanier Banting Secretariat.

For 2015-2016 fiscal year.

Date modified: