Evaluation of the Regenerative Medicine & Nanomedicine Initiative

Final Report 2013

June 2013

Canadian Institutes of Health Research
160 Elgin Street, 9th Floor Address Locator 4809A
Ottawa, Ontario K1A 0W9 Canada
Canadian Institutes of Health Research

Acknowledgements:

This report was authored by Patrick MacGuire, Lead Evaluator, and Joanne Tucker, Junior Evaluator

The evaluation was carried out, from design to dissemination, by the Evaluation Working Group: Patrick MacGuire, Joanne Tucker, Eric Marcotte, and Susan Lalumière

Cover photo courtesy of National Research Council of Canada

Table of Contents

• Executive Summary
• Background
• Key Findings
• Conclusions and Recommendations
• Methodology
• Appendix

Executive Summary

CIHR’s Regenerative Medicine and Nanomedicine Initiative (RMNI) is a strategic investment of over $80M to support multi/transdisciplinary and high–impact research in the fields of regenerative medicine and nanomedicine. The initiative was launched in 2004 and has now ended, with the final funding competition held in 2010. CIHR’s decision to discontinue support for RMNI was made on the basis that research in the area of regenerative medicine and nanomedicine has matured to a point that it is highly competitive in the Open Operating Grant Program (OOGP) and can receive funding through the joint CIHR–NSERC Collaborative Health Research Projects Program (CHRP).

This evaluation assesses the performance of RMNI and the outcomes of CIHR’s investment in this initiative. The evaluation also examines the design and delivery of the initiative and provides findings that can inform future CIHR programming. In keeping with CIHR’s requirements to evaluate under the Treasury Board of Canada Secretariat’s 2009 Policy on Evaluation, issues related to relevance are also addressed including the continued need for RMNI and the sustainability of funding for researchers in the fields of regenerative medicine and nanomedicine.

Key Findings

This evaluation finds that RMNI has made a significant contribution to the Canadian health research enterprise in its targeted fields since its launch in 2004.

• RMNI–funded researchers have been responsible for 34% of Canadian publications in regenerative medicine and 21% of Canadian nanomedicine publications between 2004 and 2010. Furthermore, the scientific impact of the publications produced by RMNI–funded researchers, as measured by the Average of Relative Citations (ARC), was above the Canadian average and demonstrates that the initiative has been both selecting and funding research excellence over this period.
• The program design and delivery of the initiative are viewed as successful from the perspectives of researchers and RMNI’s partners and stakeholders.
  • Successful applicants to RMNI express consistently high levels of satisfaction with the peer review process compared to CIHR benchmarks; and,
  • Funding partners point to the initiative as an example of excellence in terms of the dedication and expertise of CIHR’s strategic initiative lead for RMNI.
• RMNI researchers have successfully leveraged grants and awards from CIHR and other research funders as a result of holding initiative funding. For every dollar invested in an RMNI catalyst grant, researchers leveraged $5.22 in other grants/awards; for team grants, this figure is $1.44 per dollar invested.
• For every $100K invested in an RMNI grant, 1.4 refereed journal articles were published and 2.1 research staff and trainees were supported.
• Around half of RMNI research grants (46%) resulted in patents/licenses, compared with a benchmark figure of 18% for CIHR’s Open Operating Grants Program (OOGP). Thirty–eight percent of RMNI grants resulted in intellectual property claims.

When considering the relevance and continued need for the initiative as well as the sustainability of funding for researchers in regenerative medicine and nanomedicine:

• The objectives of RMNI are in alignment with the Government of Canada's Science and Technology Strategy and CIHR’s 2009–2014 strategic plan and recent federal budgets continue to affirm the government’s commitment and role of CIHR in supporting advanced research and health research of national importance.
• RMNI–funded researchers expressed concern about future support for their projects and teams if the initiative were not renewed by CIHR. This was particularly the case with regard to multi/transdisciplinary research in regenerative medicine and nanomedicine. Partners were concerned that RMNI’s absence would result in a slowdown of research in both fields.
• International evidence shows that other countries continue to view regenerative medicine and nanomedicine as having critical importance. Worldwide publication activity in both fields has been increasing annually over time, particularly in nanomedicine and annual Canadian publication growth in both fields correlates highly with world trends. Regenerative medicine is, for example, a national health research priority in Singapore and the country is home to the Institute of Bioengineering and Nanotechnology, one of seven research institutes supporting public sector biomedical R&D. Singapore has attracted leading researchers from the United States and Europe to its institutions and laboratories, and in 2010, led the world in the scientific impact of publications in regenerative medicine and nanomedicine.
• This evaluation does however provide evidence that these once emerging fields have grown significantly. At a macro level, global publications in regenerative medicine and nanomedicine (combined) have grown from an annual rate of 3,381 in 2002 to 17,905 in 2010 – a growth rate of 430%. A similar picture can be seen in Canada where publications in regenerative medicine and nanomedicine have grown from 100 in both fields in 2002 to 628 in 2010, a growth of 530%. In contrast, Canadian health research publications increased from 15,679 in 2001 to 20,700 in 2009 – a growth of 32%.
• It is also apparent that RMNI–funded researchers have been successful in leveraging grants and awards after receiving initiative funding, including in CIHR’s highly competitive investigator–driven Open Operating Grant Program (OOGP).
  • Among RMNI nominated principal investigators, 35 (52%) received OOGP grants (as a nominated principal investigator) and seven (10%) received Canada Research Chairs after having received RMNI funding.
  • Nine (13%) RMNI nominated principal investigators had received support through the Collaborative Health Research Projects Program (CHRP) after being funded by the initiative, and there has been a $15M investment into the Centre for Commercialization of Regenerative Medicine (CCRM), which includes RMNI–funded researchers among its lead scientists and advisory group.
  • Fewer than one in four RMNI–funded researchers say they would be unable to sustain their research program in the absence of future RMNI funding opportunities.

In addition to providing insight into the performance of the initiative itself, the following evaluation findings can inform decision making and assessments regarding current and future CIHR programming, particularly for programs aimed at funding multi/transdisciplinary research and research teams:

• The importance of ensuring good team coordination for large research teams through a hired project coordinator and/or trainee(s) and using regular videoconferencing to overcome barriers of geographical distance that can have a negative impact on team collaboration.
• Being involved in multi/transdisciplinary projects brings many benefits to research trainees; however, some trainees expressed concern that this can also potentially disadvantage future careers if academic institutions or employers are seeking those who have become more specialized in a single field.
• RMNI funding opportunities were intended to support the creation and enhancement of teams. Findings show that 83% of RMNI–funded teams involved some members who had worked together previously, although in no cases had all team members worked together previously. These findings demonstrate that few teams are formed purely as a result of responding to a funding opportunity; it is far more likely that existing teams who have worked together previously will be expanded or enhanced with new members.
• Both RMNI researchers and partners identified a need for an annual meeting for researchers with the purpose of making connections between research teams and exchanging information about research being conducted, including the management of research.

Conclusions

RMNI has made significant contributions to building Canadian research capacity and knowledge creation in the fields of regenerative medicine and nanomedicine. The program has been effectively designed and delivered.

The discontinuation of the initiative raises questions around the sustainability of research in these fields. Evidence from this evaluation is encouraging in that it shows a record of success among RMNI–funded researchers in leveraging other existing grants and awards. There may also be further opportunities for strategic funding for researchers in regenerative medicine and nanomedicine, for example through CIHR’s significant investments in networks for the Strategy on Patient–Oriented Research (SPOR) and the Epigenetics Strategic Initiative as well as funding for large scale research projects under the Personalized Medicine Initiative.

However, in the absence of RMNI funding, it will be important for CIHR to monitor Canada’s competitiveness in regenerative medicine and nanomedicine. If the country’s competitiveness declines, CIHR should assess the health of both fields in Canada through an examination of Canadian investment in this type of research as well as tracking the subsequent careers of RMNI principal investigators and trainees. CIHR should also offer direction to the regenerative medicine and nanomedicine research community on applying to other CIHR funding opportunities and initiatives providing support in these fields.

Recommendations

1. Implement a communication strategy aimed at researchers working in the fields of regenerative medicine and nanomedicine that offers direction on applying to other CIHR funding opportunities and initiatives providing support in these fields.
2. Conduct regular assessments with international benchmarks to determine the relative global position of Canada in the fields of regenerative medicine and nanomedicine in the absence of RMNI. If Canada’s competitiveness declines, ensure regular environmental scanning takes place to assess the ongoing health of both fields in Canada. This would include examining the Canadian investment in these research areas and tracking the subsequent careers of RMNI principal investigators and trainees.
3. Ensure that future designs of programs relating to teams of researchers or networks take into account findings from the evaluation. It may, for example, be unrealistic to design programs that are expected to fund ‘newly formed’ teams, and there could be merit in including stronger requirements for dedicated research coordinators to aid success.

Management Response

Background

The Regenerative Medicine and Nanomedicine Initiative (RMNI)

The Regenerative Medicine and Nanomedicine Initiative (RMNI) is one of CIHR's major strategic initiatives. The goal of the initiative is to support multi/transdisciplinary and high–risk, high impact research approaches in the areas of regenerative medicine and nanomedicine. Between 2004 and 2010, RMNI has invested $82.3 Million in funding to researchers through two mechanisms: team grants and catalyst grants.

Within CIHR, RMNI is co–led by the Institute of Neurosciences, Mental Health and Addiction, the Institute of Genetics, and the Institute of Musculoskeletal Health and Arthritis. It also involves many of CIHR’s other Institutes and branches, as well as external partners such as the Juvenile Diabetes Research Foundation, the Canadian Space Agency, and the Stem Cell Network

RMNI Workshops and Meetings

RMNI has worked closely with a range of organizations such as Health Canada, the Natural Sciences and Engineering Research Council, and the National Research Council of Canada to sponsor workshops and meetings on topics of common interest. These meetings bring together experts and stakeholders from different domains, aligned along common themes relevant to the fields of regenerative medicine and nanomedicine, to form connections between fields, disciplines, and backgrounds. From 2003 to 2008, RMNI co–organized on average three workshops per year, with a total operational commitment of approximately $1 Million from all sources, including internal and external partners.

Photo: Image of Nanorods – Photo courtesy of Prof. Warren Chan, University of Toronto.

Evaluation Purpose

This evaluation is designed to provide valid, insightful and actionable findings about the performance of RMNI for CIHR’s senior managers, strategic leads and program management. As well, the evaluation will provide findings that can inform decision making and assessments regarding current and future CIHR initiatives and programs. The evaluation is also designed to meet CIHR’s requirements to the Treasury Board Secretariat (TBS) under the 2009 Policy on Evaluation. An evaluation of RMNI was requested by CIHR’s Scientific Council at their planning retreat in March of 2010.

Key Findings

Knowledge Creation in Regenerative Medicine and Nanomedicine

One of the primary objectives of CIHR’s Regenerative Medicine and Nanomedicine Initiative (RMNI) is to fund research projects with the potential for generating high impact results to ensure Canada’s strong and growing presence in the fields of regenerative medicine and nanomedicine. To help assess the extent to which this objective has been achieved, a bibliometric analysis¹ was conducted on publications produced by the full population of RMNI–funded researchers (N=295) and non–funded applicants to RMNI (N=143), grouped according to RMNI funding received/applied for (team and catalyst grants), and the top 16 productive countries in the two targeted fields (including Canada). Publications relevant to regenerative medicine and nanomedicine were retrieved through a search query using specific U.S. National Library of Medicine Medical Subject Headings (MeSH). In addition, publications in core journals of regenerative medicine and nanomedicine were also included in the analysis (see Methodology section for a full description of the bibliometric analysis).

Global and National Publication Rates in Regenerative Medicine and Nanomedicine

To contextualize the performance of RMNI in terms of knowledge creation and scientific impact, it is useful to first briefly consider global and national publication rates for the fields of regenerative medicine and nanomedicine. Results presented in Figure 1 show a significant increase in the global number of publications in both fields over the period of 1997–2010, particularly in nanomedicine. Specifically, papers in both fields (combined) have grown from an annual rate of 3,381 in 2002 to 17,905 in 2010 – a 430% increase. Furthermore, results from a recent bibliometric study (Observatoire des sciences et des technologies, 2010) on ten research fields relevant to the mandate of RMNI’s co–lead Institute, the Institute of Neurosciences, Mental Health and Addiction, found that regenerative medicine and nanomedicine were among the fastest growing fields of those compared, both worldwide and in Canada.

As shown in Figure 2, publication activity in both fields in Canada has increased over time as well with growth rates similar to trends observed worldwide. A correlation analysis showed that annual publication production trends are highly correlated between the two fields as well as between Canada and the world (p<0.05). Canadian papers in both fields (combined) have grown from 100 in 2002 to 628 in 2010 – a growth of 530%. In contrast, the number of Canadian health research papers produced annually increased from 15,679 in 2001 to 20,700 in 2009 – a growth of 32%.

Source: RMNI Bibliometric Data on World Publications

Source: RMNI Bibliometric Data on Canadian Publications

Top Countries in Knowledge Creation and Scientific Impact

Figures 3 and 4 depict the publication volume (larger spheres indicate more publications produced), the Average of Relative Citations (ARC), and Specialization Index (SI) of the top 16 productive countries in regenerative medicine and nanomedicine for 2004–2010, the period of RMNI’s implementation (see Methodology section for full description of bibilometric indicators and Appendix for values by country). Overall, Canada ranks 9th on the total number of publications produced and 12th on specialization in each field – below the world average of one. For citation impact, Canada scored above the world average in each field ranking in the middle of the top 16 countries for regenerative medicine with a tie for 8th place (ARC of 1.15) and ranking 6th overall in nanomedicine (ARC of 1.04).

Results show that over the period 2004–2010, the United States was the world leader in knowledge creation in these fields, accounting for close to 40% of world publications produced in both regenerative medicine and nanomedicine. For citation impact, the United States leads in nanomedicine (ARC of 1.22) but ties for 4th in regenerative medicine behind Sweden (1st), Switzerland (2nd) and Singapore (3rd).

It is interesting to note that over the period of 2004–2010, Singapore has made significant contributions in terms of scientific impact (ranked 3rd in ARCs in both fields) and was the world leader in specialization in each field (particularly in nanomedicine with an SI of 3.21) based on production of a relatively low number of publications. A brief case study of Singapore’s approach to funding health research is presented in Figure 5.

Figure 3 – Scatterplot of ARC and SI for Top 16 Productive Countries in Regenerative Medicine 2004–2010

Source: Bibliometric Data on Top 16 Productive Countries

Figure 3: long description

Figure 4 – Scatterplot of ARC and SI for Top 16 Productive Countries in Nanomedicine 2004–2010

Source: Bibliometric Data on Top 16 Productive Countries

Figure 4: long description

Methodological Note: Regenerative Medicine and Basic Science Stem Cell Publications

Supported regenerative medicine fields within the scope of RMNI include tissue engineering, rehabilitation sciences as well as stem cell research pertaining to regenerative therapies. As the focus of RMNI was on translating regenerative approaches to health applications, papers on basic stem cell research were excluded from this analysis. However, a number of international comparator studies include all stem cell research under the regenerative medicine category. For example, a recent 2011 Thomson Reuters bibilometric study on global publications in regenerative medicine – that included basic science stem cell papers within their analyses – showed that these publications accounted for approximately 55% of the data set. This suggests that, had the RMNI bibliometric study included these papers, regenerative medicine publication volume would have doubled overall. The 2011 Thomson Reuters study also gives an opportunity to see how the inclusion of all stem cell research would have altered Canada’s ranking in the field among countries common to both studies – from 8th to 5th for citation impact (a tie with the United Kingdom); and from 9th to 8th for publication volume.

Although the Thomson Reuters study differs somewhat from the RMNI analysis in terms of the calculation and types of several bibliometric indicators used, the set of countries included and the time period covered, comparisons on relative rankings between countries common to both studies proved to be highly correlated (r=0.969, p<0.05 for publication volume rankings and r=0.864, p<0.05 for citation impact rankings) and supports the validity of the findings for the bibliometric analysis of regenerative medicine presented in this study. It also illustrates the relative strength of basic stem cell research in Canada, given the increased citation impact for Canada when these papers are included in the study.

Figure 5 – Singapore’s Approach to Health Research Funding

• Ranked 1st in specialization index and 3rd in average of relative citations in both regenerative medicine (~600 papers) and nanomedicine (~1,000 papers) over the period 2004-2010. For 2010, Singapore was the top ranked country in ARCs in both fields.
• Regenerative medicine is a national health research priority and the country is home to the Institute of Bioengineering and Nanotechnology, one of seven research institutes supporting public sector biomedical R&D. Singapore’s health research system is sustained by diverse funding sources across the public (63%) and private sectors (37%) with most funding schemes being commercialization driven.
• Provides an environment that is welcoming to intellectual property with favourable business and immigration conditions (low corporate taxes, 10 year tax exemptions and open immigration policies) that attract a high number of foreign investors and multinational companies.
• Has attracted leading researchers from the UK, USA, Sweden, Germany, and Japan to its laboratories and institutes.
• The Biomedical Research Council (BMRC) of the Agency for Science, Technology and Research (A*STAR), one of Singapore’s government funders, boasts a number of research facilities, most of which are available to all Singapore researchers regardless of research funding source, therefore reducing operating costs for basic research.

Source: Bibliometric data on Top 16 Productive Countries; Marjanovic & Chonaill (2010)

RMNI–Funded Researchers’ Contributions to Knowledge Creation

Having considered the global and national context, we turn now to the contribution of RMNI to knowledge creation in regenerative medicine and nanomedicine. As shown in Figures 6 and 7, RMNI–funded researchers were responsible for close to one–third of Canadian publications in both fields over the period of 2004–2010. In terms of each field, RMNI–funded researchers were authors on 34% of Canada’s total number of regenerative medicine publications and 21% of nanomedicine publications, percentages not unexpected given RMNI’s total overall funding success rate of 26%.

Source: Bibliometric Data on Canadian and RMNI–Funded Researchers (N=295)

It should be noted that the bibliometric analysis in this evaluation includes data for regenerative medicine and nanomedicine publications produced by researchers funded by RMNI at some point after the initiative’s inception. Although the period of publication used throughout the bibilometric analysis aligns with the overall lifecycle of RMNI (2004–2010), funded researchers may have published articles in either field prior to, or after having concluded, their RMNI grant. As a result, direct attribution between RMNI funding and publication data cannot be made.

Scientific Impact of RMNI–Funded Researchers Compared with Canadian Averages

Evidence from this evaluation demonstrates that RMNI has been attracting and funding excellent researchers in terms of scientific impact of their publications.

As shown in Figure 8, publications in regenerative medicine produced by RMNI–funded researchers generally have a higher scientific impact (based on ARC) than the average for Canadian health researchers in this field. Those funded through RMNI catalyst grants generally achieved higher average citation scores than both Canadian and team grant–funded researchers, particularly over the period 2009–2010. It should be noted that the overall average citation scores for Canada presented in Figures 8 and 9 were calculated based on publications with at least one Canadian author and as such also includes papers authored by RMNI researchers.

With regard to nanomedicine (Figure 9), publications produced by both RMNI–funded team and catalyst researchers generally achieved higher average ARC scores than those for Canadian health researchers over the period 2006–2010. Within this field, RMNI team grant researchers tended to outperform both Canadian and catalyst grant researchers in terms of their scientific impact, achieving a peak score in 2010 with an ARC of 2.24.

Source: Bibliometric Data on Canadian and RMNI–Funded Researchers (N=295)

Source: Bibliometric Data on Canadian and RMNI–Funded Researchers (N=295)

When the two fields are combined, both RMNI team and catalyst grant–funded researcher publications achieved an overall ARC score of 1.39 and 1.57 respectively, compared to an ARC of 1.11 for Canada over the period of 2004–2010 (p<0.05²). Similarly, bibliometric data shows that researchers funded through CIHR’s Open Operating Grant Program (OOGP) produce papers with higher impact than Canadian health research publications. Specifically, supported papers³ published over the period of 2001–2009 by 1,125 researchers who received funding from OOGP between 2000 and 2007 resulted in an overall ARC score of 1.51, above the overall ARC of 1.24 for Canadian health research publications over the same time period⁴ (p<0.05⁵).

Knowledge Creation and Impact of RMNI–Funded Compared to Non–Funded Researchers

Bibliometric results presented in Table 3 reveal that RMNI–funded researchers achieved higher citation impact scores than unsuccessful applicants in both fields although in only one case was a statistically significant difference observed. Furthermore, funded researchers published approximately 2.7 times the number of papers in total than unsuccessful applicants when grant type and research fields are combined. Of note, unsuccessful applicants are researchers who submitted RMNI applications that were rated through peer review as fundable but were never successful in receiving initiative funding.

For RMNI team grant–funded researchers, the citation impact of their publications in regenerative medicine over the period of RMNI’s implementation (2004–2010) was 1.29 compared to 1.27 for unsuccessful team grant applicants. For nanomedicine, team grant–funded researchers achieved an ARC of 1.48 compared to 0.96 for unsuccessful applicants. However, differences in ARCs between the two groups in either field were not statistically significant. Team–funded researchers also published 3.1 times more regenerative medicine publications and 1.8 times more nanomedicine papers than unsuccessful team grant applicants.

For catalyst grant researchers, the average citation score over the six year period was 1.88 for funded researchers compared to 1.00 for non–funded researchers in regenerative medicine (ARC differences between groups not statistically significant). For nanomedicine, RMNI–funded catalyst researchers had an ARC score of 1.28 compared to 0.62 for non–funded researchers (p<0.05⁶). In terms of publication volume, funded catalyst researchers published 3.9 times more regenerative medicine papers and 3.7 times more nanomedicine publications than unsuccessful catalyst grant applicants.

Publications Produced from RMNI–Funded Research

Results presented in Table 4 show the average number of publications resulting from RMNI grants. Overall, RMNI team grants resulted in an average of 20.8 refereed journal articles published compared to 2.0 for catalyst grants (p<0.017) and an average of 3.4 books/book chapters compared to 0.2 for catalyst grants (p<0.017).⁷

Results should be viewed in context of differences between grant types: the majority of catalyst grants surveyed had expenditures between $140–$260k, were two to three years in duration, and involved one or two researchers. In contrast, team grants surveyed had expenditures of $1–2M, involved an average of eight researchers, and were mostly five years in duration at time of survey.

As the majority of nominated principal investigators surveyed indicated that their RMNI–funded research was biomedical (see Appendix for profile of surveyed research), a benchmark comparison on publications resulting from biomedical researchers funded through CIHR’s Open Operating Grants Program (OOGP) was conducted. Overall, OOGP–funded researchers produced an average of 8.1 refereed journal articles and 1.0 books/book chapters per grant. The average amount committed to an OOGP grant included in the benchmark comparison was just over $300k and the majority of grants (66%) were three years in duration. Of note, the OOGP grants were awarded over the period of 1991–2008 and hence historical and contextual factors such as variation in funding amounts over time may have a limiting effect on the comparability of OOGP and RMNI supported research.

To account for the differences between RMNI team and catalyst grants in terms of dollars expended, duration, and number of researchers involved, the average number of journal articles published per grant was normalized by dividing the total number of articles per grant by grant expenditures and duration⁸ at time of survey as well as the number of researchers involved at time of application (Table 5). As a result, catalyst grant researchers produced an average number of 1.3 journal articles per $100k as compared to 1.4 articles for team grants (differences between groups not statistically significant). Additionally, 1.1 articles per year of grant were produced from catalyst researchers versus 4.4 for team grant researchers (p<0.017) and controlling for team size resulted in team grants producing 2.8 articles per researcher versus 1.4 for catalyst grants (p<0.017).⁹

Normalization applied to biomedical OOGP grants in terms of dollar amount committed (over grant lifespan) resulted in an average of 2.6 journal articles published per $100k while 2.3 journal articles were published per year of grant (Table 5).

RMNI Return on Investment

A ‘return on investment’ analysis was conducted on a sample of 26 RMNI grants (based on dollar amount expended at the time of survey) on several key metrics: involvement of researchers, staff and trainees, refereed journal articles produced, and funding leveraged.

As shown in Table 6, in terms of return on investment per $100k, catalyst grant researchers produced 1.2 articles, involved 4.3 research staff/ trainees and leveraged 1.2 grants/awards. For every catalyst grant dollar invested, researchers were able to leverage $5.22 dollars in grants/awards. With regards to team grants, $100k invested resulted in 1.4 articles, 1.9 research staff/trainees and 0.3 leveraged grants/awards. For every dollar invested, team grant researchers leveraged $1.44 dollars in grants/awards.

Benchmark data on ‘return on investment’ for CIHR’s Open Operating Grants Program (OOGP) shows that biomedical researchers awarded OOGP grants over the period of 1991–2006 produced 2.8 articles and trained 2.6 research staff/trainees per $100k (Table 6).

This type of comparison between team and catalyst grants should be treated with caution given contextual factors and potential confounds related to differences between funding tools and areas of research supported. It does however provide insight into the results of return on investment at a basic level for a strategic initiative and its funding mechanisms and provides a benchmark for future CIHR evaluations and studies.

RMNI’s Influence on the Development of Regenerative Medicine and Nanomedicine

A majority of RMNI–funded researchers feel that the initiative has had a positive influence on the development of regenerative medicine (80%) and nanomedicine (55%) in Canada (Figure 10). Fewer expressed the same opinion about the initiative’s influence on both fields internationally (54% regenerative medicine; 35% nanomedicine). It should be noted that over one third of researchers indicated “don’t know/not applicable” responses in terms of RMNI’s influence on nanomedicine, both in Canada and abroad. Furthermore, the majority of funded researchers interviewed indicated that they were not sufficiently aware of the initiative’s influence outside of their own research and could only comment on general areas in which RMNI was impactful. As such, the data presented in Figure 10 should be treated with caution.

Figure 10 - Influence of RMNI on the Development of the Fields of Regenerative Medicine and Nanomedicine

Source: Survey of RMNI–Funded Researchers (N=72)

Figure 10: long description

RMNI–Funded High Impact Research Projects

Photo: RMNI Case Study: Prof. John Pezacki (pictured in back) - Photo courtesy of CIHR.

The impacts of health research are far broader than what can be measured through bibliometric analyses of publications. To provide a more in–depth analysis of the wider impacts and benefits of RMNI–funded research, three case studies of projects that demonstrated high impact, breakthrough results are described in detail below.¹⁰

For each of the case profiles, the overview section provides the context and background of the research project including the research issue(s) being addressed as well as the objectives of the projects. The impacts section details the successes that were achieved, while the factors that led to success are presented under pathways to results. The role of RMNI funding provides insight into the importance of initiative support to these projects and researchers.

Understanding and Manufacturing Quantum Dots for Biological & Medical Imaging

Warren Chan
Associate Professor, Institute of Biomaterials and Biomedical Engineering
University of Toronto

Overview

Materials, systems, and particles smaller than 100 nanometers (nm) can have unique optical and electronic properties. These allow researchers to engineer new tools to probe biological systems, and to detect and treat diseases. Quantum dots are one type of nanotechnology that emit light of different colors by changing the dots size from 2 to 8 nm. As such, they may be ideal contrast agents for biomedical imaging of diseases in the body, tissues, or cells. Prof. Chan’s team originally set out to develop quantum dots for tumour targeting but their initial studies showed that targeting was inefficient, as only about 3% of qdots would enter the tumour. As a result, the team shifted focus to understanding how the surface chemistry, size and shape of a nanoparticle affect targeting. This advancement in knowledge would enable the rational engineering of nanoparticles to target diseases.

Impacts

Photo: Quantum Dots – Photo courtesy of Prof. Warren Chan.

The team was able to demonstrate that the size, shape, and surface chemistry of nanoparticles affect their tumour targeting efficiency. As well, the group demonstrated the need to quantify the amount of nanoparticles targeting tumors. While quantum dot technology was not translated, as initially proposed, the outcome of the research had a major impact in the nanomedicine community with one of the published studies having been cited over 1000 times and four other studies receiving over 100 citations since their publication. Additionally, Prof. Chan’s team received many requests for nanomaterials during the course of the project and the group created the Canadian biotechnology company Cytodiagnostics to commercialize nanomaterials. Currently, the company generates positive revenues and the team’s quantum dot nanomaterials are sold through Sigma–Aldrich globally as well as a number of distributors in many countries around the world.

Pathways To Results

A team of researchers with diverse expertise in biomedical engineering, medical biology, pharmacology, and pathology brought different perspectives to solving research problems. Strong leadership and support from Prof. Chan including the ability to change the team’s direction when needed as well as the willingness of collaborating professors to allocate time to mentor a core group of students (up to 42 in total) were factors that contributed to success.

Role Of Rmni Funding

RMNI provided a mechanism to fund a nanomedicine project in technology and health that would lie outside the boundary of most research funding programs in Canada. The work produced from the RMNI grant has been a major part of Prof. Chan’s career and accounts for approximately 30% of his total research publications.

Improving Corneal Transplantation through Tissue Regeneration & Femtosecond Laser Technology

Isabelle Brunette
Professeure titulaire, Département d'ophtalmologie
Université de Montréal

Overview

Dysfunction in the endothelial layer of the cornea leads to blindness and severe pain. Currently, endothelial dysfunction is the leading cause for corneal transplantation, being responsible for 42% of the 50,000 corneal transplantations performed every year in North America. The purpose of Dr. Brunette’s project, which involved the Département d'ophtalmologie at the Hôpital Maisonneuve–Rosemont in Montréal, QC, the Institut national de recherche scientifique (INRS) in Varennes, QC and the Laboratoire d’organogénèse experimentale (LOEX) in Québec, QC, was to improve the functional results of corneal transplantation for endothelial dysfunction through the use of tissue engineering and femtosecond laser technology.

Impacts

Photo: Microcavitation bubbles generated by femtosecond laser ablation in human cornea – Photo courtesy of Dr. Isabelle Brunette.

Through RMNI funding, Dr. Brunette’s team successfully built on a $20 Million Canada Foundation for Innovation award to develop and adapt a femtosecond laser into a surgical suite at Montréal’s Hôpital Maisonneuve–Rosemont. The laser cuts only the layer of donor tissue needed for transplantation (instead of the entire cornea as in standard transplantations) and with exact matching dimensions. The team also set out to use tissue engineering technology to grow patients’ own endothelial cells in culture for transplantation back into the patient’s eye. Preclinical studies are quite conclusive that the procedure will reduce the demand on eye banks for donor corneas and positively impact access to eye care due to decreased exclusion criteria for donor tissue. Furthermore, it will eliminate the risk for rejection since the patient’s own cells will be used to tissue engineer the corneal transplant. Dr. Brunette and her team are the only researchers in the world who have been successful in culturing cells from sick patients with Fuchs corneal dystrophy without genetic manipulation. The team was also the first to demonstrate the regenerative potential of these cells by using them to tissue engineer a new cornea that was successfully transplanted in a living eye. A socioeconomic benefits analysis conducted by a pharmacoeconomist from the group demonstrated that the proposed techniques may provide better results at a similar cost (due to improved clinical outcomes, improved recovery time, and reduced waiting times) compared to traditional corneal transplantation procedures. The results of the research have also been published in veterinary journals demonstrating how corneal transplant methods developed for humans can be applied to animal care. Dr. Brunette has received international attention for her work.

Pathways To Results

Success was credited to having access to staff working in the institutions where the research is being conducted and innovative researchers who could work in a trans/multidisciplinary environment. Collaboration between a range of expertise including clinicians, surgeons, and other researchers also helped advance the project.

Role Of Rmni Funding

The research could not have been carried out without a trans/multi–disciplinary team involving ophthalmologists, physicists, tissue engineers, economists, and clinicians and the RMNI team grant provided the resources necessary to bring these different skillsets together.

Applying CARS Spectroscopy to Improve the Study of the Molecular Determinants of Disease

John Pezacki
Senior Research Officer, Steacie Institute for Molecular Sciences

Overview

Early diagnosis has the potential to increase survival rates for people with chronic diseases. Sensitive molecular imaging techniques based on biophotonics can be used to detect the early signs of disease far before signs are present in a given tissue. Prof. Pezacki’s team, which involved researchers working in chemistry, biophotonics, and molecular imaging, set out to develop an innovative microscope using Coherent Anti–Stokes Raman Scattering (CARS), a non–linear optical spectroscopy that involves sending laser pulses down a microscope every femtosecond – one millionth of one billionth of a second – to "fingerprint" the molecular vibrations of cell components and create images of them. The CARS technique enables the study of the molecular determinants of disease without the use of dyes or other labeling agents that are invasive and destructive to cells and tissues. The project also involved an exploration of clinical applications of the CARS microscope in the early diagnosis of disease.

Impacts

Photo: Label–free multimodal CARS microscopy of an atherosclerotic rabbit aorta – Photo courtesy of Prof. John Pezacki.

During the initial phase of the team’s work, it was discovered that there was no suitable existing CARS hardware for their biomedical imaging approach. To deal with this issue, the team's technology development group, led by Dr. Albert Stolow, simplified a CARS technique developed at Harvard University and created a more cost efficient approach that could be applied beyond a controlled laboratory environment in locations such as hospitals, clinics and doctor’s offices. More than 20 peer reviewed papers have been published just from the team at the NRC alone and papers based on the research supported by the RMNI grant are still being published including a review in Nature Chemical Biology summarizing the recent applications of CARS microscopy towards understanding important diseases such as hepatitis C virus infections. The team’s success also led to a commercial collaboration with Olympus, a multinational company specializing in cameras, research and clinical microscopes. The team used an existing microscope that is sold by Olympus as a base for the development of their CARS microscope technology (which functions as an attachment). Of significance, Olympus had previously been unsuccessful in its attempts to add CARS functionality to its microscope. Since the team’s technology was compatible with an Olympus device, a partnership was created and the CARS microscope is now sold as an add–on to the Olympus FluoView FV1000–MPE microscope. An NRC–Olympus CARSLab Microscopy Facility was launched in 2009 to educate the wider biomedical community on the benefits of this technology and hence help to provide the Canadian health system with cost–effective state–of–the–art medical diagnostic technologies.

Pathways To Results

The sharing of expertise in a multidisciplinary team and the inclusion of students in the project due to RMNI funding (whom the lead researchers accessed through their adjunct professorships) were significant factors that contributed to success. The involvement of the end users – the clinicians – at the beginning who provided input on the development of the CARS microscope helped the team to better understand their end users’ needs and focus their efforts more efficiently.

Role Of Rmni Funding

Prof. Pezacki emphasized the importance of the RMNI funding mechanism as it enables multidisciplinary teams to be brought together more efficiently than having to apply for multiple grants to achieve the same results. The RMNI funding also enabled Prof. Pezacki to assume a leadership role in the collaboration at NRC and include students in the project. Overall, the team grant sped up the research process and strengthened the research outputs by bringing the innovators, the tool builders, and the end–users together at the start.

Source: Survey of RMNI–Funded Researchers (N=72)

Although a total of six RMNI–funded projects were selected for case studies based on their high impact results (with three profiled in this report), a large majority of RMNI–funded researchers feel that their research resulted in major achievements (Figure 11). As is shown in Figure 12, a range of factors were identified by researchers as enabling or hindering the success of their projects. Several enabling factors identified are core to the program theory and intent of the initiative, including the multi/transdisciplinary nature of the research projects and the involvement of end users early in the research process. As one catalyst grant NPI noted: “le projet n'aurait pu voir le jour sans une approche multidisciplinaire. On a beaucoup appris l'un de l'autre [et] les expertises étaient complémentaires et essentielles.” Far fewer factors were identified by researchers as inhibitors to the success of their funded projects; those identified related to a lack of renewal funding and, in the case of a few team grant projects, having insufficient funds to engage researchers full–time or conduct knowledge dissemination activities.

Figure 12 – Identified Factors That Enabled and Hindered the Success of RMNI-Funded Projects

Factors Enabling Success

• Group of researchers collaborating as a team
• Multi/transdisciplinary nature of the research
• Personal qualities of team members
• Leveraging of additional funding and support
• Involvement of student trainees
• Involvement of end users early in the research process
• Development of partner/stakeholder relationships including industry support

Factors Hindering Success

In reference to RMNI team grants:

• Lack of renewal funding
• Insufficient funding amount to engage researchers full-time or carry out knowledge dissemination activities

Source: Interviews with RMNI-Funded Researchers (N=23); Interviews with RMNI Case Study Participants (N=29).

Research Team Collaboration

As outlined in CIHR’s 2009–14 strategic plan, the Health Research Roadmap (CIHR, 2010), one of the organization’s core values involves the promotion, encouragement, and appreciation of collaboration among researchers in Canada and internationally. As noted, one of the explicit objectives of RMNI is to fund the creation or further development of research teams undertaking collaborative trans/multidisciplinary research that will lead to enhanced approaches to understanding and resolving regenerative medicine and nanomedicine health issues.

RMNI funding opportunities were intended to support the creation and enhancement of teams (projects involving at least three researchers) and evaluation findings (Table 7) indicate that the majority of RMNI–funded teams were enhanced through funding as 83% involved some members who had worked together previously. According to interviews with RMNI team and catalyst nominated principal investigators¹¹, in many cases, a prior history of collaboration among team members was viewed as a significant factor in the success of their project and in several instances the idea for their RMNI project originated from discussions between members of their team.

The meaning of ‘creating’ a team is also open to some interpretation. Findings show that in no case had the entire team worked together previously, demonstrating that RMNI has not simply been refunding existing teams that had already been formed for previous projects.

RMNI Multi/Transdisciplinary Research

Interdisciplinary teams have been described as “the defining feature of the scientific endeavor in the twenty–first century” (Kessel et al., 2008). The findings presented in Figure 13 confirm that from an RMNI researcher perspective, there is a value–add to this type of collaboration in terms of research outcomes.

For a majority of researchers, operating in a multi/transdisciplinary team enabled them to achieve research outcomes that would not have occurred without this type of collaboration (84%). Almost all RMNI–funded researchers would participate in this type of research again (97%) and encourage others to do so (95%) based on their experiences.

As researchers who have applied to undertake multi/transdisciplinary research, this group would be expected to have generally positive views on the benefits of that approach going into their project. The overwhelmingly positive responses at the end of the projects show that being funded by RMNI is likely to have reinforced these initial views.

RMNI researchers’ opinions on team collaboration were also generally positive with 86% reporting that their collaboration was effective and 80% indicating that their RMNI grant facilitated more collaboration with researchers from different disciplines than would have occurred through other grant funding (Figure 14).

Figure 13 – RMNI-Funded Researchers’ Opinions on Multi/Transdisciplinary Research

Source: Survey of RMNI–Funded Researchers (N=64^*)

* RMNI–funded researchers involved in research teams (projects with three or more researchers involved)

Figure 13: long description

Figure 14 – RMNI-Funded Researchers’ Opinions on Team Collaboration

Source: Survey of RMNI–Funded Researchers (N=64^*)

* RMNI–funded researchers involved in research teams (projects with three or more researchers involved)

Figure 14: long description

Key Factors for Effective Collaboration

RMNI–funded researchers and case study participants who were interviewed noted various factors that enabled and/or hindered effective team collaboration (Figure 15). For example, the importance of ensuring good team coordination for large teams through a hired project coordinator and/or trainee(s) was noted as extremely important since one of the key challenges faced in these projects was the amount of time required for NPIs to effectively manage their group.

As one RMNI team grant NPI, whose team involved many researchers and trainees, explained:

Additionally, using regular video teleconferencing to overcome barriers of geographical distance was also frequently mentioned by researchers as a key factor to ensuring effective collaboration.

Figure 15 – Identified Factors That Enabled and Hindered Effective Team Collaboration

Factors Enabling Effective Collaboration

• Regular group communication/meetings
• Coordination of teams via trainees and/or a hired project coordinator
• Setting common goals and clear objectives
• Personal qualities of team members
• Identification of appropriate team members
• Previous collaborations among team members
• Close geographical approximation of team members
• Use of videoconferencing software such as Skype to overcome geographical dispersion of team
• Presence of complementary expertise
• Mutual respect for team members and disciplines

Factors Hindering Effective Collaboration

• Required level of effort or adaptability across team members
• Required understanding of different disciplinary languages represented on the team
• Required time to manage the team
• Geographical dispersion of team members limiting face-to-face interactions

Source: Interviews with RMNI–Funded Researchers (N=23); Interviews with RMNI Case Study Participants (N=29)

International Collaboration Rates

Recognizing the importance of working at the international level, the CIHR Act states that “Canada should be an internationally acknowledged leader in contributing to the global advancement of health research" (Canadian Institutes of Health Research Act, 2000, p.1). To measure the extent of international scientific collaboration on regenerative medicine and nanomedicine publications authored by Canadian and RMNI–funded researchers over the period of 2004–2010, the number of papers with at least one author with a foreign country address was divided by the total number of papers to arrive at a percentage of international collaboration.

Results presented in Table 8 show that international collaboration rates for papers published by RMNI–funded researchers were higher than the world percentage in both fields but lower than Canada, particularly in nanomedicine. Both RMNI funded team and catalyst researchers achieved higher international collaboration rates in regenerative medicine than in nanomedicine. Canadian researchers achieved a high international collaboration rate of 56% in regenerative medicine (ranked 2nd in the world) but ranked lower in terms of nanomedicine with a collaboration rate of 41% (placing 8th among the top 16 productive countries in the field).

Knowledge Translation

Knowledge translation (KT) is a fundamental part of CIHR’s mandate and is a dynamic and iterative process that includes the synthesis, dissemination, exchange and ethically–sound application of knowledge and can occur in a variety of ways, including the commercialization of research findings.

Results presented in Table 9 show that the most common type of knowledge user or stakeholder group involved in RMNI research was other researchers and academics (excluding study stakeholders) (46%) followed by study stakeholders (formally listed in the grant application) (38%), health system/care practitioners (35%), patients/consumers of health care (27%), and industry (27%). Furthermore, results show that these groups also had the greatest level of involvement across all stages of the research process for RMNI supported research.

In addition, the most common knowledge user or stakeholder groups involved in RMNI research having the greatest level of involvement were also the most common groups who were influenced to some or great extent by the results of the research (Table 10): other researchers/academics (73%), study stakeholders (46%), industry (38%), as well as health system/care practitioners (31%).

Photo: Scanning Tunnelling Microscope – Photo courtesy of National Research Council of Canada.

Commercialization and Research Outcomes

RMNI grants resulted in a sizable proportion of commercialization–related outcomes in the context of CIHR benchmarks. Key achievements in terms of commercialization (Table 11) include the 46% of RMNI grants that produced patents/licenses and the 39% resulting in intellectual property claims. Benchmark data on biomedical research funded under CIHR’s OOGP between 1991 and 2006 reveals that 18% of grants resulted in patents/licenses while intellectual property claims resulted from 13% of OOGP grants. In addition, a large proportion of RMNI grants led to research related outcomes including research findings/knowledge creation (100%), new research method (92%), new theory (50%) and new practice (39%).

Capacity Development

Photo: Images of outgrowing axons from an injured mouse peripheral nerve, growing (from top to bottom) without (control group) or with a local electrical stimulation protocol – Photo courtesy of Dr. Douglas Zochodne, University of Calgary.

CIHR's mandate includes a duty to build the capacity of the Canadian health research community through the development of researchers and the provision of sustained support for scientific careers in health research (Canadian Institutes of Health Research Act, 2000, p.5). CIHR supports capacity development directly through training grants and awards such as the Strategic Training Initiative in Health Research (STIHR) and the Vanier Canada Graduate Scholarships. Capacity development is also supported through funding for research projects that involve students, research staff, and technicians. For RMNI, the development of capacity in the fields of regenerative medicine and nanomedicine is one of the initiative’s key anticipated outcomes.

Involvement of trainees was identified as a key factor for success by RMNI–funded researchers and case study participants. A review of CIHR grant files for the 26 RMNI grants surveyed revealed that over half of RMNI funds expended were used to pay for the salaries of students and non–students.¹² Specifically, 55% of RMNI funds expended from catalyst grants was used to support salaries for these individuals involved compared to 63% from team grants (for a total of 63% across all surveyed grants).

Results presented in Table 12 indicate that RMNI has contributed to the development of research capacity, particularly through its team grant funding mechanism (93% of team grants involved 11 or more staff/trainees). The average number of research staff and trainees involved in RMNI team grants was 27.7 compared to 7.0 for catalyst grants (p<0.05¹³). In contrast, benchmark data on biomedical OOGP grants awarded between 1991 and 2006 reveals that an average of 7.9 research staff and trainees were involved per grant. Furthermore, an average of 6.6 PhD students and 5.3 undergraduate students were involved per RMNI team grant (Table 13).

Multi/Transdisciplinary Research Training

According to CIHR’s 2009–2014 strategic plan (CIHR, 2010, p.13):

Over the next five years, CIHR will sustain a healthy research foundation by:

1. Training, attracting and retaining the best talent in health research;
2. Providing increased focus on trans–sectoral and multidisciplinary training; and
3. Preparing young researchers for non–academic labour markets.

To solicit the opinions of trainees involved in RMNI–funded projects on the value of the multi/transdisciplinary training received as well as the influence that their training has had on career advancement, an online discussion forum was held with a total of 13 trainees representing seven RMNI grants (four team grants and three catalyst grants).

All trainees who participated in the forum indicated satisfaction with the training they received and would recommend multi/transdisciplinary training to others. Most trainees reported that training from multiple mentors exposed them to a wide variety of different perspectives which enriched their research by providing different points of view while a few trainees reported that this type of collaboration improved their ability to communicate their research in ways that were understandable to all members of their team. To further illustrate the type of experiences and skill sets achieved through involvement in multi/transdisciplinary projects, an RMNI catalyst grant NPI offered the following description of the nature of training received by a PhD student in statistics who was involved in their project:

Almost all trainees stated that the single greatest challenge in their research environment was learning new techniques but that this challenge was largely mitigated by the diversity found within their team and the multi/transdisciplinary expertise available on hand. While most trainees remarked that there were no distinct disadvantages to participating in this kind of training and that their experience was valuable, several mentioned that multi/transdisciplinary training isn’t necessarily valued by all employers and that it is sometimes difficult to find employment when industry or academia are looking for candidates whose work is based within one primary discipline. An evaluation of CIHR’s Strategic Training Initiative in Health Research (STIHR) similarly found that interdisciplinary skills were reported by respondents as valued, however, the marketability of these skills was unknown (CIHR, 2008).

RMNI trainees with career pursuits outside of academia suggested that a training model that required the acquisition of skills more suitable to the needs of the workforce including industry, through internships or other models being applied elsewhere (such as laboratory rotations at the start of training, a practice common to graduate programs in the United States), would be a more suitable and effective approach for trainees with similar career aspirations.

In terms of the influence of RMNI training on career paths and opportunities, most trainees mentioned that their career plans did not change as a result of their experiences but several respondents did note that they went on to pursue further involvement in multi/transdisciplinary research as a result of their training. One team grant trainee explained that:

Leveraging of Additional Funding and Support

One indicator used in the assessment of research outcomes for CIHR programs and initiatives is the extent to which funded researchers leverage additional and/or subsequent funding. As part of the evaluation, NPIs were asked whether their RMNI grant had contributed to the attainment of other grants and awards funding. The majority of NPIs surveyed (85%) indicated that their RMNI grant helped them and/or members of their team leverage other funding. Sources of funding obtained, in order of most common, were from the Natural Sciences and Engineering Research Council (NSERC), CIHR, the Canadian Foundation for Innovation (CFI), and provincial government organizations.

Results presented in Table 14 show that RMNI team grant researchers leveraged an average of 4.7 grants and awards versus 3.2 for catalyst projects (p<0.05¹⁴) and received approximately one quarter of grants and awards from CIHR. Of note, contextual factors related to the differences in funding mechanisms may account for differences in leveraging results such as team grants having longer durations and involving more researchers and trainees as compared to catalyst grants (see Tables 12 and 16).

Researchers who participated in interviews noted that RMNI funding facilitated publications and helped build their reputations within the fields of regenerative medicine and nanomedicine which enabled them to secure additional funding. A few researchers mentioned their RMNI grant had helped them establish preliminary results that supported the need for additional research while others explained that RMNI funding helped develop partnerships with industry and other researchers and moved their research into commercialization stages.

RMNI Partnerships and Collaborations

Since its inception, RMNI’s goals and activities have been established through close collaboration and partnership with 11 CIHR Institutes and branches as well as 20 external organizations (see Appendix for full list of those involved). Collaboration with CIHR’s Institutes and branches was intended to reduce overlap and duplication in research funding through the sponsoring of joint funding opportunities.

For example, on RMNI team grant competitions, most funding partners typically contributed less than the average cost of one full grant. The multi–partnered competitions thus allowed all partners to leverage their funding significantly, with most partners identifying two to four relevant grants supported through each funding opportunity (i.e., the multidisciplinary nature of the research appealed to multiple partners). Even in cases where partners were making a significant contribution, it was more efficient from a program delivery perspective to run one large competition rather than several smaller ones. As noted by an RMNI–funded researcher and stakeholder, “ what RMNI did was that they were able to create a call that could pool money so you could get support through RMNI and there would be funding from a whole bunch of different agencies that, on your own [research] project, you would not be able to easily secure at one–time to support one project.”

External partners and stakeholders, such as other Canadian research funding agencies and government departments, provided not only funding to invest into research supported through the initiative but also support (both financial and in–kind) to organize joint workshops, meetings and symposia. As well, Canadian researchers (some funded by RMNI) working in the fields of regenerative medicine and nanomedicine provided expertise and advice in the design of the initiative.

To determine the success of RMNI’s external partnerships and collaborations, interviews were conducted with six individuals associated with RMNI who represented the following types of involvement: funding partners, workshop organizers, and content experts. RMNI partners/stakeholders all described their involvement in RMNI as successful, expressed satisfaction with the relationship and stated their intention to remain engaged with RMNI should the initiative continue.

One factor frequently identified by interviewees as having contributed to the success of their partnership was a positive relationship with the Associate Director of RMNI who was described as flexible, accessible, and collaborative. Other factors mentioned included strong management and administrative support of the initiative, RMNI’s clear mandate as well as having a history of prior partnerships with CIHR.

In terms of benefits received, funding partners reported that their RMNI partnership helped their organization access researchers, leverage their own funding and capitalize on CIHR’s established processes such as peer review. A few funding partners also stated how the partnership allowed their organization to develop relationships with other organizations and helped increase their profile among the research community and the general public. For example, one interviewee whose organization contributed funding through RMNI explained that:

Photo: 2010 RMNI Funding Announcement – Photo courtesy of CIHR.

While funding partners benefited from the partnership, they also noted that it had some risks. For example, there may not be any RMNI applications of interest to their organization or the applications of interest may not be successful in receiving RMNI funding. One of the interviewees commented that it would be helpful if RMNI had provided their organization with regular updates on the projects for which they were contributing funding and that there was a lack of clarity concerning what RMNI’s role or responsibilities were once a grant was awarded.

None of the interviewed partners reported having a similar partnership with other organizations or initiatives but many reported that their organization could benefit from a similar partnership model with other organizations. However, some doubted that the success achieved with RMNI could be accomplished elsewhere. They felt that this type of coordinated relationship required a level of commitment or reciprocity from both sides of the partnership and that not all organizations and/or initiatives are able (or willing) to do so. One interviewee also commented that the scope of RMNI was not something that existed elsewhere.

RMNI Workshops

Approximately $1 Million dollars of RMNI’s total investments was expended on holding a total of 14 workshops and meetings on topics of common interest in regenerative medicine and nanomedicine (see Appendix for full list of workshops held). These events were coordinated by RMNI in conjunction with several CIHR Institutes and branches as well as other government departments and agencies and were intended to bring together experts and stakeholders from different domains, aligned along common themes, to form connections between fields, disciplines, and backgrounds. Some RMNI workshops covered a specific theme and were by invitation only whereas other workshops were open to all interested in attending.

Approximately 33% of RMNI–funded researchers surveyed had attended at least one RMNI workshop. Results presented in Figure 16 indicate that the workshops were most useful in providing learning opportunities and presenting relevant information pertaining to research.

Figure 16 - Usefulness of RMNI Workshops for Researcher Attendees

Source: Survey of RMNI–Funded Researchers (N=24^*)

* RMNI–funded researchers who attended at least one RMNI workshop

Figure 16: long description

The majority of RMNI–funded nominated principal investigators interviewed revealed that they had not attended any of the workshops due to a lack of awareness of the events, not having enough free time to attend, a lack of interest, or someone else from their RMNI team had attended. RMNI nominated principal investigators who had attended described the workshops as beneficial in terms of networking, learning, and sharing ideas with other researchers. A few researchers interviewed described the workshops as not particularly useful to their research – they felt it was difficult to relate to the array of projects, and that workshops were driven by policies or the identification of priorities rather than by research topics.

Results should be interpreted with caution, however, as RMNI workshops varied in terms of purpose and theme and respondents were asked to provide their opinions on the overall usefulness of the event(s) they had attended in facilitating a number of outcomes, some of which may be more applicable than others depending on the nature of the workshop held.

RMNI partners/stakeholders interviewed who had been involved in the workshops indicated that the events helped open the dialogue between the different funding agencies and players involved in regenerative medicine and nanomedicine. As a result, interviewees indicated that there was increased coordination and consolidation of resources and policies/guidelines. This was also described as something that was new and unique to RMNI. For example, an RMNI partner/stakeholder stated:

When asked how RMNI could be improved, several NPIs interviewed commented that the workshops should be promoted more and both researchers and partners suggested holding an annual meeting for funded RMNI researchers with the aim to make connections between research teams and exchange information about research being conducted, including the management of research.

RMNI Design and Delivery

Researcher Satisfaction with RMNI Peer Review

Levels of satisfaction with the peer review process provide a researcher perspective on the efficacy of RMNI program delivery. Overall, RMNI–funded researchers¹⁵ reported a consistently high level of satisfaction and low level of dissatisfaction across five main aspects of the peer review process (Figure 17).

To situate the findings within the wider CIHR context, a comparison with results obtained through the 2011 CIHR International Review (IR) survey was undertaken. To ensure respondents were matched as closely as possible, a sample of IR respondents was selected based on the following criteria: respondents had indicated at least a biomedical (Pillar 1) research focus, were successful in their application for CIHR funding over the past five years, and whose application(s) for funding included team and/or catalyst grants. Of note, IR survey respondents were asked to respond with reference to all CIHR programs to which they applied. As a result, they could have submitted applications to other programs in addition to team and/or catalyst grants and therefore their opinions may not be directly attributable to either type of funding mechanism.

Results of the benchmark comparison (Figure 17) reveal that, apart from one aspect of the process, just over half of IR respondents reported satisfaction with CIHR peer review. Although results show a greater proportion indicated satisfaction, at least one third of IR respondents however were dissatisfied with each aspect of the process. In contrast, few RMNI–funded researchers were negative in their assessments of peer review and this finding provides further indication that the RMNI peer review process has been effectively delivered.

Figure 17 – RMNI-Funded Researchers’ Satisfaction with Peer Review

Source: Survey of RMNI–Funded Researchers (N=72); 2011 CIHR International Review Survey (N=152)

Figure 17: long description

Figure 18 – Suggested Improvements to Peer Review

• For team grants, focus more attention on the funding history of an applicant as well as the project management elements (including the facilitation of team collaboration)
• Limit the number of applications a single researcher can submit to RMNI
• Make the instructions clear and concise

Source: Interviews with RMNI–Funded Researchers (N=23)

Generally, interviewed RMNI–funded researchers described the peer review process used to assess their application as satisfactory. Notwithstanding their overall satisfaction, interviewed researchers did express some concerns about certain aspects of the process, in particular the peer reviewers themselves. For instance, some felt that some reviewers may not always have a full understanding of the proposed project or noted that the peer reviewers' judgments were not consistent from one reviewer to the next. When asked about possible improvements, a few researchers suggested that the team grant review process focus more attention on an applicant’s funding history while one recommended a greater focus be placed on project management elements of team grant proposals (Figure 18).

Challenges in Reviewing RMNI Applications

To help determine whether there were challenges in the effective delivery of the RMNI peer review process in terms of reviewing multi/transdisciplinary and high risk grant applications, an interview was conducted with one of two RMNI peer review Committee Chairs and one of three RMNI Deputy Directors. Of note, the role of a CIHR Committee Chair is to provide oversight with respect to the review process; however the Chair does not provide ratings for applications. The responsibilities of a CIHR Deputy Director include directing the delivery of competition and scientific peer review services for an assigned set of CIHR funding programs.

In terms of challenges with peer review, the RMNI Chair noted that high risk grant applications submitted by new independent researchers without a prior history of grant funding were likely to be rated and ranked lower by the peer review committee and speculated that this was an issue that extended beyond RMNI to CIHR overall. Another challenge identified by the Chair concerns the effect that a lack of familiarity of committee members with smaller, emerging areas such as regenerative medicine and nanomedicine can have on the review process. According to the RMNI Chair, committee members who are less familiar with newer research areas and who are negative towards their view of an application tend to weight the assessments of the external reviewers¹⁶ less than those of the internal reviewers¹⁷. The RMNI Chair also suggested that if committees are reviewing multidisciplinary applications, the committee should reflect the multidisciplinarity of the types of applications they receive, and noted that the RMNI committee did not reflect this at the time he/she served as Chair.

Although recruiting multidisciplinary peer reviewers was noted as a challenge by the RMNI Deputy Director and that matching the expertise of a group of typically three peer reviewers (two internal reviewers and a reader¹⁸) to a grant application involving multiple researchers working in different areas is difficult, this was mitigated to the extent possible by obtaining complementary expertise through the use of reviewers with multidisciplinary backgrounds along with specialists and researchers residing outside of the country (who could provide a different perspective to a review). Furthermore, researchers funded by RMNI were recruited as reviewers to ensure that multidisciplinarity was reflected in the committee. As well, the use of RMNI–funded researchers as reviewers enabled assessments of not only the science of grant proposals but also the alignment of the proposed research to the objectives of the funding opportunity, the novelty of the research and the need for the research in Canada.

In terms of improvements to the peer review process, the RMNI Deputy Director suggested that, irrespective of feasibility and burden, an increase in the number of internal reviewers assigned to multidisciplinary team grant applications (up to five reviewers in total) would help ensure that sufficient expertise is available to cover the variety of disciplines involved.

RMNI Alternative Delivery Mechanisms

Considering that all RMNI–funded researchers interviewed felt that their projects were successful and that the vast majority of researchers were satisfied with the trans/multidisciplinary nature of their research and the RMNI peer review process (at least 74% reported satisfaction across various aspects), few researchers had suggestions for how the initiative could be changed or replaced.

Photo: Operating Theatre with Integrated Femtosecond Laser System – Photo courtesy of Dr. Isabelle Brunette, Maisonneuve-Rosemont Hospital.

Furthermore, several RMNI case study researchers noted that a particular strength of the initiative’s design is that applicants are not required to engage in private sector or industry partnerships as a condition of funding. One RMNI case study researcher noted that while Canada has created the infrastructure that allows commercialization to occur, there are few Canadian R&D companies to partner with due to the R&D branches of many organizations being located outside of the country. Furthermore, they noted that a considerable time commitment may be required to negotiate issues such as intellectual property rights.

The small number of interview participants (RMNI researchers and partners) who did provide some critical opinions about RMNI’s design commented that the targeted research areas of regenerative medicine and nanomedicine may be too broad in scope and questioned why both fields were chosen as priority areas for the initiative. Similarly, a few interviewees felt that RMNI lacked definition and that the initiative needed a short–term and long–term vision rather than trying to support all types of research such as high–risk, clinical and translational research.

In terms of suggestions for improvements to RMNI, funded researchers recommended additional support for research through increasing the funding envelope for the initiative, the success rate for competitions, and the number of funding competitions held. Of particular concern to RMNI researchers, particularly those involved in teams, was the need for continued funding. As one funded team grant researcher explained: “The prospects for me are that in two years our RMNI funding will expire and the year after that, the Stem Cell Network […] will also run to its conclusion. So, in three years from now, it’s quite possible that all of the networking that we’ve established may unravel without some sort of continued new funding to maintain it and to get to the next level.”

Of a total of 77 grants awarded, only one project was successful in receiving renewal funding from RMNI. To address this issue, several researchers suggested that funding be either continued through RMNI or through alternative programs with some suggesting the creation of a small scale funding program to keep RMNI projects moving forward and/or to support RMNI research that was completed by a single researcher until their next large grant.

Photo: Image of Macrophages Ingesting Quantum Dots – Photo courtesy of Prof. Warren Chan, University of Toronto.

A few funded researchers also noted that for some projects, results will not be available for a few years after the grant has ended. This was particularly salient for Catalyst Grant projects as these may require additional funding before evidence–based grants can be received from CIHR or other agencies/foundations. As one catalyst grant NPI noted: “When we have ideas which are new, we don’t have a cumulative result from a few years; it’s very difficult to get an operating grant from CIHR. So that RMNI [catalyst] grant is suitable to fund new ideas with novelty but not much results.”

Several researchers and a few RMNI partners suggested that CIHR increase the amount of support available to researchers through the development of partnerships with federal and provincial funding agencies such as NSERC and the Fonds de recherche du Québec Santé (FRSQ), other countries such as the United States where the California Institute for Regenerative Medicine (CRIM) co–funds international collaboration and Industry.

A few interviewed researchers commented about the possibility of using the CIHR Open Operating Grants Program (OOGP) as an alternative to RMNI. Generally, they felt that this would not be a viable alternative given the typical Operating Grants' size, limited funding envelope/low success rate, and scope (does not fund the same sort of non–conventional research). While it was not considered an alternative by RMNI–funded researchers, CIHR Operating Grant funding was described as complementary and the first step to developing research ideas and applying for larger team grants such as those offered through the initiative.

In terms of the possibility of having RMNI grant applications reviewed through relevant standing peer review committees under CIHR’s OOGP program in place of a separate committee for the initiative, the RMNI Deputy Director interviewed noted that for strategic initiative funding opportunities, peer reviewers are recruited based on the needs of the applications received in order to better match reviewer expertise to a proposal. In contrast, peer review committees used for the OOGP are standing panels held biannually and the Deputy Director stressed that the degree of variation in the types of RMNI applications received would require a reviewer recruitment period much longer than the six month duration between OOGP competitions. Furthermore, the RMNI Deputy Director indicated that the multidisciplinarity of RMNI applications renders them unsuitable for review by OOGP panels where biomedical research proposals submitted tend to be unidisciplinary.

Need for RMNI

In 2008, Canada’s Science, Technology and Innovation Council (STIC), with an endorsement by the Minister of Industry, recommended regenerative medicine as a Canadian health and life sciences sub–priority within the Government of Canada's Science and Technology Strategy (Science, Technology and Innovation Council, 2011). Furthermore, the CIHR 2009–2014 strategic plan (CIHR, 2010, p.16) states:

Photo: Image of Cells Growing on a Polymer Sphere – Photo courtesy of Dr. Christopher Yip, University of Toronto.

Through its partnerships and support of multi/transdisciplinary teams and high risk, high impact research projects at the interface of health and technology in the fields of regenerative medicine and nanomedicine, the objectives of RMNI are in alignment with Canadian and CIHR health research priorities. Additionally, the most recent federal budgets affirm the government’s commitment, and the role of the Canadian Institutes of Health Research, in supporting advanced research (Government of Canada, 2012, p.268) and health research of national importance (Government of Canada, 2011, p.153–154).

Publication Growth in Regenerative Medicine and Nanomedicine

Although regenerative medicine and nanomedicine were fields emerging at the time of RMNI’s implementation, bibilometric results presented earlier (Figures 1 and 2) provide evidence that these fields have grown significantly in terms of publication output since the initiative’s inception. At a macro level, global publications in regenerative medicine and nanomedicine (combined) have grown from an annual rate of 3,381 in 2002 to 17,905 in 2010 – a growth of 430%. A similar picture can be seen in Canada where publications in regenerative medicine and nanomedicine have grown from 100 in both fields in 2002 to 628 in 2010 – a growth of 530%. Growth rates for Canadian publication volume correlates highly with the rest of the world in both fields however its specialization still falls below the world average.

As revealed through the bibliometric analysis, RMNI–funded researchers have appeared as authors on approximately 34% of Canada’s total publication output in regenerative medicine and 21% in nanomedicine over the period of 2004–2010. Furthermore, publications from RMNI–funded researchers resulted in higher citation impact scores as compared to Canadian averages in both fields. If the initiative is not renewed and no alternative funding mechanisms are available to continue supporting research in these fields, its absence may have a considerable effect on Canada’s competitiveness worldwide, especially with another Canadian strategic funding initiative, the Stem Cell Network, reaching the end of its funding cycle in 2015.

Alternative Sources of Funding for RMNI Researchers

Nearly all RMNI–funded researchers and case study participants interviewed noted that the initiative appears to be filling a void and that their project would not have achieved the same level of success through other sources of funding or initiatives. In particular, researchers felt that currently, there is no or limited funding in Canada for research teams, high–risk or early stage research, and projects at the interface between technology and health.

As one RMNI team grant NPI noted:

Furthermore, one catalyst NPI stressed that “the type of research that we do is to make tools for imaging; they are very valuable tools (hundreds of requests for them each year). Yet, I have very little chance of being successful in a regular CIHR operating grant competition because we are not driven by biological questions; we are driven by the creation of tools to help other researchers.”

When asked whether they could obtain alternative sources of funding for their research in the absence of RMNI, 42% of funded researchers surveyed felt they could do so in Canada and half indicated that they would be able to sustain their research program if the initiative is discontinued (Figure 19).

An analysis of subsequent funding found that, after having received their RMNI grant, 35 NPIs (52%) went on to receive OOGP grants (as a nominated principal investigator) and seven (10%) were awarded Canada Research Chairs. Furthermore, the Collaborative Health Research Projects Program (CHRP) has subsequently funded nine (13%) RMNI nominated principal investigators, and there has been a $15M investment into the Centre for Commercialization of Regenerative Medicine (CCRM), which includes RMNI–funded researchers among its lead scientists and advisory group.

Although a sizable percentage of researchers indicated that they could obtain support and maintain their research if RMNI is not renewed (evaluation results show that half of RMNI–funded NPIs go on to receive CIHR operating funds), researchers and partners expressed concern that RMNI’s absence would result in the slowdown of research in both fields, a reduction in the amount of new technology produced within Canada as well as the disbanding or reduction of research teams supported including the trans/multidisciplinary linkages established.

Furthermore, RMNI funding partners expressed concern that the absence of the initiative would result in a loss of research in their field and would reduce the ability of their organization to leverage resources. Funding partners also expressed concern that the absence of RMNI would indicate to their stakeholders that research in regenerative medicine and nanomedicine is no longer perceived as important.

Figure 19 – RMNI-Funded Researchers’ Opinions on the Impact of the Absence of RMNI

Source: Survey of RMNI–Funded Researchers (N=72)

Figure 19: long description

Conclusions and Recommendations

Conclusions

RMNI has made significant contributions to building Canadian research capacity and knowledge creation in the fields of regenerative medicine and nanomedicine. Over the period of 2004–2010, researchers supported through the initiative authored close to one–third of Canadian publications in both fields, produced more publications than unsuccessful applicants, achieved higher citation impact scores as compared to Canadian averages and achieved higher international collaboration rates than the world average in each field. At a national level, Canada ranked in the middle of the top 16 productive countries in each field in terms of papers produced and achieved higher citation impact scores than the world average. However, Canada’s specialization in regenerative medicine and nanomedicine was below the world average over the period of RMNI’s lifecycle (2004–2010).

RMNI funding also enabled the support of a considerable number of research staff and trainees. Furthermore, trainees report satisfaction with their involvement in multidisciplinary research. However, they expressed concern over the extent of the demand for individuals who work in more than one research field. RMNI funding also facilitated the conduct of multidisciplinary research collaboration that funded researchers felt was highly satisfactory and valuable and RMNI grants resulted in a sizable proportion of commercialization–related outcomes in context of CIHR benchmarks. Furthermore, a high percentage of RMNI–funded research involved and influenced stakeholders including other researchers/academics, industry and health system/care practitioners.

The program has been effectively designed and delivered as reflected by the level of satisfaction reported by funded researchers and partners/stakeholders. Funded researchers report a high level of satisfaction with the RMNI peer review process in light of CIHR benchmarks and partners and stakeholders viewed their participation in the initiative as successful and expressed an intention to remain involved if RMNI were to continue. Researchers had varying opinions on the usefulness of RMNI workshops; however partners and stakeholders felt the events were successful in facilitating collaboration and communication between funding agencies and key organizations involved in regenerative medicine and nanomedicine. The objectives of RMNI are in alignment with the Government of Canada's Science and Technology Strategy and CIHR’s 2009–2014 strategic plan and recent federal budgets continue to affirm the government’s commitment and role of CIHR in supporting advanced research and health research of national importance. Finally, the majority of researchers and partners/stakeholders indicated that the design of the initiative is appropriate and successful.

The discontinuation of the initiative raises questions around the sustainability of research in these fields. Evidence from this evaluation is encouraging in that it shows a record of success among RMNI–funded researchers in leveraging other existing grants and awards. There may also be further opportunities for strategic funding for researchers in regenerative medicine and nanomedicine, for example through CIHR’s significant investments in networks for the Strategy on Patient–Oriented Research (SPOR) and the Epigenetics Strategic Initiative as well as funding for large scale research projects under the Personalized Medicine Initiative.

However, in the absence of RMNI funding, it will be important for CIHR to monitor Canada’s competitiveness in regenerative medicine and nanomedicine. If the country’s performance declines, CIHR should assess the health of both fields in Canada through an examination of Canadian investment in this type of research as well as tracking the subsequent careers of RMNI principal investigators and trainees. CIHR should also offer direction to the regenerative medicine and nanomedicine research community on applying to other CIHR funding opportunities and initiatives providing support in these fields.

Recommendations

1. Implement a communication strategy aimed at researchers working in the fields of regenerative medicine and nanomedicine that offers direction on applying to other CIHR funding opportunities and initiatives providing support in these fields.
2. Conduct regular assessments with international benchmarks to determine the relative global position of Canada in the fields of regenerative medicine and nanomedicine in the absence of RMNI. If Canada’s competitiveness declines, ensure regular environmental scanning takes place to assess the ongoing health of both fields in Canada. This would include examining the Canadian investment in these research areas and tracking the subsequent careers of RMNI principal investigators and trainees.
3. Ensure that future designs of programs relating to teams of researchers or networks take into account findings from the evaluation. It may, for example, be unrealistic to design programs that are expected to fund ‘newly formed’ teams, and there could be merit in including stronger requirements for dedicated research coordinators to aid success.

Management Response

Methodology

Consistent with Treasury Board policy and recognized best practices in evaluation¹⁹, a range of methods – involving both quantitative and qualitative evidence – were used to triangulate evaluation findings. This was to ensure that the evaluation findings were robust and credible and that conclusions drawn about program performance are valid.

Bibliometrics: RMNI

A bibliometric analysis was conducted by Observatoire des sciences et des technologies (OST) on articles published between 1997–2010 in regenerative medicine and nanomedicine by 295 RMNI–funded researchers, 143 non–funded RMNI applicants – both groups including NPIs and co–grantees – as well as researchers worldwide. The bibliometric data presented in this study was drawn from the Canadian Bibliometric Database (CBD^TM) built by the OST using Thomson Reuters’ Web of Science (WoS). Given that Thomson’s databases do not have a subject classification at the level of individual papers, OST used the U.S. National Library of Medicine Medical Subject Headings (MeSH), which relies on a controlled vocabulary to assign a medical domain to each paper indexed in the PubMed database.²⁰ The following search queries were used to retrieve papers in Regenerative Medicine and Nanomedicine:

Regenerative Medicine MeSH Headings

Adult Stem Cells
Bioartificial Organs
Embryonic Stem Cells
Fetal Stem Cells
Liver, Artificial
Multipotent Stem Cells
Organoids
Pancreas, Artificial
Pluripotent Stem Cells
Regenerative Medicine
Skin, Artificial
Stem Cell Transplantation
Tissue Engineering
Tissue Scaffolds
Tissue Therapy
Totipotent Stem Cells

Regenerative Medicine Journals

Artificial organs
Biomaterials artificial cells and artificial organs
Biomaterials artificial cells and immobilization biotechnology
Biomaterials medical devices and artificial organs
Cell stem cell
Cloning and stem cells
International journal of artificial organs
Journal of artificial organs
Journal of tissue engineering and regenerative medicine
Neural regeneration research
Regenerative medicine
Stem cell reviews
Stem cells
Stem cells and development
Tissue engineering
Tissue engineering and regenerative medicine
Tissue engineering part a
Tissue engineering part b–reviews
Tissue engineering part c–methods
Transactions american society for artificial internal organs
Wound repair and regeneration

Nanomedicine MeSH Headings

Fullerenes
Lab-On-A-Chip Devices
Microfluidic Analytical Techniques
Microfluidics
Nanocapsules
Nanomedicine
Nanostructures
Nanotechnology

Nanomedicine Journals

Digest journal of nanomaterials and biostructures
IEE proceedings–nanobiotechnology
IEEE transactions on nanobioscience
IET nanobiotechnology
International journal of nanomedicine
Journal of biomedical nanotechnology
Microfluidics and nanofluidics
Nanobiology
Nanomedicine
Nanomedicine–nanotechnology biology and medicine
Nanotoxicology
Nature nanotechnology

The query using MeSH headings and journal articles retrieved 51,953 Regenerative Medicine papers while the Nanomedicine query resulted in a total of 48,382 papers retrieved.²¹ The total set of regenerative medicine and nanomedicine papers comprised 0.7% of all Canadian health research papers. Furthermore, 8% of all papers authored by RMNI researchers (funded and non–funded) were captured through the search query.

Indicators

Number of publications: The number of scientific papers with authors from a country, as found in authors’ addresses.

Average of Relative Citations (ARC): Based on the number of citations received by papers over a two–year period following publication year. Thus, for papers published in 2000, citations received between 2000 and 2002 are counted. Self–citations are excluded. Citations are counted until the end of 2010, which means that papers from 2009 have an incomplete citation window of only one year. The number of citations received by each paper is normalized by the average number of citations received by all papers of the same specialty — as defined by US National Science Foundation classification of journals²² — and research domain, hence taking into account the fact that citations practices are different for each specialty and domain. When the ARC is greater than 1, it means that a paper or a group of papers scores better than the world average of its specialty and domain; when it is below 1, those publications are not cited as often as the world average.

Specialization index (SI): The relative intensity of publication of a country in the areas of nanomedicine and regenerative medicine relative to the intensity of the world in the area. An SI value above 1 means that a country is specialized in the research area compared to the world average, while an index value below 1 means the opposite.

International collaboration rate: The relative intensity of scientific collaboration between countries. The rate is calculated by dividing the number of papers with at least one author with a foreign country address by the country’s total number of papers. A country’s international collaboration rate is generally determined by its size, i.e. larger countries collaborate less that smaller ones.

Network analysis: In order to visualize the collaborative ties between institutions or countries active in the research area, a network analysis was performed using the UCINET and Netdraw softwares. These softwares allow the creation of 2–dimensional networks of co–authored papers. The size of the edges (lines) between each of the nodes is determined by the number of co–authored papers between the two entities. A threshold of numbers of papers written in collaboration is fixed in each of the figures in order for the network to be clearer.

Bibliometrics: OOGP

Benchmark data on average of relative citation scores (ARC) was obtained from a bibliometric study conducted for the 2012 evaluation of CIHR’s Open Operating Grants Program by the Observatoire des sciences et des technologies (OST). The analysis included 20,287 supported papers²³ published over the period of 2001–2009 by a randomly selected sample of N=1,125 researchers (from a population of 3,236) who received OOGP funding between 2000 and 2007. Citation impact data was also captured on all Canadian health research publications produced over the period of 2001–2009.

Quantitative Survey: RMNI–Funded Researchers

An online survey was conducted by EKOS Research Associates Inc. on a sample of RMNI–funded researchers who were awarded funding over the period of 2004–2009 that provided quantitative data used to address a range of evaluation issues and questions. Where appropriate, survey questions were matched to those found in both the 2011 CIHR International Review survey as well as CIHR’s Research Reporting System, an end of grant survey, to allow for benchmarking. A total of N=295 researchers were identified as having been funded by the initiative. Researchers only funded by RMNI in 2010 (N=18) were excluded due to these grants having just started at time of survey. Furthermore, catalyst grant co–grantees (co–principal investigators, co–investigators, and collaborators) appearing on grants with less than three researchers (N=13) were excluded as they were not eligible to respond to research team collaboration questions. Finally, two deceased NPIs, and four co–grantees involved in an RMNI grant with one recently deceased NPI were removed and N=8 NPIs and one co–grantee declined participation. The remaining population of funded researchers received an invitation to participate with a total of 72 respondents (26 NPIs and 46 co–grantees) completing the questionnaire.

Quantitative Survey: OOGP Research Reporting System

Data were drawn from reports submitted by biomedical researchers funded under the Open Operating Grants Program (OOGP) using CIHR’s end of grant Research Reporting System (RRS); a 2009 pilot study of the RRS that targeted grantees whose authorization to use funds expired between January 2000 and June 2008 (N=457); and data on publication outputs from the full launch of the RRS in 2011 (N=104 responses were included, all submitted by February 2, 2012). Before combining the data between the two RRS data sources, validation of responses was conducted, including checking for differences in demographic profile between respondents to the pilot and the full survey. No significant issues were identified in this regard.

Quantitative Survey: CIHR 2011 International Review

Data was analyzed from an online survey conducted by Ipsos Reid between November 5th and December 5th 2010 as part of CIHR’s 10th Year International Review to examine satisfaction with the peer review process among researchers. To help ensure respondents were matched as closely as possible to researchers supported by RMNI, a sub–sample of IR respondents (N=152) were selected by the Evaluation team based on the following criteria: respondents had indicated at least a biomedical (Pillar 1) research focus, were successful in their application for CIHR funding over the past five years and whose application(s) for funding included team and/or catalyst grants.

Document Review

This covered a review of relevant CIHR and Government of Canada documents as well as health research related reports. It also included previous evaluations and studies conducted by CIHR and its Institutes.

Discussion Forum

An online discussion forum was hosted through Interactive Tracking Systems Inc. (itracks) with a total of N=13 RMNI trainees. The forum took place over a period of five days with participants logging in to the forum once or twice a day to respond to moderator questions and engage in discussion with others. Key discussion areas included the value of training and mentoring received through RMNI, the value of a trans/multidisciplinary training environment, and career advancements.

Key Informant Interviews

Semi–structured in–depth qualitative interviews were conducted by R.A. Malatest & Associates Ltd. with a stratified random sample of RMNI–funded researchers (N=23) and purposive sample of N=6 partners and stakeholders (selected based on type and extent of involvement with RMNI) to provide evidence to address a number of the evaluation issues and questions. Two additional interviews were conducted by the CIHR evaluation team with an RMNI peer review Committee Chair and Deputy Director to address issues with assessing multi/transdisciplinary and high risk, high impact RMNI grant applications.

Case Studies

Based on discussion with RMNI management, six case studies of RMNI–funded projects demonstrating high impact breakthrough research were purposively sampled to provide detailed qualitative narratives of highly impactful research outcomes to address several evaluation issues and questions. Key informant interviews were conducted by R.A. Malatest & Associates Ltd. with a total of N=29 participants: RMNI principal investigators and researchers, students, stakeholders and knowledge users. Due to length considerations, three profiles of RMNI projects are presented in this report and were selected for inclusion based on their illustration of a multitude of varying high impact results.

Administrative Data Analysis

The CIHR Electronic Information System (EIS) is designed to collect and store data on all applicants to CIHR programs. Data obtained from EIS, along with RMNI program records, were used in the analyses of administrative and financial information pertaining to RMNI, its funded grants as well as an analysis of subsequent CIHR grants held by RMNI–funded researchers, for example.

Limitations

In keeping with best practices in program evaluation, the limitations of this study are noted below, together with the strategies that were employed to mitigate them.

Bibliometrics: RMNI

Bibliometric analysis has been criticized on the grounds that estimates of publication quality based on citations can be misleading and that citation practices differ across disciplines and sometimes between sub–fields in the same discipline (Ismail et al., 2009). This is a particularly salient issue for RMNI, with an objective to fund multi/transdisciplinary research in the fields of regenerative medicine and nanomedicine. To mitigate this, measures of other outputs are also used in this evaluation to assess knowledge creation as a result of the initiative. A case study approach is also taken to assess highly impactful research conducted as a result of RMNI funding.

Secondly, it should be noted that the bibliometric analysis in this evaluation included data for regenerative medicine and nanomedicine publications produced by funded RMNI researchers at some point after the initiative’s inception. Although the period of publication used throughout the bibilometric analysis aligns with the overall lifecycle of RMNI (2004–2010²⁴), funded researchers may have published articles in either field prior to, or after having concluded, their RMNI grant. As a result, direct attribution between RMNI funding and publication data cannot be made. With further development of CIHR’s Research Reporting System, where researchers list publications produced as a result of the grant that can then be linked directly to bibliometric data, this type of analysis should become available for future CIHR evaluations.

Bibliometrics: OOGP

The OOGP bibliometric analyses used to provide benchmark data for the evaluation is based on data for publications produced by OOGP researchers (NPIs) while supported by these grants. While this method is commonly accepted based on an assumption that these grants are a significant contribution to research output (e.g., Campbell et al, 2010), an outright attribution between grant and publication bibliometric data cannot be made. Also, due to budget limitations, a sample of OOGP researchers funded at least one year between 2000 and 2007 (N=1,125) was randomly selected from a population of 3,236 for analysis rather than selecting all researchers within that group. The total sample size was adequate for the analysis, and there is no reason to expect that the universe of all funded researchers would be different from the selected sample.

Quantitative Survey: RMNI–Funded Researchers

The foremost limitation with regards to the use of a survey methodology is that it relies largely on self–reported data and/or memory recall from respondents. Furthermore, RMNI awarded a total of 77 grants and although ones awarded in 2010 were excluded from the survey, data was captured on a relatively small number of grants (26 in total) which limits the ability to generalize results to the initiative overall as well as within its two funding mechanisms. Furthermore, the relatively low number of grants and researchers sampled prevents the ability to make statistical conclusions including observed differences between RMNI results and benchmark comparison data.

Quantitative Survey: OOGP Research Reporting System (RRS)

Data collection in the RRS end of grant survey ‘Pilot study’ involving OOGP funded researchers was halted before the fourth wave of invitations were sent out. Similarly, OOGP researchers responding to the current version of the RRS have until October of 2012 to complete their report, meaning that a full sample was not available. Among the completed reports, data quality checks are still ongoing and only the responses related to knowledge creation were available to be included in the analyses. Also, in relation to estimating the numbers trained/supported by OOGP, there could be double counting since trainees could be involved in multiple projects with different nominated principal investigators.

To mitigate against the possibility that the OOGP RRS samples may not be representative of the overall population of OOGP researchers, a comparison of demographic variables of the two RRS sets of data with the OOGP population was conducted. This suggested that the two incomplete samples were broadly representative of the overall universe of researchers. The variables compared were: pillar, language and region, with differences between the samples and the population of around 5%.

Quantitative Survey: CIHR 2011 International Review

To help ensure respondents to the 2011 CIHR International Review survey were matched as closely as possible to the sample of RMNI survey respondents, an IR sample was selected based on the following criteria: respondents had indicated at least a biomedical (Pillar 1) research focus, were successful in their application for CIHR funding over the past five years and whose application(s) for funding included team and/or catalyst grants. As IR survey respondents were asked to respond with reference to all CIHR programs to which they applied to, they could have submitted applications to other programs in addition to team and/or catalyst grants and therefore their opinions may not be directly attributable to either type of funding mechanism.

Case Studies

The sampling of projects to be included in the case study analysis was purposive with only exemplary cases being selected. Also, only a small number of projects were selected due to budget and timing constraints. As with all qualitative data, these findings are not generalizable to a wider population but are used instead for illustrative purposes only.

Appendix

RMNI Partner and Stakeholder Organizations

RMNI has partnered with a number of CIHR Institutes, branches, and divisions on joint funding programs and collaborative workshops, meetings and symposia, including:

• Institute of Neurosciences, Mental Health and Addiction (RMNI co–lead)
• Institute of Genetics (RMNI co–lead)
• Institute of Musculoskeletal Health & Arthritis (RMNI co–lead)
• Institute of Aboriginal Peoples' Health
• Institute of Aging
• Institute of Cancer Research
• Institute of Circulatory and Respiratory Health
• Institute of Infection and Immunity
• Institute of Musculoskeletal Health and Arthritis
• CIHR Ethics Office
• CIHR Knowledge Translation Branch

RMNI also has served as a nucleus for engaging external partners. A number of voluntary health organizations (VHOs), non–governmental organizations (NGOs), government agencies, and Networks of Centres of Excellence (NCEs) have joined in sponsoring joint funding opportunities, including:

• ALS Society of Canada
• Canadian Space Agency
• Canadian Stroke Network
• Foundation Fighting Blindness
• Heart and Stroke Foundation of Canada
• Jacob's Ladder
• Juvenile Diabetes Research Foundation International
• Neuroscience Canada
• Ontario Neurotrauma Foundation
• Stem Cell Network

In addition to funding research, RMNI has worked closely with a large number of government departments and agencies to sponsor workshops and meetings on topics of common interest. CIHR's partners in these planning and development activities include:

• Alberta Heritage Foundation for Medical Research
• Alberta Ingenuity
• Canadian Federation of Biological Societies
• Environment Canada
• European Research Area and Canada Initiative
• Health Canada
• Industry Canada
• National Research Council Canada – National Institute for Nanotechnology
• Natural Sciences and Engineering Research Council of Canada (NSERC)
• Social Sciences and Humanities Research Council of Canada (SSHRC)

RMNI is also part of CIHR's contribution to the Cancer Stem Cell Consortium (CSCC). The CSCC is designed to enhance research into cancer stem cells, and its partners include the Ontario Institute for Cancer Research, Genome Canada, the Canadian Foundation for Innovation (CFI), the Stem Cell Network, and research organizations in Canada and California.

Regenerative Medicine and Nanomedicine Initiative (RMNI) Logic Model

Regenerative Medicine and Nanomedicine Initiative (RMNI) Logic Model: long description

Profile of RMNI–Funded Research and Researchers

Survey results (Table 16) show that researchers and research supported through RMNI were mainly biomedical (CIHR pillar 1) and RMNI researchers funded under either mechanism tended to be mid to senior investigators. In terms of dollar committment, most catalyst grants tended to fall within the $150–300K range (83%) while team grants were mainly valued at $1.0–1.5M. Catalyst grants mainly involved one to three researchers at time of application while number of researchers involved in team grants ranged from 4–6 (57%) and 10–13 (21%).

References

Campbell D., Picard–Aitken M., Côté G. et al. (2010). Bibliometrics as a performance measurement tool for research evaluation: The case of research funded by the National Cancer Institute of Canada. American Journal of Evaluation, 31; 66–83.

Canadian Institutes of Health Research Act, S.C. c. 6 (2000) [ PDF (260 KB) - external link ].

CIHR (2010). Health Research Roadmap: Creating innovative research for better health and health care.

CIHR (2008). Strategic Training Initiative in Health Research (STIHR) 2001–2006: Evaluation report.

Government of Canada (2012). Jobs, growth and long–term prosperity: Economic Action Plan 2012, The Budget Speech [ PDF (4.84 MB) - external link ].

Government of Canada (2011). The Next Phase of Canada’s Economic Action Plan: Low Tax Plan for Jobs and Growth [ PDF (4.84 MB) - external link ].

Ismail S., Nason E., Marjanovic S. & Grant J. (2009). Bibliometrics as a tool for supporting prospective R&D decision–making in the health sciences: Strengths, weaknesses and options for future development. Rand Corporation, Santa Monica, CA.

Kessel, F., Rosenfeld, P.L., & Anderson, N.B. (Eds.) (2008). Interdisciplinary research: Case studies from health and social sciences. Oxford. University Press.

Marjanovic, S., & Chonaill, S. N. (2010). Health and medical research in Singapore: Observatory on health research systems [ PDF (1.25 MB) - external link ].

Observatoire des sciences et des technologies (2010). Bibliometric analysis of INMHA–related research, 1997–2008.

Science, Technology and Innovation Council (2011). State of the nation 2010: Canada’s science, technology and innovation system.

Thomson Reuters (2011). A bibliometric analysis of regenerative medicine [ PDF (472.5 KB) - external link ].

U.S. National Library of Medicine (2011). Fact sheet: Medical subject headings.

Evaluation Unit
Resource, Planning and Management Portfolio
Canadian Institutes of Health Research (CIHR)

160 Elgin Street, 8th Floor
Address Locator 4809A
Ottawa, Ontario
Canada K1A 0W9

Phone: 1–613–941–2672
Toll Free: 1–888–603–4178
Fax: 1–613–954–1800
Web: Canadian Institutes of Health Research

Date modified:

2013-09-27