Canadian Longitudinal Study on Aging Program Evaluation
Final Report

April 2020

At the Canadian Institutes of Health Research (CIHR), we know that research has the power to change lives. As Canada's health research investment agency, we collaborate with partners and researchers to support the discoveries and innovations that improve our health and strengthen our health care system.

Canadian Institutes of Health Research
160 Elgin Street, 9th Floor
Address Locator 4809A
Ottawa, Ontario K1A 0W9

This publication was produced by the Canadian Institutes of Health Research. The views expressed herein do not necessarily reflect those of the Canadian Institutes of Health Research.

Acknowledgements

Special thanks to all participants in this evaluation – survey respondents, and key informant interview participants and Longitudinal Study Experts. Also, thank you to those who supported the evaluation: Janice Remai, Karen Croteau and Marianne Coriveau (Goss Gilroy Inc.), the Institute Teams and Scientific Co-Leads from the Institutes of Aging (IA) Dr. Jane Rylett, and Population and Public Health (IPPH) Dr. Steven Hoffman as well as Dr. Yves Joannette, the former Institute of Aging Scientific Director, who completed his term in July 2019.

The Canadian Longitudinal Study on Aging Evaluation Team

Jean-Christian Maillet, Alison Croke, Jayme Stewart, Angel Mackenzie, Michael Goodyer, and Ian Raskin.

For more information and to obtain copies, please contact: Evaluation@cihr-irsc.gc.ca.

Table of Contents

• List of Figures
• List of Acronyms
• Executive Summary
  • Program Overview
  • Evaluation Objective, Scope and Methodology
  • Key Findings
  • Recommendations
• Program Profile
• Description of Evaluation
  • Evaluation Purpose, Scope and Context
  • Evaluation Methodology
• Evaluation Findings
  • Relevance
  • Design and Delivery
  • Performance
• Conclusions and Recommendations
  • Recommendations
• Appendix A - Figures
• Appendix B
  • Methodology
  • Limitations
• Endnotes

List of Figures

• Figure 1: CLSA logic model
• Figure 2: Breadth and depth of CLSA measures
• Figure 3: Extent to which CLSA data fulfills a need within the aging research community
• Figure 4: Extent of Researchers’ agreement with application process
• Figure 5: Extent of Researchers’ agreement with decision process
• Figure 6: CLSA administrative costs as a percentage of total program expenditures
• Figure 7: Method of awareness of CLSA data
• Figure 8: Non-CIHR funds leveraged by CLSA
• Figure 9: Non-CIHR funds leveraged by category
• Figure 10: Number of applications to access CLSA data by researchers and trainees
• Figure 11: Number of trainees working on projects using CLSA data
• Figure 12: New collaborations resulting from CLSA data access
• Figure 13: Collaboration statements
• Figure 14: Number of CLSA publications
• Figure 15: Comparator studies used in environmental scan
• Figure 16: Response rate
• Figure 17: Profiles of successful applicants
• Figure 18: CLSA affiliated researchers in survey sampling frame
• Figure 19: Demographic information

List of Acronyms

Executive Summary

Program Overview

Since beginning operations in 2008, the Canadian Longitudinal Study on Aging (CLSA) follows approximately 50,000 Canadian men and women between 45 and 85 years old until 2033. The platform takes an integrative approach, examining healthy aging through a number of different lenses with investigators collecting information on the changing biological, medical, psychological, social, and economic aspects of people’s lives. The goal of the CLSA is to understand the complex interplay between physical, social, and psychological determinants of healthy aging. The CLSA has completed baseline data collection, the first follow-up wave and is currently completing the second-follow up. Since funding began in 2002, the CLSA represents a total CIHR investment of $80.6 million, with additional partner contributions. The current funding phase for the CLSA ends in 2021.

Evaluation Objective, Scope and Methodology

The CLSA program evaluation assesses the relevance, design and delivery, and performance of the platform, covering the period of 2009-10 to 2018-19, using a range of methods and sources to triangulate evaluation findings. The objective of this evaluation is to provide management with timely actionable evidence that will help inform CIHR decision-making and planning regarding the activities and investments for the next phase of the CLSA.

Key Findings

Relevance

The CLSA is positioned to meet the continued need for the collection of data to support multidisciplinary research related to healthy aging. Population projections estimate that over the next 20 years, approximately 25%, or almost 10 million Canadians, will be aged 65 years or older. There are many health, social, and economic implications associated with an aging population that the CLSA is uniquely positioned to address given its longitudinal nature, objectives, sample size, and multidisciplinary data.

The CLSA’s objectives are aligned with CIHR’s mandate of supporting initiatives that will lead to the improved health of Canadians as well as a strengthened healthcare system. The CIHR Act (S.C. 2000, c6) aims among other objectives, at “addressing emerging health opportunities, threats and challenges and accelerating the discovery of cures and treatments and improvements to health care, prevention and wellness strategies.” Finally, the CLSA is aligned with Government of Canada priorities to promote healthy aging and the advancement of pan-Canadian collaboration on health innovation as outlined in Ministerial mandate letters for the Minister of Seniors and Minister of Health, respectively.

Design and Delivery

To date, the CLSA business model has been implemented as designed. Research protocols were peer reviewed in 2009 and 2014. In 2014, peer review of the second phase of funding assessed that the CLSA was well designed, implemented and positioned for longitudinal research. Additionally the committee concluded that the selection of core biomarkers were scientifically sound and that the platform had the right expertise to carry out the biomarker analyses. It is worth noting that the CLSA’s infrastructure is becoming increasingly outdated, which puts the platform’s operations at risk without new dedicated infrastructure funding.

Both the governance of the CLSA and CIHR’s oversight of the CLSA were deemed appropriate, but may need to adapt as the platform evolves. Scientific experts see a key opportunity for the CLSA to adjust its governance model as the platform shifts from implementation to focus more on data utilization and scientific productivity in order to reflect the current phase and future desired state of the platform. Although key informants believed that CIHR’s oversight function was deemed appropriate, a few interviewees expressed the need to clarify and communicate the role of CLSA’s Scientific Director co-leads with respects to initiative leadership and oversight within CIHR. Interviewees also perceived a need to incorporate a mechanism to obtain independent scientific advice for CIHR on the performance of the CLSA.

There is an immediate need to develop a succession plan for CLSA’s scientific leadership; but this is complicated by the lack of incentives available to CIHR and the CLSA limiting their ability to develop such a plan. Scientific experts encouraged CIHR to examine incorporating incentives to attract and retain future scientific leaders to the CLSA including ensuring stable funding for both infrastructure and operations, and allowing CLSA’s scientific leadership to have preferential access to the platform’s data.

Sustainability of the CLSA remains a challenge given the misalignment between the current funding model and the platform’s data collection waves. The CLSA’s current funding operates on a 5-year cycle, while the platform’s data collection waves run every 3 years. This, coupled with the fragmentation between CIHR’s funding for CLSA operations and the Canadian Foundation for Innovations (CFI) funding for infrastructure poses a risk to the CLSA’s sustainability.

Performance

The CLSA is currently achieving its expected immediate outcomes as evidenced by the implementation of a communication plan that raises awareness of the platform, facilitates participant retention and promotes data availability. However, it is worth noting that opportunities exist for the CLSA to improve its effectiveness particularly as it pertains to data availability. CIHR leverages the CLSA’s efforts by promoting access to the CLSA data to the broader research community through Catalyst Grant competitions.

The CLSA has successfully leveraged funding from non-CIHR sources for the duration of both directed grants. The platform was successful in leveraging 15% of its total operating budget from non-CIHR sources for the duration of the first directed grant between 2009 and 2015 and 22% of its budget for the period corresponding to the first three years of the second directed grant (2015 and 2018).

The CLSA is helping to increase capacity for research on aging through data accessibility. Since 2014, there have been 229 requests to access the CLSA and the platform is seeing an increase of applications from both researchers and trainees. Collaborations in projects using CLSA data are predominantly within academia. However, an opportunity exists to increase data promotion and access with stakeholders in the public and private sectors.

Available evidence demonstrates that the CLSA is making progress toward achieving its intermediate outcomes. Specifically, the platform is advancing knowledge as demonstrated through peer reviewed publications, with an annual average of 16 publications between October 2014 and September 2018. Additionally, the CLSA is establishing linkages with provincial health databases and key research initiatives including Canadian Urban Environmental Health Research Consortium (CANUE) and the Canadian Consortium on Neurodegeneration and Aging (CCNA).

Finally, with a participant retention rate of 94.5%, the CLSA has exceeded its anticipated rate of 92% in the period corresponding to the end of the baseline data collection and the end of the first follow-up wave.

Recommendations

The evaluation makes four recommendations aimed at improving the performance of the platform to achieve its expected results.

Recommendation 1:

CIHR should better position the CLSA to achieve its intended outcomes by aligning future funding with remaining data collections waves; ensuring the ongoing relevance of the data to be collected; and facilitating the full utilization of the platform following data collection.

Recommendation 2:

CIHR should consider modifying its governance structures for the CLSA to: better support the platform’s knowledge mobilization and scientific productivity outcomes; clarify the roles of the CIHR co-leads with respect to initiative leadership and oversight; and develop an independent mechanism to obtain scientific advice.

Recommendation 3:

CIHR should increase the promotion and access of CLSA data to international researchers, and stakeholders in the public and private sectors.

Recommendation 4:

CIHR should ensure that measures are in place to safeguard the continuity of the CLSA’s scientific leadership for continued sustainability and legacy of the platform, which could potentially include incorporating incentives as part of succession planning, within the limits of existing policies.

Program Profile

Program Description

The Canadian Longitudinal Study on Aging (CLSA) is a unique longitudinal data platform that follows 51,338 Canadians between the ages of 45-85 (at inclusion) for at least 20 years. A core set of data are collected from all participants, including measures on demographic and lifestyle/behaviour, social, anthropometric, psychological, socio-economic, and health status. Over 30,000 CLSA participants (comprehensive cohort) provide core information though in-home interviews and additional in-depth information (e.g., diet, medication use, chronic diseases, and sleep patterns) gathered through physical examinations and biological specimen collection. The remaining over 20,000 participants (tracking cohort) are a sample from across the 10 provinces and provide core information through telephone interviews (for more information on CLSA measurements please see Figure 2). After the initial 5-year baseline data collection period (2010-2015), CLSA participants are followed up with every three years. To date, the CLSA has completed the first follow-up wave and two-thirds of the second.

Overall the objectives of the CLSA are to:

1. To examine aging as a dynamic process;
2. To investigate the inter-relationship among intrinsic and extrinsic factors from mid-life to older age;
3. To capture the transitions, trajectories and profiles of aging; and
4. To provide infrastructure and build capacity for sustained high quality research on aging in Canada.

The CLSA’s scientific management team (SMT) is composed of the Principal Investigator (PI) Dr. Parminder Raina (McMaster University) with Dr. Christina Wolfson (McGill University) and Dr. Susan Kirkland (Dalhousie University) as Co-Principal Investigators (Co-PIs). The SMT is supported by more than 160 researchers and collaborators across 26 institutions in Canada. The data repositories and the activities of the CLSA are managed centrally at McMaster University.

With its exceptionally rich source of both alphanumeric data and biological samples, the CLSA data will help inform how Canadians age and the genesis of age-related disorders. Ultimately, the data from the CLSA will be used to:

• Prevent diseases and improve health services;
• Better understand the impact of non-medical factors, such as economic prosperity and social changes, on people as they age;
• Improve health policy and inform government programs and services; and
• Generate new knowledge on interrelated biological, clinical, psychosocial and societal factors that influence disease, health and well-being.

Data Sample Access

As a research platform, the CLSA makes its data available to both researchers and trainees, while protecting the privacy and confidentiality of study participants. Currently, the CLSA has up to four data applications deadlines each year. Applications are received through Magnolia, an online data access system, and subsequently, the Data and Sample Access Committee (DSAC) conducts an administrative and statistical review to ensure completeness and feasibility. The SMT reviews DSAC recommendations and upon approval, the applicant is notified and an agreement between the CLSA and the researcher’s host institution is prepared. Once this agreement is signed and ethics approval is obtained by the applicant, the data is released .The entire process from application to data release may take up to six months.

The CLSA operates with a partial cost recovery model. Access to alphanumeric data costs $3,000 for researchers based in Canada, and $5,000 for researchers based at institutions outside of Canada. Graduate students using data for thesis research and postdoctoral fellows using data for postdoctoral research are eligible to have the fee waived. For images and complex data, additional fees of $1,000 apply.

Program Resources

While activities leading up to the development of the CLSA started as early as 2001, funding for the development of the protocol and the methodology took place between 2002 and 2008, representing an initial investment of $4.3 million. The CLSA began operations in 2008 and received a total of $76.3 million from CIHR, through two directed grants,^{Footnote 1}specifically for baseline data collection, the first and second follow-up waves and for the analysis of a set of core biomarkers. To date, the CLSA represents a total investment $80.6 million for CIHR.

Description of Evaluation

Evaluation Purpose, Scope and Context

The purpose of this evaluation is to provide CIHR senior management with valid, insightful and actionable findings regarding the following:

• Needs addressed by the CLSA and the platform’s alignment with CIHR and the Government of Canada priorities;
• Effectiveness of the design and delivery of the program in supporting the achievement of intended outputs and outcomes; and
• Achievement of the program’s expected outputs, and immediate and intermediate outcomes.

The evaluation covers the period covers the period from 2009-10 to 2018-19 and is the first evaluation of the platform since the platform commenced operations in 2008. The evaluation was committed to as part of CIHR’s 2018-19 Evaluation Plan and designed to meet CIHR’s requirements to the Treasury Board of Canada Secretariat (TBS) under the Policy on Results and the Financial Administration Act. It is important to note that this evaluation of the CLSA is being conducted as the platform prepares to enter its third wave of data collection, scheduled to begin in April 2021. The findings of this evaluation will help inform CIHR’s decision-making and planning regarding the activities and investments for future phases of the CLSA.

The COVID-19 Pandemic

During the drafting of this evaluation report, the World Health Organization (WHO) declared COVID-19 to be a pandemic on March 11, 2020.^{Footnote 2} Almost immediately, the CLSA responded to protect the health and safety of their participants by suspending in-person data collection on March 12, 2020.^{Footnote 3} Despite these challenges, the CLSA was quick to seize the opportunity to investigate how the pandemic was affecting aging adults in Canada. Although the CLSA’s response to the pandemic fall outside the scope of the evaluation, several activities are worth noting.

In the very early stages of the pandemic, the CLSA developed a questionnaire examining how older adults were coping with coronavirus pandemic. This questionnaire focused on the physical and mental health impacts on participants, as well as their ability to access to health care services.^{Footnote 4} In October 2020, the CLSA received a $4M investment from the Government of Canada COVID-19 Immunity Task Force to assess the seroprevalance of COVID-19, along with information on risk factors, healthcare use, and the psychosocial and economic impacts of the coronavirus pandemic.^{Footnote 5} Starting in December 2020, CIHR in collaboration with the Public Health Agency of Canada (PHAC) will be investing $1.2M to support the creation of a COVID-19 module for the CLSA. This module will consist of the collection of additional data on COVD-19 as the pandemic evolves and the development of electronic datasets, which will be made available to researchers through the CLSA platform.^{Footnote 6}

Evaluation Methodology

Evaluation Questions

The evaluation addresses the following specific questions.

Relevance

1. To what extent is there a continued need for the CLSA?
   • 1.1. To what extent does the CLSA continue to address a demonstrated need in the aging research landscape?
   • 1.2. To what extent is the CLSA aligned with federal government and CIHR priorities?
   • 1.3. To what extent is the CLSA aligned with federal roles and responsibilities?

Design and Delivery

2. To what extent have the design and implementation of the CLSA been effective and efficient?
   • 2.1. To what extent has the CLSA developed national infrastructure for the collection, storage, management and dissemination of multidisciplinary data?
   • 2.2. To what extent has the CLSA implemented an appropriate and effective governance structure?
   • 2.3. To what extent has the CLSA been delivered in a cost-efficient manner?
   • 2.4. Is the current CLSA budget and funding model appropriate for sustainability?

Performance

3. To what extent has the CLSA achieved expected outcomes? (For more information see the CLSA Logic Model; Figure 1)
   • 3.1. Has the CLSA achieved its expected outputs?
   • 3.2. To what extent has the CLSA achieved expected immediate outcomes?
   • 3.3. To what extent has the CLSA made progress forward the achievement of expected intermediate outcomes.

Evaluation Approach

The evaluation employed both quantitative and qualitative data collection methods and analyses. Consistent with TBS guidance and recognized best practice in evaluation,^{Footnote 7} multiple lines of evidence were used to triangulate evaluation findings. This included document review, an environmental scan of comparable national and international longitudinal/cohort studies (n = 5 studies) and, a survey of researchers and trainees who successfully applied to access CLSA data (n = 131). There were also key informant interviews with CIHR Senior/Program Management (n = 4), CLSA Scientific Management Committee and sub-committees members, local site investigators (n = 10), and, CLSA partners including other funders, host universities and Government of Canada partners (n = 5). At the request of CIHR Senior Management, interviews were conducted to gather scientific advice on the CLSA including with former members of the CLSA International Oversight Committee (n = 2), international experts in longitudinal studies (n = 7) and representatives from international funding agencies (n = 4). Additional details about the methodology are provided in the Appendix B.

Limitations of this Evaluation

Most evaluations face limitations that have implications on the validity and reliability of the findings. The following outlines the key limitations encountered and the mitigations used to help ensure that the evaluation results can be used with confidence to inform program decision making.

It was difficult to find longitudinal studies that were directly comparable with the CLSA, both in terms of sample size, geographic diversity, healthcare context and breadth and depth of measures. As a mitigation strategy, the five studies selected for in-depth analysis collectively provided a representation of CLSA’s core design elements (e.g., large sample size, national scope, healthcare context).

In addition, representatives interviewed often did not provide details or declined to answer certain questions pertaining to the funding of their respective studies. As a result, comparisons of funding models and study sustainability was based on readily available information, making direct comparisons with the CLSA challenging.

The survey targeted a distinct population of researchers and trainees (n = 131) who successfully applied to access CLSA data. The contact information for the survey sampling frame was obtained from the Approved Project Summaries section of the CLSA website. Due to confidentially concerns from the CLSA management team, the evaluation was not able to obtain a sample of researchers and trainees who were unsuccessful in applying to obtain CLSA data. Although there was a good response rate to the survey (55%), only 3 of the 72 respondents who completed the survey self-identified as trainees. Given that there was no way of identifying trainees from the overall sampling frame it was not possible to determine if trainees represented a larger proportion of the respondents. As a result of the low trainee response rate, the analysis of the survey focused on responses from Researchers. It should also be noted that of the 69 researchers who participated in the survey, 22 were affiliated with the CLSA as either members of the SMT, local site investigators or members of governance committees.

Scientific experts interviewed had varying levels of familiarity with the CLSA. Although background materials were provided to inform respondents of the implementation, governance, funding and management of the CLSA, most interviewees were not expected to invest extensive time to understand the details of the study and CIHR’s funding mechanism to prepare for the interview. As a result, some experts found it difficult to comment in detail on some of the aspects of the CLSA and the Canadian research landscape. As a results, scientific experts anchored their comments and advice to CIHR on their own experiences related to longitudinal/cohort studies or health research funding in their jurisdiction.

Evaluation Findings

Relevance

Key Findings:

There is a clear and continued need for the CLSA

There is a clear and continued need for data to support multidisciplinary research related to healthy aging given the aging demographic in Canada. A review of program documentation revealed that over the next 20 years, population projections estimate that 25% of Canadians will be aged 65 years or older, and increase from current estimates of 23%. This means that by 2036, it is estimated that almost 10 million Canadians will be aged 65 years or older.^{Footnote 8} There are many health, social, and economic implications associated with an aging population, such as housing, health care costs, and other necessary supports. However, there are very few large-scale cohort studies investigating the transitions and trajectories of healthy aging. The CLSA is a unique initiative in the Canadian research landscape that positions Canada to meet the continued need for multidisciplinary data on healthy aging.

Key informants perceive the CLSA as a unique initiative in the Canadian research landscape given its longitudinal nature, objectives, and sample size. With its psychosocial, economic, and biological/medical components, the data can be used by researchers from a range of disciplines. The ongoing relevance of the study is ensured through CLSA’s working groups and advisory committees involving various subject matter experts. The platform has the capacity to address many research questions and inform decision-making, which fulfills an important gap in aging research necessary in the context of an aging population. Further, survey respondents supported that the CLSA fulfills an important need for aging research in Canada (M = 4.82 out of 5, SD = 0.61; Figure 3).

The CLSA’s objectives are aligned with both federal government and CIHR priorities

Through its policies and publications, the GoC has demonstrated a continued need to support Canada’s aging population. In 2009, the beginning of the CLSA’s first directed grant, the Special Senate Committee on Aging published Canada’s Aging Population: Seizing the Opportunity, which examined several issues of aging in our society, including promoting active living and well-being, health care promotion and prevention, and various health care needs.^{Footnote 9} More recently, in 2018, the Standing Committee on Human Resources, Skills and Social Development and the Status of Persons with Disabilities reported on Advancing Inclusion and Quality of Life for Seniors, which examined among other objectives, “How the government can improve the overall quality of life and well-being for seniors including community programming, social inclusivity, and social determinants of health.”^{Footnote 10} The federal government’s commitment to the aging population is further evidenced by the Minister of Health’s priority to advance pan-Canadian collaboration on health innovation,^{Footnote 11} and by the establishment of the Ministry of Seniors in 2018, which includes promoting healthy aging among its priorities.^{Footnote 12}

CIHR shares a commitment by the Government of Canada to using Gender-Based Analysis Plus (GBA+) to develop and assess effective policies and programs. Although the CLSA was developed and implemented before the development of the GBA+ analytical tool, the platform was designed such that the tracking cohort was representative of the Canadian population.^{Footnote 13} The comprehensive cohort is said to be national in scope, however it was not designed to be representative of the Canadian population owing to the fact that the sampling strategy was centered on proximity to the CLSA’s data collection sites, located across Canada. It is worth noting that through its Catalyst Grant: Analysis of CLSA Data competitions, CIHR encourages the use of CLSA data to support research that incorporates sex- and gender-based analysis (SGBA).

Consistent with the broad objectives of the Tri-Agency Equity, Diversity and Inclusion (EDI) Action Plan, the CLSA strives for a barrier-free access to its data by the research community. Specifically, the review process by the CLSA is iterative in nature and is geared towards ensuring all researchers and trainees may access its data. Additionally, by waving its cost-recovery fees for trainees (i.e. postdoctoral fellows and graduate students), the CLSA is actively removing potential access barriers to encourage data access and the timely completion of their research.

The CIHR Act (S.C. 2000, c6) acknowledges the importance of supporting initiatives that will lead to the improved health of Canadians as well as strengthen the healthcare system, and among other objectives, aims at “addressing emerging health opportunities, threats and challenges and accelerating the discovery of cures and treatments and improvements to health care, prevention and wellness strategies.”^{Footnote 14} The CLSA aligns with CIHR’s current strategic priorities due to the importance of the aging demographic and the need to better understand and improve health outcomes of this segment of the Canadian population. For the strategic plan in place during the period of the evaluation, Roadmap II, under “Strategic Direction 2: Mobilizing Health Research for Transformation and Impact” and Section “2.1: Refreshing Health and Health System Research Priorities,” the CLSA aligns with Research Priority A, “Enhanced patient experiences and outcomes through health innovation,” Research Priority C, “Promoting a healthier future through preventative action,” and Research Priority D, “Improved quality of life for persons living with chronic conditions.”^{Footnote 15} The objectives of the CLSA are aligned with the priorities of several CIHR initiatives including the Community-Based Primary Health Care, Personalized Health, Environments and Health and Canadian Epigenetics, Environment and Health Research Consortium.

Design and Delivery

Key Findings:

The CLSA business model has been implemented as designed

The implementation of the CLSA began in 2008, with the development of its research protocols and business model which were peer reviewed in 2008 and 2014. In 2014, the peer review committee was asked to assess the appropriateness and expected scientific benefits of a proposed set of core biomarkers. Overall, the 2014 peer review committee assessed that the CLSA was well designed, implemented and positioned for longitudinal research. The committee found that the selection of core biomarkers to be scientifically sound and that the CLSA team had the expertise to carry out the analysis of these biomarkers.

A review of program documents and key informant interviews demonstrate the CLSA successfully completed participant recruitment in July 2015 and that the platform developed the needed infrastructure, implemented robust standard operating procedures and a Quality Management System (QMS) that have allowed for the timely completion of baseline data collection. It is worth noting that CLSA’s infrastructure is becoming increasingly outdated, which puts the platform’s operation at risk without new dedicated infrastructure funding.

In order to be successful in achieving its objective of increasing the capacity in Canada for research related to aging, a key feature of the CLSA’s business model is to establish a process through which the platform is able to make available its multidisciplinary data to the research community. Overall, 68% of the researchers surveyed did not experience any barriers to acquiring and using CLSA data. Furthermore, researchers surveyed agreed that their applications were processed within the stated timelines (M = 4.45 out of 5, SD = 0.81; Figure 4), that the application review process was fair (M = 4.31 out of 5, SD = 0.86; Figure 5) and that they received CLSA access to the data within six months of their application approval (M = 4.53 out of 5, SD = 0.74; Figure 4). Finally, researchers were satisfied with the CLSA’s data access fees (M = 4.0 out of 5, SD = 1.02).

The CLSA governance model is consistent with those of similar longitudinal studies but may need to adapt as the platform matures

The CLSA’s current governance model was peer reviewed in 2014. Based on the scan of the governance structures of other longitudinal studies, the CLSA’s governance structure was most closely aligned with the governance models of the Irish Longitudinal Study on Aging (TILDA), the Canadian Partnership for Tomorrow Project (CPTP) and the Longitudinal Aging Study Amsterdam (LASA). The common features among the studies include:

• The use of an external body composed of international experts from multidisciplinary backgrounds;
• Separate bodies for study management and scientific direction;
• The use of working groups to embed subject matter experts within the study (e.g. knowledge translation, communications, ethical/legal); and
• Oversight by funders.

Overall, key informants perceived the CLSA’s governance model as effective, clear, inclusive and transparent. There was an openness on the part of some key informants to changing the governance structures as the study evolves. This view aligned with those of the scientific experts. Some experts believed the current governance structures to be too complex and it was sometimes difficult to discern the difference between the mandates of committee structures such as the CLSA Advisory Council and the Scientific Advisory Board. Overall, scientific experts believed that an opportunity exists for the consolidation of the multiple governance bodies to better support the CLSA as it shifts from implementation to data utilization, scientific productivity and knowledge mobilization.

Overall, CIHR’s oversight of the CLSA was deemed appropriate

From 2009 to 2013, CIHR’s oversight of the CLSA included an International Oversight Committee (IOC). The mandate of the IOC, as per its terms of reference, was to provide CIHR with expert advice as well as independent ongoing oversight and evaluation of the CLSA according to internationally recognized standards of excellence for science and leadership. The IOC was composed of five members with internationally recognized scientific and leadership expertise relevant to the CLSA. During its term, the IOC produced three reports (2010, 2011 and 2012) for CIHR on the CLSA’s progress which covered CLSA’s implementation of research protocols and participant recruitment strategy, CLSA’s governance and management along with critical components of the study such as the core set of biomarkers.

In 2013, as the first CLSA directed grant was ending, the IOC was dissolved as CIHR was realigning its organizational structures to better support its initiatives. This realignment included implementing a formal governance strategy for all of its major initiatives, including the CLSA. In 2015, CIHR’s Science Council approved CLSA’s new governance strategy for the CLSA, whereby CIHR oversight of the program would be performed by two Scientific Director (SD) co-leads and the Director General (DG) of CIHR’s Initiative Management and Institute Support Branch (IMIS). Under this new oversight structure, the SD co-leads also have the responsibility to provide leadership for the CLSA as a CIHR-funded initiative. By switching to this oversight model, CIHR lost its mechanism to obtain independent external advice on the performance and management of the CLSA that was previously provided by the IOC.

Although key informants believed that CIHR’s oversight function was deemed appropriate, there were a few interviewees that expressed the need clarify and communicate the role of CIHR’s SD co-leads with respects to initiative leadership and oversight within CIHR. Furthermore, some key informants believed that an opportunity exists for CIHR to receive more independent scientific advice on the performance of the CLSA.

There is a need for the succession planning of CLSA’s scientific management team

As part of the application process for the second directed grant, the CLSA was tasked with developing a succession plan for the CLSA SMT. An analysis of the relevant program documentation reveals that succession planning remains an ongoing risk. In its attempts to develop a succession plan, the SMT has identified the following barriers in their annual reports to CIHR:

• CLSA researchers are required to apply for data access regardless of the responsibility of leading and managing multiple components of the platform;
• There are no incentives in the current model to take on the enormous responsibility of managing the CLSA platform; and,
• The lack of incentives compromise the sustainability of the CLSA as the leadership will have to make a choice between finding resources for the CLSA and their own program of research.

Key informants and scientific experts both noted that the lack of a succession plan is a risk to the platform’s sustainability. Scientific experts went further to observe that the development of incentives is key to mitigating this risk. When asked to consider possible incentives, the scientific experts identified stable funding, for both infrastructure and operations, as well as allowing CLSA’s scientific leadership to have preferential access to the platform’s data,^{Footnote 16} would potentially help attract and retain future scientific leaders to the CLSA.

The current CLSA funding model possesses significant challenges to the platform’s overall sustainability

A longitudinal study’s funding model is one element that can potentially pose a challenge for long-term sustainability. In an attempt to compare CLSA’s funding model to those of similar longitudinal studies, the analysis of the environmental scan revealed most studies, with the exception of the Baltimore Longitudinal Study of Aging (BLSA), an intramural study led by the National Institute on Aging with the National Institute of Health (NIH) as a single funder, all other studies had multiple funders, and most operating funds were obtained through a 5-year grant cycle.

With the exception of the BLSA, all studies examined in the environmental scan experienced challenges related to sustainability. In the case of the Australian Longitudinal Study on Aging (ALSA), funding sustainability was a chronic issue over the course of the 20 years of data collection. The consequence for the lack of funding sustainability led to both interruptions and irregular data collection waves and limitations around data access.

An analysis of program documentation revealed the challenge faced by CIHR in appropriately funding the CLSA for the duration of the second directed grant. In March 2013, CIHR initially approved $26.2 million over five years (2015-16 to 2019-20), however CLSA conveyed concerns CIHR’s funding allocation did not align timing of the data collection waves, thereby leaving the platform with a significant shortfall. The CIHR allocation that was planned for five years was in fact required for three years of operations. This shortfall was confirmed by the international peer review committee during the review of the second directed grant application in July 2014. Committee members identified a funding gap of approximately $15.4 million between the funds available and the amount required by the CLSA to conduct its operations. As a result, in September 2014, CIHR subsequently approved an increase in the funding for the second directed grant.

An additional challenge to the platform’s sustainability is the lack of alignment between the CIHR’s 5-year directed grant cycle and the CLSA’s 3-year follow-up data collection waves. This lack of alignment raised concerns that any changes to CIHR’s financial landscape could potentially leave the CLSA with interruptions in data collection. It is important to note that CIHR has been proactive in working to align operational funds with CLSA’s data collection waves. To this end, in 2017, CIHR approved an ad-hoc $11.2 million extension to complete the platform’s second follow-up wave by March 2021 thereby ensuring that any future grant renewal would align with CLSA’s data collection requirements.

Key informants and program documents highlight the risk to the platform’s sustainability due to fixed funding levels over the duration of the operational grants. This current model contributes to a widening gap between current funding levels and increasing operational costs, such as increases in salary/benefits, inflation, fluctuations in the exchange rate of the Canadian dollar and a better than expected participant retention rate, currently at 94.5%.

A key risk to CLSA’s sustainability is its increasingly outdated infrastructure. Currently, CLSA’s funding for operations and infrastructure is fragmented between CIHR and the Canadian Foundation for Innovation (CFI), which have grant competition cycles that do not align. A review of program documents cited the lack of funding alignment between CIHR and CFI as a key factor putting the platform as whole at risk. Indeed, some key informants perceived the fragmentation of funding as an opportunity for CIHR and CFI to work together to ensure a more comprehensive approach for funding the CLSA’s operations and infrastructure.

When asked to consider CLSA’s sustainability as the platform is set to enter its next phase, most scientific experts agreed that a clear commitment to funding and longer funding cycles are important factors in order to fulfill the CLSA’s objectives. In addition, scientific experts acknowledged the importance of peer review to continue to assess whether the CLSA is achieving milestones and quality standards. Although most experts did not specify a preferred grant duration, most agreed that future CLSA funding should continue to align with the platform’s three-year follow-up waves.

The CLSA is being delivered in a cost efficient manner

Evaluation findings indicate that the CIHR is delivering the CLSA initiative in a cost efficient manner. The ratio of direct program administrative costs to the total program expenditures and the proportion of an initiative’s budget that is expended both speak to how efficiently a program is being run. The evaluation found the ratio of direct administrative costs^{Footnote 17} to total program expenditures to be low, remaining between 3.23% and 1.38% since 2014-15, which correspond to the CLSA’s second directed grant (Figure 6).

CLSA representatives interviewed perceived that the platform was being delivered in a cost-efficient manner. They cited the comparatively low cost of data collection ($455 per participant) and the centralization of the platform’s data infrastructure and management activities, as examples of efficient resource utilization.

Performance

Key Findings:

The CLSA has implemented a communication plan to raise awareness of the platform and the accessibility of its data

The CLSA developed and updated its communication plan throughout the duration of both of its directed grants. The three major objectives of this plan were to engage and ultimately retain participants, raise awareness of the CLSA platform with researchers, strategic partners and Canadians at large, and to promote the awareness and availability of CLSA data to researchers.

When examining the various components of CLSA’s communication plans and comparing them to those of other longitudinal/cohort studies, the CLSA was most similar to the TILDA and CPTP studies in that they all shared the wide dissemination of study reports, lay audience summaries and presentation and all three studies have a social media presence. Furthermore, when examining the performance of the various components of CLSA’s plans a review of program documentation demonstrated an increase across CLSA’s social media metrics, in particular YouTube (views up 5,720 from 2017-2018), Twitter (Followers up 258 from 2017-28) and Facebook (Likes up 26 from 2017-18). In the reporting period between October 1, 2017 and September 18, 2018 the CLSA has begun tracking Website visits (48,960 total visits) resulting in a total of 155,509 page views within the CLSA website. The most visited pages included the CLSA homepage, Participant Contact Information Update, Researcher’s Section and Data Access Section. Finally, a key milestone in CLSA’s communication strategy has been the release of the CLSA Report on Health and Aging in Canada: Findings from Baseline Data Collection 2010-2015. As part of the promotion of the report, the CLSA hosted an event on Parliament Hill on May 22, 2018.

When focusing on the promotion of data availability, key informants highlighted that the CLSA relied on the CLSA website, webinars, social media and newsletters to promote data availability, and similarly program documentation highlighted CLSA’s effort to update its website to promote CLSA approved projects in the Researcher section. A survey of researchers who accessed CLSA data found that only 7% of researchers became aware of data availability through CLSA’s communication products as compared to 17% through University communication products and 15% through CIHR Communication products (Figure 7). This data highlights an opportunity for the CLSA to improve the effectiveness of its communication strategy as it pertains to the promotion of data availability.

CIHR primarily promotes data access, knowledge exchange, cross-learning through its Catalyst Grant Competitions focused on the analysis of CLSA data

To help the CLSA seize the opportunity in generating new knowledge and building research capacity, several CIHR institutes and external partners, such as the Réseau québecois de recherche sur le vieillisement (RQRV) and the Canadian Space Agency (CSA), have partnered together to launch Catalyst Grants focused on the analysis of CLSA data (2016 and 2018). These one-year $70,000 grants are designed to catalyze and support research efforts of Canadian researchers across all of CIHR’s Institutes by accessing data from the CLSA.

In 2016, CIHR received a total of 87 applications and awarded 25 Catalyst Grants. Of these 25 grants awarded, 8 recipients (32%) were researchers affiliated with the CLSA as either members of the SMT or local site principal investigators.

Respondents from the CLSA SMT/investigators and partners categories noted the importance of CIHR’s Catalyst Grants in helping to support the success of the CLSA through by stimulating interest and promoting the platform’s data among the broader Canadian research community.

The CLSA has successfully sustained the leveraging non-CIHR funding sources

As part of the CLSA’s funding agreement with CIHR, the platform is required to leverage a minimum of 15% of its total operating budget from non-CIHR sources. In examining CLSA’s partnership data, CLSA partners could be broadly categorized into three categories: public sector,^{Footnote 18} private sector^{Footnote 19} and in-kind contributions for which a cash value was assigned.^{Footnote 20}Overall, the platform was successful in leveraging 15% of its total operating budget from non-CIHR sources for the duration of the first directed grant between 2009 and 2015 and 22% of its budget for the period corresponding to the first three years second directed grant (2015-2018; Figure 8).^{Footnote 21} Upon closer examination, the increase in the leveraging of non-CIHR funding between the first and second directed grant appears to be attributed to the increase of in-kind contributions from $1.5 million between 2009 and 2015 to $4.4 million between 2015 and 2018. Additionally there was an increase in public sector funding from $2.5 million in 2009-2015 to $3.0 million in 2015-2018 (Figure 9).

There is a perception from key informants that stakeholders such as health charities, philanthropic organizations and private industry either do not have the interest or the mandate to invest in longitudinal studies. These findings are supported by CLSA’s partnership data, particularly where funds leverages from the private sector dropped from $70,000 in 2009-2015 to $5,000 in 2015-2018 (Figure 9). Furthermore, some key informants observed that while the CLSA has successfully leveraged the required funds, the partnership work required to meet the 15% leverage requirement in future funding opportunities will be difficult. A subset of interviewees attributed the difficulty of leveraging funds to a limited partnership ecosystem whereby the CLSA is competing with other research initiatives for limited funds.

The CLSA is helping increase research capacity for aging-related research

Building research capacity is a key objective of the CLSA. The key means through which the CLSA can achieve this objective is through the availability and access to the platform’s data. In 2014, the CLSA began making its alpha numeric datasets available to researchers as the platform wound down the baseline data collection phase and prepared for the first follow-up wave.

Between 2014 and July 2019, the CLSA approved 229 data access applications. A closer examination of the trends over this time period demonstrates that data access applications by researchers steadily increased from 5 in 2014 to 52 in 2019. Similarly, data access applications by trainees increased from 3 to 28 over the same time period (Figure 10).

Researchers at all career stages have applied to access CLSA data. Of the researchers surveyed 47% identified as mid-career investigators, 28% as senior investigators and 25% as early-career investigators. Additionally survey data demonstrated that researchers of all career stages reported having trainees as part of their research team who worked with CLSA data. Researchers reported post-health degree fellows (n = 53), doctoral trainees (n = 31) and postdoctoral fellows (n = 26) as accounting for the majority of trainees on research teams (Figure 11). Similar results for trainees were observed in the analysis of the 2016 Catalyst Grant competition. Specifically, 18 of 19 grants reported having trainees as part of the research team with 67% of the trainees being at the doctoral or master’s level.

Taken together, these findings demonstrate that the access to CLSA data appears to be increasing the capacity in the Canadian research landscape.

Researchers who have accessed CLSA data have established research collaborations

Researchers who are accessing CLSA data are establishing new collaborations in Canada. Of the researchers surveyed, 74% reported having established new collaborations with researchers within their discipline while 61% have reported establishing new collaborations with researchers in other disciplines within Canada. Researchers are also establishing collaborations with researchers outside of Canada, with 31% of researchers surveyed reporting collaborations within their discipline outside of Canada and 19% reporting collaborations in other disciplines outside of Canada (Figure 12). Overall, researchers surveyed agreed that gaining access to CLSA data facilitated collaborations with other researchers in the aging community (M = 4.41 out of 5, SD = 0.74) and that the collaborations were productive in terms of advancing knowledge (e.g. peer reviewed publications, presentations) (M = 4.43 out of 5, SD = 0.75; Figure 13).

Although collaborations between researchers outside of academia (i.e., heath charities, private and public sectors) are occurring, these types of collaborations are occurring less frequently than collaborations within academia. Specifically, of the researchers surveyed, only 8% reported establishing collaborations with federal policy makers, 8% with provincial policy makers, 13% with health charities and only 2% with the private sector (Figure 12). This finding aligns with the perceptions of some key informants, across respondent categories, that the awareness and use of the platform’s data beyond academic circles is an important priority. Taken together, these findings indicate that stakeholder groups outside academia are not capitalizing on the scientific opportunity of the CLSA as it pertains to improving health policy, informing programs and services for both government and non-governmental organizations, and potentially informing discoveries and post-market research in the private sector.

The scientific experts consulted highlighted that the CLSA data is less accessible for those in non-academic community. For instance the 6-month turnaround time to access the platform’s data is unlikely to meet the needs of policy-makers. There are opportunities for the CLSA to increasingly promote the availability of CLSA data with international scientists, policy-makers in order to truly maximize the platform’s research potential and ensure that the CLSA is having an impact on health services, policy and the health of Canadians at large. Furthermore, scientific experts agreed that developing open-access datasets is a key mechanism that can help increase data accessibility and usability among international researchers and stakeholders outside of academia.

The CLSA supports the advancement of knowledge through increase accessibility to high quality data

The CLSA is advancing knowledge predominantly through peer reviewed publications. A review of the CLSA’s annual reports to CIHR and accompanying bibliographies reveal that between 2014 and September 2018 the CLSA averages 16 publications per year (Figure 14). It is important to note that as a result of the continued increase of applications for CLSA data access, in the years 2017 and 2018, the CLSA began reporting peer reviewed publications that use CLSA data, as validated by CLSA’s pre-publication review committee. Prior to 2017, peer review publications reported by the CLSA were a mix of protocol validation studies and studies using CLSA data. In addition to the publications reported by the CLSA, an analysis of the end of grant reports for the 2016 Catalyst Grant demonstrates that 8 of the 19 grants analyzed reported having a total of 19 publications.

Although still in a relatively early stage of scientific productivity, CLSA publications are starting to have an impact beyond academia. CIHR analyzed CLSA’s peer reviewed publications (2016 and prior) (Figure 14) through a matching process using a proprietary database which assessed publications observable influence beyond academia (OIBA). The analysis revealed that CLSA publications had been cited by clinical trials (n = 5), grey literature and government reports (n = 2) and textbooks (n = 3). These results are consistent for publications across all of CIHR’s programs for the same time period.

The CLSA is establishing linkages with health databases and research initiatives

In an effort to enhance the data collected by the CLSA and maximize the impact of research findings through the analyses of these data, the CLSA has worked diligently in establishing linkages with health databases as well as other relevant cohort and research initiatives. A review of program documentation demonstrates that the CLSA has been successful in establishing linkages with provincial healthcare insurance numbers on 94% of participants. Furthermore, in collaboration with the Canadian Institute for Health Information (CIHI), the CLSA is actively pursuing other provincial database linkages to obtain data on physician billings, hospitalizations, prescription drugs and mortality files. These linkages are intended to enhance and validate participant data being collected at CLSA sites. It is worth noting the challenges that national cohort studies, such as the CLSA and the CPTP experience, when trying to link national studies with provincial databases. Challenges include the prohibition of cross-provincial data transfer, lacking of resources and variability in the quality and completeness of administrative health data.

Additionally, the CLSA is working to establish linkages between the platform and other research initiatives. Notably, the CLSA has established strategic linkages with the Canadian Urban Environmental Health Research Consortium (CANUE) and Health Canada databases. Both these linkages are instrumental in helping elucidate the effect of climate change and social and material deprivation on the health of aging. Recently, the CLSA and the Canadian Consortium on Neurodegeneration and Aging (CCNA) partnered on a CIHR Team Grant exploring the impact of big data on dementia. The project entitled Broad and Deep Analysis in Neurodegeneration (BRAIN) will track the longitudinal trajectory of dementia across several datasets to develop algorithms for delineating cognitive decline in the CLSA dataset.

The CLSA is successfully retaining participants

An examination of CLSA’s annual report (covering the period of October 1, 2017 to September 30, 2018) reveals that the CLSA participant retention rate from baseline data collection to the end of the first follow-up wave was 94.5%, exceeding its anticipated retention rate of 92%. Both program documents and key informant interviews attribute the successful retention rate to the implementation of CLSA’s communication plan through newsletters, scientific cafes, and the distribution of Christmas and birthday cards.

Another key component of CLSA’s participation strategy involves monitoring emerging issues that could impact participant retention. For instance, the CLSA annual report highlights participation rates could be affected as more participants enter long-term care facilities. As the CLSA develops a mitigation strategy, the platform continues to work with its appropriate governance committees to balance its retention strategies without overburdening participants with information or repetitive contact.

Conclusions and Recommendations

Relevance

The evaluation concludes that there is a continued need for the CLSA in the Canadian research landscape. By virtue of the size of its sample and breadth of data, the CLSA is a unique initiative positioned to meet the continued need for the collection of data to support multidisciplinary research related to healthy aging. Most researchers surveyed agreed to a very great extent that the ability to gain access to CLSA data fulfills an important need for aging research in Canada (85%).

The CLSA is well aligned with Federal Government priorities and CIHR roles and responsibilities. The CLSA’s objectives are aligned with CIHR’s mandate of supporting initiatives that will lead to the improved health of Canadians and a strengthened healthcare system, as well as several priorities outlined in CIHR’s Roadmap II, which was CIHR’s strategic plan during the period under review. Additionally, the CLSA aligns with government priorities to promote the healthy aging of Canadians and advance the pan-Canadian collaboration on health innovation as outlined in Ministerial Mandate Letters and other government publications.

Design and Delivery

As per its initial design, the CLSA has 12 years, and 4 data collections waves remaining. With the third funding opportunity scheduled to launch in spring 2020, CIHR should ensure that all data being collected continues to be relevant and consider if additional data, including biomarkers, can and should be collected to reflect the current state of science and government priorities.

To maximize CIHR’s investments in the CLSA, it would be important for CIHR to begin planning, with relevant stakeholders, for CLSA’s legacy to ensure that the platform and its data continue to be a relevant tool for the Canadian research landscape well after CIHR’s investments have sunset. In 2017, CIHR sought an $11.2 million extension from Science Council to align the operational funding to CLSA’s data collection waves. Future funding opportunities should continue to align with the platform’s 3-year data collection waves. Scientific experts advised that CIHR should demonstrate a long-term commitment to funding the CLSA and that peer review should continue to inform future funding decisions.

The CLSA governance model was perceived as working well, and closely aligned with other comparable longitudinal studies both nationally and internationally. Some key informants and scientific experts observed that the model was too complex, and saw an opportunity to modernize the governance structures as the CLSA moves away from the participant recruitment, implementation, and baseline data collection phase to a phase focused on data access and usability by the research community as well as public and private sector stakeholders.

Some interviewees expressed the need to clarify and communicate the role of the CIHR SD Co-Leads with respects to initiative leadership and oversight of the CLSA. Additionally, some interviewees believed that CIHR should receive ongoing independent scientific advice on the performance of the CLSA.

Overall, the lack of available incentives represents a key risk in the succession planning of the CLSA scientific leadership. Scientific observed that there is an opportunity for CIHR to focus on developing appropriate incentives to facilitate CLSA’s ability to recruit and retain scientific leaders.

Performance

The CLSA is currently achieving its immediate outcomes, particularly in the area of capacity building as evidenced by the increasing requests to access CLSA data among both researchers and trainees. CIHR’s Catalyst Grant competitions are cited as contributing to capacity building by raising awareness and access to the CLSA’s data by the broader research community.

The evaluation also found evidence that the CLSA is making progress towards achieving its intermediate outcomes. Specifically, the CLSA is establishing linkages with provincial databases and other related research initiatives such as CANUE and the CCNA. The platform has also been extremely successful in retaining participants with a rate of 94.5% of participants retained from the time of baseline data collection to the end of the first follow-up wave.

Although the CLSA is still early in its scientific productivity phase, it is starting to have an impact in the area of advancing knowledge, as evidenced by an average of 16 peer reviewed publications annually since 2014, with some of these publications having some observable influence outside of academia. Both key informants and scientific experts believe that raising awareness of the availability and accessibility of CLSA data beyond academia is an important priority, particularly for stakeholders in the public and private sectors. These perceptions are further substantiated by survey data indicating that CLSA data usage and collaborations by researchers are happening predominantly within the academic research community. Scientific experts believe that an opportunity exists for the creation and promotion of open-access datasets.

Recommendations

The evaluation makes four recommendations to improve the performance of the platform to achieve its expected results.

Recommendation 1:

CIHR should better position the CLSA to achieve its intended outcomes by aligning future funding with remaining data collections waves; ensuring the ongoing relevance of the data to be collected; and facilitating the full utilization of the platform following data collection.

Recommendation 2:

CIHR should consider modifying its governance structures for the CLSA to: better support the platform’s knowledge mobilization and scientific productivity outcomes; clarify the roles of the CIHR co-leads with respect to initiative leadership and oversight; and develop an independent mechanism to obtain scientific advice.

Recommendation 3:

CIHR should increase the promotion and access of CLSA data to international researchers, and stakeholders in the public and private sectors.

Recommendation 4:

CIHR should ensure that measures are in place to safeguard the continuity of the CLSA’s scientific leadership for continued sustainability and legacy of the platform, which could potentially include incorporating incentives as part of succession planning, within the limits of existing policies

Appendix A - Figures

Figure 1: CLSA logic model

Figure 2: Breadth and depth of CLSA measures

Source: The Canadian Longitudinal Study on Aging

Figure 3: Extent to which CLSA data fulfills a need within the aging research community

Figure 4: Extent of Researchers’ agreement with application process

Figure 5: Extent of Researchers’ agreement with decision process

Figure 6: CLSA administrative costs as a percentage of total program expenditures

Source: Award expenditure and administrative costs data obtained from CIHR Finance

Figure 7: Method of awareness of CLSA data

Figure 8: CLSA administrative costs as a percentage of total program expenditures

Source: The Canadian Longitudinal Study on Aging Partnership Data

Figure 9: Non-CIHR funds leveraged by category

Figure 10: Number of applications to access CLSA data by researchers and trainees

Figure 11: Number of trainees working on projects using CLSA data

Figure 12: New collaborations resulting from CLSA data access

Figure 13: Collaboration statements

Figure 14: Number of CLSA publications

Appendix B

Methodology

Additional details about the multiple lines of evidence used in the evaluation are presented in this section. They included a document and data review, an environmental scan of national and international longitudinal/cohort studies, a survey of researchers and trainees who successfully applied for CLSA data access, in-depth interviews with KIs and the solicitation of advice from scientific experts.

Document Review

A document and data review was conducted to provide context to the evaluation and help address several evaluation questions related to program relevance, design and delivery and performance. Documentation reviewed included CLSA program literature, meeting summaries and minutes of governance bodies including the CIHR-CLSA Oversight Committee, and CIHR Science Council.

Environmental Scan

An environmental scan of Canadian and international longitudinal studies was conducted to help address evaluation questions related to the design and delivery of the CLSA. A long list of 10 longitudinal studies was profiled based a series of criteria to ensure comparability to the CLSA. The long list was subjected to additional screening criteria to select the final comparator studies (Figure 15).

The environmental scan consisted of a review of publically available information (e.g. study websites, methodology documents, and annual reports). This document review was complimented by interviews conducted with a member of the research team for each study. The interviews allowed for the capture of factual and descriptive information that was not readily available in the public domain (e.g. the study’s funding and business model, lessons learned, and best practices).

Figure 15: Comparator studies used in environmental scan

Source: Goss Gilroy Inc. Canadian Longitudinal Study on Aging Evaluation – Key Informant Interviews and Environmental Scan Report, 2019.

Survey of Researchers and Trainees who Accessed CLSA Data

A survey of researchers and trainees who accessed CLSA data was included to investigate opinions with respects to the continued need, design and delivery and performance of the platform. The survey included closed and open-ended questions as well as demographic information about the survey participants.

All administrative data for the sample were pulled from the CLSA website Approved Project Summaries Page. Duplicated data were removed. The survey was sent to 131 successful applicants of CLSA data, and 72 completed the survey, for a response rate of 55% (Figure 16). Based on preliminary analysis on 3 of the 72 respondents identified themselves as trainees (Figure 18). Given the small response rates from trainees, the analysis focused only on the responses of researchers (n=69). It is worth noting that 32% of the final sample were CLSA-affiliated researchers (Figure 18).^{Footnote 22}

The sample disposition is presented by language preference and gender in Figure 19 .

Figure 16: Response rate

Source: Canadian Longitudinal Study on Aging Survey Results Report

Figure 17: Profiles of successful applicants

Source: Canadian Longitudinal Study on Aging Survey Results Report

Figure 18: CLSA affiliated researchers in survey sampling frame

Source: Canadian Longitudinal Study on Aging Survey Results Report

Figure 19: Demographic information

Source: Canadian Longitudinal Study on Aging Survey Results Report

Solicitation of Advice from Scientific Experts

In March 2019, at the request of CIHR Senior Management, solicitation of independent scientific expert advice (through key informant interviews) intended to inform the next phase of the CLSA, was added as a line of evidence in the evaluation. The scope of the advice was focused on overarching issues related to the relevance and usability of the CLSA; the feasibility of the next phase of the CLSA given current resourcing; how the next phase of the CLSA could be built upon; and the overall value of the CLSA to Canada. Respondent categories included former members of the CLSA IOC (n=2), International Scientific Experts (n=7) and representatives from heath research funding agencies (n=4).

Limitations

The following limitations and mitigation strategies have been identified for this evaluation:

Date modified:

2022-03-21