Appendix 2: Evaluation questions, indicators and data sources

Midterm Evaluation of the Pandemic Preparedness Strategic Research Initiative

Table of contents ]

Issue and Question Indicators Data sources
1. Effectiveness of priority setting
1.1 To what extent were the PPSRI research priority setting processes appropriate, in terms of timeliness, mechanisms, and inclusivity of a) research areas and b) groups of researchers? Stakeholders' views of appropriateness
Researchers' perceptions of COI
Key informant interviews: Section 2
Researchers survey Q4-9
1.2 How successful were the research priority setting processes – to what extent did they arrive at the most appropriate set of priorities for Canadian research in pandemic preparedness? Stakeholders' views of appropriateness
Researchers' ratings of priorities
Key informant interviews: Section 2
Researchers survey: Section E
1.3 To what extent was national and international duplication of effort avoided, and complementarity enabled? Stakeholders' views of appropriateness
Researchers' ratings of priorities
Key informant interviews Q2.2,
1.4 How necessary, appropriate (in terms of role and composition) and effective was the Task Group? Stakeholders' views of appropriateness and effectiveness
Researchers' perceptions of fairness, COI
Key informant interviews Q2.3
2. Effectiveness of partnership development
2.1 To what extent has the PPSRI been successful in building national and international partnerships? Which partnerships have been most and least successful? Are there partnerships that should be developed but have not been? Stakeholders' views of effectiveness Key informant interviews Q3.1-3.3
2.2 What have been the impacts of partnerships on: a) coordination and integration of national and international research programming; b) resource leveraging; c) research duplication and complementarity? Stakeholders' views of impacts
Documentation of partnerships, resources leveraged
Key informant interviews Q3.1-3.3
Review of documentation
2.3 To what extent have the partnership structure and reporting strategy been effective in supporting the ongoing initiatives of partners, including the PPSRI? Stakeholders' views of effectiveness Key informant interviews Q3.4
3. Appropriateness of program design
3.1 To what extent is the suite of activities and funding programs offered through the PPSRI allowing the achievement of program objectives? Which components are most and least successful? Stakeholders' views of appropriateness,
Researchers' ratings of incentive features
Uptake/quality of response to program components
Key informant interviews Q4.1
Researcher survey Q3
Administrative databases
3.2 Is the overall strategy of strategic competition effective in ensuring that the most promising research is funded? Would an alternative strategy including the enabling "front-runners" (recognized leaders in the relevant fields) to continue to pursue relevant PP research) be more effective? Is this the most effective strategy to ensure short and long term capacity development? Stakeholders' views of effectiveness
Researchers' views of effectiveness
Key informant interviews Q4.1
Researchers survey Q21-23
3.3 How effective was the communication strategy used to launch the funding opportunities? Stakeholders' views of effectiveness
Researchers level of awareness and ratings of effectiveness
Key informant interviews Q4.2
Researcher survey Q1, Q3, Q10
4. Achievement of desired funding opportunity outputs
4.1 To what extent have the funding programs generated expected and/or desirable uptake from the relevant research communities?
4.2 To what extent do the sets of funded projects cover the intended field of program objectives? Which objectives streams are more and less well represented across funded projects?
Stakeholders' views of uptake and coverage
Per competition and per program stream: No. of applications, no. of grants, success rate (including successful-unfunded and funded), amounts awarded; disciplinary diversity of applications and grants; national and international collaborations involved in applications and grants;
Comparisons on the above variables between successful and unsuccessful applicants
Key informant interviews Q5.1-5.2
Administrative databases
4.3 To what extent will the funded projects result building in research capacity in areas related to pandemic preparedness (e.g., training of students, redirection of research foci, new investigators on teams) Stakeholders' view of capacity development opportunities
No. of students and fellows involved in applications and grants, by level per program stream
No. of investigators newly involved
Evidence of change of research foci
Key informant interviews Q5.3, 5.4
Researcher survey Q13,14, 19, 20
Administrative databases
5. Success of PPSRI networking and KT activities
5.1 To what extent to which the PPSRI has successfully facilitated communication and networking among researchers involved in pandemic preparedness research? Have all relevant teams and individuals been provided with networking opportunities, and what has been the uptake? Stakeholders' views of success
Researchers' ratings of networking
Documentation of participation in networking activities
Key informant interviews Q6.1
Review of program documentation
5.2 To what extent are communication and networking producing the expected results in terms of enhanced collaboration and increased capacity? Stakeholders' views of success
Researchers' ratings of networking impact
Key informant interviews 6.2
Researcher survey Q11,12,15
5.3 To what extent has the groundwork been put in place for effective knowledge translation to occur (e.g., inclusion of end-users, KT plans)? Stakeholders' views of success
Proportion of funded projects with KT plans
Degree and type of involvement of research users
Key informant interviews Q5.4
Researcher survey Q17-18
Administrative databases

next section ]

Date modified: