Mitigating Bias in the Peer and Merit Review Processes: Transcript

Title

Welcome to Module 2: Mitigating Bias in the Peer and Merit Review Processes. This is the second module in the Excellence in Peer and Merit Review learning series.

A Few Notes Before You Begin

For the best experience, use Google Chrome or Microsoft Edge on a Mac or PC. Some accessibility features may not work in other browsers. Most narrations in this module match the on-screen text, and closed captions are provided when they differ. Terminology in this field continually evolves. Content may be updated periodically, as appropriate.

Navigating this Module

Use the playbar to pause and resume playback, navigate between slides, or mute and unmute audio. You can also toggle closed captions, browse the full table of contents, and collapse or move the playbar.

Introduction

This module is designed to help you understand how systemic and individual biases can affect the peer review process. The learning objectives of the module are to:

  1. Describe where in the peer review process bias is likely to occur.
  2. Describe techniques to interrupt biases and to microaggressions in peer review.
  3. Encourage a diverse, inclusive, and culturally safe research ecosystem.

A Word of Self Care

Everyone understands bias from a unique position based on their own lived experience. For some people, the topic of bias in peer review may be relatively new. Others may experience bias, microaggressions and discrimination every day. As you proceed through the modules, take time to reflect on your physical and emotional state as you react to the information provided. Press pause or take breaks as often as you need to. The playback will start again where you left off. If you feel unable to complete the course due to the impact of its content, use the button on your screen to access supportive resources.

Mental Health Support

Objective 1

References

Bias in the Peer Review Process. Bias can happen in many ways during the peer review process. This section explores three areas of the peer review process where bias is likely to occur:

  1. letters of recommendation and reviews;
  2. traditional metrics; and
  3. evaluations of the research subject and methodology

Letters of Recommendation and Review 1

Bias can be evident in the language and tone of recommendation letters and meetings.

Letters of Recommendation and Review 2

For instance, letters for women in medical school may be shorter, include "doubt raisers", and be less focused on achievements compared to men's letters. They often contain more "grindstone adjectives" such as "hardworking", "conscientious" or "dependable". Letters for men include more standout adjectives such as "excellent", "superb", or "outstanding".

Letters of Recommendation and Review 3

Similarly, recommendation letters for Black medical students may use fewer standout words and more negative terms like "competent". These biases can affect how applicants are perceived and their chances of success. A thorough review focuses on relevant criteria and avoids biased comments based on language or location.

Traditional Metrics 1

Granting agencies fund research based on excellence, but what if the criteria used to judge excellence are biased?

Traditional Metrics 2

Reviewers often rely on metrics like years of experience, publication numbers, and impact factor, assuming they are fair. However, these metrics can be biased, especially when they don't consider the applicant's context.

Traditional Metrics 3

Factors like research area, career stage, publication country, and topic can influence the quantity and types of research outputs. This doesn't mean the researchers or their projects are less deserving of funding or less excellent.

DORA 1

Recognizing these factors, the federal funding agencies have signed the San Francisco Declaration on Research Assessment (DORA). Use the button on your screen to explore DORA, a global initiative to support the development and promotion of best practice in the assessment of scholarly research.

Explore DORA

DORA 2

In adopting DORA, the agencies recommend that reviewers consider the value and impact of diverse research products (including datasets, tools, standards, software, partnerships, community service and knowledge mobilization products) in addition to research publications.

DORA 3

Reviewers are also asked to consider a broad range of impact measures including qualitative indicators such as community engagement, meaningful mentoring, considerations of multiple ways of knowing, and influence on policy and practice.

Multiple Choice 1

Which of the following statements best describes an approach to evaluating research impact that aligns with DORA?

  1. Research impact is primarily assessed based on the number of publications an author has in high-impact factor journals.
  2. Research impact is solely determined by the number of citations a paper receives.
  3. Research impact is measured by considering the significance of the research question, the quality of research methods, and the actual impact of the research on the field.
  4. Research impact is evaluated based on the prestige of the institution where the research was conducted.

Option c) reflects the holistic approach advocated by the San Francisco Declaration on Research Assessment (DORA), which encourages considering various factors beyond traditional metrics like journal impact factor or citation counts when evaluating research impact. Click anywhere on your screen to continue.

Incorrect response:

While your answer acknowledges traditional evaluation metrics, such as the number of publications in high-impact factor journals or citation counts, it's important to recognize the broader perspective advocated by DORA. Research impact should be assessed holistically, considering factors such as the significance of the research question, the quality of research methods, and the actual impact of the research on the field. Relying solely on metrics like journal impact factor or institutional prestige may overlook the true value and impact of the research. Keep in mind the multifaceted nature of research assessment as promoted by DORA. Click anywhere on your screen to continue.

Research Subject and Methodology 1

Reviewers might doubt projects with unfamiliar topics, subject groups, or methods. They often prefer familiar topics and may unconsciously be wary of new ideas.

Research Subject and Methodology 2

For example, a study involving people with cognitive disabilities in data collection could be rejected because reviewers question their ability to gather data and the extent of the study's impact given the population involved. This is a clear case of ableist bias, as well as bias related to methodology and research users.

Research Subject and Methodology 3

This type of bias often results in unfair rejection of work by scholars from racialized backgrounds. These scholars tend to tackle innovative topics involving human subjects or dynamics. They also commonly work with racialized groups and may use theoretical frameworks like critical race theory or post-colonial theory, which reviewers from different fields may not be familiar with.

Bias Related to Ways of Knowing and Doing 1

First Nations, Inuit and Métis Peoples have unique cultures, histories, and priorities. They often use unique, non-Western ways of understanding and sharing knowledge, depending on their context, location, and goals. Researchers working with Indigenous Peoples and communities recognize this distinctions-based approach and the diversity between and within Indigenous communities. They have developed research methods grounded in Indigenous ideologies like relatedness and connectedness, as well as traditional knowledge, to benefit their communities.

Bias Related to Ways of Knowing and Doing 2

Western research is often seen as the gold standard for defining research excellence, and Indigenous researchers face obstacles when peer reviewers undervalue their methods and expect them to cite non-Indigenous sources, or to translate oral knowledge into written form.

Bias Related to Ways of Knowing and Doing 3

For instance, an Indigenous researcher might emphasize "letting stories speak for themselves" as their analysis method, a land-based gathering as a way of knowing, and prioritize "knowledge transfer" over mobilization.

Bias Related to Ways of Knowing and Doing 4

These methods may not match Western ideas of academic rigor and rationality but are equally valuable scholarly methods.

Addressing Anti-Indigenous Bias

To address anti-Indigenous bias, we must recognize the importance of Indigenous self-determination and self-governance in research, Indigenous ways of knowing and doing, Indigenous research ethics, and ensuring Indigenous applicants do not face systemic barriers in the peer review process.

Multiple Choice 2

When reviewing research conducted by or relating to Indigenous scholars and communities, what is a critical consideration to ensure inclusivity and respect for diverse knowledge systems?

  1. Applying Western-centric evaluation criteria to maintain consistency in peer review standards.
  2. Acknowledging the distinct cultures, histories, interests and priorities of First Nations, Inuit and Métis Peoples, including their unique rights, laws and governments.
  3. Emphasizing the need for Indigenous researchers to conform to established scientific paradigms to ensure credibility.
  4. Including studies conducted by non-Indigenous scholars in Indigenous-led research to ensure objectivity.

The correct answer is b): This option highlights the importance of respecting and valuing the variety of Indigenous perspectives and knowledge systems, which helps promote inclusivity and equity in the peer review process. Click anywhere on your screen to continue.

Incorrect feedback: While your response demonstrates some understanding of the complexities involved in evaluating Indigenous research, it's important to recognize the broader perspective needed for inclusivity and respect for diverse knowledge systems in the peer review process. Remember, taking a distinctions-based approach to research may require different approaches when working with First Nations, Inuit or Métis partners. Embracing diverse perspectives enhances the robustness and inclusivity of the academic community, promoting equity in research evaluation. Click anywhere on your screen to continue.

Objective 2

References

Ways to Mitigate Bias. Dealing with bias is an ongoing process that doesn't stop after peer review. We must first recognize that we all have biases and that they also exist in research institutions, affecting how research and its benefits impact people living in Canada. Then, we need to identify and reduce our own biases through reflection and action. The following techniques help counter shortcuts and assumptions that might unfairly affect the peer review process.

Self-Assessment

Before peer review begins, identify bias through self assessment. An Implicit Association Test helps demonstrate the differences between people's stated beliefs and their internal attitudes. It helps people reflect on their attitudes towards gender identity, sexual orientation, race, disability, age and other biases. Before taking any implicit association test, ask yourself what kind of results you expect to receive.

Action Plan

After becoming aware of biases you may hold, create a short action plan to mitigate the influence of these biases. Action plans can be simple:

  1. write down the thought, attitude or action you would like to change.
  2. write down one or two strategies that you will use to make the change.
  3. at the end of the peer review cycle, reflect on what happened when you used these strategies.

Strategies When Reviewing

When reviewing application packages, there are several strategies to help you mitigate bias. In this section, we'll explore how to:

  1. Take the time
  2. Reflect and reflect again
  3. Embrace different ways of knowing
  4. Review the application as a package.
  5. Focus on application content, not career path
  6. Notice the words that you use, and that others use
  7. Embrace the principle of "nothing about us without us," and
  8. Rethink the effects of innovation

1. Take the Time

Block off sufficient time so that you can evaluate each application without distraction. Studies have shown that evaluators who were busy or distracted by other tasks gave women lower scores than men for the same written evaluation of job performance. Our brains are less likely to rely on short-cuts if we take the time to absorb and analyze information. If possible, review your assessment on a different day for any potential biases.

2. Reflect and Reflect Again 1

Self-reflection can help identify whether individual biases are operating during the peer review process. Periodically evaluate your judgements and consider whether evaluation biases are influencing your decisions.

2. Reflect and Reflect Again 2

Ask yourself questions such as:

2. Reflect and Reflect Again 3

Stereotype replacement can also help identify whether individual biases are operating during the peer review process. This process consists of thinking about a stereotype that you hold and consciously replacing it with accurate information.

2. Reflect and Reflect Again 4

For example: Smaller institutions may be perceived to be less innovative, when in fact these institutions also offer innovative, diverse and valuable perspectives on research. Ask yourself, if this application came from a larger, more familiar institution would it give the application more merit?

3. Embrace Different Ways of Knowing and Doing 1

First Nations, Inuit and Métis Peoples understand their own needs and are in the best position to govern their own knowledge as it relates to their unique needs and governing structures. Before beginning your review, reflect on the importance of Indigenous self-determination and self-governance in research. Look for evidence of leadership and/or that meaningful collaboration with Indigenous People has been prioritized in the proposal.

3. Embrace Different Ways of Knowing and Doing 2

Ask if the research adopts a distinctions-based approach. A distinctions-based approach should be used to recognize the unique cultures, histories, and priorities of First Nations, Inuit, and Métis Peoples. This approach recognizes the diversity between and within Indigenous communities. 'Pan-Indigenous' approaches that use the umbrella term "Indigenous Peoples" as a specific population and do not disaggregate data must have a clear and sound rationale for the approach.

3. Embrace Different Ways of Knowing and Doing 3

Look for evidence of Indigenous self-governance or data sovereignty as described in the:

Or other principles of Indigenous self-governance as determined by the Indigenous communities.

3. Embrace Different Ways of Knowing and Doing 4

Use the buttons on your screen for help in applying the Tri-Council Policy Statement on the Ethical Conduct for Research Involving Humans (TCPS-2) Chapter 9 guidelines, as well as any other agency specific frameworks, such as SSHRC's Guidelines for the Merit Review of Indigenous Research.

4. Review the Application as a Package

Consider the evaluation criteria and review all components of the application package equally. This helps to mitigate bias related to the language used in reference letters, the applicant's career progression and perceived research impact.

5. Focus on Content, Not Career Path 1

When reviewing a CV or publication record, prioritize the merit and societal impact of the research over the career trajectory. This reduces biases related to researcher productivity and expertise.

5. Focus on Content, Not Career Path 2

Emphasize the quality and impact of research, knowledge mobilization, and contributions to the research ecosystem instead of counting publications or considering citation factors and awards. This approach supports individuation, which involves evaluating the applicant as a unique individual rather than relying on group-based assumptions.

5. Focus on Content, Not Career Path 3

When the Applicant Themselves is Under Evaluation… embrace diversity. Not everyone follows the same career path or uses the same leadership style, yet, across industries, it is increasingly recognized that transferrable skills, inclusive leadership and emotional maturity are keys to success.

5. Focus on Content, Not Career Path 4

In situations where the applicant is being evaluated more than the research, as with early career researchers or training awards, look for evidence of transferrable skills or parallel experiences that demonstrate the candidate's ability to meet the evaluation criteria.

6. Notice Words You and Others Use 1

As you are reviewing the application package, reflect on the words that are being used to describe the applicant and their research. Do reference letters contain outstanding adjectives such as "best", "excellent" or "exceptional" or use more grindstone adjectives such as "competent" or "hard worker"? Reflect on how those words affect your perception of the applicant.

6. Notice Words You and Others Use 2

Does the language used to describe the applicant or their work suggest gender, racial, ableist or other assumptions? If providing a written review, re-read it to screen for loaded words, assumptions, or value judgements.

7. "Nothing About Us Without Us" 1

When reviewing application packages, ask: Does research designed to benefit people from underrepresented groups include them as key decision-makers and participants in all aspects of design, data collection and results analysis? When considering research impact, use disaggregated information to identify data gaps and reflect on the degree to which focusing on underrepresented groups fills these gaps.

7. "Nothing About Us Without Us" 2

The English form of this slogan was popularized in disability activism during the 1990's and has since expanded to other interest groups and movements. The approach supports research that is relevant and of benefit to the community, while mitigating risks as described in Chapter 2 of the TCPS-2.

8. Embrace the New or Innovative

Consider the effects of undertaking innovative research on the researcher and question institutional biases towards various research topics. Funding something new, different or innovative may be seen as risky, but everybody learns when researchers investigate new topics or try innovative methodologies. Are innovations truly not fundable, or does a traditional understanding of what is an acceptable topic make it so?

Multiple Choice 3

When reviewing funding applications, which of the following strategies can help mitigate bias and promote inclusivity?

That's correct! Reflecting on personal biases and assumptions that may influence evaluation decisions can help promote inclusivity. Click anywhere on your screen to continue.

Incorrect feedback: Choosing this option may inadvertently perpetuate biases that favor established researchers, overlooking the value of diverse perspectives and expertise across career stages. Try again after reflecting on the principles of inclusivity and equity in the review process, and considering all applicants equally based on the content of their applications rather than their career status. Click anywhere on your screen to continue.

During Group Discussions

Embrace diversity and create a dynamic that welcomes peer reviewers as equals. This helps to mitigate potential bias in decision-making about applications and ensures that committees are inclusive for all peer reviewers and applicants alike.

Tips to Mitigate Bias 1

Strive for bias-free group interactions by using gender neutral language (e.g. "the applicant") when referring to people or personal relationships, unless otherwise specified. Practicing deep listening. Consider sharing the floor with others to provide everyone on the committee an opportunity to speak.

Tips to Mitigate Bias 2

Responding quickly and appropriately to transphobic, homophobic, sexist racist, ableist or ageist comments, making it clear that actions or comments that are derogatory or dismissive are unacceptable. Participating actively should the topic of equity, diversity and inclusion arise. Explicitly and respectfully naming factors that can impact researchers from underrepresented groups. Communicating in both of Canada's official languages.

Objective 3

References

Learn to address microaggressions. If you are concerned about potential bias during group discussions – in yourself or other reviewers – it is important to address it. Raise the issue during the committee discussion or communicate concerns with your peer review committee Chair or agency staff at any time.

What are Microaggressions?

Microaggressions are brief and common verbal, behavioural, or institutional actions that play into stereotypes or discrimination against a group of people, often from underrepresented groups. It's important for you to understand and be able to recognize microaggressions as they can often manifest during group discussions.

The Impacts of Microaggressions 1

Microaggressions are a consequence of larger ("macro") systems of discrimination, oppression, power, and privilege. Microaggressions can send a message that peer reviewers from underrepresented groups do not belong on peer review committees.

The Impacts of Microaggressions 2

Although often difficult to respond to, addressing microaggressions helps to mitigate both individual and systemic bias. Each individual may have their own strategies for addressing microaggressions depending on their own identity and lived experience.

An Example of a Microaggression

An international postdoctoral candidate presents their research findings at a campus lunchtime seminar. At the beginning of the question-and-answer period, a faculty member says "I have no comments on your research, but you may want to work on your presentation skills. With that kind of accent, it's impossible to understand the point you are trying to make." Taken in isolation, one instance of microaggression can seem like a minor event; however, members of underrepresented groups often experience the same microaggression repeatedly over time, producing adverse emotional, social, psychological and health impacts. This can, in turn, affect their researcher productivity, and sense of inclusion.

Step One: Consider 1

How to Address Microaggressions. Now let's explore steps you can take when witnessing a microaggression. Step One: Decide whether a response is warranted. The person best positioned to judge whether a sentence, tone, or behaviour is a microaggression is the person who receives it. It is not the intention behind a phrase or action that counts, it is the impact on the person receiving it.

Step One: Consider 2

Take a moment to decide if the speech or action was a microaggression. If you are a bystander, try to take your cues from the person at whom the speech or act was directed.

Step One: Consider 3

Weigh the potential costs when deciding whether to respond. Speaking up can be an act of allyship and solidarity, but it may further marginalize the person who experienced the microaggression.

Step Two: Respond 1

Microaggressions can be subtle and not always perceived as conflict. It is important to consider the agency's policies and procedures on dealing with conflict during merit review when deciding how to respond. Take time to review these guidelines if you have not already done so.

Step Two: Respond 2

It is possible to respond to microaggressions using a variety of techniques. The method you choose depends on your comfort level, knowledge, experience, the degree to which you know other review committee members, your role on the review committee, and the context in which the microaggression occurred.

Step Two: Techniques

There are many ways to respond to a microaggression both as a bystander or as someone on the receiving end. You could:

Step Three: Debrief

Regardless of whether you choose to respond, take time to reflect on the situation. If possible, discuss it with a trusted colleague or friend.

Committing a Microaggression 1

If someone states that you have committed a microaggression during the peer review process, pause. It is easy to dismiss microaggressions as someone being too sensitive or holding a grudge, but microaggressions are part of a pattern of bias that repeats itself over and over. It is important to stop and reflect.

Committing a Microaggression 2

By listening and apologizing for the negative impact on another person regardless of intention, you hold yourself accountable to the peer review committee and to the principle that research be equitable, inclusive, and diverse. Debrief following the incident to reinforce the commitment to addressing microaggressions fairly.

Conclusion

In Summary

After completing this module, you should understand that:

Next Steps

Consider taking the following actions if you would like to learn more about the topics raised in these modules:

Survey

Before concluding this module, please complete this form to assist the agencies in tracking uptake and improving the quality of our learning products.

Thank You

Thank you for completing this learning module. Input your name to generate a completion certificate.

Additional Resources

If you have questions or would like to learn more about bias, please contact a representative at the respective agency or explore the resources linked on this screen:

Date modified: