From Questions to Confidence: Mastering Evidence-Based Methods with Dr. Romina Brignardello-Petersen

Dr. McGavock: [00:00:00] Welcome back to the Emerging Scholar Podcast. I'm your host, Dr. Jon McGavock from the University of Manitoba, and I'm thrilled to have you back for season two. Last season, we took you on a journey through your first year of research training, covering everything from crafting great research questions to understanding the foundations of both quantitative and qualitative methods.

Dr. McGavock: We met incredible scholars, explored fundamental concepts, and built a roadmap to help you navigate your early research career. This season, we're going deeper. If season one was about laying the foundation, season two is about mastering the craft. We're taking those methods we introduced and really unpacking them, diving into the nuances, the challenges, and exciting new possibilities that each method has to offer.

Dr. McGavock: Throughout this season, you'll meet a remarkable team of both established and emerging scholars with expertise across a diverse array of approaches, from natural experiments that reveal causal insights into real-world [00:01:00] settings, to qualitative inquiry grounded in social justice, the complexities of implementation science, and yes, my personal favorite, causal inference.

Dr. McGavock: Each episode will introduce you to researchers who are pushing the boundaries, asking bold questions, and making novel and important scientific discoveries. First, we're celebrating Canadian research excellence, showcasing incredible work happening right here at home. Second, we're emphasizing the critical importance of patient engagement in shaping research that truly matters across a range of scientific methods.

Dr. McGavock: And finally, we're championing curiosity, the essential spark that drives all great science and keeps us asking what if and why. Whether you're designing your first research study, wrestling with methodological challenges, or simply passionate about rigorous and meaningful research, this season is for you.

Dr. McGavock: So let's dive in, stay curious, and continue building the skills that will shape your own scientific journey Welcome back everyone to the Emerging Scholar Podcast. I'm your host, Dr. Jon McGavock from the Children's Hospital Research Institute of [00:02:00] Manitoba. In one of our bonus episodes last year, we had the privilege of speaking with Dr.

Dr. McGavock: Gordon Guyatt, the pioneer who coined the term evidence-based medicine and fundamentally transformed the way we integrate research evidence into clinical practice. Today, we're continuing that conversation about rigorous methods that underpin evidence-based decision making by welcoming one of Dr.

Dr. McGavock: Guyatt's esteemed colleagues and mentees from McMaster University, Dr. Romina Brignardello-Petersen. Dr. Brignardello-Petersen is an associate professor in the Department of Health Research Methods, Evidence and Impact at Mac. Her research focuses on developing and refining methods we use to synthesize evidence and make evidence-informed decisions in healthcare.

Dr. McGavock: With training that spans dentistry, biostatistics, clinical epidemiology from institutions in Chile, Toronto, and McMaster, she brings a truly international perspective to this work. What makes Dr. Brignardello-Petersen's work particularly important for the emerging scholar is her expertise in systematic reviews and the GRADE approach, the Grading of Recommendations Assessment, Development and Evaluation system.

Dr. McGavock: These are gold standard methods for [00:03:00] synthesizing research evidence and assessing the certainty of that evidence, the skills that are absolutely essential, whether you're planning your first research project, writing a literature review, or developing clinical practice guidelines. We discuss the critical distinction between different review types, systematic reviews, narrative reviews, scoping reviews, as well as play-by-play for how to conduct a rigorous systematic review, and most importantly, how to use the GRADE approach to help us communicate the certainty of the evidence to decision-makers that emerges from that systematic review.

Dr. McGavock: For those of you planning your first systematic review or wondering whether this methodology is right for your research question, this episode will provide you with the foundational knowledge you need. So whether you're a graduate student contemplating your first literature review, a medical trainee trying to make sense of conflicting evidence, or a resident or fellow planning a systematic review as part of your own scholarly work, this episode is for you.

Dr. McGavock: Let's welcome Dr. Brignardello-Petersen to the Emerging Scholar Podcast[00:04:00]

Dr. McGavock: Okay. Welcome back to the Emerging Scholar podcast. We are here with maybe our fifth scholar luminary from McMaster University, Dr. Romina Brignardello-Petersen. Welcome to Emer- Emerging Scholar. Can you tell us a little bit about yourself?

Dr. Brignardello-Petersen: Thank you for having me. A little bit about myself, well, you already said I'm based at McMaster, which is the place where evidence-based medicine was born.

Dr. Brignardello-Petersen: So I came to Canada because of that. I'm originally from Chile. I was a practicing dentist. I noticed I didn't really know much about why we did the things we did other than we were told to do so in dental school. So got interested in research, learned that Canada was the best place to learn it. Moved here for my master's, then PhD, then postdoc, and here I am 15 years later.

Dr. Brignardello-Petersen: So that's m- pretty much the short version of my story.

Dr. McGavock: Birth by [00:05:00] fire, coming to a very cold climate Hamilton, Ontario, not known for the best winters, and just right into the heart of evidence-based medicine as a dentist, which is fairly rare.

Dr. Brignardello-Petersen: Yes. Actually, I studied at U of T. It's funny, I always tell the story that I wasn't good enough to study in McMaster, yet for some reason I'm good enough to teach at McMaster.

Dr. Brignardello-Petersen: But I made it into the waiting list for my master's. I'm still waiting. I never heard back from them. But I did go to U of T, and I did the clinical epidemiology program, and I was the first-ever dentist to be in that program. Awesome. Uh, that program is only for people who have clinical degree, but I was the first-ever dentist to be interested in that.

Dr. McGavock: Wow.

Dr. Brignardello-Petersen: Very rare at that... in 2010. Yeah.

Dr. McGavock: The... Dr. Hertzel Gerstein gave me the exact same s- scenario. Did not get into the clinical epi program at Mac, had to study at U of T for a little while, [00:06:00] and then they brought him back as a professor.

Dr. Brignardello-Petersen: It highlights how we learn from our mistakes at McMaster. We're always open to correct course.

Dr. McGavock: And the door is never closed. I think a really important teaching moment here for the residents is that you were a practicing physician, you're reading the evidence, you're trying to apply it in your clinical practice every day, and instead of just applying it blindly, you're asking yourself, "Where did this even come from?"

Dr. McGavock: Do you have an example there that really triggered that thought of, "I need to figure out more about how this evidence synthesis is created"?

Dr. Brignardello-Petersen: So when I was a dentist, I find myself just doing what felt very repetitive tasks. So it felt like every patient was pretty much the same. I worked at an emergency room, and it felt like it was, like, three or four things that we were doing, mostly driven by that's all we were told we could do.

Dr. Brignardello-Petersen: The- those were all the resources we had. But very rarely we would actually stop and think what we were doing or why we were [00:07:00] doing it. So once you start really looking into articles, at least in dentistry back then, and learning a bit about how you have to appraise these studies, you realize there are quite important limitations, and this is not as trustworthy as my professors might have thought that they should be.

Dr. Brignardello-Petersen: So then you get into the rabbit hole of, like, okay, how should we be making these decisions really based on evidence? What are the principles? And that's when you come across systematic reviews. The principle of for you to make decisions based on evidence, you need to know what all the evidence is, and systematic reviews allow you to do that.

Dr. Brignardello-Petersen: So it was, like, a natural evolution to learn and see that systematic reviews are a good investment of your time and effort as a user and as a producer.

Dr. McGavock: So, like, you kind of pull back the curtain. And being a critical physician, right? Someone who's there and questioning, like, "Why are we doing this?" And then, "How does this get generated?"

Dr. McGavock: Summaries and evidence base [00:08:00] that wasn't really guided by the best science.

Dr. Brignardello-Petersen: Unfortunately, dentistry has always been a bit behind medicine, right? So when you think about evidence-based medicine, it started in the early '90s, and then dentistry has always taken some time to catch up. So all of the advances that have made it to the world of evidence-based medicine, all the medical doctors who already had an idea that this is what we have to do, this might be the best way of doing it, dentists were five, 10 years behind.

Dr. Brignardello-Petersen: So it was disappointing, and sometimes it continues to be disappointing how we're always playing catch-up. But things are getting better, just like in every discipline. People are trying. People do want to learn, and it's our job to teach them how to do better as, again, users of evidence and also as people who produce evidence.

Dr. McGavock: Yeah, and it looks like you're starting to produce some really good evidence in this space as well, just looking [00:09:00] through your Google Scholar and your CV. So congratulations on making that jump from pulling teeth to pulling papers, I guess we could say. For novel trainees coming into this space, we may have heard the terms narrative reviews, scoping reviews, and systematic reviews.

Dr. McGavock: Can you give us a sense of how those are different and why a systematic review is closer to what we're looking for evidence-based medicine?

Dr. Brignardello-Petersen: Sure. These are all different types of evidence syntheses, and they all have their function. A narrative review, I usually like to explain it as the equivalent of a book chapter, something that you would go to when you want to start learning about a topic that you don't know anything about.

Dr. Brignardello-Petersen: For example, every time that I start working with a new group, let's say we're going to work in these new guidelines to address potential treatments for gout. If I don't know much about gout, my first choice would [00:10:00] be to go and find a good narrative review that tells me what is gout, how is it diagnosed, what are the characteristics of the people who are more likely to get it, what are the different treatment options, and so on and so forth.

Dr. Brignardello-Petersen: So narrative reviews are pretty good at giving you background knowledge that you need to have. Of course, the more advanced you are in your clinical career, the more background knowledge you already have. When you're training, you need to start getting familiar with everything. A narrative review might actually be very useful for that.

Dr. Brignardello-Petersen: They have their use. But of course, just like a book chapter, if you've written some of those, you kind of get to pick and choose what it is that you put in it, right? Yeah. You get to pick and choose what you cite as references or not. It is prone to some sort of biases based on your experience, your knowledge, and so on.

Dr. Brignardello-Petersen: So they're good for that purpose, but not so much for evidence-based decision making. Now, the scoping reviews [00:11:00] are a more reproducible type of evidence syntheses, but are the first step that we take when we want to start learning what is out there in a specific research area. For example, if I don't know what people have investigated, and I'll give you a dental example.

Dr. Brignardello-Petersen: In patients who have a dental infection, what is the evidence available about all of the possible antibiotic treatments, and who has conducted that research, in which countries, in which type of patients, like older patients, younger patients? Which specific type of infections? Which specific type of antibiotics?

Dr. Brignardello-Petersen: Have they measured? It kind of maps the evidence that's out there to tell you, oh, there are 20 papers that have actually addressed the effects of amoxicillin versus traditional penicillin. But it doesn't tell you what are the effects of [00:12:00] amoxicillin versus penicillin. It's just a map of what's out there.

Dr. Brignardello-Petersen: And we do need those maps because it tells you what are the potential gaps in the literature where it might be wise to actually start doing research that nobody has done. So it has that function in that way. But systematic reviews, getting to the actual main topic, are evidence syntheses that address a specific Question.

Dr. Brignardello-Petersen: The most traditional ones addresses questions about intervention. So what are the, uh, effects of A versus B on a specific type of patients? And then what you do, I'm assuming we're gonna talk about it with more details, but you collect all of the studies that have addressed that question, you put them together, you appraise them, and you come up with an answer, and the answer is, these are the effects and that outcome.

Dr. Brignardello-Petersen: So a very specific question with a very specific answer. And of course, their example is about intervention, but you could [00:13:00] also ask questions about what are the diagnostic properties of- Mm ... this specific test, or what is the prevalence of a disease in the specific population? So basically a specific question with a very specific numerical answer that you get through the literature, and you make a conclusion that considers the limitations of the evidence.

Dr. McGavock: In an ideal world, you'd have a testable hypothesis or an answerable question. Whereas with a narrative review, it's less of a question, it's more I wanna introduce you to this topic, and this is my point of view. The scoping review might be Would you start with a research question or would it be a very broad question like what has been done, what hasn't been done, where should the field go?

Dr. McGavock: I would assume that you would have a hypothesis that you could test within that systematic review and meta-analysis.

Dr. Brignardello-Petersen: So in systematic reviews, we do not test hypotheses.

Dr. McGavock: Oh.

Dr. Brignardello-Petersen: What we are trying to do is to find what [00:14:00] is the best estimate of effect. So if you want to put it back into statistical terms, it's not, uh, hypothesis testing, but it's actually estimation effect with its confidence interval what we are after.

Dr. Brignardello-Petersen: Uh, as systematic reviewers, we start with a very open mind, and we don't really know nor care very much about whether what we're going to find it's a benefit or a harm or if it's precise or imprecise. What we actually want to know is once we put all of the evidence together, what is the best answer to my question of what are the effects of A versus B?

Dr. Brignardello-Petersen: And that best answer is reflected by, I don't know, if the outcome is mortality, a risk ratio- Mm-hmm ... with its 95% confidence interval, and we make conclusions and we make statements based on that risk ratio and that confidence interval, and we absolutely [00:15:00] avoid words like statistically significant when we're talking about that effect estimate, and it's one of the things that we pay a lot of attention to.

Dr. McGavock: And now I need to go back to the systematic reviews that I have published to see whether or not I've actually started them with a research question or whether I've gone so far as to say we had a hypothesis, and that's why we're doing this podcast because I'm learning as much as everyone else.

Dr. McGavock: So what makes a systematic review systematic?

Dr. Brignardello-Petersen: One word that I repeat a lot when I'm teaching about systematic reviews is reproducible. That is the key to all research. Basically, the word systematic means you're following a process with specific methods that [00:16:00] you have thought out and implemented and described well enough that anybody who follows the exact same process as described should come to the same conclusions.

Dr. Brignardello-Petersen: That you are systematic means reproducible at every stage, from question formulation to deriving eligibility criteria for studies based on those questions that you created to where you're gonna search, how you're gonna search, how you're gonna treat the different articles that you come up with, which data you're gonna take from every article, how you are going to analyze the data, how you will assess if the studies have any methodological limitations, what other limitations of the evidence you will consider, and what is your conclusion when you put all of that together.

Dr. Brignardello-Petersen: So all of the steps on the systematic review have methods that are reproducible and that are [00:17:00] pretty well standardized today.

Dr. McGavock: When you're reading a very well-done systematic review or you're conducting one yourself, all of those steps are included in how you report what you did, so that reproducibility is relatively easy for the next person coming in.

Dr. McGavock: So every element of the search strategy, every place you did that search, all the hits that you came up with, how you did the screening, all of those methods need to be reported for this to be a high-quality, systematic scientific effort. Correct?

Dr. Brignardello-Petersen: Reporting what you did is one of the first steps to ensuring the trustworthiness of what you have done.

Dr. Brignardello-Petersen: Even though reporting checklists are not necessarily meant to improve quality, they tell you what you should write but not what the appropriate way of doing it is. They clearly help us ensuring that we're considering all of the things that we do need to be considering and reporting. And a [00:18:00] well-reported systematic review, it's more likely to be a good methodological quality systematic review and therefore more useful for whatever purposes people are using it.

Dr. Brignardello-Petersen: So reporting, it's absolutely important. But we're lucky that just like in trials, people have checklists. Like for example, for randomized trials, people have the CONSORT checklist that helps them with the reporting. We have the PRISMA checklist- Mm-hmm ... that tells you all of the different things that you need to report in a systematic review.

Dr. McGavock: So let's go from the start. Let's say we have a trainee coming in, and they're deciding they want to do a systematic review. The first place would be the question, and how would we start to formulate that question?

Dr. Brignardello-Petersen: The question is always the most important. So absolutely just like you would do for a primary study for a trial or for an observational study, we use the framework, the PICO framework, P for population, I for [00:19:00] intervention if it's an intervention systematic review, C for comparator, and O for outcome or an analogous framework depending on whether it's a diagnosis question, prevalence question, and so on.

Dr. Brignardello-Petersen: So the first thing is let's make sure that we have a question that has all of these elements explicit, and with sufficient detail that we all understand what we're talking about. I would say you have your trainee that comes through the room, and my first question to- for them would be who is going to be our senior clinical expert that is going to make sure that what we're doing makes clinical sense?

Dr. Brignardello-Petersen: 'Cause I can make sure it makes sense from a methods perspective, but from a clinical perspective, we need to have someone who absolutely knows what we're talking about.

Dr. McGavock: Right. So if you're looking at pediatric obesity or pediatric type 1 diabetes, we need someone there who's an endocrinologist who can help us formulate that, every element of that PICO.

Dr. Brignardello-Petersen: [00:20:00] 100%. We need content expertise in every systematic review that we do, even if the topic seems familiar. Even now for me today, like I cannot in good faith be the content expert for dental papers 'cause I have not practiced in 15 years. I'm so outdated that we need-

Dr. McGavock: Practicing people who are out there who can define it.

Dr. McGavock: And we just started tackling one. There's new technology out in type 1 diabetes. We're looking at new closed loop technology where you have an insulin pump talking to a continuous glucose monitor and bringing endocrinologists who are working in that space who understand the I. The I could be seven different things.

Dr. McGavock: In this case, what was really interesting is the C could've been five or six different things. So are we defining those? Like, what is included as the right comparator? What is the wrong comparator? Or how clearly do we define those interventions in that population? Because the more narrow we get, the harder it'll be to answer that question with a systematic review.

Dr. McGavock: But the [00:21:00] broader we get, it'll be a very muddy, very messy question. Do you have any guidance on how tight we should make that PICO framework?

Dr. Brignardello-Petersen: That is an excellent question, and it's the whole dilemma of we want to be as broad as possible to be clinically helpful, but not too broad that we are putting things together that don't make sense.

Dr. Brignardello-Petersen: So it's like the whole debate of you're mixing apples and oranges. That was the criticism of systematic reviews when they started appearing in the early '90s. Why would you put them together? The response to that would be like, "Well, but I am actually interested in the effect of fruits, not apples and oranges," right?

Dr. Brignardello-Petersen: And the counter-argument to that is what if apples and oranges do have different levels of taste satisfaction and moisture and whatever it is that people are interested in? So it is very important to know the clinical area. [00:22:00] I would say that the broader you go in a question, the more work you will have in your systematic review.

Dr. Brignardello-Petersen: You want to strike the right balance between answering a question that makes clinical sense but also that it's feasible to answer. How do we do that? In this debate of lumping versus splitting things, we try to ask ourselves if we lump two things, do we really think that these are exchangeable in practice?

Dr. Brignardello-Petersen: For example, for COVID-19, we had some interleukin-6 block, receptor blockers. We were debating whether we would treat them like both tocilizumab or sarilumab, whether they were the same and they were just called interleukin receptor blockers, or whether they were two things that we had to treat separately.

Dr. Brignardello-Petersen: The answer to those questions are usually it depends. And again, there are so many things that from the clinical perspective we need to be [00:23:00] knowledgeable of to determine what it depends on. But basically, every time that we are very confident that the two things are absolutely different, meaning they do have very different effects- Then you should keep them separate.

Dr. Brignardello-Petersen: If there is actually a chance that they might have more or less the same effect on the intervention, we start with a broader approach. So when we are doing our review, it comes to the step of data analysis, and if we actually see that the studies do seem to differ from one another, we go and explore and say, "Could it be that actually these two blockers were different and we shouldn't put them together?"

Dr. Brignardello-Petersen: We do have methods to explore that formally and to proceed depending on what we see. If we're absolutely sure they're different, of course you separate, but if not, we try to go as broad as possible so that way we enhance the applicability of the answers that we're [00:24:00] given.

Dr. McGavock: Yeah. I'm thinking I'm always gonna bring it back to a diabetes example 'cause that's kind of where I work.

Dr. McGavock: You know, you could say, "Let's take a look at all glucose-lowering medications." And then the clinicians are like, "Whoa, whoa, whoa, whoa." We are making significant clinical decisions here about different types and even different classes like hypertens- antihypertensives. Same thing. You'd never pool all of those because a really important clinical decision is which one of those are we gonna choose?

Dr. McGavock: Then also, I... hopefully we'll talk about this later, but from a patient perspective too, they may be asking, "Well, I don't wanna know whether all of these are gonna reduce my blood sugar. I'm really interested in whether this one is better than that one on this outcome." I think from a provider standpoint but also from a patient standpoint, wherever there's that equipoise or that need for that differentiation, you should be differentiating it in your systematic review

Dr. Brignardello-Petersen: Absolutely.

Dr. Brignardello-Petersen: The question needs to make sense from a clinical perspective. That's the first criterion that the question needs to pass. Even if [00:25:00] it's a perfectly methods question, like it has all the P, the I, the C, and the O, but if it doesn't make sense from a clinical perspective, it's a no-go.

Dr. McGavock: So then that takes me into the outcomes.

Dr. McGavock: We have options for binary outcomes, counts, right? Yes or no, dead or alive, disease or not, complication or not. And then we also have continuous variables that we could look at, so quality of life scores, diabetes distress scores, A1C, blood pressure. How do you make those kind of decisions when you're determining how precise you make that outcome?

Dr. Brignardello-Petersen: For the outcomes, our approach should be what are the outcomes that people want to know about when making those decisions? And by people, we mean primarily the patients, not necessarily the clinicians, 'cause sometimes those don't match, right? We teach our students your outcomes should be decided about what are the outcomes [00:26:00] that are the most critical or the most important outcomes for decision-making from a patient perspective.

Dr. Brignardello-Petersen: And so that's the primary criterion, and it doesn't matter if that's a dichotomous or that's a continuous outcome or if it's gonna be too difficult for me as a systematic reviewer to do the data analysis. Those are all secondary considerations. It's about what is important for decision-making. Yeah, I'll leave it there.

Dr. Brignardello-Petersen: I could elaborate more, but I would like to emphasize that's the main thing that we always pay attention to.

Dr. McGavock: Do you bring patient partners in to help you determine which outcomes are important or should be included?

Dr. Brignardello-Petersen: We should, right? We absolutely- We should ... should. Yeah. I know your question was whether we do it or not.

Dr. McGavock: Yeah.

Dr. Brignardello-Petersen: So I've been lucky [00:27:00] that a lot of the systematic reviews that I do are tied to guideline projects.

Dr. Brignardello-Petersen: And so as part of those efforts, we do have a group of experts that include patient partners, and the choice of the outcomes are absolutely informed by what these patient partners are telling us. W- when you ask a clinical expert about what are the potential important outcomes, they can come up with a list of 25, 30 outcomes easily.

Dr. Brignardello-Petersen: Yeah. Right? And so when we do guidelines and systematic reviews to support guidelines, we need to prioritize the top seven, for example. So that prioritization usually happens involving patient partners and considering their perspective.

Dr. McGavock: Then lastly, it's the quality of the evidence or the certainty of the evidence that you will be interpreting later, so then you also pick the study designs.

Dr. McGavock: And so in this case, depending on which study designs you choose to include Will help you with determining the certainty [00:28:00] of that evidence. Recommendations for which studies you should be including within your systematic reviews and how you make those decisions.

Dr. Brignardello-Petersen: For the study design, it depends on what your question is.

Dr. Brignardello-Petersen: We know, for instance, that when you have an intervention question, by default the best type of study to answer those are randomized clinical trials. Mm-hmm. Right? We know that is the case. And you might remember that whole hierarchy of evidence where we have, like, trials at the top. Except that it was a bit misinterpreted because no- very few people made it clear.

Dr. Brignardello-Petersen: But this is just for intervention questions, right? Because when we're talking about a harm question, you cannot have a randomized trial, so the best you're gonna have, the top of your pyramid, it's going to be perhaps a prospective cohort study.

Dr. McGavock: Mm-hmm.

Dr. Brignardello-Petersen: So the decision about which studies to include depends on, first we always try to include the ideal type of study to answer that specific question, [00:29:00] and we always say we are going to include them.

Dr. Brignardello-Petersen: But there might be situations where we know that we will not find anything, and we don't want to go through the exercise of a systematic review to end up saying that there are no trials. And so with our content experts, we are able to anticipate whether there will be trials or not. If we think there will not be, then we say, "Okay, so we will also include non-randomized studies of interventions that could give us some answer."

Dr. Brignardello-Petersen: And of course, we know that just by default it will be more prone to biases, and that's okay. It's still better to know what those studies are saying and acknowledge those limitations explicitly than to say we know absolutely nothing about this because we're refusing to look at that evidence. What is the ideal, and then if you anticipate the ideal is not available, what is your next best bet?

Dr. Brignardello-Petersen: And going for that so that you [00:30:00] are informing your audience.

Dr. McGavock: I'll ask a question maybe a little bit loaded, but, like, thinking about something like red meat, right? I was involved in one on non-nutritive sweeteners where you might have some randomized controlled trials where you're reducing red meat, and then you'll have a thousands of prospective cohort studies.

Dr. McGavock: I've seen people include both. So you would have some of the randomized controlled trial data Maybe some natural experiment data where for whatever reason red meat was not available in one area and then it was available, and so you'd have that level of non-randomized experimental evidence, and then you have a boatload of observational data.

Dr. McGavock: When would you decide to include all of this in the systematic review? I've seen people do that and I'm never really sure why.

Dr. Brignardello-Petersen: The way I would do it is that I would always try to make sure to have the best available evidence. So if I know that there will be three trials and that they have very few patients.

Dr. Brignardello-Petersen: Mm-hmm. I know already that [00:31:00] there will be so many limitations that when I assess how trustworthy that evidence is, my answer is going to be it's low certainty or low quality. If I know that I happen to have so many large observational studies with thousands of patients, and again granted they're gonna have their limitations.

Dr. Brignardello-Petersen: But maybe all of them together are still gonna give us a body of evidence that is more trustworthy than just the three trials. That's when you would go for including everything. When you're considering in the ideal world with unlimited resources, you would just take everything. Right. And you would see, okay, this is how much I trust in the body of trials.

Dr. Brignardello-Petersen: This is how much I trust in the body of observational studies. And then you would m- make your conclusions based on that. But when we're considering feasibility, we stop at the point where we think that going even further will not give us any more trustworthy evidence.

Dr. McGavock: And then [00:32:00] you have to call your shot.

Dr. McGavock: So you have to summarize all of those, your research question w- using the PICO format, the types of studies you're including and excluding, and then you have to register it. And why are we registering our systematic reviews now?

Dr. Brignardello-Petersen: Well, same reason why we are registering our trials, right? So to avoid any biases.

Dr. Brignardello-Petersen: To make sure that we are doing what we said we were going to do, and that we cannot manipulate the methods or choose methods that might give an answer that may be more closely aligned with what anybody in the team might want. That's one reason why we register them. It also has other potential benefits, like whenever a student of mine comes and they tell me they want to do a systematic review on a specific topic, the first thing that we tell them to do is go see if [00:33:00] someone else is doing it already.

Dr. Brignardello-Petersen: Because if they are, there's no point in duplicating the efforts. If you're going to be almost done and there is going to be the publication that came out from this group, then it's going to be really hard for you to disseminate your systematic review, and it's gonna be an, a not so good investment of resources.

Dr. McGavock: So you need a really good question, then go back, check and make sure no one else is doing that same question. Although in my experience, I have found that sometimes when we develop the question, we realize a few other people have tried it and they've done it very poorly Which I will transition to now.

Dr. McGavock: One of the things that I've found is they don't always search the largest possible resources to find the clinical trials or to find the observational studies of interest. I guess once you've registered in PROSPERO, your next step is to create a search strategy and then identify which information sources you're gonna go with.

Dr. McGavock: And in your paper here that we're gonna reference in the show notes, you give a really nice list [00:34:00] of your different options. We have general medicine, then specialized, then ones that are specific to certain regions. And so how have you gone about choosing those different information sources within your systematic review?

Dr. McGavock: And do you have any guidelines for residents for what to choose and maybe what not to choose?

Dr. Brignardello-Petersen: In the ideal world, we would search everywhere, right? When you read old papers about systematic reviews, the advice would be like, "Search everywhere," and if you're not searching for gray literature, it's a sin.

Dr. Brignardello-Petersen: There are, of course, sources where we absolutely need to search. So the largest electronic databases like Medline or Embase, we should always be searching on those. Then depending on the specific topic of your clinical question, there might be other databases that are mandatory for you. So for instance, if my systematic review has to do with acupuncture, I can already anticipate that there- Mm-hmm

Dr. Brignardello-Petersen: might be many trials [00:35:00] that were done in China. And so looking on Chinese databases, it's a must because if I do a systematic review that only includes databases that index journals or just the biggest journals from there, it's going to result in me missing a lot of articles. When we do systematic reviews about values and preferences or acceptability or questions that are answered through qualitative studies, we need to search in databases that are specific for those So I would say always search in the largest electronic databases.

Dr. Brignardello-Petersen: Evaluate if there is a need for another electronic database depending on your question. The value of gray literature depends on the specific field that you're dealing with. Most of the time, what we end up finding in gray literature are smaller studies that are not very well [00:36:00] reported. Yeah. They have fewer patients.

Dr. Brignardello-Petersen: Yeah. So when you put them together with what you find from the largest electronic databases, they don't change your conclusions a lot. And so many of the times they feel like, "Was this effort worth it?" But that's where the content expertise, once again, will play a big role. If your content experts can tell you like, "Absolutely, this drug was assessed by the pharmaceutical company, and they only published the reports on their website, and they never really published them in a peer-reviewed journal."

Dr. Brignardello-Petersen: Of course, if you have the time and effort, go ahead. It doesn't hurt. But most of the time we do have limited resources

Dr. McGavock: In my experience, the best way to start this is to talk to your librarian. There's usually a [00:37:00] librarian with expertise in this area in terms of creating the search strategy, knowing where to look. In my field, SPORTDiscus will have some randomized control trials, and one we just did about cycling infrastructure, there were a number that were published in transportation journals.

Dr. McGavock: And so there's a whole transportation literature that includes data on cycling. And if people have looked at natural experiments with pre/post controlled changes to the built environment, many of those get published in a transportation journal that do not follow the same reporting guidelines, but have a wealth of natural experiments.

Dr. McGavock: And if you don't include them in your search strategy, you miss 30% of the published literature out there. Thanks in large part to our librarian, we discovered all these transportation journals to search for. So once you've created your search strategy, you get 1,000 or so hits. How do you determine what gets included in the data extraction and then potentially in the analysis?

Dr. Brignardello-Petersen: So you follow your [00:38:00] eligibility criteria, which are in alignment with your question that you did at the beginning. So just to give you one example, we can be talking about a specific antibiotic as a intervention, amoxicillin. But your eligibility criteria has to be a bit more specific and say, for instance, is it any regimen, any doses given, like absolutely any regimen of amoxicillin or a specific one, right?

Dr. Brignardello-Petersen: So for each of your PICO elements, you create eligibility criteria, and then after you have your results from your search, you have reviewers who look first in title and abstract and see if the title and abstract seem relevant or not. If yes, they move it to the next step. If not, it's excluded at that point.

Dr. Brignardello-Petersen: And then all of those articles that after title and abstract seem relevant, we collect the full texts, and we evaluate whether the article meets all the eligibility criteria. And we include in [00:39:00] our systematic review all of the papers that meet all of the eligibility criteria and give us data for at least one of the outcomes that we are interested in.

Dr. Brignardello-Petersen: And when we have that-- once we have that, what we do is we get the data that we need from those studies, and usually that data, again, is very related with our specific questions. So we extract characteristics of the specific population, so like the demographics, age, t- maybe clinical severity of the condition that we're dealing with, characteristics of the intervention of the comparator, how the outcomes were measured, and outcome data.

Dr. Brignardello-Petersen: For example, for mortality, out of the people who received this intervention, how many died? What was the total, and so on. We do not extract the conclusions that- Mm-hmm ... make our paper much better or try to make our conclusions more relevant than what they are. And so the conclusions are not really helpful, even though I still see them in some [00:40:00] systematic reviews.

Dr. Brignardello-Petersen: But we are not supposed to use any of those. It's just data about the methods and the results of the studies.

Dr. McGavock: Then when you've extracted that data, we use a template called Rayyan, and Rayyan helps us double-blind screen, so two independent reviewers. Wherever there's papers that do not align, like whether one person excluded, one person accepted, that gets pulled somewhere else.

Dr. McGavock: A third person comes in and helps them do that. And then once we extract the data, then we have to make some hard decisions about should we be pooling all of this information in the same meta-analysis? Are there criteria that you go by? Mm-hmm. In one case, you say mortality. That's easy, right? You have an outcome, it's gonna be very easy to analyze.

Dr. McGavock: In other cases, it might have been measured differently, and then you may have to choose whether or not you meta-analyze that or how you would meta-analyze that. Is that correct? What are some of the things that you go through? And also in that context too, harms, so adverse events. That's another thing that we would [00:41:00] always extract from our systematic review.

Dr. McGavock: But in terms of the analysis, how do you go about determining whether you're gonna pool all that to meta-analyze it?

Dr. Brignardello-Petersen: That is an excellent question, and it goes back to our conversation about whether we go broad or more narrow, right? So the narrower you go, the more specific you are in terms of your criteria to pool The more comfortable you're going to be because you don't feel like you're mixing apples and oranges.

Dr. Brignardello-Petersen: Sometimes those decisions result in meta-analysis that have very few studies, very few patients. There's just so much you can say about that. Most of the time we take a broad approach to pooling. So in terms of the outcomes, for instance, you were bringing up mortality. But then there are outcomes like quality of life that people measure in many different ways, or function that can be measured in many different ways.

Dr. Brignardello-Petersen: The first step that we follow to decide whether we're pooling [00:42:00] those studies together, assuming everything else is okay in terms of population intervention comparator, is with the content expertise we determine are all of these different papers measuring the same construct? Is this paper that is measuring function going or trying to capture the same thing that this other paper that used a different scale that's measuring function, and is that the same that the other one?

Dr. Brignardello-Petersen: Mm-hmm. And in many cases the answer is yes. So like with quality of life, you can have an SF-36 scale or an EQ-5D, or sometimes you have the WOMAC quality of life or function. But even if you sometimes have different scales, they're all trying to measure the same construct. And then if that is the case, then we should be pooling them together, and we have statistical strategies that we use to do that.

Dr. Brignardello-Petersen: But the main question again goes back to the content [00:43:00] expertise of are all of these studies getting at the same question? If the answer is very likely yes, then we are comfortable pooling them.

Dr. McGavock: Then we have to make a call about if one scale goes to 100 and the other scale goes to 35, how are we gonna analyze the differences between the intervention and control arms?

Dr. Brignardello-Petersen: We absolutely consider that and- Yeah ... uh, we do need to make some transformations and some tricks here and there, but we can always pool more than we initially think we could pool. The more systematic reviews we do, the more we realize you can always pool more than you thought you could. With the right methods, in this case statistical expertise, such as what are the statistical tricks to deal with the scales having different ranges, perhaps different directions.

Dr. Brignardello-Petersen: Of course we consider all of that, but we absolutely can pool them.

Dr. McGavock: So now in addition to your content expert, your librarian, your methodological expert, you also need [00:44:00] a statistical expert who's got experience analyzing this data. So it's a team of at least- It takes a village It takes a

Dr. Brignardello-Petersen: village. It takes a village.

Dr. Brignardello-Petersen: It's a teamwork, absolutely.

Dr. McGavock: This isn't something you're gonna do on the side of your desk. So now it's in. You've meta-analyzed the data. Now we need to assess the certainty of the evidence, and I think this is where GRADE comes in. And so you've been intimately involved in creating and promoting GRADE, which is a tool for grading evidence.

Dr. McGavock: Can you tell us a little bit about what GRADE is and why that's important?

Dr. Brignardello-Petersen: Sure. So the way we introduce GRADE when we teach is we usually show this slide that has the meteorologist, where they're saying something like, "I think there is a 40% chance that it's gonna rain, but only a 10% chance that we know what we're talking about."

Dr. Brignardello-Petersen: That 40% chance of rain is usually what we make analogous to the answer we get from a meta-analysis, the risk ratio from mortality with its confidence [00:45:00] interval. But then the 10% chance that we know what we're talking about is what we make analogous to what we call the certainty or the quality of the evidence.

Dr. Brignardello-Petersen: Whereas we try to assess systematically and communicate to people how much we truly trust what we have in front of us. We cannot just take at face value those numbers that we have in front of us. Many of the studies might have had limitations in terms of their study design, and we assess them at high risk of bias, for instance.

Dr. Brignardello-Petersen: Even if we have all of the trials, it so happens that there were only three or four, and there were only a total of 100 people in some disciplines. And that, again, we're not as confident in that situation as if I told you, "Oh, we have a meta-analysis with 20 studies and 3,000 patients." Right? Mm-hmm. So, so what we have in front of us with the meta-analysis can also have [00:46:00] limitations, which we call limitations in the body of evidence.

Dr. Brignardello-Petersen: We need to assess these systematically so that we can come up with what we call a level of certainty. GRADE provides us this methodological framework for us to reproducibly assess all of the potential limitations of the evidence and end up telling our readers of the systematic review whether this evidence is high certainty, moderate certainty, low certainty, or very low certainty.

Dr. Brignardello-Petersen: And of course, our conclusions are going to be different depending on which level of certainty we are dealing with.

Dr. McGavock: Which is super important because you might say your highest level of ev- evidence is a systematic review and meta-analysis of randomized control trials that have been done in the last 15 years, but the quality of that review is only as good as the quality of evidence that's inside of it.

Dr. McGavock: Pooling a bunch of bad data does not make it better. [00:47:00]

Dr. Brignardello-Petersen: I guess the nuance there would be that the quality of the systematic review itself is somewhat independent from the quality of the evidence, right? I can do a perfect systematic review where just because of the field that I'm working on ends up having very low quality evidence.

Dr. Brignardello-Petersen: But my systematic review, my process is still reproducible. It minimized all biases. It minimized all errors. It, it's perfectly done and reported. So my review is a good quality review. But the evidence, it's very low certainty. And it's my job as a good quality systematic reviewer to actually call it out and say, "This is very low certainty evidence."

Dr. Brignardello-Petersen: Of course, if I do a terrible job at my systematic review, like nothing else matters 'cause I shouldn't even be looking at- Right ... those results. But the certainty of the evidence absolutely tells us the limitations in terms of [00:48:00] what conclusions we might make. This specific procedure reduces mortality, or this specific procedure may reduce mortality.

Dr. Brignardello-Petersen: Those are two different statements. Uh, and we make those intentionally in systematic reviews after using GRADE to assess the certainty of the evidence.

Dr. McGavock: And so that'll influence whether that's ready for clinical application or give the clinicians who are reading this a sense of how ready they are to maybe apply that to their clinical setting, to the bedside.

Dr. Brignardello-Petersen: So yes, we're giving them the best conclusions we can in terms of what are the effects of the interventions. And of course, they will put, uh, all of that together with any other important factors for their own situation, like their patient's values and preferences. Mm-hmm. Depending on where they practice, if there are any resource or [00:49:00] acceptability limitations.

Dr. Brignardello-Petersen: And based on all of those factors, they will make a decision. But of course, their decision might be very different if my systematic review is telling them, "This drug for sure reduces mortality," versus I'm telling them, "It seems to reduce it, but I have very low certainty in that, so it could go either way."

Dr. McGavock: So there are five things that go into this GRADE evaluation, five domains, and there's limitations to the body of evidence in five different categories. So there's limitations to the study design, inconsistency, imprecision, indirectness, and lastly, publication bias. Do we wanna do a little bit of each of those to see how they line up to give you a sense of what the GRADE or the certainty of the evidence is?

Dr. Brignardello-Petersen: Absolutely. [00:50:00] So for risk of bias is the limitations of study design is the synonym of risk of bias. So it's the one we all know. Other people call it methodological quality. If the studies have limitations in how they have been conducted to the extent that we believe that the effect estimates that the studies are giving us are biased, then of course we are less confident in whatever it is that we get from our meta-analysis.

Dr. McGavock: And there's usually a, some type of graph at the end of your systematic review meta-analysis about the risk of bias in each different domain of the trial design, right?

Dr. Brignardello-Petersen: Yes. So there are tools that help us assess the risk of bias for each of the studies that we're including, and these tools are organized in domains.

Dr. Brignardello-Petersen: So for example, in a trial, the first domain has to do with randomization, whether participants were appropriately randomly allocated to the groups. The readers of a review should be able to see a figure where you see what is the risk of [00:51:00] bias of all of those studies. And of course, if 90% of the evidence in my meta-analysis comes from studies at high risk of bias, then I'm not as confident as I would be if it's just 10%.

Dr. McGavock: That's kind of standardizing something we're familiar with. Inconsistency, how do we define that?

Dr. Brignardello-Petersen: So inconsistency has to do with disagreements across studies. The easiest way for me to understand that is, let's say you have 10 studies, and out of the 10 studies, five tell you that the drug reduces mortality and five tell you that it increases mortality.

Dr. Brignardello-Petersen: And now you put all of this together in a meta-analysis and your pooled estimate is in the middle. No effect. And your answer is that it doesn't do anything to mortality. I, I can see from your reaction that's already making you uncomfortable.

Dr. McGavock: Yeah,

Dr. Brignardello-Petersen: no. That it doesn't make sense. We can understand why we're seeing what we're seeing, but that doesn't make sense, and [00:52:00] you're not confident on that pooled estimate.

Dr. Brignardello-Petersen: Whereas if, on the other hand, you had all of the 10 studies telling you that it reduces mortality or all of the 10 studies telling you it, it has a trivial effect-

Dr. McGavock: Mm-hmm ...

Dr. Brignardello-Petersen: then you would trust your pooled estimate more, right? So disagreements across studies make us uncomfortable about this pooled estimate.

Dr. Brignardello-Petersen: And so whenever we have disagreements across studies that we cannot explain, then we do what we call rate down the certainty of the evidence because of inconsistency.

Dr. McGavock: So two things come to mind here. One, randomized control trials where one or two have humongous effects And seven or eight have negligible or very small effects.

Dr. McGavock: And your final point estimate for the intervention is influenced by those two that are kind of outliers. Would that influence your consistency or your inconsistency estimate?

Dr. Brignardello-Petersen: It would. [00:53:00] And it's what we call a small study effect when you have like one or two smaller studies that are changing your whole picture.

Dr. Brignardello-Petersen: So when we address inconsistency, what we do is to see like, of course, there might be some trials that seem slightly different from the other ones, but the question that we ask ourselves, "Is this making a big difference or not?" In the first example I gave you, I told you there are five here, five there, pooled estimate is in the middle.

Dr. McGavock: Mm-hmm.

Dr. Brignardello-Petersen: Of course, it's making a difference. The five here and the five there are doing something so that the pooled estimate is in the middle. But if you have a situation where you have eight studies that are giving you a message and then two that seem off, but the pooled estimate is still pretty much consistent with those eight studies, they're not really making a difference in your conclusion.

Dr. Brignardello-Petersen: So that wouldn't be important inconsistency. Whereas again, if those two studies [00:54:00] are actually changing your conclusion, this is important inconsistency and we should act accordingly.

Dr. McGavock: Sometimes the analysts working on our study will rerun the analyses and remove outliers And see what happens to our estimate.

Dr. Brignardello-Petersen: It's interesting. Whenever I teach about inconsistency to my students, the first slide that we always use when we see an outlier-

Dr. McGavock: Mm ...

Dr. Brignardello-Petersen: check if these are mistakes. 'Cause a lot of the times they are. Okay. Whenever you see something off in a meta-analysis, the first thing that I always go do and I ask my students to do is check if we made a mistake here in extracting the data.

Dr. Brignardello-Petersen: Sometimes something as little as, I don't know, mistakenly including a standard error- Mm-hmm ... instead of a standard deviation- Mm-hmm ... can be the problem. Or a typo. You just wrote like a two instead of a three or something. Mm-hmm. The first one is check if there are mistakes. But when we do see outliers or studies that are [00:55:00] inconsistent, the first thing that we go do is explore what might be the reason.

Dr. Brignardello-Petersen: Sometimes the reason might be those reasons that we were talking about in the beginning. Perhaps I did go too broad. Perhaps I should not put together all the interleukin 6 receptor blockers. Right. And so what we do is what we call a subgroup analysis, where we separate them and we say, "Okay, so what happens if we actually put the studies apart, the ones with the tocilizumab and separately the ones with the sarilumab?"

Dr. Brignardello-Petersen: And if it seemed like, yes, that was actually the cause, then the course of action is, okay, we should not gone as broad. We should keep the analysis separate for each of them.

Dr. McGavock: Right.

Dr. Brignardello-Petersen: So I would say more than let's remove outliers is let's figure out why this is happening. Of course, if we cannot find a reason and we think that this one study is having too much impact, we might still have some techniques to [00:56:00] deal with that.

Dr. Brignardello-Petersen: Sometimes we change our statistical analysis model. We've noticed that sometimes the random effects model that serves us so well in most instances might not be the most appropriate in this circumstances, so we do have strategies.

Dr. McGavock: Thinking about the last systematic review that we did here, it could be related to the comparison group, right?

Dr. McGavock: If we're looking at a certain diabetes medication or a certain diabetes technology, if the comparison group is very closely related to that, we're gonna see a smaller effect size. But if the comparison group was anything else in care, you could see huge effect sizes because you have a number of people that are not using any technology at all included in the comparison group.

Dr. McGavock: And so when they're compared to someone who's adopting a new technology, they could see a massive effect. The duration of the intervention was one. If you're only using it for two months and then you're comparing that to someone who's been using it for 8 or 12 months, you could see massive differences there.

Dr. Brignardello-Petersen: Exactly. And in those cases, the specific [00:57:00] package of interventions w-within the comparator or the duration of the intervention are what we call subgroup variables. These reasons that we already knew upfront might be a problem. And so when we see these studies that are inconsistent, those are the first reasons that we go explore if they're actually causing the inconsistency.

Dr. McGavock: So check out the inconsistency. Do not remove the outliers. You'll probably find a story in there and then think maybe about a subgroup analysis that's clinically relevant. Mm-hmm. Uh, what is imprecision?

Dr. Brignardello-Petersen: So imprecision is related with the statistical uncertainty, which in the case of a meta-analysis is reflected by the confidence interval of the pooled estimate.

Dr. Brignardello-Petersen: So we have our pooled estimate for our outcome. Let's say we're talking about pain, and so the mean difference in pain is minus two, and the confidence interval goes from [00:58:00] minus three to minus one. When we deal with inconsistency, we kind of assess, is this confidence interval too wide to the extent that we are not confident in what we're saying?

Dr. Brignardello-Petersen: Mm-hmm. Do I think that a minus three on a scale of 10 is a pretty similar conclusion that a minus one that was the other extreme of my confidence interval? If the answer is yes, then considering statistical uncertainty, I'm still fairly confident. But if we were looking at a confidence interval that goes from minus seven to plus three, now we're talking about, well, there is an, a very large reduction of pain on one side.

Dr. Brignardello-Petersen: There is an increase on the other side. Definitely too much statistical uncertainty that is imprecise.

Dr. McGavock: Which affects your certainty of that evidence, and it could be related to, in the case of a dichotomous [00:59:00] outcome, you just don't have enough people who had the outcome, even though there are multiple randomized trials.

Dr. McGavock: Or in this case, the tool that you're using to measure it is very broad, somewhat messy, so you could get a range of values, and so that's gonna influence your precision in the point estimate for that intervention.

Dr. Brignardello-Petersen: Correct. It's, uh, the sample size. It's always something like how many patients you have in total in your meta-analysis.

Dr. Brignardello-Petersen: It's always related with your precision. And like you said, for the dichotomous outcomes, how many people are experiencing the outcome will also be reflected in your pooled estimate confidence interval.

Dr. McGavock: And what's indirectness?

Dr. Brignardello-Petersen: Indirectness, it's a term that is the opposite of applicable or generalizable.

Dr. McGavock: Hmm.

Dr. Brignardello-Petersen: So indirectness, it's basically lack of applicability to your target question. So an example that I always [01:00:00] use is that we are doing a systematic review that has to do with antibiotic prophylaxis before any tooth extraction And so we have this broad question about any tooth extraction, and we go and search for trials.

Dr. Brignardello-Petersen: It so happens that most of the trials that we find have been done in patients who are undergoing wisdom tooth extraction, not just any. So these trials meet the eligibility criteria. In my systematic review, just because of the gaps in the research world, I just have a very specific sub-sample within my question.

Dr. Brignardello-Petersen: So if now I believe that this meta-analysis answer will not apply to everybody, then this is an indirectness problem. And of course, the way we assess it, again, the content expertise. I should know, like, okay, why do I think this might not apply to any tooth extraction? [01:01:00] And the reason would be, well, 'cause extractions of wisdom teeth are much more complex than other extractions.

Dr. Brignardello-Petersen: They usually require a bigger surgery and osteotomy, and it's so different than just extracting a front tooth because of trauma or periodontal disease, for instance. Any lack of applicability where you think that what you found in your review might not necessarily apply to your whole target or your whole broader systematic review question.

Dr. McGavock: Where I see that in my world of physical activity is the majority of studies looking at physical activity and mortality or any kind of hard endpoint are always done in populations who are over 55. If someone's asking me, "Is this a universal finding?" I said, "Well, if you're 20 We don't actually have data on that right now.

Dr. McGavock: The other place was during the COVID pandemic. Early on, we didn't have any data on kids. You could say there's a 95% [01:02:00] protection against COVID if you're vaccinated, but that was in these really select populations. And so in both those cases, does that fall into the indirectness where if you're trying to apply this broadly, if the studies are only in limited samples of highly restricted individuals, you have to wait a little bit before you can make that call as to whether it applies to your patient population?

Dr. Brignardello-Petersen: It applies as long as you have a very good rationale to believe that it doesn't apply. Mm-hmm. So your example had age involved, and I would agree with you absolutely that the recovery from a 20-year-old patient might not be the same as a 60-year-old patient. But perhaps if we were talking about a 45-year-old patient, they're closer to what you have in the studies.

Dr. Brignardello-Petersen: Perhaps the effect modifiers, things like lifestyle or their healing process is different. And therefore, even though you might not have evidence necessarily in people age [01:03:00] 45, you can be fairly confident that the evidence applies when it comes from those that are perhaps older than 55. But again, that's a content call.

Dr. McGavock: Right.

Dr. Brignardello-Petersen: We always try to be very cautious with those. You brought up the pandemic too. Again, at the beginning, we were all synthesizing evidence from other diseases. I remember a lot of systematic reviews about MERS at the beginning or the systematic reviews we did about ventilation techniques. They were not just for, specifically for patients with COVID.

Dr. Brignardello-Petersen: And then of course, we would assess like this, this is the closest population we have to what we're looking for.

Dr. McGavock: Right.

Dr. Brignardello-Petersen: How much evidence do we truly have that this would not apply as such? Many times we are actually okay. So to give you another example, I worked once in a guideline for bleeding disorders where we were looking for evidence in people with von Willebrand disease.

Dr. Brignardello-Petersen: Very limited evidence. But our [01:04:00] guideline panel was confident that we could use evidence from people with hemophilia. Hmm. Where there was much more evidence. And they felt confidence to the point that they didn't think it was necessary to rate down the certainty of the evidence for indirectness because they thought that whatever was happening in the interaction between the condition and the drugs we were looking at was not going to be different in one group versus the other.

Dr. Brignardello-Petersen: So even though it reads like a completely different diagnosis, this wasn't indirect enough for us to rate down. So it's a content expertise call.

Dr. McGavock: I really like that. If there isn't a reason why you can't rule it out, then it's possible that it applies.

Dr. Brignardello-Petersen: And it's better to use that than to use nothing, right?

Dr. McGavock: Exactly. Equipoise falls in that category, I would think. Lastly, publication bias.

Dr. Brignardello-Petersen: Publication bias is when we [01:05:00] fail to find studies that have been done for different reasons. We either fail to find them or they have never been published. So it's this whole phenomenon that most of us have experienced that when you do a study that is not interesting in terms of the findings, or you are not able to recruit a large group of people, it's so much more difficult to get it published- Mm

Dr. Brignardello-Petersen: than if you actually found a statistically significant result. And so we're always aware that smaller studies with negative results are harder to publish- Mm ... and therefore are also harder to find.

Dr. McGavock: Mm-hmm.

Dr. Brignardello-Petersen: And so publication bias has to do with the extent to which we believe that those studies exist for our clinical question to the point that they would change our conclusion.

Dr. McGavock: So the studies exist, the results are there, they just have never made it into the literature that we searched. And so we actually do a [01:06:00] test for that at the end to see whether or not we think there was some degree of publication bias based on the results we have in our study. Do you have faith in those publication bias estimates?

Dr. Brignardello-Petersen: Unfortunately, we're not able to do the tests most of the time because we need a minimum number of studies, right? So we say that you should not be using those statistical tests for publication bias when you have less than 10 studies 'cause they don't have enough power to detect it. So we do them when we can.

Dr. Brignardello-Petersen: But again, I've repeated so many... I, I feel like I'm repeating too much, but content expertise. Right. Your experts truly know what is the likelihood that pharmaceutical companies have done studies that they didn't publish and that they're keeping from you, or that they did 10 studies and that they only published one.

Dr. Brignardello-Petersen: And they know when you should be suspicious about those things. So as systematic reviewers, we do our best to [01:07:00] find everything that we can find. The more we suspect those studies might exist, the more efforts we make into going into gray literatures. But sometimes we're not successful, and again, we need to make those decisions based on the likelihood that this is happening if we don't have enough studies to do the tests.

Dr. McGavock: Awesome.

Dr. McGavock: I think we covered a ton. If you were gonna give a message to trainees who are thinking about doing systematic review with meta-analysis, what would you say? Where should they start? Any other starting points you think we should look at?

Dr. Brignardello-Petersen: Getting a team. The team is the most important. I think we talked about different types of expertise.

Dr. McGavock: Mm-hmm.

Dr. Brignardello-Petersen: And so getting your team together. Make sure that you have your content expert if you are not the content ex- expert. Make [01:08:00] sure you have a methodologies friends, and make sure you have a buddy who will be screening and extracting data- Yeah ... and doing all of these things because it's very unlikely that your content expert or your methods expert will be doing those things- No.

Dr. Brignardello-Petersen: Yeah ... with you, right? So find your librarian, find your content expert, find your methodologist, and get people to help you.

Dr. McGavock: Awesome. Is there anything coming up in this area that we should be aware of or, like, paying attention to that you find really exciting?

Dr. Brignardello-Petersen: So there are a lot of methodological papers about how to use these principles for assessing the evidence.

Dr. Brignardello-Petersen: What we covered today was just related to how you assess it in a systematic review, but GRADE also has this other side of how you take the evidence and move it to clinical decisions. And so lots of methods work, lots of complexity. So CORE GRADE is an initiative by Dr. Guyatt [01:09:00] where we have written a series of papers that present the basic principles that people should know.

Dr. Brignardello-Petersen: And if you know those, you will get by 95% of the time and don't need any more complexity in your life. And these papers are very user-friendly. They are being published currently at the BMJ. Okay. I think we're on paper four at this point. They've been published one weekly for the past couple of weeks. I would encourage people if they want to learn more about assessing how trustworthy the evidence is, CORE GRADE will be an excellent resource.

Dr. McGavock: That's awesome. We're recording this in May 2025, two weeks before Dr. Guyatt is giving the Henry Friesen lecture here in Winnipeg. So we will link to those papers in this, and we'll have a follow-up interview with him to go into more detail about that.

Dr. Brignardello-Petersen: Oh, I'm sure he will love it. He's all over social media with his LinkedIn videos and promoting the CORE GRADE papers.

Dr. Brignardello-Petersen: It's-- We absolutely believe that things need to be [01:10:00] made easier- Mm-hmm ... for systematic review producers and also users. So CORE GRADE will be an excellent resource.

Dr. McGavock: And clinician scientists absolutely need to know about this stuff because you're making everyone's life easier to make the best decisions at the bedside so that you have these tools in your hand.

Dr. McGavock: So that's one of the reasons we're doing this podcast. I'm so grateful that you came on to share this. We'll make sure we link to that. And we'll also think about doing another podcast on network meta-analyses. If there are residents out there, whether you're at Mac or not, Dr. Brignardello-Petersen is potentially there to support you.

Dr. McGavock: Trainees at Mac for sure can probably reach out to you to say, "I'm interested in doing something. Can we partner?" And then hopefully there are content experts around in your area as well that you can find to help you with your first systematic review and meta-analysis. So thank you again for this. This was amazing.

Dr. McGavock: Really in-depth dive into this. I wish I would've done this two years ago myself so that, uh, that would've informed some of the systematic reviews that I did. I look forward to sharing this with everyone else. So thanks again.

Dr. Brignardello-Petersen: Thank you for having me.[01:11:00]

Dr. McGavock: What an incredibly valuable conversation. A huge thank you to Dr. Romina for joining us today and for breaking down the complex world of systematic reviews and GRADE methodology, as well as walking us through that groundbreaking paper that was published in twenty twenty-five in the American Journal of Epidemiology.

Dr. McGavock: My favorite takeaway from this might be the opening line that Canada is the best place for research. I could not agree more. For those of you who are listening who are thinking about conducting your first systematic review, there are a few key takeaways for you to remember. First, not every question needs a systematic review, so make sure that your question is appropriate for this method.

Dr. McGavock: Second, don't go it alone. We heard time and time again through this interview how important it was to have content experts, methodologists, even potentially members of the patient partner group that you're interested in translating this evidence for. Don't go it alone. There are excellent tools available to you and resources [01:12:00] from the Cochrane Handbook to GRADEpro.

Dr. McGavock: A series of really important papers just came out in the BMJ outlining how to use GRADE. Those are all in the show notes. And third, remember that the goal isn't just to synthesize evidence, but it's to assess the certainty of that evidence so that decision-makers have a better opportunity to translate that into clinical practice guidelines and use them on the bedside.

Dr. McGavock: This episode really did connect beautifully with our conversation with Dr. Guyatt and was essentially a chapter two of maybe four chapters. Together, these two episodes conduct part of an arc on evidence-based medicine. I think we have a few more chapters to go that I look forward to digging into. And I highly recommend that if you enjoyed this, go through the two papers that are in the show notes in-- published in the American Journal of Epidemiology about systematic reviews, an introduction to methods, and how to interpret them.

Dr. McGavock: Thank you again for joining us on the Emerging Scholar Podcast. Until next time, keep asking those great questions, keep pursuing rigorous methods, and keep [01:13:00] building the evidence base that will improve both health and healthcare for us all.

Dr. McGavock: The Emerging Scholar podcast was recorded in the UMFM studios located at the University of Manitoba in Winnipeg, Manitoba, Canada. This is Treaty One territory, the original homeland of the Anishinaabe, the Ininew, and on the national homeland of the Red River Métis. Manitoba is also homeland to the Anishininew, the Dakota Oyate, and the Dene peoples.

Dr. McGavock: We acknowledge that our water is sourced from Shoal Lake 40 First Nation. As a settler scientist working in health research, I acknowledge TRC Call to Action Number 18, that health inequities between Indigenous and non-Indigenous people are the direct result of colonization. Winnipeg is the center for reconciliation in Canada, and where we are speaking from today is at the crossroads of Canada's history and Canada's future.

Dr. McGavock: We are three [01:14:00] kilometers away from one of the most sacred meeting grounds for Indigenous people on the plains and within Turtle Island. We are also several kilometers away from one of the last residential schools in the province. We are working collectively on reconcili-action with youth, elders, and community leaders to heal from the trauma of colonization, past and present, honor the original treaties, and create a Canada that addresses and eliminates the inequities and injustices of the past and present to create a future for Indigenous children to live and thrive.

Dr. McGavock: On behalf of the team at the Emerging Scholar podcast, we want to thank Osani Balkaran for the great production for our opening and closing themes. We also want to thank UMFM 101.5, especially our executive producer, Jared McKetiak, for technical assistance and production on this project. We also want to thank all of our guests for the discussion and the insight that they've provided.

Dr. McGavock: Once again, thank you for listening. Be sure to subscribe to the Emerging Scholar podcast on [01:15:00] Apple Podcasts, Spotify, and umfm.com.