Design Principles for Data Analysis with Dr. Lucy D'Agostino McGowan

Dr. McGavock: [00:00:00] Welcome to the Emerging Scholar Podcast, season one. We hope you enjoy this series of great conversations with leading medical researchers that's designed for graduate students, medical trainees, residents, and fellows to help you on your journey towards your first scientific project. Welcome to episode four, season one of the Emerging Scholar Podcast.

Today, we are fortunate to have Lucy D'Agostino McGowan, who is an assistant professor in the Department of Statistical Sciences at Wake Forest University. She's also the co-host of a great podcast called Casual Inference, which she does with Ellie Murray. The reason we're talking to Dr. D'Agostino McGowan today is that, uh, she published a paper and kind of white paper in 2023 called Design Principles for Data Analysis that I thought was absolutely perfect for, uh, young scholars to read, to go through, to figure out [00:01:00] what is the best way to approach your design and your data management as you launch into that first analysis.

And so before you even talk to a statistician, I think it's critical for you to understand what you're gonna do with that data. And so this interview is designed around that paper, and I hope you look forward to it, and it really helped me and my students understand how to approach our data sets as we launch into one of our first projects Okay. Welcome everyone. Uh, I'm here with Dr. Lucy D'Agostino McGowan, and, uh, we're here to talk about design principles when analyzing your data for a research project. Lucy, do you wanna introduce yourself and tell us how you got here to us today?

Dr. D'Agostino McGowan: Yeah, that sounds great. Thanks for having me. Um, so I am an assistant professor in the Department of Statistical Sciences at Wake Forest University.

And my research, I have kind of a couple different arms, but broadly I study causal inference and [00:02:00] causal inference type techniques. And then I also have an arm that looks at kind of human data interaction as, uh, it's sort of framed as, which sort of comes from the human computer interaction that came from some of the early like user interfaces on computers that people were thinking about.

But this idea of human data interaction is sort of how people interact with data, how they analyze data, things like that. So my research in that area, um, started, I did a postdoc at, at Johns Hopkins under Jeff Leek in their data science lab and, and since have continued working with Roger Peng and Stephanie Hicks who, um, both were at Hopkins, now Roger's at UT Austin.

Uh, but anyways, that's sort of where this all originated with the idea being trying to think about kind of how data analysis takes place, how data analysis products are consumed, and how we could maybe take, uh, what we've learned in other product type industries to be able to apply to this area. So we've been thinking a lot about if you could like effectively design studies with certain principles in mind to align between the, you know, person [00:03:00] doing the analysis and the consumer of the analysis, or maybe even the person who's kind of proposing the analysis to begin with.

Um, and these things are sort of supposed to go in tandem with like the statistical type of design decisions that are made, uh, in terms of what statistics you would use to actually analyze your data. So that's kind of where I came from in this, for this little area anyways. So that- Yeah, I'm happy to be here.

Dr. McGavock: Yeah. It's so great to have you. And so that's where this, uh, beautiful paper came out, uh, that was published this summer, 2022 on, uh, design principles. And I guess the humans designing the study and then the humans reading the study are quite variable, even though the data is kind of black and white, and maybe your code you can show to everyone, but all the decisions in between, you kind of need some principles to go by, right?

Dr. D'Agostino McGowan: Yes. And I think there's even like ability for there to be extra people in the middle there. You know, there's maybe the PI that comes up with the idea for the study, but then perhaps the analyst is different, you know. So there, there are all these intermediaries along the way. And so the big idea that we had for [00:04:00] sort of writing down what these principles were, were to try to drive alignment between all of the potential parties involved, you know, so that everyone was on the same page with the question that was being asked and, and what data was being used and what, what were the goals.

Was this meant to be something that answered like a single question with a single data set, or was this supposed to be this exhaustive analysis that maybe had tons of different sensitivity analyses as part of it? Was it meant to be something that could be reproduced by someone in the future, or was it just supposed to be a quick analysis that could kind of inform some decision that needs to be made in the short term but maybe wouldn't be repeated?

So that was the big idea that we sort of- Came at this question with and how we ended up coming up with these potential pr- principles with the idea of if you could optimally design something where you were able to communicate clearly between all of the different stakeholders, whether it be the person interested in the initial question or the person consuming the analysis at the end, you know, b- by like reading the results sort of to get [00:05:00] everybody on the same page with what the goals were so that you could ultimately decide whether or not you agree with what ends up being said.

Dr. McGavock: Right. And that's perfect. And I think the segue for me would be to say to trainees that often people have come to you at the end of their project when they haven't- Yes ... done any of this thinking beforehand, and it's nice to have some guideposts and some definitions to refer back to, to say, "This was the analysis that we want, we intended to do, and this is why we intended to do it.

Now here's where it goes." Can you maybe talk to us about the importance of engaging with you as a biostatistician or a data scientist at the outset of the project? And, uh, what kind of questions should we be coming to you with before we even start thinking about testing our hypothesis?

Dr. D'Agostino McGowan: Yeah, this is a great question.

I'm biased, but I do think that the earliest you can involve your analyst or your biostatistician in the process, the better. Um, and that's because, you know, aside from even these design principles that, that we've outlined in, in our paper, just trying to understand The [00:06:00] types of questions that are possible to be answered with a given data set or, um, a, you know, the, a given kind of perspective study.

Statistically, just mathematically based on like the reality of how mathematics works, you know, you need to sort of think through how data's collected in order to make sure that you can actually answer the types of questions you're trying to answer. So what I always recommend is to get your biostatistician in the door as soon as you have the question.

And then the thing that I request as a biostatistician is what is that question? And I want you to be really specific when I'm asking you what the question is. So often, most of my work is in causal inference, so we're trying to understand, you know, how some cause affects some outcome. And I think that this is broadly, uh, again, my bias might be showing, but I think most questions in medicine are intended to be this type of causal question where you have some exposure of interest and you wanna know how that impacts some outcome.

And so when you come to me in our first meeting, the first thing I want you to do is define both of those things very, very clearly. So when you say that your [00:07:00] exposure is, you know, maybe you're interested in studying like diabetes drug, I, what I want... I wanna know like which drug you wanna look at and what you wanna compare it to.

Like, do you wanna look at sulfonylureas versus metformin, or do you wanna look at some insulin, or do you wanna look at like what people are doing at, at baseline? Do you wanna look at a new drug? So have very specifically what that exposure is. And also things like dosing, because if you want this to be a good clean study, we want everybody to have the same exact exposure in order to be able to establish the potential effects there.

So both like what is the exposure? What's the measurement of this exposure? How is it measured? When is it measured? So that's all very, very important. And then similarly for the outcome. So if you come and say that outcome is mortality, well, what do you mean by that? Is, are, are you measuring the time to death?

Based on some window? Are you measuring, you know, how are you going to collect that information? Are you going to use health records or something else? Or, or maybe you're gonna do like progression if you're gonna measure something like that. How are you gonna define whether someone progresses? And so [00:08:00] having as many specifics on just these two things can really help.

And then, uh, on top of that, so, so kind of defining what your question is, exactly what you mean, and exactly how long you intend to follow these, these different pieces, the exposure and the outcome, I think can help to be able to get you to the right place in terms of being able to get to the right analysis.

And then of course, the next question's gonna be: How is the treatment gonna be assigned? So is this something... or the exposure gonna be assigned? Is this gonna be like a randomized study where that's the easiest one as a statistician to handle, although, uh, maybe the hardest as a, uh, the person designing the study because it takes a lot of time.

And so that... and, and if it is gonna be randomized, how will the randomization take place? Or are you going to be looking kind of at, um, data that already exists and try to sort of suss out these relationships in, uh, observational studies, which is where most of my research ends up being. And if that's the case, then I want you to describe to me the relationships that you think exist between other variables and that exposure that you very carefully [00:09:00] defined and the outcome that you very carefully defined.

And so this is where, uh... and you mentioned this offli- offline before, so maybe this is something that your folks are familiar with. But like drawing the DAG or the picture of the relationships that you posit exist between this very clearly defined exposure and this very clearly defined outcome, such that we could actually capture that true causal effect if we were able to capture kind of all these other variables that we think are on the causal pathway of these other pieces.

Dr. McGavock: And I think in our own lab, whenever someone comes in with a question, we say, you know, it's an observational study that we have, and we have four years' worth of data. The biostatistician is always like, "Well, you're the expert in this area." What's really helped us speed the translation has been the DAG. So saying to the statistician, "Okay, here's the worldview.

We think we're s- interested in this cause or this exposure, and we have data on this outcome, but there's all these other factors." And so by drawing it out, I think the analyst has a really good sense of like, "Okay, let me think about how we can design that."

Dr. D'Agostino McGowan: Yes.

Dr. McGavock: And so, yeah. Yes. Where, how has that [00:10:00] influenced you, and what elements should be going into a DAG that a trainee should be thinking about when, when they sit down with you for the first time?

Dr. D'Agostino McGowan: Yeah. So I think broadly you want to have any factor drawn on your DAG that's going to both have some influence on your exposure and some influence on your outcome. And I tend to like people to draw these as broad as possible. So even if these are not all factors that we're gonna ultimately need to adjust for, kind of anything that you think might either cause or be like a cause of a cause on your exposure, and then similarly the same for the outcome, with arrows drawn between them for kind of how these different pieces interact.

But all of that I think is really important. The big thing that I find a lot of folk, uh, asking about these i- is what if we have these cyclical relationships? So DAG stands for directed acyclic graph. They can't, can't draw cyclical relationships, but I find a lot of the times when I'm working with clinicians, um, there's this question of, well, well, we know that, for example, as your sodium increases, your [00:11:00] proteinuria increases, which then increases your hypertension, which then maybe increases your so...

I don't know. So you have these relationships- Yeah. Yeah ... that sort of sound cyclical. And the way that I've always explained this is that any cycle that you think exists on your DAG is actually just due to not accounting for time. And so if you, once you put time on there, uh, you no longer see these cycles.

And so the other thing when people are drawing these DAGs, when I'm working with folks kind of at early stages, I try to have the time also kind of incorporated there because you can't have a cause that happens after an effect, right? Everything needs to happen prior. So being able to like draw these pictures that are sort of time scaled of we think that having increased sodium increases maybe your cholesterol, which then incre- whatever it might be, but sort of having it time ordered to show the effects that you actually think are happening mechanistically, and then we can incorporate those in our models to make sure we adjust for them appropriately.

So anything [00:12:00] that happens prior to your exposure, any kind of thing you have data collected on prior to exposure, these are all potential factors that could show up in your DAG. So if you have access to a data set that has certain variables in it, you should consider that set of variables, anything that happens before your exposure, as things that could show up in your DAG.

And then kind of after drawing it out to... You would wanna then decide to include things that are... Your, your, your analyst can probably help you tell, figure out which things you need to make sure you include, uh, when you're adjusting for your final.

Dr. McGavock: And then I would al- just add that maybe some of those variables are being listed as a confounder when they're actually a moderator or a mediator, right?

And it's- Yes ... it's making it more confusing when it's actually more straightforward if you just thought about it in a different way

Dr. D'Agostino McGowan: Yeah. If you draw the picture where you have arrows from causes to effects, it's very easy to see, you know, mediators versus confounders because a mediator's gonna fall in between with the arrow in and arrow out versus a confounder will have two [00:13:00] arrows out.

So once you draw the picture, it becomes a little more obvious what you need to adjust for. And the other thing that's great is, you know, often your statistician pro- is certainly not gonna be the expert necessarily in the subject matter, but sometimes they have worked on several studies in this area and have a little bit of a sense.

So I've also worked on projects where, you know, something like age, for example. In most medical research, y- age is really important to think about when you're trying to adjust for most things. And so if I'm looking at a DAG and age doesn't show up, I might suggest, like, have you considered age as a potential confounder here?

And then we can sort of have a back and forth and discuss whether or not it should be included. So I think the other nice thing about drawing your assumptions is that it allows for this conversation to take place that historically kind of before I was doing more of these pictures, you know, I might fit a model and show it to my collaborator, and they might just sort of their eyes kind of glaze over 'cause they just see a bunch of coefficients and, you know- Yeah.

Yeah ... it doesn't mean as much to them. It's harder to know exactly what my assumptions are when you're just looking at my model output, as opposed to [00:14:00] looking at a picture that actually has arrows. You know, statistics in general has no way of determining direction of effects. So it can show that, um, you know, variable A and variable B are correlated, but it can't show whether A caused B or B caused A.

It just knows that the two are correlated. So, um, so giving you this model output doesn't actually give you all the information that you might get for s- from seeing the actual picture of what I think is happening.

Dr. McGavock: Yeah. And in my experience, someone comes in and says, "Well, we need to adjust for age and sex and income and, uh, puberty status and whatever, because that's what Smith et al did in 1942, and we've just been following that- Yes ever since," but without seeing the modeling- Forever ... right, you don't really know. So with that model though, I land on the first principle that you talk about, which I'm gonna label as feasibility, but data matching So does your data- Yes ... match with your DAG? So do you wanna tell us a little bit about that and, and does that sometimes lead to, "Look, we can't even answer this question"?

Dr. D'Agostino McGowan: [00:15:00] Yes. Part of why, too, you wanna get with your statistician early so that you don't, you know, end up getting pretty far down, uh, a path and then, uh, realize that actually the question that you're interested in isn't possible to answer with the data that you have. So yeah, this data matching principle basically just gets at how well does the available data match what you need, the data that would be needed to investigate the question of interest.

And so ways that this could go wrong could be that when you've defined this specific exposure and the specific outcome that I requested that you define when you walked, you know, into my office, uh, it turns out that your dataset doesn't actually have that exposure or that outcome in it. You have something at a different kind of a level of granularity, in which case you wouldn't be able to answer a question about that particular exposure.

And then the other way that this can go wrong is if you've drawn your picture and there's a really important confounder, and you don't actually have access to data on that confounder, there's not really a way to be able to adjust for it. Now, if that's the problem, there are, [00:16:00] like, sensitivity analyses and things that we can do quantitatively to try to get back at what maybe would've happened if you knew something about that confounder, so if you had some sense of it.

But yes, this data matching question, I think often if it's possible, I like actually even to have a physical, like if this is a, a retrospective type study where the data already exists, like in an EHR or something, to have an actual, like, printout of the first page of your little dataset so I can see what is in front of...

When you say you have data, is it really on the patient level or is it aggregated? I can see kind of what k- types of variables you really have. Are these, you know, or are they, um, at, are, are they continuous type variables or have they been dichotomized or, you know, something along the way to sort of get some sense of what actual numbers are gonna be put into the equations that we're gonna be fitting.

So yeah, that data matching one is crucial to try to get it early on to make sure that you're actually able to answer the question That you'd like to answer, or whether you just need to tweak the question a little bit to match what you have available.

Dr. McGavock: Right. And I think for residents that's probably the [00:17:00] answer, is if you really like this question, if you've got your DAG on one side and your table on the other and you're trying to match them up and do check marks, there's only three check marks and you're like, "Well, this- Right question is not gonna be answered with this data set, and I've only got so much time, and this is my data set," then it may be it's, okay, well, what could we answer, right? And you kind of go back- Yes ... and say, "Let's do the data matching and reframe our question and redo the DAG." So you've got four, I'm gonna say five main principles after data matching in your design principles, and the first one is- Yes exhaustive. So can you tell me what an exhaustive analysis is?

Dr. D'Agostino McGowan: Yes. So this is trying to get at basically, did you use kind of multiple different methods to try to answer the same question? And so I think an example where an analysis might not be exhaustive would be if you had one straightforward model that you were intending to fit, you weren't gonna look at it dif- different ways.

Um, maybe if you have, like, a randomized study, for example, you really just need to fit this very simple [00:18:00] model 'cause you're sort of relying on randomization, abstracting away some of the other things that you might need to check for if it were in an observational setting. So, like, an example I like to give for an exhaustive study is I had worked on one where we were looking at the difference in, um, diabetes drugs for this group of veterans.

And we were using kind of several different techniques. We did matching and weighting and, and actually a couple different versions of weighting, and we also changed the, our definition of the outcome just to make sure that we weren't just seeing our effects due to our choice of an, an, in the analysis, that we felt like these little minute choices were changing our ultimate result, then we might need to look a little closer into what the result really, really meant.

And so I think this is sort of a little bit to, just to get on the same page. If, as a resident, if you're working on a study that has a short timeframe, perhaps it's not gonna end up being exhaustive, and that's okay. Just to s- but to, like, make sure that the analyst is, understands that that's the purpose and sort of that everybody's on the same page with what this analysis is intended to be as opposed [00:19:00] to what it could be otherwise.

Dr. McGavock: It may not be exhaustive now, but maybe it's the beginning of something exhaustive.

Dr. D'Agostino McGowan: Exactly. That's the other thing that I think is so important 'cause we, what we tried to frame with these is that for actually all of the principles, even ones that do sort of feel like they could have some positive or negative associations, you know, they're not meant to be value statements.

And so if something is not exhaustive, it doesn't necessarily mean it, it's worse. It just means the goal of this particular study was not to be exhaustive. And if we were to continue trying to answer this question, then perhaps we would do an exhaustive analysis. So I think that that is a big piece of what we're trying to demonstrate.

Dr. McGavock: And in the exhaustive, you're trying to reframe the way you design it so that you're addressing different sources of bias in each type of analysis, and then summing the, the gross output gives you a bigger sense of what's actually true.

Dr. D'Agostino McGowan: Right. Exactly. Or even just seeing your choice of method isn't influencing your final result.

[00:20:00] So it might not be that you necessarily have a different answer for each of the different methods. You, you hope that it's the same answer for all the different methods, and you're just trying to see how sensitive your, your analysis is to each of them.

Dr. McGavock: Can you explain switching the outcome and why you did that?

What was the outcome? What did you switch it to and, and why, why'd you do that? I l- I love that- Yeah. So we- ... I love that approach.

Dr. D'Agostino McGowan: Actually, I said outcome, but I meant exposure. So we- Okay ... we, we, um, we switched the exposure. One of them was a persistent... Let's see if I can rem- the acronyms were PENNER versus PER.

So it was persistent exposure versus persistent exposure. I'm gonna look this up right now. Do you mind if I look this up live- Yeah, go for it ... so I can get the right-

Dr. McGavock: And I think for me, this is something that I'm really interested in right now under the broader concept of triangulation

Dr. D'Agostino McGowan: Yeah

Dr. McGavock: And so I, I know this isn't triangulation, but I like the idea of telling residents, "Look, just because we ran a survival analysis, [00:21:00] because we ran a logistic regression, and you got this relative risk or you got this output, it doesn't mean it's, it's real," right?

Dr. D'Agostino McGowan: Yeah.

Dr. McGavock: And we can- That's right ... we can do some checking on whether or not the way we designed this influenced the result, but it's actually on us to be exhaustive, I think.

Dr. D'Agostino McGowan: Yeah.

Dr. McGavock: So maybe you're not gonna do that, but within my research group, we're gonna make sure this is exhaustive so that the story we tell at the end is really robust, and it's not just you interpret it for yourself.

Dr. D'Agostino McGowan: Right. And it's not just, like, maybe due to some, you know, idiosyncrasy. That was the real thing we were trying to make sure we didn't end up being.

Dr. McGavock: And I feel like being critical when looking at your results. So if you have an exciting positive result that's going in the direction you wanted it to and you had this hypothesis and you think this is gonna be something that a journal really wants to see, rather than rolling with it, being critical about it and saying, "Okay, is this just a, a signal in the noise that we [00:22:00] designed by analyzing it in this way?"

Yes. And then doing that deeper dive. And so I think that is on us as research groups, and that's why you bring in statisticians because I think they're also more critical than we are in terms of saying, "Well, I'm not sure we're 100% on this."

Dr. D'Agostino McGowan: Okay, I found the definitions here. So the exposure that we initially defined was that you needed...

So the exposure, we were looking at diabetes drugs, and, um, you needed the default analysis with a PENR, which is persistent exposure not required. So that would be that, like, it's sort of like an ITT type of anal- an intention to treat type of analysis where at the start of the study you had the exposure.

But you might have switched off of it, so you may have switched medications kind of moving forward. But we analyzed it, like, the, in this kind of ITT type of framework where what you had or were taking at the beginning was what the group that you were defined in for the remainder. And then we had the persistent exposure required was that we [00:23:00] did that as a sensitivity analysis to make sure that we weren't seeing totally different effects once we sort of required that you stayed in the exposure group that you, you ultimately stayed in the exposure group.

So sort of more like a per protocol type analysis in the clinical trial framework. So there you could splice that back into that- Yeah. ... in that other slot. Yeah. Yeah. But yeah, so I... But I think that this is, like, you know, the, the whole idea. If those were radically different estimates, then perhaps we would have to think about why that would be the case.

Like, what would be driving it such that you would get these really different estimates depending on how you defined your exposure.

Dr. McGavock: That's great. Um, I also think about- Yeah ... reexamining the population you selected in your observational cohort. And I think about collider bias- Yeah ... right? And you're like, "Okay, well, we'll on- we'll restrict this only to people who we have an A1C on," right?

Or, "Only people- Yeah ... for whom we have the drug on." And then you get these results where you're like, "Well, that goes against everything we've been taught."

Dr. D'Agostino McGowan: Right. And you would have to decide, yeah, the only way that would be valid is if that was the only subgroup that you were trying to [00:24:00] then generalize the result to.

So if you're trying to generalize to a whole population, then that would definitely get you the wrong thing.

Dr. McGavock: Yeah, and that's where the obesity paradox comes from, those types of analyses. S- Right? So the second one is skeptical, so testing alternative theories.

Dr. D'Agostino McGowan: Yeah, so this I think exhaustive and skeptical kind of go hand in hand.

This is sort of if you... It's the framework of also kind of, uh, the sensitivity analyses, but it's sort of, like, from the opposite perspective, where now you're trying to posit different ways that you could have seen the results. So the exhaustive would be you're gonna look at it a whole bunch of different ways and make sure that your result seems consistent.

And the skeptical would be you try to find alternative explanations for the result that you saw and see if you can break the results, essentially, if you, if you had this other explanation as opposed to the kind of effect that you're observing. In the paper, we, we used an example. Um, there was this, like, study from back in, in the '90s.

This H- Harvard Six Cities study came out that was looking at the relationship [00:25:00] between, I, I believe it was mortality and exposure to particulate air pollution in these different cities. So there was this, like, very clean graph that came out of this study where it basically looked at that, like, cities that had more pollution, the mortality was much higher, and it looked like basically a straight line.

So this study came out, and of course, like, the EPA and others wanna validate that that's true, right? That they, they wanna make sure there's not some other confounder that's making these cities look different than the others. And so they did this. They had a very skeptical analysis where they released this special report that had, like, you know, the h- I, I guess, I think it was the Health Effects Institute was the one that commissioned this.

But they had hundreds of pages of trying to find alternate explanations. And ultimately, actually, I don't, I don't think they actually came to any other explanation. But the idea was that here they're trying to see is there something else that might have driven this. Again, as a counter to that, so if you randomize your analysis, then perhaps if you give a treatment that was randomly [00:26:00] assigned, you might not need that much of a skeptical...

You know, in theory, if your study's large enough, there really would likely not be alternative explanations, assuming that you have adherence to the, the drug that people are given and that randomization isn't broken in other ways. You know? Then in that case, maybe you wouldn't need to have a very skeptical analysis.

Dr. McGavock: So what I would put in there, and what we're trying to implement a little bit in our work, is the concept of negative exposures and negative controls. So would you put- Yes ... that as a, a method within the skeptical piece? Because you're using the same dataset, and you're like- Right ... maybe we have some residual confounding associated with socioeconomic status.

So is there something that's not on the causal path but is related to unmeasured confounding? And if we see that signal, then we know, okay, something's up.

Dr. D'Agostino McGowan: Yes. I think that's a great way to, to be able to explore something skeptically, to be able to look at negative controls and see, you know, would we have just seen a big effect because for any exposure that we had had for this particular outcome.

And for things, you know, these [00:27:00] negative controls, you often try to find something that you think there really is no relationship. There should be no relationship. And so you could use that as kind of like a baseline.

Dr. McGavock: Mm-hmm.

Dr. D'Agostino McGowan: Yeah.

Dr. McGavock: In our- That's great ... in our DOHaD life course work, the shared genetics and shared environment, no one ever has that in administrative data, right?

Yeah. And it's just this huge elephant in the room.

Dr. D'Agostino McGowan: Yeah.

Dr. McGavock: It's like, well, is there a way that I can dissect that out with the data that I have? And I think you can. Yeah. It's just being skeptical, right?

Dr. D'Agostino McGowan: Yes. And then once you've established, you know, if you could quantify what you think some unmeasured confounder like that, if you could quantify the impact you would've expected should you have been able to measure it, your statistician, it's pretty straightforward to be able to incorporate that into your ultimate effect.

So basically to adjust the observed effect for some confounder as long as you could quantify how the relationship between that confounder and the exposure and that confounder and the outcome. And so these types of skeptical analyses, even if you never are able to get at the actual confounder of interest, they can be really powerful because they can help [00:28:00] you determine the magnitude of effect that would be needed to change your results.

Dr. McGavock: And then for me, the important part for residents who are listening is that if you come in and say, "We will do a skeptical analysis," that means I need to find a negative exposure and/or a negative outcome. Yeah. So when I'm asking for the data and I'm extracting the data, it'll save you months in advance if you plan for it- Yeah rather than think about it right at the end. Yes.

Dr. D'Agostino McGowan: Yes.

Dr. McGavock: So that's huge. Um, and the- Yes ... so the third one is clarity. I really like that one. So can you expand on that?

Dr. D'Agostino McGowan: Clarity basically does the analysis. Does it summarize the data in a way that's very clear to the person on the other side of the... And so I think, uh, uh, an analysis that has, um, you know, excellent clarity might have a single figure that really describes what the take-home message is.

And I think that sometimes, you know, this again is based on how you design the study because sometimes based on the study design, it's just not possible to get a clear answer from the, [00:29:00] from the analysis. But if it's, you know, designed in such a way that you're making use of some of these things like, you know, negative controls or other techniques to be able to get at these kind of cleaner, uh, type of effects, then you really could potentially get at some clarity.

And so having upfront that the goal is to have this single clear result, I think can, uh, help drive choices that you make in, in other pieces of your analysis process.

Dr. McGavock: It's not a graphical abstract, right?

Dr. D'Agostino McGowan: Yes.

Dr. McGavock: It's not like you- Yes. It's not

Dr. D'Agostino McGowan: that-

Dr. McGavock: You're not pitching. No. You're not pitching your results.

Dr. D'Agostino McGowan: No.

Yeah, I said a plot, but I mean, like, the data.

Dr. McGavock: Right. Yeah.

Dr. D'Agostino McGowan: It's not the users.

Dr. McGavock: The knowledge translation tool that comes out, you know, at the head, at the start of your paper there. For me though, I think it also is important to think if this is a negative result, what does a good figure look like to tell people?

The variation in this thing is so huge. What would that graph look like- Yeah ... to show people, look, this is a challenging question to, to tackle?

Dr. D'Agostino McGowan: Actually, what you just described there, you know, showing there's two [00:30:00] different ways too, because if, if you're seeing some like huge variation, it's, it's not always that the result is necessarily negative, it's just that it's inconclusive.

And so I think being able to communicate that via a figure where you can sort of show. And, and even like some of the, you know, negative controls actually could help you because you could show if it was something where it's not a problem with the size of the data, it's a problem really with like the ability to answer this specific question, then you could show that, you know, this data set could have answered this question with some kind of efficient measure.

But the question we chose to try to ask is just not enough information to know for sure if the, if it's the directionality of the effect. For example, if you look like it's negative, but your confidence interval is super wide.

Dr. McGavock: Which then helps the next generation of residents come in and say, "Okay, maybe we shouldn't go down- Yes this path," right? Or- Yes ... maybe we need to do it a lot differently so we're not trying to reproduce crap, which is your fourth one, which is reproducibility, which is really key. Yes. So can you talk about that?

Dr. D'Agostino McGowan: Reproducibility is essentially, I think maybe that's the one that most people are the most familiar with because it's gotten a lot of, uh, attention [00:31:00] in the past decade or so.

But the big picture is just could someone who was not the original producer take your analysis and reproduce it? Um, this is a little different than like replicability, which would be like could someone just redo the whole thing and get the same result? The reproducibility is-- So the replicability would be like could someone with like brand-new data but the same study question replicate what you saw, which is a really important piece, but maybe doesn't have a lot to do with your study design because as long as you report what you did, someone else might be able to replicate it.

Reproducibility is different in that it requires that along the way are recording all of the steps that you're taking such that you would allow somebody to take that same output and like be able to recreate it. And so if the intention of a study is to be reproducible, which I think a lot of times in academic medicine we are trying to do reproducible research, then your analyst would need to know upfront that you're gonna need their code and that you're gonna want to make the data be in a form that could be shared, you know, maybe in a de-identified way if that was the intention, and that, you know, the analysis steps are clearly [00:32:00] recorded, and you're not just throwing spaghetti at the wall and trying a whole bunch of different analyses and then just showing the final one.

You know, if it's really meant to be reproducible, every step is recorded along the way. And so it's, uh, I think aligning on that principle is important really early on so that everybody's on the same page with what the intention of the final product is.

Dr. McGavock: Yeah, and more and more you're being asked to put your code in an appendix at the end.

Dr. D'Agostino McGowan: Yes.

Dr. McGavock: And if you're using administrative data that you can't make publicly available, then having a flowchart and saying, "You know, we excluded this, we used these codes, this is how we got here. So if you put in an application, you try to get access to that data, you follow our flowchart, and then you use this R code."

You should get the same. Yeah. Same thing. Yeah. Now, one thing that you didn't bring up in the paper but I've seen other people do is you kind of hold back like 15, 20% of your data if you've got a really large data set. And you try to- Yeah ... actually reproduce it yourself on that same data set. Is that something that you think people should be thinking about?

Dr. D'Agostino McGowan: Yes. So this paper, um, the intention here was [00:33:00] sort of to have principles that are complementary to things that are statistically good ideas, but are not meant to sit on top of what has already been recommended, um, for a good statistical design. So this was sort of like some of these other things that people take into account when they're trying to design something to sort of align between your expectations.

And I think that what you've described there, this holding data out in order to validate a model, is a very good statistical recommendation. And I think the only reason it doesn't show up here is just because it's been written about a lot in more of the statistical design. But I think most statisticians would tell you that tends to be a good idea to be able to make sure, especially, you know, if you're building a prediction model, to be able to show that it holds outside of just your little sample that you built it on.

So getting your estimates in one population and then testing them in another to make sure that they hold outside of the little sample that you have. And this is a good idea in general, but, you know, some things that it provides is it, it basically protects you against overfitting your models [00:34:00] such that they really only are useful within your little subset of data, but not useful to people broadly.

So yes, that is a very good idea-

Dr. McGavock: Right ...

Dr. D'Agostino McGowan: to do.

Dr. McGavock: And then I'll just add your thoughts about triangulation. So maybe moving beyond your own data set if you don't have some variables, but you can maybe answer it with another data set. Is that something you recommend, or have you done that? Do you have an example of using that approach?

Dr. D'Agostino McGowan: Yes. I haven't used the term triangulation, so I'm gonna say what I think, and then you can tell me if I'm interpreting it right. I was on a study several years ago. We were looking at-- It was like this protocol in the ICU to see if it helped with delirium. And it was a big observational study that was conducted at Vanderbilt.

Actually, it was like multi-center, but we were at Vanderbilt at the time. And one of the measures that we were missing was the severity of illness. We had it at a select number of institutions. We had it basically at predominantly the academic institutions had this measure in their data, but the majority of other institutions didn't.

And we [00:35:00] thought that there probably were things that were a little bit different about the academic medicine institutions, so we didn't wanna just i-include that or only impute, for example, based on that variable with these other institutions. But you can imagine if your outcome is anything like that has to do with how severe your disease is, then this is probably an important variable to measure.

So this triangulation piece that, that we were just talking about, we were able to basically do one of these sensitivity analyses where we could estimate what we think that the impact of this was based on. We used a couple different data sources to estimate this. So we looked at like the literature in general and the prevalence of this particular outcome in the ICUs, and then we also used our subset of data that didn't have this variable.

We looked at the prevalence within our exposure groups. And so we didn't require-- Because we were worried that this was gonna kind of have some kind of differential effect on the outcome, depending on if you were at an academic institution or not, we felt like that piece wasn't that useful because we only had data on the subset.

[00:36:00] But we thought that the subset probably did give you some information about just the prevalence of this variable, this disease severity variable based on the exposure to th-this treatment that we were interested in. So anyways, we were able to quantify basically how Big we thought that this effect was on the exposure.

So and in order for it to be a confounder, it would have to affect both the exposure and the outcome. And then we could do this tipping point sensitivity analysis where we could say, "Okay, we think that it's this effect on the exposure based on this other data," so we sort of triangulated what this effect potentially was.

And we were looking at a directional change, and so we said in order for our exposure to actually change direction, so not to no longer be positive, but actually be bad, harmful, this disease severity variable would need to have this magnitude of an effect with the outcome. And so we were able to kind of quantify that, and then we could state, like in this particular case, that would've, it would've had to be like an enormous effect in order to tip the analysis all the way to the point where it, it wasn't likely that that was the case.

And so this let us have more confidence in [00:37:00] the effect that we saw. Does that kind of get at what you're thinking of in triangulation?

Dr. McGavock: That's part of it. Yeah. The definition, so like Deb Lawlor has published a paper back in 2016 about epidemiological, uh, ideal- clinical ideological triangulation, where they talk about if you have a certain source of bias that you can't really deal with in your data set- Yeah can you find another data set that has a different sources of bias, maybe a different population? And then you run analyses on both data sets, and you see whether, you know, to what degree do those results line up. Um-

Dr. D'Agostino McGowan: I see. Yes.

Dr. McGavock: Their most... I don't know if it's their most famous, but one I really like is they had this really robust association between breastfeeding and child health outcomes, right?

In the UK and- Yes ... ALSPAC cohort. But breastfeeding is really tightly correlated with income and wealth and- And socioeconomic ... socioeconomic status- Yeah ... and that kind of thing. And in another country though, where breastfeeding is more common, and ex- extended breastfeeding is more common, and you have a [00:38:00] much different sociodemographic profile, right?

If it holds up in that cohort as well, right?

Dr. D'Agostino McGowan: Yes.

Dr. McGavock: It's a robust finding. But if it doesn't hold up in that cohort, then there's something, some bias in the ALSPAC cohort that we're not picking up with the data. And so it's- Yes ... it's looking at the broader results from two different studies where you've been really- Oh,

Dr. D'Agostino McGowan: yeah

Dr. McGavock: really strategic- Okay ... about trying to Capture that bias Find Yeah

Dr. D'Agostino McGowan: Yeah. Find a study that's uniquely different in ways that you think might defy... That's great. Yeah. That's a great idea. What was the result of that study? Were they able to-

Dr. McGavock: I think it wa- Was

Dr. D'Agostino McGowan: it robust?

Dr. McGavock: Well, no, I think it washes away the- Yeah the breastfeeding effect- The effect ... is really about socioeconomic status and being able to do it. Yeah. Yeah. So, um- Yeah ... and then that group is also just pioneers in MR, applying different Mendelian randomization approaches, and so let's say you have a study result in one case, can you then go, it's not always possible, but design a Mendelian randomization study within, like the UK Biobank, for example.

[00:39:00] And now you're using- Yeah ... an instrumental variable analysis, and that's ob- obviously supposed to be unbiased, right? But it'll, it gives you kind of a, a different estimate using that different study design. So I'm really excited about, you know, exploring those- Yeah ... opportunities and trying to integrate those different study designs.

It's more challenging, takes more time, but it's more exciting in terms of trying to get to the truth.

Dr. D'Agostino McGowan: Yeah, that's great. I think that's a great idea. That if you have access to more than one dataset, I mean, even just showing that things reproduce in more than one dataset, yeah, that gives you much more confidence in the, kind of whatever the ef- effect is that you observed, so.

Dr. McGavock: And I, I think just to close, like, for us with trainees, and I'd love to hear your thoughts on this, I want trainees to think, "Is this really true?" Not, "Am I gonna get a publication?" Right?

Dr. D'Agostino McGowan: Yes.

Dr. McGavock: Or like, "Is this advancing my career?" It's no, like, "Do I actually believe this?"

Dr. D'Agostino McGowan: Yes. "

Dr. McGavock: And is it gonna influence the way that I practice medicine?"

Right? Your thoughts?

Dr. D'Agostino McGowan: I think that's definitely true. We don't need any more noise [00:40:00] in an already noisy-

Dr. McGavock: We don't need more mess.

Dr. D'Agostino McGowan: Yeah.

Dr. McGavock: Yeah.

Dr. D'Agostino McGowan: Yeah.

Dr. McGavock: So.

Dr. D'Agostino McGowan: Yeah.

Dr. McGavock: So that was fantastic. Thank you so much for walking us through that. We're gonna attach this, uh, paper and any other resources you have. Maybe you could send me an email, and we'll put it in our, uh, little repository of resources.

Yeah. If you're a podcast enthusiast, you also have, uh, a wonderful podcast. You wanna tell us about that before we close out?

Dr. D'Agostino McGowan: Yeah. I co-host, uh, Casual Inference, where we're, uh, sponsored by the American Journal of Epidemiology, and we try to get into some of these causal type questions. We interview a lot of, um, experts in the field and try to think about how to best answer causal questions, and also we try to keep things casual, hence the name, so that it's easy for people to kind of jump in and understand.

So we're so welcome new listeners always.

Dr. McGavock: Yeah. It's, it's casual. It's fun. Uh, it has a Canadian link. I believe your co-host, Kaiser- Yes, Ellie. Yeah ... Ellie's Canadian, so. Ellie's

Dr. D'Agostino McGowan: Canadian.

Dr. McGavock: For all you Canadians out there, please tune in. That's fantastic. And we'll, uh, we'll hopefully cross paths [00:41:00] one day soon.

Such a great paper, and you're doing great work. I'm so happy that you took time to, uh, answer some questions today. Thank you.

Dr. D'Agostino McGowan: Yeah. Thanks for, thanks for organizing.

Dr. McGavock: That was our conversation with Dr. Lucy D'Agostino McGowan on design principles for data analysis. Be sure to join us on episode five, where we're joined by Dr.

Hailey Banack to learn more about DAGs, what they are and how you can use them. Thanks for listening. The Emerging Scholar Podcast was recorded in the UMFM studios located at the University of Manitoba in Winnipeg, Manitoba, Canada. This is Treaty 1 territory, the original homeland of the Anishinaabe, the Ininew, and on the national homeland of the Red River Métis.

Manitoba is also homeland to the Anishininew, the Dakota Oyate, and the Dene peoples. We acknowledge that our water is sourced from Shoal Lake 40 First Nation. As a settler scientist working in health research, I acknowledge TRC Call to Action Number 18, that health [00:42:00] inequities between Indigenous and non-Indigenous people are the direct result of colonization.

Winnipeg is the center for reconciliation in Canada, and where we are speaking from today is at the crossroads of Canada's history and Canada's future. We are three kilometers away from one of the most sacred meeting grounds for Indigenous people on the plains and within Turtle Island. We are also several kilometers away from one of the last residential schools in the province.

We are working collectively on reconcili-action with youth, elders, and community leaders to heal from the trauma of colonization, past and present, honor the original treaties, and create a Canada that addresses and eliminates the inequities and injustices of the past and present to create a future for Indigenous children to live and thrive.

On behalf of the team at the Emerging Scholar Podcast, we want to thank Osani Balkaran for the great production for our opening and closing themes. We also want to thank UMFM 101.5, especially [00:43:00] co-producer Jared McKetiak for technical assistance and production on this project. We also want to thank all of our guests for the discussion and the insight that they've provided.

Once again, thank you for listening. Be sure to subscribe to the Emerging Scholar Podcast on Apple Podcasts, Spotify, and umfm.com.