Predictive Epidemiology with Dr. Laura Rosella

Dr. McGavock: [00:00:00] Welcome to the Emerging Scholar Podcast, season one. We hope you enjoy this series of great conversations with leading medical researchers that's designed for graduate students, medical trainees, residents, and fellows to help you on your journey towards your first scientific project. Welcome everyone to episode eight of the Emerging Scholar Podcast.

Today we are interviewing Dr. Laura Rosella from the Dalla Lana School of Public Health at the University of Toronto. She currently holds a CRC chair in population health analytics and is also a PI for the Temerty Center for AI Research and Education in Medicine. Today we're interviewing Dr. Rosella because she's an expert in the second topic around epidemiology we're interested in, which is predictive epi.

She's an expert of applying machine learning to administrative health [00:01:00] data sets. Today we talk to Dr. Rosella about what is predictive epidemiology, how can machine learning help us, and what is the future for AI within administrative health data. Dr. Rosella is not only a expert in this area, she is a wonderful educator and describes things in a really great way.

It's really an honor to have Dr. Rosella on. She's one of Canada's leading epidemiologists and, uh, and scientists in this area, and we look forward to understanding a little bit more about what the values of AI can bring to administrative health data research Okay. Welcome back to The Emerging Scholar. My name's Jon McGavock from the University of Manitoba, and it is my distinct pleasure to have a colleague and someone I really admire in the Canadian epidemiological landscape, Dr. Laura Rosella. Laura, welcome to the podcast, and do you wanna introduce yourself for us, please?

Dr. Rosella: Yes, absolutely. It's so great to be here. Laura Rosella. I am an associate professor in epidemiology at the Dalla [00:02:00] Lana School of Public Health at University of Toronto, and also at the Temerty, uh, Faculty of Medicine. Couple other roles I have that are relevant for today, I'm the education lead for the Center for AI in Research, Education, and Medicine at the Temerty Faculty of Medicine at U- of U of T, and also the associate director of education and training at the Data Sciences Institute, which is a new institute that's been developed at University of Toronto.

Dr. McGavock: Excellent. I find that no matter what grant I'm writing, when I'm referencing people in the field or some of the most important data that's come out of Canada, your name is almost on everything that I reference. So, uh, you're quite accomplished, and you have this broad range of, uh, topics that you've, uh, you've delved into, so congratulations on that.

Dr. Rosella: Thank you. Hopefully that's, uh, a good thing.

Dr. McGavock: I think it's a great thing, and it's always phenomenal how well written and how well done, uh, the research is, so I really admire that. So today we wanted to talk about predictive epidemiology, which is something I know very little about. I [00:03:00] understand three large groups of epidemiology.

There's descriptive epidemiology, and we've had a podcast on that. There's causal epidemiology. We're gonna have a podcast on that. What I find the work that you do falls really nicely in line with this predictive epidemiology, and I would have never been trained in that, you know, predictive epidemiology.

So can you maybe explain a little bit about what that is and maybe give us an example of where your path started in predictive epidemiology?

Dr. Rosella: Yeah, absolutely. And let me just say you're not the only one. I- in epidemiology training, the predictive part of it usually isn't formally covered. I think in recent years it's been more of a focus because of the machine learning popularity, et cetera, but, you know, it's not really integrated in a lot of s- core epidemiology training.

So for those that are listening, if they feel they haven't been exposed to it, that's actually the more typical experience. For myself, where I got introduced to it was working in my master's and then subsequently my PhD. I worked with a really brilliant physician, a [00:04:00] scientist named Doug Manuel, who many of you may know.

He's now based out of Ottawa, and he was very fascinated with predictive models. Because he was a clinician, he saw how they were, they were being used in medicine, and for many of you, the most popular example is the Framingham algorithm. So going to your doctor's office, primary care, there would be some calculation done on your likelihood of developing cardiovascular disease, and then based on that probability, which was turned into a nice picture format, they would decide on what interventions would be best for you.

And underlying all of that work is a predictive model. That model was developed on the Framingham cohort, and, you know, sort of late '70s, uh, and early '80s, the first iterations of that model became developed. And the investigators at the time thought if we could just predict who's going to have cardiovascular disease, we can intervene on them and prevent their outcome.

And they did an amazing job, and it became very [00:05:00] popular, and they made it very accessible. This is something I think we've lost a bit, uh, in that they gave tools that were really simple to look at. You could actually just put some numbers in, look at a graph, and know what to do. And so that's the original clinical prediction model that really started a lot of work in this area.

So my supervisor at the time and myself started talking and said, "You know, we don't have anything like this for population health." We have this example in clinical medicine where you're predicting an individual patient in front of you and what to do about that patient, but imagine you can do this for a group of people, for populations, and imagine you can run it on routine surveillance data, then this could be a really important tool for helping make decisions at the population level.

So that's how I started on it. My entire PhD was devoted to prediction modeling in the diabetes context, and I dove into a deep hole of the methods that we use to develop prediction models, and more importantly, how we validate them to make sure they're doing what we think they're doing.

Dr. McGavock: So my [00:06:00] understanding would be it's empirically designed or scientifically driven algorithms for clinicians to use at the bedside or for policymakers to use when making decisions about how best to help the people or the person sitting in front of you at that time.

Dr. Rosella: Yeah.

Dr. McGavock: Yeah. And I think COVID made that very relevant, too.

Dr. Rosella: Absolutely. And one other way to think about it is, you know, when there's only one or two factors that influence someone's risk, you don't need a model. You don't need an algorithm. It's sort of like this is present and this is absent, then I know what to do.

Once you start getting to three, four, five, it becomes very difficult to estimate risk simply by just collecting these things one at a time. So these types of models, what is very useful about them is there are methods to add weights to all those factors, and then you can take into account multiple factors into risk assessment, and so that's the real need to actually go into the modeling part of it.

Dr. McGavock: Yeah. So it's not as simple as adding up you're a certain age, you have diabetes, you smoke, yes or no. It's a lot [00:07:00] more complicated than that.

Dr. Rosella: It's a lot more complicated, and actually it's very hard for humans to do that calculation in their head because all of us have an idea of which one might be more or less important, and we have a very hard time understanding how these things interact with each other.

So it's actually very hard for us to assign the probability just based on looking at a patient or looking at these values.

Dr. McGavock: So how do we decide what goes in that model? And, um, maybe I heard for your PhD you were in Winnipeg, uh, shout out to my- Yeah ... hometown. Um, and Winnipeg has a certain, you know, working with the Manitoba Centre for Health Policy and large administrative data, there are only certain things available in that data set.

So can you maybe describe for us how you make those decisions about what goes into the risk prediction model that you're trying to create?

Dr. Rosella: Yeah, absolutely. So the first and most important thing to recognize when you're even thinking about predictive modeling is that the way you build those models is completely different than a way you would build a model, for example, if, if you have a causal question or if you're doing descriptive epidemiology, and [00:08:00] this is why it's really important to differentiate the goal of your analytic task.

So I teach my students regression. Lots of people learn, uh, regression. When you're studying epidemiology, it's a very important tool. You can use regression completely differently if you're doing a predictive model versus a causal model versus a description. You may not even use regression in that case, or you might just be doing a simple adjustment or standardization.

So that's the most important thing. The goal of a prediction model is to maximize your ability to accurately predict what's going to happen, not to explain it and not to actually focus in on one particular exposure that you might be interested in or treatment. So the first thing you would do when you're trying to determine what factors go in the model is really to understand all the variables that might explain or might account for the variation that you see in the outcome, and there's lots of ways people go about doing this in terms of, you know, looking at the literature.

Now, there are some data-driven ways which are a bit controversial in [00:09:00] terms of really just plowing through thousands of variables and seeing which ones are actually explaining the variation that we see. They're not causal. Um, and then there's some combination of those types of approaches, but- You mentioned, uh, that I spent time in Manitoba, and the really important part of that is with any time you're developing a prediction model, most important thing is that there's a validation strategy.

So what does that mean? You develop the model ideally in one setting or in one data set, and then you take that model out and you bring it and see if it works in another setting. So in my context, I developed a model using linked administrative data in Ontario, and then I came to Manitoba and said, "Does the model actually work here?"

And going out of the sample is very important because when you're doing predictive models, they get really good at learning their own data that you're working in. And if you don't take it out, you can actually over-predict and fine-tune it so well to the data that you're seeing that it actually doesn't work.

And we wanted this model to work across Canada and across the world, [00:10:00] really, in other settings, so you have to take it out. This idea of training, what we call, on one data set and validating on an external data set is very important. And some people do that within a data set. They chop it up. But the most rigorous validation, of course, you can imagine would be to actually take it in a different setting and see if it holds up, and that's what, what I did during my PhD and what we do in all our predictive model approaches.

Dr. McGavock: So what were you trying to predict, and then what variables did you put in there? And just maybe linking this to descriptive and causal epi, did you draw out a theoretical framework or some type of working model that you thought you needed these sources of data to put into your algorithm or into your model?

Dr. Rosella: Yeah, great question. So we were trying to predict, uh, the incidence of type 2 diabetes to inform diabetes prevention. And interestingly, we were using survey data linked to administrative data, so we could tell from this administrative data who developed type 2 diabetes based on validated algorithms. And from the survey data, we were able to get risk factor [00:11:00] information like height and weight, physical activity, age, all of these types of variables which we know are important for diabetes.

In terms of what we put into the models, I definitely did look at the literature and looked at what variables we knew from the epidemiologic literature were important for type 2 diabetes. We also had randomized control trials on interventions to prevent or delay diabetes, which involve physical activity interventions, diet, et cetera.

There are some pharmacologic interventions, but we didn't include that. So we looked at the literature, and I would not say that we had a theoretical orientation in the same way that you would when you're asking a causal question. So when you're asking a causal question, say you're looking at one specific risk factor, let's say smoking, and its risk on developing type 2 diabetes, you would draw your causal diagram or your DAG.

You would look at what other variables are explaining that association. That's not the typical approach you use in a predictive model. You would actually look at all the variables which could [00:12:00] contribute to risk and try and look at those variables in a systematic way. And so we did that looking at reviews of the literature to come up with a set of variables.

So not a purely data-driven approach, which we can talk a bit later that some people use nowadays. And so based on those variables, we built our model. And the other really interesting thing about predictive models is that how do you know whether your model's working well? The metrics of performance for predictive models are completely different than the metrics of performance on a causal model.

So you look at things that measure something called discrimination, which is how well your model ranks who's at high risk and who's at low risk. If I had 1,000 people here and I put them in order from lowest to highest risk of diabetes, how well is my model doing that? So we have measures that do that.

And then there's something called calibration, which is how well observed and predicted agree. Meaning if I say risk, 20 people have risk of 20%, how well does that actually agree? Is it actually 20%? And both of those are actually very important. If you've never done prediction modeling, you [00:13:00] probably have never heard those terms before.

Dr. McGavock: I have never. They're

Dr. Rosella: all...

Dr. McGavock: Yes. But here's what I do hear. Receiver operating curves

Dr. Rosella: Yes.

Dr. McGavock: And sensitivity and specificity. Yes. And I think anyone here, like, has written an exam in med school or in their residency has heard those things. So how does that relate?

Dr. Rosella: Exactly. So very closely, actually. So sensitivity and specificity is this concept of does the test say positive or negative, and then what's the truth?

It's the same thing with predictive models, except you have to do this across the entire population, so it gets a bit more sophisticated. But essentially, my model is giving a certain person a predicted probability, and I observe a predicted probability on average in the population for people at this risk level.

And based on that, you can say whether or not our model's getting it right or whether or not our model's way off. And the ranking of putting people from lowest to highest risk, essentially also observing who got diabetes and who didn't and ranking them, that's basically the ROC curve, the area under the ROC curve.

That's how we determine [00:14:00] discrimination of a model. And, uh, calibration, we often look at this graphically, actually. We take observed and predicted across the population. We divide it into lots of different subgroups and say, "How well are we matching up in terms of what we predicted these groups of people's risks to be and what we actually observe in the data?"

So in my example, I did that in Ontario, and then I went and I took my model out and I did that in Manitoba, and I said, "How's the model performing?" But what's really interesting is that when you do that, we call this testing phase, where you have all your variables in your model that you think are building your ideal model, and then you realize that actually some variables that you know clinically might be really important actually may be causing problems with your model.

You might be over-predicting. And so you have to take some of these variables out, and then you see that your model is actually performing better on these metrics. This is where it's very different from other parts of epidemiology because you might think, "Well, you can never take out that variable because it's an confounder."

Well, confounding has really [00:15:00] no relevance in the concept of prediction. You're trying to maximize your ability to accurately predict whether or not this person develops a condition. A confounder is related to a pr- a focal exposure, which you don't have in a predictive model. So you kind of have to tune your model in very specific ways until you get it working just right.

And there were some surprises when I was developing, and we've done tons of models since then, which variables go in and out based on model performance.

Dr. McGavock: Okay, so it's March Madness time, right? Yep. If I correctly predict 64 wins out of 64 games, I'm great. What does over-prediction mean?

Dr. Rosella: So really, it's over-fitting.

So it's when you are predicting so well to your own data set, your own sets of patients. So in the context of March Madness, that's it. Those are the teams, and that's everybody. But for you, when you're doing research, you may have access to data on 1,000 patients, and then next you might be applying it to a different group of patients going forward.

[00:16:00] Or then there's next year and next year and the year after, right? Patients you haven't even seen yet. So overfitting is when your model predicts so well in the data set that you developed it in, but then you take it out and it actually starts falling apart. There's this bias-variance trade-off they call it, and which is very confusing because bias and variance are terms that have different meanings depending on the context.

Uh, I prefer sort of noise to signal problem- I like that, yep ... which is every little blip in your data set, your model starts fitting it and learning it. And so it becomes really good at that particular data set it's used to seeing. But as soon as you take it out, it falls apart. I mean, I think people are seeing this now actually in all of these models that are being popularized in AI models where even self-driving cars, which are actually just a bunch of predictions, they know their track really well, but as soon as you take it out, they start falling apart.

So that's the big problem with prediction that we actually try to mitigate by [00:17:00] testing in different validation sets and importantly, external testing. What ends up happening is that term that you thought was so important was actually just a nuance in your particular data set, and by taking it out, you actually have a better performing model.

Dr. McGavock: Right. Like if everyone who got heart disease in Framingham was eating at a certain restaurant on, uh, Dorchester Street, it goes into your model and, and it enhances it really well, but then you put that same restaurant in an algorithm in Dallas, Texas Um, in the Dallas Heart Study, and you realize, uh, it doesn't predict at all.

Yep. Okay. I get it. It doesn't

Dr. Rosella: predict at all.

Dr. McGavock: Yeah. I had a question. When I was in Texas and I was learning about the Framingham study, I asked about physical activity. I'm like, "How is physical activity not in Framingham?"

Dr. Rosella: Yeah.

Dr. McGavock: And it just didn't carry enough weight relative to everything else, was kind of my understanding.

But can you explain that, how some variables where you think, like, "This should totally be in the model," we wouldn't put in the model for certain reasons?

Dr. Rosella: Yeah, it's a really good question 'cause this came up for diabetes as well. We know clinically that physical activity is so important, and that- that's one of the big [00:18:00] cautions with predictive models, is that you can't make any inference about which terms are in there and not in there in terms of their implications for disease.

Because it really is just a mechanical aspect of the model, which makes some people really uncomfortable. But if you understand how these models work and you use them properly, I think that, um, it's not as bad as people think. But that being said, sometimes really important terms that you know are important clinically don't end up in the model.

This also happens with really important biomarkers. And, you know, this came up. You know, we used BMI, which we know is not necessarily the best measure of excess weight. There are other more sophisticated measures. But from a predictive point of view, it does really well. So what's happening there? So couple things.

One is There is a limit to prediction. So once you put in all of these important terms and you have interactions, and you start predicting very well, you know, 90% accuracy, 95% accuracy, the [00:19:00] incremental value of these terms become less and less. So that's often what happens, is that you've explained all the variation already.

Layering on top some measure isn't actually adding any, uh, additional value and actually can start this overfitting problem that I was just talking about. But the other piece is, uh, and this came up w- in our previous work as well, some variables are measured with a lot of error. So physical activity, especially if it's self-reported, may have a lot of error in it.

And what that means is when you have misclassification, and it can be random or it can be systematic in one direction or, or the other, it makes it very hard to predict anything on it. And you can just think about this on your day-to-day life. If I have to ask you, you know, some very detailed question about what you did yesterday at 2:00, and you're like, "Well, maybe I did this or that, or I might be off," and that's the basis of your prediction, you can see how it's not gonna be useful and it adds noise to your model.

So some of these terms just have poor measurement characteristics, making [00:20:00] them very noisy. Generally speaking, that's not a good term for a predictor in a prediction model.

Dr. McGavock: That's fantastic. That totally helps me understand, um, w- why we don't include some of these measures. And someone might come in and say diet, and I can just tell you right away, it's the noisiest variable you'll ever put in your, your model.

It's a great

Dr. Rosella: example.

Dr. McGavock: Yeah. I

Dr. Rosella: mean, it's so hard to make any prediction based on diet. Very hard.

Dr. McGavock: Okay. So statisticians in my group have now started to encourage us to use machine learning approaches. And so in your PhD, I think you selected the, the variables that you had in the survey. Now we have these machine learning capabilities.

Can you explain a little bit about why we're going to this? And is this-- is it fair to call this an unbiased approach? Or would you prefer- Yeah ... a machine learning approach?

Dr. Rosella: Yeah, I think that bias is possible with all of these approaches, so I wouldn't use the term unbiased. But I will say the main reason...

Let's think about what machine learning is and why it does well for a prediction task. So based on everything I just said, [00:21:00] you know, we're not really developing a causal model, we are not thinking about confounders in the same way. What machine learning does really well is it goes through a lot of variables and identifies which one can explain an outcome in a predictive sense, not in a causal sense.

And so for a predictive task, you can see why it works well. And there's... I mean, machine learning is a very broad term, and there's lots of versions of it, but, uh, so I'm speaking in generalities here. For prediction, machine learning actually does work well, but the caveat is you need to have the data for it to really perform well.

So when you have very structured data... So the data I, I was mentioning before, survey data, it's very structured. We didn't have a lot of variables. There were maybe 18 to 20 that we were really looking at in detail, and we ended up with, you know, 10 to 12. The advantage of a machine learning approach, which generally speaking is not parametric.

What does that mean? It doesn't rely on statistical distributions, meaning that you have to really specify what type of distribution the data comes from. [00:22:00] It can handle a lot more interactions. It can just generally handle more data. So if you don't have a lot of data, there is no reason to use a machine learning approach.

And that's really important because sometimes I see these comparisons, and they're kind of like, why would you compare these models? If your data's very structured and you don't have a lot of variables, there's no reason to believe machine learning will do well. Where machine learning is doing well is in the prediction space where there's lots of data, so lots and lots of variables.

I'm talking thousands of variables, not tens and twenties. And because you're not constrained-

Dr. McGavock: Not just-

Dr. Rosella: Yeah ...

Dr. McGavock: lots of data, meaning lots of participants or people-

Dr. Rosella: Correct ... in the cohort,

Dr. McGavock: but lots of participants-

Dr. Rosella: Lots

Dr. McGavock: of- ... plus lots of information about them.

Dr. Rosella: Exactly. Okay. So if you have a million people but 10 variables, there's not going to be a big performance advantage to using machine learning in that case because the, you know, machine learning really is about patterns between inputs and outputs.

You still only have, uh, an output, which in this case we would call that supervised [00:23:00] learning because you have an outcome you're trying to predict, diabetes, and you have inputs, and you have 10 variables. Even if it's on the million people, still only 10 variables. Right. It's gonna learn very quickly from those 10 variables, and you're probably gonna get a very similar, uh, result to a parametric regression, which works really well.

But when you have 1,000 variables, like some of the health administrative databases that you have access to in Winnipeg and now through digital health records, and I won't even talk about unstructured data just yet. While some of the traditional approaches aren't as flexible. It really is about flexibility.

So if I can incorporate more variables and I can do more interactions between those variables, I'll get a little bit closer to that perfect prediction that I'm aiming for because I'm able to capture those complexities which I can't using some of these other approaches. So I think that's why machine learning's becoming so popular from a prediction point of view.

We have more data than we used to have, a lot more. A lot more of our health data is being digitized, and so we have the opportunity to [00:24:00] realize some of those performance gains, which we didn't in the Framingham Cohort. That was a very s- well-collected, structured cohort study. There were lots of variables, uh, f- especially for that time, but not like what we're seeing now.

That's where we're going.

Dr. McGavock: So my introduction to this was a Scandinavian paper that came out five years ago, uh, looking at, they said five classes of diabetes. So they had genetic data, they had behavioral data, they had administrative data, they had clinical data. It was just unbelievable What they had. Yep.

And so they then said, "Okay, we're gonna throw all this data into the computer," put that simply. And is that unsupervised, where they don't know- Yep ... what's gonna come out?

Dr. Rosella: Correct.

Dr. McGavock: And the term latent comes out, uh, a few times, so maybe you can explain that. Yep. And so this is an unsupervised model, so maybe define that for me, and then things that we would not have seen clinically, right?

That's the import- That was the important discovery there for me, was that people have classified diabetes as Type 1, autoimmune, Type [00:25:00] 2, lifestyle. Uh, that's super general, I'm- I apologize, but that's kind of how we approached it. And they said, "Well, look we're seeing when we do a machine learning approach that incorporates all of this together."

So can you- Yeah ... explain a little bit about how they... Not to talk about the paper specifically, but, um-

Dr. Rosella: No, no, no. It's a great example. And, um, and so first I'll explain the difference between supervised and unsupervised. So supervised machine learning is when, as I mentioned, you have a labeled outcome. So I know what I'm trying to predict.

I'm trying to predict cardiovascular disease, diabetes, sepsis, and I have labeled data, meaning I know the variables that are going in. The process of developing the machine learning model is to put those together and say which combinations of these inputs are best explaining my output which I'm labeled.

Unsupervised learning, which a- again, is becoming very popular, although its direct applications in clinical and public health are- remain a bit uncertain, but there's some examples, especially in the areas of public health surveillance. But unsupervised learning is saying, "I don't have an outcome. I don't have [00:26:00] five types of diabetes.

I'm just saying, 'Here's a group of patients with diabetes. Can you take the data and cluster it in a way that puts them into groups based on all the data you have?'" They're more similar within these groups than they are between the groups. And there's lots of interesting examples of that in surveillance to say, "Well, this looks like something interesting, and this looks like something interesting."

And then there's a semi-supervised learning, which they take those clusters, and then maybe they put them into another model and they predict mortality. And maybe they found that this class of diabetes actually is at higher risk of death than this class of diabetes. So there's all kinds of interesting applications.

Or response to certain medications compared to others. Exactly. That was fascinating. And that's what you're seeing now. Now, the worry about those approaches is that of course you're going in without a particular outcome that you're interested in. Is what you're finding real, or is it, is it even clinically meaningful?

So a lot of these studies that come up with these clusters, the next question is, is it clinically meaningful? [00:27:00] And that's why I mentioned the semi-supervised, 'cause then you might take those clusters and actually try and understand how they relate to outcomes which are meaningful.

Dr. McGavock: So that's an appropriate application of whatever cluster comes out of that machine learning

Dr. Rosella: approach.

Exactly.

Dr. McGavock: Right.

Dr. Rosella: And it's different than sort of predictive modeling, but... And I'll also say, I'll go back to the over fitting concept that we're so worried about. Is it just that Scandinavian data set? Are the clusters that we found just unique to the data there?

Dr. McGavock: Mm-hmm.

Dr. Rosella: Ideally, again, you would take it out and see if you repeated the exercise in another data set, would you get patients looking similar in those clusters?

Or even do you wanna define those clusters in a more specific way and see if it's reproducible? So that's another concern that people have when doing those approaches.

Dr. McGavock: Okay. So to transition to the ethical piece Uh, I've been told that a systematic review is only as good as the quality of the studies that go into the systematic review.

I would expect the same to be true for a machine learning approach, that the approach is only as [00:28:00] good as the data that you have. There's a lot of controversy right now about algorithms to predict kidney disease and the use of ethnicity. In a lot of these data sets, we don't even have ethnicity or maybe some social factors.

And so there are some limitations to this, and maybe a machine learning or AI could maybe perpetuate some of these inequities or some of these racial biases. So is that something we worry about and, and how do you address that?

Dr. Rosella: Absolutely, and I would say we worry about it with all types of prediction models, and there's a couple reasons.

So one is when you develop a model, it's learning based on the data, and if the data-generating process or the structural factors that are underlying our society that led to that data being the way it is, it's gonna learn it. So there's nothing special about these methods that protect it against those biases.

In fact, why we worry about it even more is if it's then used for decision-making, we're going to actually propagate those biases and create a bigger [00:29:00] problem than there was to begin with, potentially. So the concerns are real, and they're serious. So what do we do about them? One thing that's really important when we do the validation I was talking about, and this is something that we do in my lab all the time, is we don't just validate our models externally to make sure they work, but we do a subgroup validation across important groups that may be important for whatever use case we're looking at or just the population in general.

So socioeconomic groups, geography, et cetera. And our question that we're testing for there is we've developed a model, looks like it's working well. Does it perform the same across subgroups? And this subgroup validation is critical, and in fact, I would like to see if some of these algorithms are starting to get regulated and FDA approval, which some of them are.

You know, that should be part of the approval, that actually you've showed that there's not differential performance by certain subgroups. But, you know, that's just knowing that there's a problem. Um, one very famous paper by Ziad Obermeyer [00:30:00] showed algorithmic bias among Black Americans using administrative data because they systematically do not have access to healthcare because of the insurance system in the US.

And because of that, the model was showing some odd outcomes which were simply reflecting the bias that exists for Black Americans getting health insurance. And again, we're gonna use that for decision-making. There's some serious concerns. So there's a lot of concerns that need to be addressed here. I personally believe we will, and- Advocate for a lot more regulation needed in this space before it's actually used for decision-making, before these things are approved.

And then these are some of the things that you can address through regulation. But the subgroup assessment, and then you mentioned the data quality piece. I mean, that's true for anything. Really anything we do in epidemiology- Yeah ... surveillance, we need better data quality, and it doesn't get enough funding or attention.

So we need to stop just kind of ignoring the fact that [00:31:00] the data quality, enhancing our data collection so we capture all the social determinants of health, for example, something we really shouldn't still be talking about in 2023. We really should be capturing all the social determinants of health so that these can go in our models and we can understand the impact across important subgroups.

These are all things that need, uh, further attention as well. But I will say- Yeah ... one more plug for epidemiologists- Yeah. ... which is, like, great. Um, one of the most important things I've learned, because when I work on, especially in the machine learning world, and I collaborate with computer scientists and lots of people that are computationally phenomenal, what epidemiologists bring to the table is our understanding of bias and our understanding of equity, and clinicians bring this as well.

So these model development teams, when you do have an epidemiologist and a clinician working alongside computer scientists, for example, the bias pieces come out a lot sooner, and we worry about them sooner. We try and measure it. We try and address it. We try and make sure it's incorporated in the model.

So just a plug for if you're thinking, "I'm never [00:32:00] gonna work in machine learning," maybe you're not running the models, but if you can be on the development team, you will offer a critical perspective on ensuring those models are more fair and less biased.

Dr. McGavock: Yeah, and maybe a plug for social epidemiologists or for people with- 100% lived experience to say, "Hey, wait a second. Let me tell you something about that data."

Dr. Rosella: Let me tell you about how I got into that data set- ... or why I'm not in that data set-

Dr. McGavock: That's great ... which is really

Dr. Rosella: important.

Dr. McGavock: Yeah, and you do talk about the importance of doing this in teams, which is maybe a nice transition to you've created two massive teams, AI for Health and T-CAIREM.

Do you wanna explain a little bit about that and then, uh, what the goals are for those teams, and if you're a trainee or a new investigator, uh, how they can get involved?

Dr. Rosella: Yeah, absolutely. And so big teams have created these teams, so, uh, I'll acknowledge that. I'll talk about AI for Public Health first. So this is a health research training platform, and so we got funding for six years to develop a series of training opportunities to enhance AI [00:33:00] training for people interested in really applying this in the public health or population health context.

So what we're trying to do is we recognize that these technologies are rapidly advancing, and our training, uh, in university programs is doing great, but it's, you know, it's hard to turn those things around in terms of having the competencies to be able to work in this area. And I have to tell you, our motivation was really about the betterment of society, which sounds kind of cheesy, but if we are not able to apply the best technology, the best approaches for even talk about public health surveillance, you know, which there's a, a lot of AI applications now for public health surveillance, for prediction, for understanding heterogeneity in subpopulations.

If we start falling behind, this affects population health. So we really want to-- We've looked at this in a very broad way. We're doing short courses. We have workshops. We're funding scholarships so that people working in this space can really upskill or be exposed to the different [00:34:00] aspects of what it means to do AI in a public health context.

This means equity and bias as well. That's a core part of our training, as well as some of the technical aspects and as well as some of the application and implementation aspects. So we have investigators across the country. Uh, we have, uh, mentors from across the country, and, uh, we're really excited about the potential impact for sort of reinvigorating and making sure that we're all aware of this training, what it can do, what it can't do.

The private sector has moved at lightning speed- Yeah ... in incorporating, and we're really worried about falling behind. Making sure that we are at the table to define how it's used responsibly and ethically and for actually improving people's health. So that's the goal of that. T-CAIREM, it was made possible by a generous donation from the Temerty family at the Faculty of Medicine at University of Toronto, and it, it's an institute for training, education, and research- For AI in medicine specifically, the director is Muhammad Mamdani, who's a really amazing person that leads advanced analytics [00:35:00] units in the hospital, and we're really focused on, again, education and training opportunities for people in medicine.

So we have developed required modules in undergraduate medicine, um, which is pretty amazing. In first year, uh, undergraduate medicine, the trainees are getting exposed to what this means for them, knowing that by the time they graduate, they'll probably have to interact with some kind of AI in their clinical practice.

Mm-hmm. Uh, we're also really focused on implementation and ensuring that there's clinical input into everything that's happening in this space. And so again, it's open to people across the country, so if there's anyone interested, you can join and be a member. And a lot of our opportunities are national, and since we've started, there are now national work groups around education, around training, around research that are forming across Canada.

So it's, it's very exciting. I have to say, we get a lot of attention from people in Europe and in the US because they don't have, uh, [00:36:00] centers quite like this, so it's pretty exciting.

Dr. McGavock: So that, and that solves a huge problem that I think some people will have if they have had no exposure to it and they know ChatGPT or whatever.

They're like, "How do I apply this to medicine?" They can, uh, come and see you and then, uh, get a little more familiar- Yeah ... with it. The other concern I, I was thinking about is do I have the right data? Like, can I actually use this in my research space or my clinical space? And so what are some of the criteria you're using?

And I, I think your center's really attractive to do this kind of work because you have amazing administrative health data or electronic health records. Same in Manitoba, plug for Manitoba.

Dr. Rosella: Manitoba's even better,

Dr. McGavock: I have to say. Like- More

Dr. Rosella: comprehensive. Yeah.

Dr. McGavock: When you're looking at like these, this is my gold standard or, you know, or these are my, this is my minimum sources of data, what are you looking for?

Dr. Rosella: Yeah. So I mean, some of the key aspects that I certainly I look for is it's really important to have that base [00:37:00] of the population. I think what's really tricky, especially people I don't think this way they understand, is sort of like what's my denominator? Who's in? Who's out? How did they get here? If you don't understand that part, it's very hard to undo, uh, all those selection biases that go into who's in your data set.

So a really solid, you know, we happen to have sort of a population-based, uh, way of thinking about admin- linked administrative data in Ontario as you do in, in Manitoba, but in a lot of other settings that doesn't exist. It might just be people that have come to a hospital or come to a practice, which it doesn't mean you can't use that data, but you really have to understand, uh, sort of who's in and who's out of that particular data set.

And the ability to link across databases is generally really important. It's very difficult to get everything you need in a comprehensive model, especially if you wanna do subgroup performance and you wanna ensure generalizability and you wanna look at fairness if [00:38:00] you aren't able to link to other databases.

I don't know of any data set that has everything all packed into one, so generally you need some ability to link across other databases. You wanna have some understanding of data quality, even if not all data elements can be validated, but you wanna have an understanding of what the quality is for at least the variables you're using.

If any of those is uncertain, like uncertain data quality, you're unsure of who you're capturing in the population, you're, you're unsure of what's being measured, how it's being measured, or if it's being measured the same way- You know, don't even start. Yeah. There's no point in trying to actually try to do anything with that data set if some of those minimum requirements aren't being met.

And maybe then your goal is to really get that data set in shape. Mm-hmm. Which I wish more people would be doing that, because once the data set's in shape, you have consistent measurement, consistent data quality, understanding of who's in and out, good collection across social determinants of health, for example, has a tremendous impact, not just on a particular project, but the thousands that will follow.

Dr. McGavock: Right. [00:39:00] So an example I ran into recently is should we use primary care data or should we use administrative health data? One has everyone who goes to primary care, the other has everyone.

Dr. Rosella: Yeah.

Dr. McGavock: One has detailed lists on, of everything that you're tracking in primary care. The other one has ICD codes for everything that you've done as you've interacted with the system.

And I think about myself as a, a mid-40s, uh, man who doesn't go to see a doctor in primary care, right? I'm missing from that. But then more importantly, I also think about the indigenous youth that we work with who are not being seen in primary care, and I think that- That's right ... is a major gap. And so that data set, it would be something we'd be concerned about.

But the other data set that has absolutely everyone from the province or the majority of people from the province is something that you'd rather work with.

Dr. Rosella: Yeah. And I think it also depends on the purpose of the analysis. And so if the purpose is to really understand a particular aspect of treatment that is only in the primary care EMR, then that's one [00:40:00] thing.

But if you're actually trying to understand a phenomenon at the population or who's getting care and who's not getting care, then you might have to sacrifice some of the clinical detail for having that full population coverage. It, it depends on what you're looking for. Obviously, you know a lot of the work I do, I'm interested in populations and how things move through populations, so I t- I tend to favor that a lot more.

But if there's something really specific you need in a particular database, then that could change your perspective.

Dr. McGavock: And for residents out there, we are so fortunate in Canada to have access to databases like this, and so please take advantage of it when you're thinking about your resident scholarly project.

Dr. Rosella: Yeah, absolutely. And we've been talking about Ontario and Manitoba. Of course, there's others that have similar shops, and Statistics Canada's been really great at, at enhancing their linkages. CIHI has a national- Yeah ... hospital database. You know, we now have linkages between the census and some hospital data.

You know, these are amazing opportunities for research.

Dr. McGavock: Awesome. You have cleared so many things up for me in the last [00:41:00] 45 minutes. Thank you so much. Uh, is there anything else I, I missed that, uh, you, you think we should talk about as it relates to, uh, AI or machine learning or predictive epi?

Dr. Rosella: Yeah. I mean, I think I'll just say one, uh, thing.

I've said it earlier, but I'll re- reinforce it. Don't feel that in order to be involved in this space that you need a lot of deep technical training. So just go into it knowing that this is a team sport, and some of you actually may love the programming aspect of it, and that's going to be your role on, on the project.

But your role may also be working with a programmer and providing that critical insight into the problem or- Sometimes we call that domain knowledge, and it's equally if not more important to play that role. So, you know, awareness is the first step, and then being involved can look like many different things.

Like you can be involved in understanding how it's used. You can be involved in the coding and doing all the validation work. You can be involved in providing input on what variables are important or what subgroups are important to look at. So [00:42:00] that's what I'll, I'll say is not a lot of people get intimidated about the actual programming part of it, and that's only one, uh, tiny piece in a way to, to get involved with this type of work.

Dr. McGavock: But if you're passionate about improving your decision-making, this is a great way to get involved in improving- Absolutely ... the landscape, so awesome.

Dr. Rosella: Yeah. Very impactful.

Dr. McGavock: Awesome. Thank you so much, Laura. Uh, that was wonderful, and, uh, hopefully there are people in your province, uh, residents in your province that are, that are looking to do this kind of work, and they'll reach out to you to, uh, to see if they can collaborate in some way.

And, uh, good luck with, uh, AI for public health and, uh, T-CAIREM. I think it's gonna really change the landscape in Canada, so thank you for leading that.

Dr. Rosella: Great. Thank you so much. Great discussion, great questions, and really nice to speak to you.

Dr. McGavock: That was our conversation with colleague and friend Dr. Laura Rosella on predictive epidemiology and the role of machine learning and where predictive epidemiology will go next.

Be sure to join us in episode nine, the first in a two-part series with Dr. Robert Platt on our [00:43:00] third type of epi, causal epidemiology. Thanks for listening. The Emerging Scholar Podcast was recorded in the UMFM Studios located at the University of Manitoba in Winnipeg, Manitoba, Canada. This is Treaty One territory, the original homeland of the Anishinaabe, the Ininew, and on the national homeland of the Red River Métis.

Manitoba is also homeland to the Anishininew, the Dakota Oyate, and the Dene peoples. We acknowledge that our water is sourced from Shoal Lake 40 First Nation. As a settler scientist working in health research, I acknowledge TRC call to action number 18, that health inequities between indigenous and non-indigenous people are the direct result of colonization.

Winnipeg is the center for reconciliation in Canada, and where we are speaking from today is at the crossroads of Canada's history and Canada's future. We are three kilometers away from one of the most sacred meeting grounds for indigenous people on the plains and within Turtle Island. We are also several kilometers away [00:44:00] from one of the last residential schools in the province.

We are working collectively on reconcili-action with youth, elders, and community leaders to heal from the trauma of colonization, past and present, honor the original treaties, and create a Canada that addresses and eliminates the inequities and injustices of the past and present to create a future for indigenous children to live and thrive.

On behalf of the team at the Emerging Scholar Podcast, we wanna thank Osani Balkaran for the great production for our opening and closing themes. We also wanna thank UMFM 101.5, especially co-producer Jared McKetiak for technical assistance and production on this project. We also wanna thank all of our guests for the discussion and the insight that they've provided.

Once again, thank you for listening. Be sure to subscribe to the Emerging Scholar Podcast on Apple Podcasts, Spotify, and umfm.com.