Designing Research Questions That Involve Real World Experiments with Dr. Bapu Jena

Dr. McGavock: [00:00:00] Welcome to the Emerging Scholar Podcast Season 1. We hope you enjoy this series of great conversations with leading medical researchers that's designed for graduate students, medical trainees, residents, and fellows to help you on your journey towards your first scientific project. Welcome everyone to episode two, season one of the Emerging Scholar podcast.

Today we are interviewing Dr. Bapu Jena. Dr. Bapu Jena not only had his own, uh, podcast called Freakonomics MD, but he's also a clinician and economist at Harvard Medical School. He holds the Joseph P. Newhouse professorship in healthcare policy within the Department of Healthcare Policy within Harvard Medical School.

His recent book called Random Acts of Medicine: The Hidden Forces That Sway Doctors, Impact Patients, and Shape Our Health, is an introduction [00:01:00] to the incredible work that he does. The reason that we brought Bapu on today is not only because he's a podcast host of Freakonomics MD and therefore has some wonderful stories to tell, but he also designed some of the most interesting, innovative, clinically meaningful questions that I've seen in my lifetime.

And so we brought Bapu Jena in to think about how do we design research questions that involve natural experiments, real world experiments, and then using those experiments to design studies within administrative data sets. It was a real pleasure today, a real honor to interview Dr. Jena. I hope you enjoy this interview as much as I did Okay. We're here with, uh, Dr. Bapu Jena from Harvard University, and I'm gonna turn it over to you to maybe introduce yourself to the, to the audience.

Dr. Jena: Thank you for having me. Uh, so I'm an economist and a, and a physician. I, uh, work clinically at Mass General Hospital, which is, uh, one of the large teaching hospitals in Boston.

I work as a [00:02:00] hospitalist there a few weeks a year, but I spend most of my time doing economics research. That's, uh, health economics research.

Dr. McGavock: And, uh, my, my understanding the trajectory was economist first, physician second. So can you give us a little history of where you started and then how you

Dr. Jena: got to it?

Oh, yeah. It was totally random, right? So I, uh, I studied, uh, biology in college, uh, because I, I knew I wanted to go to medical school, and I thought I wanted to do research as well. But I thought I might wanna do basic science research, like, you know, working in a lab. And, uh, the only reason I studied economics in college was because I thought I needed to make myself a little bit more well-rounded for medical school.

And so I thought, "Okay, economics sounds like a humanities." Turns out where I went to college, it, it's not a humanities. It's, it's like very mathematical, and it was, uh, s- completely different than what I expected. But anyway, I did it, and when I was interviewing for MD-PhD programs, this would've been, like, 1999 now, and I visited the University of Chicago, the director of that program [00:03:00] said to me, "Oh, I saw you studied economics as well.

Would you be interested in doing your PhD in economics instead?" And I was like, "Uh, does this person think I can't do a PhD in cell biology?" "Is that what's going on here?" Yeah. And so I said, "All right. You know what? I'll try." So literally that afternoon he set up some meetings with some economists at the University of Chicago, which is a, a very good place to study economics, and, uh, they sold me on it.

So I, you know, that's it. There's not much downside. If I can't do it or if I don't like it, I'll go back to my original path of, of being a basic scientist or clinician scientist. That's how it all started, and the rest is history.

Dr. McGavock: And you landed with a pretty impressive mentor.

Dr. Jena: Yeah. No, I did. Yeah, and I would say I, I don't think it would've been I couldn't have done much better, I think, if I had done a PhD in cell biology or immunology or genetics or something, so. Yeah. I think it, it turned out okay.

Dr. McGavock: Okay, so the reason we're talking today is that I feel like you are probably the best health researcher at the moment in terms of coming up [00:04:00] with great impactful ideas that you turn into these just beautiful research projects that have important, meaningful impact.

Can you give me some ideas as some of the most exciting projects you've worked on recently, just to get a sense of some of those, some of those cool ideas?

Dr. Jena: Yeah. So it's interesting when you talk about ideas because every time I have an idea, I think, "Oh, this is my favorite idea." And then I have another idea, and then that becomes my favorite idea.

So let me s- I'll tell you one of my initial favorite ideas, and that was one which, uh, I think we published in, like, 2014 or 2015. Um, and this was really my first foray, or maybe second foray, into this line of research, which is very much like Freakonomics meets medicine. And that was a study that looked at what happens to patients who are hospitalized with acute cardiac conditions during the dates of the American Heart Association and American College of Cardiology meetings.

And the thought was these meetings are large. Cardiologists leave town. [00:05:00] Often they leave large academic medical centers to go to these conventions to present research, to hear about research. But what happens to the patients who are left behind, who come into the hospital with a cardiac arrest and need cardiac care?

Do their... Do the outcomes differ? Does the care that they receive differ because there's a different composition of cardiologists who remain behind than are there typically in the year? And I thought that might actually lead to worse outcomes because I thought, "Well, staffing might be lower. Maybe the preeminent best cardiologists are out of town."

But somewhat surprisingly, we found that outcomes were, were better, and they were actually a lot better. Um, so, you know, in, in this data, 70% of people who have a cardiac arrest, these are Medicare patients, so they're older, 70% die within 30 days of having a cardiac arrest. If you happen to have a cardiac arrest during the dates of one of these cardiology meetings, uh, it falls to sixty percent.

And so it's a huge reduction in mortality. It's larger than the effect of beta blockers, statins, [00:06:00] stents, whatever it is that cardiologists might do. And so that was one of my studies that it-- and, you know, the-- it's a good example because it uses large data, it focuses on a natural experiment, in this case, people by chance have a heart attack during two different periods, when, when cardiologists are plentiful and another, uh, period when there's maybe different or fewer cardiologists around, and patients don't choose that.

So it's really as good as random. And then most of what I do has a, has an element of sort of creativity or surprising characteristic to it, and this falls in that category. And then the fourth piece, and this is true for most of my work, it's not true for all work in general, but I try to pick projects that I could explain to anybody.

They don't have to be in medicine. They don't have to be in health economics. You know, you could pick anybody off the street and explain this finding to them, and they would understand and be like, "Oh, that's kinda interesting. I wouldn't have expected that."

Dr. McGavock: That sounds like your [00:07:00] algorithm

Dr. Jena: That's the algorithm,

Dr. McGavock: right.

So let's use this as an example. One of my favorite papers, 'cause I'm a runner as well-

Dr. Jena: Mm-hmm ...

Dr. McGavock: and the story you tell about driving your wife to a 5K race and around, was it Mass Gen?

Dr. Jena: Yes, yeah.

Dr. McGavock: And you're, you're like, "I can't find parking anywhere here, and I have a parking spot. What's it like for an ambulance?"

So was that the idea? Or tell me about how that idea sprung into your head, and then what did you do? Like, what was the process of taking it from that observation in the real world and bringing it all the way to the finish line in the New England Journal?

Dr. Jena: Yeah. So it was an idea that I, I really credit my wife, because my wife was running this race, and it passed through Mass General, and she wanted me to watch her on the race route, but I couldn't because the place I was gonna park, I couldn't get to that parking space.

So I turned around and went home. And hours later, I see her at home, and she just makes this offhanded comment. She said, "Well, what happened to everybody else that needed to get to the hospital that day?" And that was like, you know, that was just a offhanded comment. No one would ever [00:08:00] think, "Oh, I'm gonna write up a research paper about this."

But I thought, yeah, that's a really good point because marathons are incredibly disruptive to the cities where they're held. I mean, they, they, they generate a lot of value for people who are running, they generate a lot of value for spectators, maybe for businesses, but they are also, you know, a challenge for people who need to get from point A to point B when a city's hosting a marathon.

And so that was sort of the impetus. Like, it was this offhand comment that she made, and literally the next day I, I went to my office and I spoke to my research assistant and I said, "Let's just get the, the dates of the 10 major marathons in the US over the last 10 years and look at a map and figure out what the zip codes are that those marathons go through.

And then let's link that information to information from Medicare, where we have data on hospitalizations and outcomes, and just let's look and see very quickly whether or not people who have a heart attack or a cardiac arrest, these are time-sensitive conditions, have worse outcomes on marathon days compared to the identical [00:09:00] days of the week in the surrounding weeks in that same city.

And let's do the same thing for cities that are close by but that are not directly affected by the marathon as sort of a control group." That figure was very easy to produce, relatively speaking.

Dr. McGavock: You already had the data.

Dr. Jena: We already had, yeah, so we had the data- Okay ... which is the hospitalization data. And, and for a lot of what I do, I use pretty standard data that, that are available to health s- researchers, like insurance claims data, national registries.

Um, so I, I haven't specialized in getting access to unique data or collecting, uh, unique data. That's not my sort of MO. It's really trying to take something that's well-trodden, where people know the data, and come up with a different question.

Dr. McGavock: So you just, you generated that figure and then... Wow.

Dr. Jena: Yeah. We generated the figure, and then we knew that there's something here because we saw that mortality rates went up by about 15%.

We know that, uh, you know, the average age of that population is 77 or 78, so they're not running marathons. Right. So these are not people who are [00:10:00] running marathons and then having a cardiac arrest because they ran a marathon. And, you know, in much of my research it, it starts with just one figure, and then you know you're either off to the races or you just need to close down.

And in that case, we knew that we were off to the races, and then we did a series of other analyses that kind of elucidated the mechanism, made sure that what we were finding was a real finding as opposed to something that's just sort of random. Uh, and that, that whole project probably took about a month, maybe six weeks.

Dr. McGavock: Wow.

Dr. Jena: And so you can, you know, I think when people do research, what they don't realize is that many ideas take the same amount of time. An idea that might end up in a prestigious journal, an idea that might end up not in a prestigious journal, difficult to publish, many of those ideas take the same amount of time.

And so that's why I really prioritize the importance of the idea over everything else.

Dr. McGavock: So I had a discussion with a grad student, uh, in my lab, Nicole, about where ideas come from. Uh-huh. And I was [00:11:00] watching, uh, uh, Comedians in Cars Getting Coffee, and-

Dr. Jena: Mm-hmm ...

Dr. McGavock: Seinfeld said, um, "Ideas are like mice just running around your room, and all you gotta do is just when you see one that you like, you just put your foot on its tail," right?

Um, but Nicole said, "No, comedians and, and Bapu, they are great observers of the world." And comedians who are really great, they're showing you something in the world that you may have missed but is obvious to you and has real impact and, and you connect with, right? So- Yeah ... you've got this process that you talk about where, um, I guess your grad students get together a couple days a week, and you talk about ideas and kind of like work them through.

Can you talk about, like, what is your process for observing the world? I have a feeling that's one of the things that kind of drives it. Um, and then how do you just weed out the ones that are good for research and the ones that aren't gonna make it?

Dr. Jena: It's an interesting question. So I didn't always think this way.

I mean, if I were to think back to when I was in graduate school, I didn't have this sort of mind, which is like looking at all these random things that I'm observing in the world and, and [00:12:00] thinking whether it could be a good research study. That sort of happened incrementally, and it started with just w- you know, one project, and probably that project was, I remember it clearly, I was sitting in the library at the University of Chicago and looking at Google.

Well, no, looking at, uh, Yahoo News. So, you know, many people, I often joke, will look at like The New York Times, Washington Post, Wall Street Journal, some of the more prestigious news outlets, and I was looking at Yahoo News. And I came across an article that mentioned something about STD rates in the elderly being on the rise, and somewhere down the screen there was an advertisement for Viagra.

And I just put those two things together, and I said, "Gee, I wonder if the introduction of Viagra, which is a huge, you know, innovation medically, might be responsible for this- ... epidemiologic finding." And so that was the first study that I did, like, looking at that issue, and it sort of whet my appetite for these types of questions, and then the cardiology meeting study was an early one.

And after that, it sort of cascaded to where I was, for the most part, [00:13:00] primarily interested, and sometimes only interested in those kinds of studies. And the question is, well, where do they come from? I don't think that they come from me being particularly creative. I think they come from me specializing in those kinds of questions and being interested in those kinds of questions.

Now, there are some things that I do I don't know if they work or not, but I do it because I think it's a proactive process, is I sit down with the people I work with, you know, probably two to three times a week, and we meet for about 45 minutes to an hour, typically when I'm driving to soccer, so it's not like it's a huge time sink.

I'm doing something else at the same time. And we just ta- talk about ideas. And I'll sit down, I'll say, "Okay, who's got an idea?" And most of the ideas that we talk about are not good, including the ones that I put forward. But it's that process of just brainstorming and hearing something and just train of thought, like, "Okay, that makes me think of this."

And then someone says, "Well, this is not really related, but you made me think of this." And we iterate on that until we eventually come up with an interesting [00:14:00] idea. And I think that process is useful for two things. One is it is the case that some of our ideas actually come from those sorts of meetings.

But the second thing is that I think if you train yourself to think like this and see the world and focus on sort of quirky questions, you'll observe that in all sorts of encounters that you have in the world. Like, I, I, I find it hard to imagine that we can train someone to transplant a heart from one human being to another, but we can't train people to come up with questions like the ones that I answer.

Right. Um, and so I, I think, I think that there is some learning by doing that is definitely helpful here.

Dr. McGavock: So that algorithm is creativity. I'll maybe go backwards. So it needs to be a pretty creative question. Yeah. You, you need to be able to imagine what a natural experiment would look like. So you talked about red- Sure red meat once. So red meat and cardiovascular disease. So how do we get a natural experiment to prove that reductions in red meat are associated [00:15:00] or causally linked to reductions in heart disease, right? So you-

Dr. Jena: Yeah ...

Dr. McGavock: so that's your second one. That needs to be there. Yeah. You need to have access to the data in some way, right?

'Cause you're not gonna be prospectively collecting it.

Dr. Jena: That's right. Mm-hmm.

Dr. McGavock: Um, and then is there sort of a, a causal model that you create to say, even if we have the data, do we have all the ways that we can make sure that this is a robust analysis?

Dr. Jena: Yeah, I think that's, that comes into the question of the natural experiment.

Like, you know, in the example that you gave, which is does red meat cause heart problems, it's hard to know because people who eat red meat probably do other things that are not healthy as well, and maybe those things cause the heart problems. And so we're always thinking, is there a clean natural experiment that essentially takes some people and makes them eat red meat by chance, and other people not eat red meat by chance?

And once you have that, I think a lot of the pieces fall together, and the only thing that remains difficult to show is, like, [00:16:00] what's the mechanism? That, that sometimes is hard. So in the, in, in the New England Journal study that you mentioned about marathons, we showed that mortality goes up on marathon days, and my hypothesis was that it was because the roads were closed.

But that's something that we actually tried to show mechanistically because we said, "All right, well, can we find ambulance transportation data?" So we got, for a subset of cities, ambulance data where we knew the pickup time and the drop-off time, and we could show that during the mornings of marathons when the roads are closed, that there were increases in transportation time.

But in the evenings when the roads reopen, those differences vanished, and it looked just like any other day. And so I think that mechanism is, is important. I will say one thing, which is I think people underappreciate the importance of the idea, and so let me give you an example. Suppose that a high school...

Uh, maybe even be more extreme. Suppose that a middle school student happens to be listening to this podcast or my, my podcast when they're with their parents in the [00:17:00] car and says, "Mom, I got this interesting idea." And the mom emails me, said, "Oh, my son or daughter had this random idea." I would personally be willing to put that middle school student as an author on the paper if they provided what I think is the scarcest ingredient.

It's that idea. Now, we, we place a lot of emphasis in research on framing questions, clear writing, rigorous methodology. In my line of work, it's, you know, good, uh, statistical work, careful programming, statistical programming. All that stuff is important, but you can find people who can help you with those things.

But if this, you know, 12-year-old suggested to me a brand-new idea that I hadn't thought of and I hadn't seen anybody think about before, that just, you know, that, they just bought a, you know, a ticket- Yeah, totally ... into the, to the show really.

Dr. McGavock: Yeah. Well, and you talked once, uh, I heard an interview where you said, "I'd love to develop a course on how to create good ideas."

What, what would be some of the things you'd teach in that [00:18:00] course for generating good ideas?

Dr. Jena: Well, I, I would say we, uh, you know, I would focus on- Myself and the students reading those ideas, which I think we can objectively say are really interesting or creative. And they might not be the most important.

A lot of stuff I work on, to be fair, isn't, you know, policy relevant sometimes, but it's just interesting and creative. You can't fault me for not having, you know, checked that box. I would say surround students and myself with those kinds of ideas, and then the, the business of the course should really be about just coming up with ideas, not spending time learning what everybody else has done, getting into the weeds of what they've done.

Not to say that that's not important. It- it's important to cross the finish line, but there's no point running the race unless you have an idea in the first place, a good idea in the first place. So I, I worry less about crossing the finish line and more about is there something that makes me wanna start this race in the first place?

And I would say it's just volume. You know, we would sit down three times a week or two times a week, and everybody would [00:19:00] just bring an idea or two ideas and, and high frequency talk about ideas. Almost like a Shark Tank, if you will.

Dr. McGavock: Right. And, and you also said once, um, a lot of times a resident, right, a trainee will come up and they'll be like, "Oh, I got this great idea.

I was on rounds with so-and-so, and they had an idea for a project." And you said, "Yeah, I, I wonder where that idea's coming from," right?

Dr. Jena: Yeah, yeah, yeah.

Dr. McGavock: Now, how does a- It's, uh- How does a trainee make that choice? Like, this is good or this is bad. What kind of advice would

Dr. Jena: you give? I would say, you know, the job of people like me and, and you and others who are, who are supposed to be mentors is to be accessible.

So I always tell students and trainees, "If you have an idea, just shoot me an email, and I'll shoot you an email back." To me, it's interesting or not interesting, and that's just, you know, that's one data point. Someone else might think it's incredibly important or not, and they might differ from what I would think.

But I think that's what we should offer them, is that if, if someone has an idea, we should be able to say, "Yeah, you know, here's what we think about it." But in general, what I tell residents and trainees is that they are exposed to [00:20:00] a lifestyle where there's just so much interesting stuff happening around them that if you just tune in to what you're seeing and think in this way, you could pick up literally multiple things a day.

And when I'm on service, I'm always seeing things that are like a source of ideas. I mean, I'll give you just one quick example. Last time I was on service, we had a patient, as many people do, who came into the hospital without COVID and got COVID in the hospital. And as a result, we were wearing surgical masks, and then we started wearing different outfits and different masks when we went into their room.

And that got me thinking, well, what if we use that as a way to figure out whether surgical masks work or not? Because it's, you know, it's random. Some people get h- COVID in the hospital, other ones don't, and we actually see the history of that evolving in real time, and we know how our behaviors change.

We could, we could figure out what's the causal effect of wearing surgical versus N95 masks and other [00:21:00] PPE in the hospital setting just by looking at something like that. So that's like, you know, that's like a random observation I made, but it could be a research paper.

Dr. McGavock: Totally. And I... The other great question you came up with, so one is Or what I hear you saying is you have an observation, is it a natural experiment, right?

And I think with your economist training, maybe you're always looking for those natural experiments.

Dr. Jena: Yes. Yeah.

Dr. McGavock: And the second one was this guy took 15 years to do this long colonoscopy RCT, and then at the end of your podcast you're like, "Well, wait a minute. W- I think we could have a natural experiment here," right?

Yeah. Which is- Yeah. ... is there another way to do this that might not take as mu- many resources? And I loved it, and I love that you put it out there to say, "Hey, if you guys have the data, maybe you should try this."

Dr. Jena: Yeah. I think, uh, and the way I would say the, the way I think about natural experiments and randomized trials is we need randomized trials, especially for something as important as that.

But sometimes natural experiments can give you valuable information until the trial is run, and also natural [00:22:00] experiments can give you information about what trials should be run and not, right? Like, if you had good natural randomization of patients to one treatment versus another, and there was some indication that a particular treatment might be effective for a disease and we didn't know that before, I would look at that information and say, "Okay, maybe I should design a randomized trial on this information"- Right as opposed to saying, "All right, I do this observational study where I show that peanuts are associated with dementia. Now let me do a randomized trial of peanuts to see if it affects dementia." Right. Like, I, you wouldn't do that because ... People would do that, but you shouldn't do that because the quality of the evidence that you're basing that decision on isn't great in the first place.

Dr. McGavock: Yeah, and if you've got the data already, you might as well think what's the best way to use that data in the most creative way possible that gives us the best kind of causal, or we approach causality with our answer.

Dr. Jena: Yes, exactly, yeah.

Dr. McGavock: Well, I know you gotta run. I'm so grateful you made time with us. Bapu Jena is the host of Freakonomics MD.

Please take a [00:23:00] chance, go check it out. Uh, it's part of the Freakonomics network. Uh, great talking to you today. All the best, uh, in the future, and, uh, hopefully we get a chance to brainstorm on some ideas one day soon.

Dr. Jena: Yeah. And I will say, anybody who listens, if you have a good idea or any idea, send it to me and we'll talk.

Dr. McGavock: Guaranteed. All right.

Dr. Jena: All right. Thanks a lot. Thank you so much.

Dr. McGavock: Really appreciate it. Bye.

Dr. Jena: All right, take care.

Dr. McGavock: You too. That was our conversation with Dr. Bapu Jena on how he and his lab design very cool research questions out of real world natural experiments. Be sure to join us for episode three where Doctors Lori-Ann Linkins and Alfonso Iorio talk to us about where to find the best evidence to guide your next research project.

Thanks for listening. The Emerging Scholar Podcast was recorded in the UMFM studios located at the University of Manitoba in Winnipeg, Manitoba, Canada. This is Treaty 1 territory, the original homeland of the Anishinaabe, the Ininew, and on the national homeland of the [00:24:00] Red River Métis. Manitoba is also homeland to the Anishininew, the Dakota Oyate, and the Dene peoples.

We acknowledge that our water is sourced from Shoal Lake 40 First Nation. As a settler scientist working in health research, I acknowledge TRC Call to Action number 18, that health inequities between Indigenous and non-Indigenous people are the direct result of colonization. Winnipeg is the center for reconciliation in Canada, and where we are speaking from today is at the crossroads of Canada's history and Canada's future.

We are three kilometers away from one of the most sacred meeting grounds for Indigenous people on the plains and within Turtle Island. We are also several kilometers away from one of the last residential schools in the province. We are working collectively on reconcili-action with youth, elders, and community leaders to heal from the trauma of colonization, past and present, honor the original treaties, and create a Canada that addresses and eliminates the inequities and [00:25:00] injustices of the past and present to create a future for Indigenous children to live and thrive.

On behalf of the team at the Emerging Scholar Podcast, we want to thank Osani Balkaran for the great production for our opening and closing themes. We also want to thank UMFM 101.5, especially co-producer Jared McKetiak for technical assistance and production on this project. We also want to thank all of our guests for the discussion and the insight that they've provided.

Once again, thank you for listening. Be sure to subscribe to the Emerging Scholar Podcast on Apple Podcasts, Spotify, and umfm.com.