This interview is part of CVB’s ongoing initiative to drive reproducible, robust, and confirmatory science to improve the research process and accelerate results for patients.
Q&A with the Authors of “White Matter Hyperintensities Are Not Related to Symptomatology or Cognitive Functioning in Service Members with a Remote History of Traumatic Brain Injury”, part of the Null Hypothesis Special Collection in Neurotrauma Reports
The Null Hypothesis Special Collection in Neurotrauma Reports is a collection of articles comprised of high-quality, well-conducted peer-reviewed studies that incorporate negative, inconclusive or replication findings in the field of brain trauma. The initiative, which is part of a collaboration between Cohen Veterans Bioscience and the Center for Biomedical Research Transparency, was initiated in 2019 and has led to the publication of eight empirical studies across brain trauma that highlight the value of all data for better understanding the biological underpinnings of brain injury, including traumatic brain injury, concussion, and spinal cord injury.
We caught up with three of the authors of one of the Null Hypothesis publications, Drs. Sara Lippa, Kimbra Kenney and John Ollinger to learn more about their findings and why initiatives like this are important for ensuring that all research findings are considered, no matter the outcome.
Chantelle Ferland-Beckham: Can you tell me a little bit about your study, what the research questions were and what the main findings were?
Sara Lippa: In this study, we looked at whether white matter hyperintensities were related to cognitive outcomes and subjective symptoms in individuals with a history of mild traumatic brain injury (mTBI). To answer this question, we administered MRIs, neuropsychological tests and different self-report measures to about 1200 active-duty service members with a history of mTBI, and then looked at whether having any white matter hyperintensities on your MRI was related to worse outcomes, whether the total number of white matter hyperintensities was related to worse outcomes and whether those with the most white matter hyperintensities (individuals with 20 or more) had worse outcomes than individuals with zero white matter hyperintensities. What we found was across 37 different measures of cognition and self-reported symptoms, no matter how we sliced and diced the data, worse outcomes were not related to white matter hyperintensities on MRI.
Chantelle Ferland-Beckham: In your opinion, how does this advance the knowledge on TBI? Does it challenge something that was previously established, i.e., that these white matter intensities are predictive of mTBI outcomes?
Sara Lippa: I’m not sure that there already was a definitive answer to that question, but there already were some published studies, mostly with very small sample sizes, that had looked at whether white matter hyperintensities were related to cognition or symptom reporting following TBI. And the findings of those studies tended to be mixed, with one study, for example, finding that white matter hyperintensities are related to memory and another finding they are related to executive functioning, but not memory. But there was still this idea out there among both patients and providers that the more white matter hyperintensities one has, the worse off one would be.
Kimbra Kenney: Just to add to that, both Sara and I are clinician researchers. And so the idea for this study came from our clinical experience and from other clinical providers. Early on, especially with mTBI, the TBI research community was looking for THE biomarker that would tell you if you had TBI. I think everyone thought that imaging would give us the answer and that clinicians would be able to use MRI changes to definitively diagnose a prior TBI, even long after that TBI-causing event occurred. As Sara mentioned, because there was some evidence out there from small sample size studies, clinicians started evaluating people with highly symptomatic TBI long after the event and there was a tendency to attribute their symptoms to these white matter hyperintensities from the TBI. So there was this impetus to look at, in a rigorous as well as a nice sized cohort, whether there was a correlation between white matter hyperintensities and mTBI symptoms and outcomes.
Chantelle Ferland-Beckham: Given that there are a lot of studies out there that are negative and non-confirmatory and that most of these types of studies don’t get published, why was it so important to publish this study?
Sara Lippa: We felt that these findings had really important clinical implications that needed to be highlighted. As Kim mentioned, we felt that it was really important that if providers discuss white matter hyperintensities with their patients, that they do so in a way that minimizes any potential iatrogenic effects. If a provider feels it’s very important to talk about the number of white matter hyperintensities on MRI, they should also make sure to communicate with the patient that these findings are nonspecific—they don’t necessarily have anything to do with their ongoing symptoms or cognitive issues and they can occur in healthy individuals. Even in those with 20 or more of these white matter hyperintensities, we don’t see a relationship between them and their outcomes. Sometimes in TBI, patients focus on their brain damage and focus on the injury. We worry that if you give them something concrete that shows this damage in their brain, that will result in iatrogenic effects. We felt it was important to try to minimize this unintended association since it’s not related.
Kimbra Kenney: If the context of these white matter hyperintensities isn’t fully explained to patients, then clinicians may inadvertently mislead their patients by not fully explaining the implications of the findings- both what they do and do not mean. I also think in general it’s important for the whole scientific community that negative studies are published so that multiple research groups don’t launch studies, with the same goals and aims, that have already been thoroughly researched but not published because of the negative results. Particularly with this topic, because the study has such a large number of patients and all of the previous reports have been based on much smaller samples, by widely disseminating and publishing these findings, this will keep the next group from pursuing the same research question.
But it is very difficult to get negative results published. I think that’s partly why we put that spin on it because the clinical implications are significant.
Chantelle Ferland-Beckham: What do you think are the barriers to publishing more findings like those highlighted in this study?
Sara Lippa: I think initiatives like this, that focus on null hypotheses, are fantastic because it is really hard to publish these kind of results. A lot of journals are very competitive, and they have tens or hundreds of submissions for every slot that they have available to publish something and it seems like the null results are less likely to garner enough interest, whether it be from the reviewers or the journal itself. One thing I have noticed with trying to publish negative results is that it’s almost like the reviewers are hesitant as well, like they almost don’t believe that you’ve really looked at everything the right way to make sure there’s not an incorrect analysis or an effect there. It’s occurred to me that the reviewers who review these manuscripts have probably published positive findings in the same area, and so they may be even more hesitant than a different reviewer to accept the manuscript. But there are a lot of legitimate reasons why you might have a null result. You could have done an experiment improperly or had too small a sample, or just your theory was not correct. But there are a lot of times when the null results are really important and it’s a tough situation when it’s hard to get these important results published.
Chantelle Ferland-Beckham: We couldn’t agree more. I am a scientist myself and I think about a lot of data that just never went anywhere. I think everybody has lab notebooks full of things that would be great to have out there so that we’re not duplicating efforts and wasting precious dollars.
Kimbra Kenney: We tend to try to bury negative findings and forget them. You asked the question: “Have you ever published negative or inconclusive findings before?” and I was trying to think of some. John, can you think of some that you’ve published because you’ve published some large series? I know that often negative findings show up in the limitations, e.g. ‘we had a small group effect size, but it can’t be used individually’. To a certain extent, at least for an imaging study, you have to focus on the group effect sizes to get your findings published. I think that the pressure to publish—and to publish positive findings—as well as the overall competitiveness of the academic field are the biggest barriers to publishing negative findings.
John Ollinger: I agree. I think Sara’s point that if you have a negative finding, you have to make sure that you didn’t do something wrong somewhere is important. If you have a positive finding, you don’t necessarily have to do that. So it takes longer to get a paper ready if it’s a negative result, just because there’s so much checking and rechecking that has to go into it.
Kimbra Kenney: I can say we selected this journal to publish because it was TBI and because it was focused on negative results. This was our first submission; we knew that we were going to submit here.