Consider whether important assumptions were tested.
Sometimes claims about the effects of health actions depend on putting together different types of evidence and making assumptions (modelling). Studies that use modelling can provide valuable information about the effects of health actions, but they can be misleading.
Explanation
When evidence is lacking or there is extreme uncertainty, models can be helpful. Modelling studies combine information from a variety of sources and depend on assumptions to compare health actions. For example, a comparison of using different diagnostic tests (e.g., two different tests for detecting cervical cancer) might require an assumption about what actions people will take based on the test results (e.g., how women with a ‘positive’ test will be treated). If it is uncertain what action people will take, it is important to consider how changing that assumption might affect the results of the comparison.
Expert judgement is often used when there is limited or conflicting evidence about something included in a model. However, it is often not clear how expert judgements were obtained, making it difficult to assess the reliability of those judgements and the findings of the modelling studies depending on them. In addition, expert judgements may be misleading due to the experts being biased or due to overconfidence.
The quality of models can be variable, and their results can be misleading. Overall, the certainty of models corresponds to the least certain evidence included in the model.
Example
Before and during the Covid-19 pandemic, there were not many randomized trials of most of the public health measures used to control the spread of infections, such as school closures. As a result, estimates of the effects of those actions were often based on models and non-randomized studies. The modelling studies often suggested different effects. For example, some modelling studies suggested that school closures could reduce transmission of the coronavirus, while others disagreed. These models depended on many assumptions (e.g., about how many parents went to work or worked at home when schools were closed or opened, what other protective measures were in place, and what proportion of infected people have symptoms) and changes in these assumptions could change the results. Because there were many assumptions and there was important uncertainty about many of them, the results of these modelling studies were very uncertain.
Remember: Whenever claims about the effects of health actions depend on assumptions, it is important to consider the basis, limitations, and uncertainty of the assumptions that are made and to test if the results would be different if there were changes in the assumptions.
- Video: Counterfactuals: Causal Inference Bootcamp. This is a Duke University video lasting 4:52 minutes.
- Video: The Credibility of the Unconfoundedness Assumption: Causal Inference Bootcamp. This is a Duke University video lasting 3:01 minutes.