Consider whether the people being compared were similar.
For a comparison to be reliable, the health actions being compared should be the only important difference between the comparison groups, at the start of the comparison. In other words, to begin with, the comparison groups should be similar overall. Randomly creating groups is the only way to avoid unknown differences affecting the results of a comparison.
Explanation
When comparing health actions, the only difference that researchers want to find is the difference in the effects of those actions. For example, if they are comparing a new treatment to an old treatment, they only want to find the difference in the effects of taking those treatments. They do not want to find the effect of anything else, such as how sick someone was when they or their doctor decided which treatment to use.
If there are important differences between the groups at the start of the comparison, apart from the health actions they will take, it is difficult to know how much of what happens in the groups is caused by the health actions being compared and how much is because of other differences. This can make the effect of a health action seem larger or smaller than it really is. It can even make a harmful action seem helpful, or a helpful action seem harmful.
There is only one way to be sure that the groups are similar overall: randomly creating them. When researchers randomly create groups, nobody decides who takes which health action – not the researchers, not the people in the groups, or anyone else. Researchers often use lists of random numbers, usually generated by computer programmes, to randomly allocate participants to comparison groups. Researchers call this process “random allocation”. They call comparisons with randomly created groups “randomized studies”, “randomized trials”, “randomized controlled trials” or “RCTs”.
Random allocation is most reliable when the allocation is ‘concealed’ (so that no one knows beforehand which participants will be allocated to which health action). If the comparison is too small, it may not help to have randomly created groups. For randomly created groups to be similar, the groups should be large enough for researchers to be sure that there are not important differences between the comparison groups due to chance alone.
Nonrandomized studies, where someone decides who takes which health action, can check that the comparison groups were similar with respect to things that are known to affect outcomes, if those things were measured. But they cannot check that the comparison groups were similar with respect to things that affect outcomes but are not known to do so, or things that were not measured.
Example
For many years, people believed that hormone replacement therapy (HRT) prevented women from having heart disease. The belief was based on comparisons made by researchers. However, the groups in the comparisons were not randomly created; the women had decided themselves whether they used HRT or not. It later became clear that the comparison groups were not similar.
When researchers made comparisons with randomly created groups, they found that HRT actually increased the chance of having heart disease. It turned out that there had been an important difference between the groups in the original comparisons other than use of HRT. The women who chose to use HRT were better-off (of higher socio-economic status) than the women who did not use HRT. Because women who are less well-off (of lower socio-economic status) are more likely to have heart disease, and they were less likely to use HRT, the difference in heart disease that was attributed to HRT was probably because of the difference in socio-economic status, not because of HRT.
HRT is still recommended for preventing hot flashes in some women going through menopause, since it does in fact prevent hot flashes and it is safe for most women. However, it is no longer recommended for preventing heart disease, and it is recommended that women who have had a heart attack, a stroke, or a blood clot (as well as women who have had breast cancer) should not use HRT.
Remember: Be careful about relying on the results of non-randomized comparisons of things people do for their health (health actions). Ask if there were important differences between the comparison groups that might have affected the results.
Primary school
- Lesson 6. Fair comparisons of treatments. In: The Health Choices Book.
- Animation: Fair comparisons of treatments – The Health Choice Book.
Secondary school
- Lesson 6. Randomly-created groups. In: Be smart about your health.
Other
- Podcast: Ep. 7 of The Health Choices Programme: “Fair comparisons”
- Podcast: Episode 2: Fair comparisons. Parents Making Informed Health Choices.
- Video: What is a randomized trial? This is a Cancer Research UK video lasting 1:54 minutes.
- Video: Not all scientific studies are created equal This is a TEDEd video lasting 4:27 minutes.
- Video: Placebo effect, control groups, and the double blind experiment This is a Simple Learning Pro video lasting 3:35 minutes.
- Video: Fair comparisons and randomization This is a British Pharmacological Society / Keele University video lasting 4:02 minutes.
- Blog: Comparison groups should be similar. Students 4 Best Evidence.
- Essay: The need to compare like-with-like in treatment comparisons. The James Lind Library.