Start with the design
Find out whether the study was an experiment or an observation. Random assignment offers stronger grounds for claims about cause. A correlation, however interesting, may reflect a third factor or a cause running in the opposite direction. Also ask who was studied, because many samples are students from a few wealthy countries.
Effect size and statistics
A statistically significant result is not necessarily important. Look at the size of the effect and the range of uncertainty around it. Small samples produce unstable estimates, and flexible choices about analysis can make false positives far more likely, particularly when those choices are not disclosed.
Replication
A large project that attempted to repeat 100 published psychology studies reproduced a considerably smaller share of the original significant results than expected. Since then, preregistration and large multi-lab projects have become more common. Give more weight to repeated, preregistered findings and treat dramatic results from a single small study with care.
Further reading
- Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716.
- Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366.