Critically Reading Scientific Papers

Scientific papers are not perfect. Even though they may be published in reputable journals, peer-reviewed or written by respectable authors, they are usually always subject to limitations. Reading scientific papers critically takes some time to develop and it definitely didn’t come to me straightaway, but practice makes perfect! It’s a key research skill every researcher and academic should enhance. Throughout my undergrad, Masters and now PhD I’ve read my fair share of scientific papers and feel I have picked up some useful pointers which I aim to share.

Validity of the paper

Firstly, check if the article is actually published in a journal or if it is a pre-print. Pre-print articles are ones that have not undergone peer-review or been published in a journal. Their purpose is to disseminate findings quickly to the public to allow for open sharing of research, while they still undergo peer-review. This is great as research should be shared quickly and open to the public, but these articles have not gone through peer-review yet so should be read with caution. If it is not a pre-print, check the journal the paper is published in. Is it a legitimate journal? Do they undergo peer-review processes for all their articles?

Check the authors of the paper and see what background they have in the research field and their credentials. This has an impact on the quality of the information and how trustworthy the author is. Has the author published other work in this field? Can you contact the author if you wanted to?

The date of publication and when the actual study was carried out is also important as it speaks to the validity of the information now. There are many studies published a very long time which still carry accurate information in today’s age, but science is always developing and new information is always being published. Therefore, when reading very old papers be cautious that the results and conclusions, which may have been correct in that time, may not be suitable now and there might be more up-to-date information. Nevertheless, don’t disregard old papers as the evolution of information is also an important aspect to consider.

Hypothesis

This is something I have seen missing in many scientific papers but is the basis of scientific research. What is the actual hypothesis they are testing? This should be clearly pre-defined in the paper as the validity of results and experiment rely on a robust testable hypothesis. The hypothesis should always be stated at the start of the study and data collection should be driven by this hypothesis. Unfortunately, in many cases researchers do this the other way round by conducting many experiments and statistics and seeing which answers are most interesting to form a hypothesis from that. Multiple statistical testing of associations is likely to bring up one that is statistically significant and some researchers take advantage of this to form their hypothesis after to publish their research. Definitely keep your eyes out, usually at the end of the introduction/background section, for a clear formulated hypothesis!

Study design

The study design is important to consider as it informs on whether the research questions are adequately answered using the correct methods. It also allows you to focus in what biases you should be looking out for. Each of these study designs are used to answer particular types of research questions. Therefore, check what study design the paper uses and determine whether it is appropriate for the research question. Is the paper investigating the effectiveness of an intervention? In which case an RCT should be used. Is the study trying to identify risk factors for a disease in which case a cohort or case-control are most ideal, but cross-sectional can be used, however, it has its limitations such as temporality, which you must consider.

Sample size

A minimum sample size should be pre-determined in the study by carrying out power calculations. Check whether these calculations have been done and whether the actual sample size meets or exceeds that number. If the sample size is too small and does not meet this number then the study will not have enough power to detect a difference when it exists. Power should be around 80-90%. However, if the sample size is larger than necessary it is, firstly, wasting resources when an accurate result can be found with a smaller sample size.

Secondly, it is unethical if more participants are being used when there is no real need. Thirdly, in the case of RCTs you are subjecting more people to the placebo and denying them better regimen. Also, a very large sample size is at risk of producing a statistically significant results due to the sheer power of the study when there actually isn’t a significant difference.

Sampling method

There are many types of sampling methods each with the their own limitations which you should consider when reading a paper. The most ideal method is random sampling as this adjusts for baseline differences in the samples and also removes selection bias. Other common forms of sampling are convenience, purposive, snowball, stratified and clustered sampling which are all non-random (and of course lots of others so if you do come across other sampling methods definitely look it up to explore the uses and limitations). Each of them have their own reasons so don’t judge it straightaway as there may be a good reason to use it. However, keep in mind the level of involvement of researchers have in choosing participants and how that can bias the sample and the results.

Ascertainment of outcome/exposure

How the study measures exposures and outcomes is important as it can lead to many types of bias. Check who is measuring the variables; is it someone who knows exactly what the research is about in which case there may be some level of interviewer/researcher bias as they elicit certain responses. RCTs should be double blind so neither the participants nor the researcher carrying out the analysis know which group the participants are in, again this reduces researcher bias.

Recall bias occurs when self-reported measurements are used as outcomes may affect the way way participants recall certain things. Hence, objective measurement tools should be used such as medical records where possible. However, this is not always possible so you should just keep in mind these limitations of the study. Also, reporting bias can occur when participants know the research outcomes and hypothesis and give answers they think will be of interest. They may also not want to reveal certain information if there it is particularly sensitive or stigmatised. Check if the study tries to eliminate these biases through confidentiality protocols, not giving away too much of the research, only what is ethically necessary and training the researchers in how to measure these outcomes/exposures etc.

The study should clearly define all the variables from the start to eliminate misclassification bias of either outcome or exposure. This is when participants are put into the group as the outcome or exposure has not been measures accurately.

Ethics

Research ethics is a huge topic which is impossible to cover in this post. Some key things you should look out for with regards to ethics is has the study been approved by an ethics committee. Informed consent should be obtained from each participant, ideally written, but in cases where this is not possible, verbal consent with a witness should be obtained. Participants should be informed of what the research is about, the benefits and risks and should be told they can leave the study at any time. The study should have methods of maintaining confidentiality and anonymity where necessary.

The study design also impacts the ethical aspects we should be looking out for. For example in RCT’s you need to check the treatment of both intervention and control group, the sample size, denied access to essential treatment etc. Ethics in research is a huge topic which I will a dedicate a separate post to in the future as it is key for researchers and academics to understand.

Statistical analysis

The paper should have a clear paragraph in the methods section outlining which statistical tests will be performed, what significance level, critical value and confidence interval will be used and what associations will be tested. Check the results section to see it matches with what they said in the methods section or have they done extra tests they didn’t mention before? If the paper carries out multiple comparisons some will end up being significant purely out of chance, therefore, the study should determine which comparisons will be made from the start and not carry out more. If multiple comparisons are to be made, check if they the study has carried out a test which accounts for multiple comparisons such as bonferroni correction.

Check if the statistical tests used are appropriate for the research question, variables and and type of data. I have linked a useful flowchart which can be used to determine which statistical test is most appropriate for the study. You need to look out for things like is the data discrete, continuous, categorical etc? Is it paired or unpaired data? Is the data parametric or non-parametric? The study should highlight this information if it is a high quality paper.

Ideally, multivariate analysis should be carried out especially if it is not an RCT as multivariate analysis can adjust for confounding factors. Without this you need to interpret the results with caution. Finally, remember that statistical tests only inform on associations and not causation. The paper should not over emphasise non-significant results or state causation.

Funding

Usually at the end of the paper there will be small section detailing funding for the study. This is important as funding received from a particular organisation may lead to funding bias. This is when the results and conclusions of the paper are made to align with the interests of the organisation that provided the money for the study. This is not always the case, of course, but it is something to be cautious of.

That was a whistle stop tour of some of the things I look out for when reading scientific papers. By no means is this comprehensive, but it gives a good starting point. The more you read, the more important factors you will begin to notice. Practice makes perfect!