How to engage in responsible, not questionable, research practices
Questionable research practices (QRPs) are research misbehaviors that threaten scientific integrity. In contrast, responsible research practices (or RRPs) refer to a set of research behaviors and ways of working that promote reproducible and sound science. This articles examines both types of practices and shows you how to make sure you’re engaging in RRPs rather than QRPs.
What are Questionable Research Practices?
Questionable research practices (QRPs) are behaviors that threaten scientific integrity. They include practices such as as only reporting certain outcomes, but may not necessarily include outright misconduct such as falsifying data. While not as severe as misconduct, QRPs can still undermine the credibility and reliability of scientific research and should be avoided at all costs.
The main reason scientists can engage in QRPs is the pressure to publish. In today’s “publish or perish” standard model, many scientists feel they need to continuously publish new or significant findings in order to obtain more funding and secure employment. To do so, they engage in QRPs and ironically compromise the foundation of rigor and reliability the scientific community is based on.
John, Lowenstein and Prelec (2012) examined the use of QPRs by surveying over 2,000 psychologists. They found a 60% engagement rate in practices such as selectively reporting studies that “worked,” not reporting all dependent measures, collecting more data after seeing if results were significant, and excluding certain data points after examining their impact. These findings are shocking and suggest that QRPs are rampant in research. The below list covers the most common QRPs.
P-hacking
P-value hacking, or p-hacking involves manipulating or tweaking data analysis until the desired, usually statistically significant result, is obtained. This practice may include adjusting variables, removing outliers, or trying various statistical tests until a p-value falls below a predetermined threshold for significance. The dangers of p-hacking are that it inflates the risk of false-positive findings and spurious correlations.
Imagine you are investigating the impact of a new drug on depression. You conduct multiple statistical tests to analyze various aspects of the drug's effects and find that one test that shows a marginally statistically significant result (p-value of 0.034). However, you had initially set a significance threshold of p < 0.01. To make the result appear significant, you continue to tweak the analysis by changing the statistical method or excluding some data points until the p-value reaches 0.01.
By doing this, you artificially inflate the significance of their findings, potentially leading to the wrong conclusions about the drug's efficacy and safety. This can be dangerous for many reasons, including lack of reproducibility or reliability of findings, contributing to the replication crisis in science, and potentially medical or public health policies that are based on flawed science.
Selective Reporting
Selective reporting, also known as “cherry-picking,” involves the deliberate choice to publish only positive or statistically significant results while failing to include negative or inconclusive findings. This practice can occur when researchers see that data for one condition or experiment is significant and only report that one as opposed to reporting their full set of analyses for all conditions or experiments.
For example, imagine you're conducting a study to determine if a specific drug has a positive effect on patients' health. You collect various health indicators, such as blood pressure, heart rate, and cholesterol levels. After analyzing the data, you notice that only one health indicator, blood pressure, shows a statistically significant improvement with the drug. You then choose to report only the positive results for blood pressure while ignoring the outcomes of other health indicators that didn't show any significant changes.
This selective reporting leads readers to believe that the drug has a more substantial and positive impact on patient health than it actually does, creating a skewed and incomplete picture of the drug's effectiveness.
The dangers of selective reporting extend beyond skewing the body of evidence—it also wastes resources and misleads other researchers. Selective reporting can also have far-reaching consequences in fields such as medicine, where treatment decisions may be based on incomplete or biased information.
Hypothesizing after the results are known (HARK-ing)
HARK-ing is a practice where researchers formulate hypotheses after examining the data, rather than before conducting experiments. This can happen when researcher see their data does not match their hypothesis, so they change it to align with the results. This practice can also involve not reporting hypotheses if they don’t fit the data, or incorporating other hypotheses from the literature as if they were part of their current ones.
This practice blurs the line between confirmatory and exploratory research, making it difficult to falsify certain hypotheses. HARK-ing also exacerbates the ongoing replication crisis in many fields because tailored hypotheses are challenging to generalize or replicate across different settings.
HARK-ing can lead to a lack of rigor and transparency, as researchers may unconsciously or consciously mold their hypotheses to fit the observed results. The consequences include unreliable findings, diminished reproducibility. HARK-ing undermines the core principles of hypothesis testing, hindering the pursuit of genuine scientific knowledge and discovery.
What are Responsible Research Practices?
Responsible research practices (RRPs) have evolved in response to QRPs and encompass a set of ethical and methodological behaviors that researchers adhere to during research. They ensure the integrity, credibility, and reliability of research outcomes. By setting clear standards and ethical boundaries, RRPs help build public trust, facilitate collaboration, and improve the quality and impact of research (for a review, see Schwab et al., 2022).
Ethical Considerations:
Ethics are paramount in any research environment. RRPs demand that researchers prioritize the welfare of human subjects, animals, and the environment when conducting research. Adhering to ethical guidelines ensures that research is conducted with respect, empathy, and consideration for potential consequences.
Every field of research typically has their own set of ethical guidelines and standards that researchers need to follow. For example, the American Psychological Association has a rigorous set of guidelines for conducting animal research and a code of conduct for psychological research. Universities and research institutions will also have their own set of ethical guidelines in place for people to follow. Always ensure you are complying with ethical standards to promote fair research.
Transparency, Reproducibility, and Openness
Transparency relates to a full disclosure of the methods, data, and findings used in research. Reproducibility is the ability for others to replicate your results using the same procedures, data and analyses. And, openness involves the active sharing of research data, methodologies, and results with the public and scientific community. All three are all part of the open science movement that aims to increase reliability, replicability and openness.
One way to enhance transparency and reproducibility in research is to pre-register your experiment before you conduct it. This is a process where you list in detail your data collection and analysis plan before you start collecting data and post it on a repository such as Open Science Framework. Another way is through registered reports, which are a type of pre-registration where your article is initially accepted for publication on the basis of the background and methodology rather than on the results section.
Both pre-registration and registered reports are vital for increasing transparency in how research is conducted and reducing the likelihood of questionable research practices, as they prevent the selective reporting of results and minimize the impact of publication bias.
Open access is another way to enhance transparency by removing barriers to accessing research findings, allowing a broader audience to have access to research. It fosters greater visibility and accessibility of research outputs, enabling more comprehensive peer review and facilitating the detection of errors or misconduct. Additionally, open access encourages data sharing and collaboration, fostering a culture of transparency and accountability within the scientific community.
Data Integrity and Management
When working in research, scientists often have access to personal or sensitive information depending on the types of studies they conduct. This could include confidential medical records, identifiable genetic data, or private survey responses. It is crucial to have robust data protection measures in place to ensure the privacy and confidentiality of individuals involved in the research.
Data integrity and management is a key RRP which requires that data is collected, stored, shared, and managed in a secure and compliant way at all stages of the research cycle. Properly managing data is critical to prevent data manipulation, fabrication, or falsification. Typically, your university or lab will have rules in place about what you can do with your data.
One way to ensure this is to create a data management plan. A data management plan is a structured outline that details how data will be handled during a research project, encompassing aspects such as data collection, storage, sharing, and preservation. It is crucial as it ensures the integrity, accessibility, and long-term usability of research data, promoting transparency, reproducibility, and compliance with ethical and legal standards (see here for an example of a data management plan).
Responsible Authorship
Responsible authorship is acknowledging all contributors that have substantially contributed to the research. Responsible authorship is important because articles need to accurately reflect authors’ contribution to promote transparency and accountability among researchers and give credit where credit is due.
Researchers can sometimes add authors to publications just because they are prominent researchers or they think their inclusion will positively impact publication outcomes, which is unethical. In addition, author disputes can often arise due to a lack of accountability and contributor roles from the beginning, which decreases the spirit of collaboration in research.
Responsible authorship includes
Avoiding guest authorship, or adding authors who didn't contribute significantly
Avoiding ghost authorship, or excluding those who did contribute
Another way to foster responsible authorship is to use the CRedIT authorship statement, which credits what each author contributed to the publication. This helps all authors involved be aware of their role and contribution to the publication and gives the appropriate credit for each involvement.
Good Mentorship
Good mentorship is critical in research to train and inspire the next generation of researchers. However, busy professors and stressed out graduate students often don’t have enough time to mentor young researchers, leading to many aspiring researchers adopting the wrong methodologies or failing to develop critical skills needed for science. What’s worse, is many established researchers may not know good research practices themselves, leading to a vicious cycle of perpetuating poor practices. Remember that scientists are created, not born, and a good mentor can make all the difference.
Being a good mentor means spending some time regularly to help young researchers and organizing helpful events to teach them good research methods. Giving them useful advice and chances to learn more are also important. Building a friendly and supportive relationship with open communication is the key to helping them become successful scientists.
Constructive Peer Review
Peer review is the main way scientific research gets disseminated, so ensuring this process is fair and constructive is key for science. When peers review each other's research, they help spot any mistakes, ensure the research methods are sound, and confirm that the findings are trustworthy. However, this process can sometimes go awry with biases in peer review, not disclosing conflicts of interest, or even overlooking major flaws.
One way to ensure peer review is fair and constructive is to establish clear guidelines and criteria for reviewers, ensuring that the assessment is based on the merit of the research rather than personal biases. Implementing a double-blind review process where the identities of both the authors and reviewers are concealed can mitigate the impact of potential conflicts of interest and unconscious biases. Encouraging constructive and detailed feedback, fostering open communication between authors and reviewers, and promoting a culture of transparency and accountability within the peer review system can further enhance the integrity and effectiveness of this critical process.
Takeaways
QRPs, such as selective reporting and p-hacking, can undermine the credibility and reliability of scientific findings, leading to inaccurate or misleading conclusions. RRPs, including transparency and reproducibility, are essential for ensuring the reliability and integrity of scientific research, fostering trust and confidence in the scientific community. Upholding these principles is a key responsibility of anyone with an interest in science, from young students to seasoned professors.
