Discerning Robust vs. Flawed Science

In today’s age, we’re constantly bombarded with persuasive headlines promoting a flood of health claims, miracle cures, and the latest research. For patients and healthcare professionals alike, it can be challenging to distinguish between robust, meaningful science, and flawed studies, or even information, that can be misleading or even harmful.

A pivotal moment that reshaped the understanding and treatment of ME/CFS to this day, was when two psychiatrists published an influential opinion piece in 1970. This piece centred on the 1955 Royal Free Hospital outbreak, which had initially been classified as a viral epidemic. The psychiatrists, noting that the illness disproportionately affected female nurses rather than male doctors, argued that the outbreak was not a viral event but rather a case of “mass hysteria.” Crucially, a very tactical, tiny footnote was overlooked and revealed that they had no evidence to back this claim. Despite this, their unsubstantiated opinion initiated a psychological narrative for ME/CFS, fundamentally altering its course. This marked a significant and detrimental shift from the World Health Organisation’s previous classification of ME as a neurological disease, paving the way for flawed theories like “deconditioning” and “illness behavior.” The impact of this single piece of writing, which was NOT even research, was profound, setting the stage for years of misunderstanding and inadequate care for the ME/CFS community. 

These theories led to designated treatments like Cognitive Behavioral Therapy (CBT) and Graded Exercise Therapy (GET), which have since been shown to cause harm to many patients, as evidenced in the recent report Treat ME (July 2025) by the Open Medicine Foundation of responses from 4000 ME/CFS patients¹.

This article, written by ANZMES, the leading National Advisory on ME/CFS, offers the guidance and tools you need to critically evaluate health information, empowering you to make informed decisions based on evidence, NOT headlines.

¹ TREATME: the Open Medicine Foundation’s Mammoth ME/CFS and Long COVID Treatment Survey Results – Health Rising

🔑Understanding the Language of Research

Before we can analyse a study, we need to understand the basic principles that underpin quality research. These terms are crucial for gauging the trustworthiness of a study’s findings.

Key Terms:

• Validity: This refers to the accuracy of a study’s findings. Does it actually measure what it claims to be measuring? For example, a study claiming a therapy, treatment, or drug improves “quality of life” must use methods that genuinely capture that complex concept, rather than just measuring something simpler, like the ability to walk a certain distance. A study that lacks validity produces misleading results. 
• Reliability: This is about consistency. If the same study were repeated, would it produce similar results? A reliable study uses methods that are consistent and repeatable, ensuring the findings are not just a one-off fluke.
• Generalisability: This is the extent to which the findings of a study can be applied to a wider population. For example, a study on long COVID only includes participants under 30, its findings may not be generalisable to the entire long COVID population, which includes all ages. A critical factor for generalisability is the Central Limit Theorem (CLT). As a general rule in statistics, a sample size of more than 30 is often considered sufficient to assume that the sampling distribution is approximately normal, allowing for the generalisation of findings to the broader population. Conversely, sample sizes smaller than 30 are typically not considered robustly generalisable based on this principle.
• Replication: The act of re-conducting an entire study, often by different researchers, to see if the original findings can be reproduced. Although single studies can seem promising, replication is pivotal to strengthen the hypothesis. If a study can’t be replicated when the same methodology is used, it may indicate that the original study was flawed in some way.  However, when results from studies using different scientific methods support each other, this strengthens the evidence base, increasing validity, generalisability and the potential for funding. 
• A Bonferroni correction is a way of making the test for significance much stricter to account for the large number of tests being performed. If you run hundreds of statistical tests, your chance of getting a “significant” result by a random fluke is very high. This correction raises the bar for what counts as a discovery, helping to ensure that a finding is truly meaningful and not just a random statistical blip. Studies that test many hypotheses without these corrections may report false positives. 

❓Questions to Ask of Any Study

Critical analysis isn’t about simply criticising; it’s about conducting a deep, systematic evaluation. Use this checklist to scrutinise the research you encounter.

1. Scrutinise the Source and Motive

Every study is influenced by a reason and a source of funding. It’s crucial to determine any potential motives and biases.

• Who conducted the study and who funded it? Look at the authors’ qualifications and professional affiliations. Are they recognised experts in the field? More importantly, was the study funded by an organisation that stands to profit from a particular outcome? It’s important to distinguish the type of funder. Unlike commercial entities that may profit from a certain finding, patient advocacy organisations (like ANZMES) fund research with the primary motive of patient welfare, not profit. Most importantly, these grants mandate researcher independence, meaning the funder has no influence over the study’s results or publication. Whereas, some individuals may publish studies on a technique that they have commercial interest in – creating not only a conflict of interest, but raising questions about their motives and therefore the conclusions drawn in the study.
• What was the underlying reason for the research? Is the study attempting to answer a genuine scientific question, or could its primary purpose be to generate income, lobby for a specific viewpoint, or persuade a group towards a certain belief system? We need to determine the motive and bias that may be present in the research undertaken. It is concerning when flawed studies are shared amongst healthcare circles via their platforms which can mislead professionals into recommending a therapy to patients without knowing the full risks.

2. Examine the Methodology

The quality of a study is fundamentally tied to how it was conducted. Some research may not benefit from a rigid, “one-size-fits-all” process. A truly evidence-based approach relies on a collective set of scientific principles, not just a “box-ticking” exercise. This requires flexibility and creativity, especially when the patient and their unique experience are central to the research. 

• How was the data collected? Was it through objective, science-based measurements, or subjective methods like self-completed questionnaires and interviews? In complex conditions like ME/CFS, self-reported data is essential to capture the lived experience of core symptoms like pain, fatigue, and cognitive dysfunction. However, robust research strengthens this subjective data by using validated and standardised questionnaires, like the SF-36 (a 36-question health survey that doctors and researchers use to get a snapshot of your overall well-being and quality of life) or the more specific ME/CFS Fatigue Types Questionnaire. These tools are specifically designed to be reliable and consistent, and a high-quality study will often cross-reference patient reports with objective markers where possible to ensure the findings are valid.

For understanding complex conditions like ME/CFS, a precision medicine approach is essential, as it moves beyond a one-size-fits-all model. The “deep dive” approach, exemplified by cases like the JenX recovery story after 18 years of severe ME/CFS, offers invaluable insights by focusing intensely on an individual’s condition, history, and experiences². These “deep-dive” studies on small, carefully selected patient groups can reveal significant findings in specific subgroups that might be overlooked in large-scale research. By focusing on individual patient nuances rather than broad generalisations, this method allows for a more personal and profound understanding of the illness’s complexities, which is often lost when dealing with large, one-dimensional datasets, often seen in fields like cancer studies.

• Who were the participants? How large was the sample size? Were they selected in a way that represents a good cross-section of the group being studied? It is important to distinguish this from some vital ME/CFS research that intentionally uses a small sample size for a highly-individualised ‘deep dive’. Given the complexity of ME/CFS, these studies are often necessary to explore specific mechanisms, like unique biomarker profiles etc. The key feature of high-quality research in this area is that the authors will explicitly state the study’s limitations eg. weaknesses, constraints, or boundaries of a study. For example, the findings may only apply to a specific patient subtype—and will not generalise them to the entire ME/CFS population. The danger arises when these detailed but narrow findings are overstated or used to promote a universal treatment or cause.
• Was the response rate sufficient? It is vital to know how many people started a study versus how many completed it. If a study begins with 20 participants but only reports on the 12 who finished, it has a 40% drop-out rate. This is a significant flaw that can create a falsely positive picture.
• Was there a control group? Without a control group for comparison, it’s impossible to know if an intervention caused the outcome or if participants would have improved anyway. For example, if noted a participants’ illness duration was less than a year, it’s possible these participants could have already been on a recovery pathway. It’s also important to include a group of participants with a longer duration of illness otherwise this weakens any causal conclusions.

² From Severe ME/CFS to Healed: Jen’s Remarkable Rinvoq ME/CFS Recovery Story – Health Rising

3. Evaluate the Data and Conclusions

The final step is to check if the claims stand up to scrutiny.

• Does the data back up the claims? Read past the headline and abstract. Do the results presented in the study actually support the strong conclusions being made, or are the findings overstated? To do this, look for a few key statistical concepts:

Understanding the p-value and Statistical Significance: 

• One of the trickiest but most important numbers in a research paper is the p-value. It refers to the significance of the results, representing the probability or the confidence we can have in the hypotheses. Its job is to help us decide if a finding is a genuine effect or just a random fluke. 
• The easiest way to understand it is to think of a courtroom trial. For example, in a trial, the starting assumption is that the defendant is “innocent until proven guilty.” In research, the starting assumption is called the null hypothesis—it assumes the treatment or intervention has no effect. The prosecutor then presents evidence to challenge the defendant’s innocence. The p-value is like a statistical summary of that evidence. 
• A small p-value means the evidence is very surprising and unlikely to have occurred by chance. In science, a p-value greater than 0.05 means we accept the null hypothesis (no effect) and below 0.05 means we can reject the null hypothesis. Because this is so unlikely, researchers reject the “no effect” assumption and declare the finding statistically significant.
• Statistical significance helps assess whether the results of a study are likely genuine rather than caused by random chance. When a result is deemed statistically significant, it suggests there’s a high probability that the observed effect is due to the treatment. Researchers usually establish a significance threshold in advance—commonly a p-value of 0.05—to determine the level of evidence required to consider the result valid.

• Was the research peer-reviewed? Reputable scientific research is published in journals that use a peer-review process. This means independent experts in the subject area have evaluated the study for quality and validity before it was published, acting as a critical filter.

However, the peer-review process is not infallible and can be subject to human error or reviewer bias. Therefore, even after a study is published in a reputable journal, it is wise to be discerning. Look for post-publication commentary, such as “rapid responses” in the ‘Responses’ tab of the article where other experts and groups may critique or strongly argue the study’s methods, findings, or conclusions. 

• Have they referenced other reputable sources? Good research acknowledges the existing body of knowledge. Be wary of studies that ignore or dismiss contradictory evidence, especially major clinical guidelines from bodies like the National Institute for Health and Care Excellence (NICE), Mayo Clinical Proceedings, and Centre for Disease Control and Prevention (CDC). For example, some researchers may dismiss established guidelines in favour of a single “high-quality” trial. What these researchers may not mention is that such a trial was later found to have many inaccuracies, prompting a correction and clarification. Some ME/CFS ‘treatments’ like Graded Exercise Therapy (GET) have a controversial background, with major clinical guidelines specifically advising against them in response to poor research quality and ethical concerns.

Why is Replication So Important?

Often in ME/CFS research, we see new studies draw the same, or similar, conclusions to those that have been published previously. Replication is crucial for several key reasons below, contributing to the overall integrity and progression of scientific knowledge³.

• Accumulation of knowledge: The results from one study alone are usually not enough to draw firm conclusions about an association. Researchers must gather evidence from several studies leading to the accumulation of knowledge over time to build an evidence base. Think of it like building a structure brick by brick – each study is a brick, and replication ensures the mortar is strong enough to support the entire wall of knowledge.
• Replication using different methods: When results from studies using different scientific methods support each other, this strengthens the evidence for a particular association. This demonstrates that the finding isn’t an artifact of a specific method but a robust, verifiable observation.
• Increased trustworthiness: As evidence grows in support of a particular hypothesis,  especially when the evidence is from different research groups, using different methods and different study populations, the more other researchers, health professionals, and the public come to trust the conclusions drawn.  

The more high-quality research there is suggesting that a particular system is involved in ME/CFS disease mechanisms (eg. immune system) the more likely it is that large funding bodies will invest in research into potential treatments in that area of the disease. A recent example of this is a study by a team of researchers at Cornell University which concluded that “immune dysregulation underlies ME/CFS pathology.” While this conclusion is not ‘new’ knowledge in itself as ME/CFS has been linked with the immune system for years, findings add to the evidence base which supports involvement of the immune system in ME/CFS disease mechanisms. 

It is also important to note, however, when evaluating research, it is crucial to recognise that replication doesn’t always guarantee validity. Sometimes, a flawed study protocol is meticulously followed by other researchers, leading to the repetition of a distorted or inaccurate conclusion. A key example is earlier research on long COVID, where poorly defined patient cohorts resulted in flawed findings that were then replicated in subsequent studies.

 ³ www.meresearch.org.uk/why-replication-of-research-findings-is-important/

Ethics, Bias, and Integrity: Why Ethical Approval Matters

Rigorous research involving humans must undergo an ethical review. This process is designed to:

• Protect Participants: It ensures that participants are not exposed to undue risk or harm and have given fully informed consent.
• Ensure Scientific Integrity: It scrutinises the study design for scientific validity and rigour.
• Limits Bias: It helps to ensure the research is conducted objectively.

Labelling a study as an “audit” can sometimes be a way to bypass this essential ethical oversight, allowing unreliable or lower-quality research to be published. For example, if 12 participants were surveyed but 20 were in the group, this means there was a 40% dropout rate. Ignoring the reasons behind dropout rates, ignores vital information that may show that the technique was not as promising as the audit may lead readers to believe. Missing 40% of the group is significant, diminishing any causal conclusions, especially if not all experiences were reported. 

Bias in Research

It is crucial that medical education relies on the latest research and adheres to the principles of evidence-based research, without personal bias or conflicts of interest. Bias can distort research findings, leading to incorrect conclusions. Here are common types to watch for:

• Responder Bias (or Volunteer Bias): It’s important to consider those who volunteer for the study may be different from those who don’t. Responder bias is the tendency for people in a study to provide inaccurate answers, often unconsciously. This can happen for various reasons, such as trying to give the answers they think the researcher wants. As a result, this bias can distort the data, making the study’s conclusions an unreliable reflection of people’s true thoughts or behaviours. 
• Selection Bias: This occurs when the participants are not chosen randomly. For example, a researcher might consciously or unconsciously select participants they believe will respond well to the treatment, creating a biased sample.
• Confirmation Bias: This is the tendency for researchers to favour, interpret, and recall information that confirms their pre-existing beliefs or hypotheses. Naturally, we can seek and interpret information that confirms existing beliefs and ignore contrary evidence for example, only following news sources you agree with. This tendency can lead to researchers downplaying negative results or over-emphasising positive ones. When researchers, or those promoting specific interventions, only cite evidence that supports their existing viewpoint and disregard contradictory findings, it exemplifies confirmation bias in action, hindering a truly objective assessment of the evidence.  A good researcher should actively seek opposing viewpoints, and assign a ‘devil’s advocate’ before any decisions.
  • Another example, some studies have been criticised for ‘outcome swapping’ where the primary measure for success was changed partway through, for example from an objective measure like school or work attendance to a subjective self-report questionnaire. If a treatment/therapy overtly encourages participants to only report positive outcomes, this is likely to lead to confirmation bias. 
• Multiple Testing Bias: In complex illnesses like ME/CFS and long COVID, researchers often test hundreds or thousands of different variables (e.g., biomarkers) at once. When so many tests are performed, it becomes statistically likely that some will appear “significant” purely by chance. To counteract this, researchers must use statistical corrections – for example, a Bonferroni correction (see key terms), to adjust their p-values.

Putting It All Together

🚩What Flawed Science Looks Like

Flawed science often shares common characteristics. Be sceptical when you see studies that:

• Avoid ethical review by labelling themselves as an “audit.”
• Have a high, unexplained drop-out rate.
• Lack a control group, making it impossible to determine cause and effect.
• Use small, non-representative samples but make broad generalisations.
• Report “significant” findings from testing many variables without correcting for multiple comparisons.
• Rely on vague or self-reported data without objective measurement.
• Ignore or dismiss major clinical guidelines and high-quality contradictory evidence.

An audit based on a flawed study can be seriously misleading to healthcare clinicians. Patients with complex chronic illnesses deserve evidence-based healthcare, and professional bodies have a duty to safeguard people from the biased promotion of interventions or treatment approaches that are unproven and do not have a scientific basis. It is the responsibility of researchers to conduct high-quality, ethical studies and the duty of healthcare professionals to critically appraise the evidence before recommending treatments. A medication would be held to much more stringent safety standards. We must demand the same for all interventions.

👍What Good Research Looks Like

In contrast, high-quality science is built on a foundation of rigour, transparency, and respect for evidence. Good research, particularly in ME/CFS, typically:

• Is Ethical and Transparent: It undergoes a formal ethical review, is transparent about its funding sources, and clearly reports and explains participant drop-out rates.
• Uses Precise Definitions: This is especially critical in ME/CFS research. Good science uses specific diagnostic criteria that require the hallmark symptom of Post-Exertional Malaise (PEM) to ensure they are studying a consistent patient group. This stands in contrast to the historical use of flawed criteria like the 1994 Fukuda definition, which did not require PEM and allowed for the inclusion of patients with different forms of chronic fatigue. This methodological flaw fueled the conflict between the biomedical and psychosocial models of the illness; by creating mixed study groups where biological signals were diluted, it created an opening for psychological theories to take hold. The correct criteria for ME/CFS Research should be in accordance with Canadian Consensus Criteria (CCC).

*If recruiting through medical clinics, Institute of Medicine (IOM) Criteria 2015 or International Consensus Criteria (ICC)  is used by clinicians for diagnosis. 

• Is Methodologically Robust and Acknowledges Limits: It employs appropriate control groups, uses sample sizes large enough to produce statistically meaningful results, or studies individual patients in depth according to the principles of Precision medicine, and applies necessary statistical corrections when testing multiple variables. When navigating the body of ME/CFS research, you will often see studies with small sample sizes due to being highly individualised. Good science acknowledges these limitations explicitly and avoids making broad generalisations about findings beyond what the data support, particularly in smaller or exploratory studies like some ME/CFS research.

It’s important to consider, specifically for ME/CFS research, a strong argument can be made that while large cohort studies are valuable, they should not overshadow the profound insights gained from longitudinal studies on individual patients or small, carefully selected groups. These studies are crucial for uncovering significant, reproducible changes and can yield consistent conclusions when different technologies are applied, providing a deeper understanding of the illness.

• Integrates Objective Data:  While valuing patient-reported symptoms, strong research seeks to validate these experiences with objective, measurable biological data whenever possible. Given the importance of patient experience, you will see a heavy reliance on self-reported data in ME/CFS research. Good science strengthens this by using validated and standardised questionnaires to ensure the data is reliable and consistent.
• Avoids Confirmation Bias: Good research actively seeks out and considers all relevant evidence, including findings that may contradict initial hypotheses, rather than selectively citing only supporting information.

Summary

Some research can fail to meet scientific standards, misrepresent a therapy’s risks, and reinforce all the criticisms of previous weak studies. While non-harmful elements of a programme may deserve a place in treatment (and are often already widely available, in free or low-cost formats) and some participants do improve (although it’s unclear if they recover), the overall approach must be scrutinised. By asking the right questions and demanding robust evidence, we can all contribute to a healthcare landscape built on a foundation of ethical and trustworthy science.

🧰Your Critical Thinking Toolkit

As a take-home, exercise your critical thinking skills by asking these questions next time you read or discuss new information:

• Who stands to benefit from this information? Who is most directly affected, and who would be the best person to consult for an alternative perspective?
• What are the strengths and weaknesses of the study’s claims? Is there a counter-argument to the conclusions presented?
• Where would we see this in the real world? Are there similar concepts or situations that either support or contradict these findings?
• When is this information acceptable or unacceptable? What is the best time to take action based on this research?
• Why is this relevant to you, and what is the underlying challenge or problem this research aims to address? Is there a need for this information today?
• How is this similar to or different from other information you’ve encountered? How does this information affect you or others, and how can you approach it safely?

The core message is this: shared decision-making is not just a best practice, it is a critical component of effective and ethical care, particularly for patients with complex, chronic conditions like ME/CFS. Patients who are well-informed and actively involved in their treatment have better outcomes and are more likely to adhere to their management plan. This is especially true for those with ME/CFS, a condition that has been historically misunderstood. By empowering patients (and their carers, when the illness is severe) with robust, evidence-based information, you are enabling them to become partners in their care. This collaborative approach, where the patient retains personal control over their health decisions, is essential for building trust and ensuring the management plan respects their unique needs and lived experience.