Facebook Twitter

Credibility and Bias

Credibility and Bias

A lot of people seem to be confused about something called “research.” So they sit down and type into Google, “tell me what I want to hear.” Then they read one or two of the top results that confirm what they already believed.

What you’re engaging in is called confirmation bias. You’re only looking for information consistent with your beliefs or desires while ignoring anything or anyone that disagrees. It seems that most people aren't interested in finding the truth. They want to be told that what they already believe is true. That’s not how research works.

Google (and most search engines) are designed to provide information you want to read. For example, if you believe the earth is flat, it’s pretty easy to search “flat earth proof,” and you get dozens of websites that say we’re living on a giant disc, or a donut, or on a plate supported by elephants and a giant turtle.

What those websites do is make arguments, sprinkled with half-truths and scientific fallacies, to try and convince you of a lie. If you go no further, it’s easy to end up believing something that’s not true.

The first problem is the source. That Facebook meme, Tik Tok video, or Reddit discussion thread, is not as valid as a double-blind placebo-controlled study published in a reputable peer-reviewed scientific journal.

They are both opinions, but the clinical study is backed up with verifiable data. It’s put together into a paper that has been reviewed, critiqued and corrected by experts in the field. Everything has been documented so other researchers can reproduce the results.

Your Facebook gif does not have the same weight of proof. It shouldn’t be a badge of honor to show ignorance or share ignorance. Instead, it’s an embarrassment that can hurt or kill people.

Doing real research is challenging. In 2003, a client asked me a simple question. “Is it true I should drink eight glasses of water a day?” A quick search online showed hundreds of health and fitness websites repeating the claim, but none of them could document why.

It took my team and me five years, reading over 240 clinical studies, wading through more than 1,800 research citations and interviewing more than two-dozen medical professionals to get a proper answer.

In March of 2008, doctors Dan Negoianu and Stanley Goldfarb released a report titled“Just Add Water,” where they looked into the eight glasses a day idea along with several others floating around the internet. What did they conclude?

There is no clear evidence of benefit from drinking increased amounts of water. Although we wish we could demolish all of the urban myths found on the internet regarding the benefits of supplemental water ingestion, we concede there is also no clear evidence of lack of benefit. In fact, there is simply a lack of evidence in general.”

Dr. Goldfarb said, “A little mild dehydration for the most part is OK, and a little mild water excess for the most part is OK. It’s the extremes that one needs to avoid...”

That was the answer. Unless you’re drinking a prescribed amount of fluid under doctor’s orders, you should let your thirst be your guide. If you’re thirsty, drink more. When you’re not, stop. It’s that simple. It wasn’t the neat little answer most people had grown up accepting, but it was the right answer, backed by research.

If you’re trying to get the facts, you need to research with reputable sources. Here are three.

For rumors, memes, or conspiracies, a great resource is snopes.com. This is what they do in their own words.

When misinformation obscures the truth and readers don’t know what to trust, Snopes’ fact-checking and original, investigative reporting lights the way to evidence-based and contextualized analysis. We always link to and document our sources so readers are empowered to do independent research and make up their own minds.”

For clinical studies on drugs, supplements, medical procedures, or treatments, your first stop should be pubmed.gov. I like to keep a dictionary handy to look up some of the big words, but it’s worth it. Here’s what they provide. `

PubMed® comprises more than 30 million citations for biomedical literature from MEDLINE, life science journals, and online books. Citations may include links to full-text content from PubMed Central and publisher web sites.”

When those aren’t enough, consider expanding your search to Google Scholar at scholar.google.com.

As of January 2021, Google Scholar looks through more than 160 million sources. Google searches articles, theses, books, abstracts and court opinions.

Once you get the studies, you’re not done yet. You still have to look at the information critically. Here are the questions you should ask.

What are researchers trying to accomplish?

Read the introduction first to learn the researcher's intent and motivation. Figure out what they're trying to uncover. Are they asking the same things you are? If they are, then read through the rest of the paper. See if the way they've set up and conducted the experiment can adequately answer those questions.

Read the abstract only after you've finished everything else. It contains a neat summary of the entire paper, but it's subject to the researchers' biases. If you read it first, it might inadvertently bias you.

How big was the study?

Testing something on 10 people won't give you as much information as testing on 1,000 people. The more subjects that are involved, the more trustworthy the findings are.

How long was the study?

You need to test something long enough for the real effects to come out. When multivitamins were tested, most companies, if they tested at all, didn't go much beyond 6-12 weeks. But when 160,000 people were tracked for more than 10 years, researchers found that Multivitamins taken daily may be dangerous to your health. People who took multivitamins died sooner than those who didn't. But it took years to discover the problem.

Who paid for the study?

In 2015, as sales for sugary beverages dropped, Coca-Cola decided the best way to hang onto customers was to shift the blame. Coca-Cola gave over 1.5 million dollars to a new nonprofit called Global Energy Balance Network. The scientists at that nonprofit were hired to argue that Americans are too overly fixated on food and that we need to exercise more.

Coca-Cola wanted to make it look like independent scientists thought their sugary drinks weren't a problem. Their funding changed the tone of the research from that organization and tainted the results. Be wary of companies trying to push their agenda.

Is it a single study or a meta-analysis?

In a single study, researchers define the experiment's scope, document what's happening and use the data to come to conclusions. A meta-analysis is a study that looks in-depth at dozens of OTHER studies. They go through the scientific literature and find the most relevant data, then put it together to see the results.

For years we've been told organic produce contains more vitamins and nutrients than non-organic varieties. Several small studies seemed to support that conclusion. That was, until a team at Stanford's Center for Health Policy did the most comprehensive meta-analysis to date of existing studies comparing organic and conventional foods.

The researchers found that organic foods did not seem to carry fewer health risks than conventional foods. They also found that there was little evidence organic foods were any more nutritious than conventional foods. While organic foods were better for the planet, they didn't provide significant benefits for people eating them.

Is the study retrospective or prospective?

A retrospective study asks subjects what they've done in the past. Things like how many times you ate, drank, or exercised. They're less expensive because they just have to ask a bunch of questions and tabulate the results. They're also far less reliable because they rely on the subject's memory. Most people can't remember everything they ate a week ago, never mind two years ago.

In a prospective study, researchers follow the subjects over time. There are multiple meetings, with far more extensive record-keeping to verify what people are doing. Instead of putting everything together over a couple of interviews, a prospective study can take years before the results are known. These are much more reliable but not done as often because of time and money limitations.

Other Study Types

A Cohort Study is typically done over a long period of time (a longitudinal study). Many take years to complete. Researchers follow participants that share a common characteristic, like a job or place where they live. Then they divide the participants by those exposed to a specific risk factor, such as smokers and non-smokers. Cohort studies are particularly valuable in epidemiology, to help researchers understand what factors increase or decrease the likelihood of developing disease.

Case-Control Studies are a type of observational study. Two groups of people who have different outcomes are identified and compared. Researchers look through historical factors for the underlying cause that led to the different results. If exposure to a particular thing is more common in one group, researchers can hypothesize that exposure might have caused the outcome of interest. Case-control studies are cheaper to conduct than cohort studies, but they also provide less evidence than a randomized control trial and rely more on the memory of people reporting past behaviors.

Cross-Sectional Studies (also known as cross-sectional analysis, transverse study or prevalence study) are observational studies. You gather all the data from a population and measure what's happening at a single point in time. They're great for describing features of a research population, measuring the prevalence of health outcomes, and understanding determinants of health. Their most significant disadvantage is that they only measure what's happening at that specific moment. By their definition, they don't provide long-term or ongoing information.

How much of a supplement, drug or treatment is given to the subjects?

In the 1990s, resveratrol was proclaimed a miracle supplement. Mice that took it lived longer. Since resveratrol was found in red wine, companies began promoting a glass a day as a healthy habit. What they failed to mention was how much resveratrol those mice were given. A person would have to drink 5 gallons of wine a day, to get the equivalent amount of resveratrol that those mice received.

Doses in the experiments may be much higher or much lower than you would experience in real life.

Answer those seven questions, and you'll be much more able to judge how valid the study is. If the answer to any of those questions is suspect, that study may not be as reliable as you might want.

Now that you know, what are you going to research?


A Note About Debates

I'm not a big fan of debates. Let me explain.

In a debate, the person who is the best debater wins. The facts are only necessary for how they can be twisted to support your argument.

A common tactic on my high-school debate team was burying the opposition in pointless drivel so they would get distracted by emotions. The opposing team would take valuable time responding to made-up outrages while ignoring the fundamental flaws in my team's arguments. It's a great way to win a debate, but a terrible way to pass on useful information.

A personal trainer is supposed to help clients get results. If a better way to do something is discovered, it's essential to see where the evidence leads. Check on sources and make sure they're credible. Evaluate the evidence. Take the time to read and understand clinical studies.

I'm not interested in a spirited debate. But I'm always happy to learn more about the evidence you've collected. When a better method or program is proven, we all succeed when we share the truth.


12 Cognitive Biases Explained - How to Think Better and More Logically Removing Bias

Call for a FREE Consultation (305) 296-3434
CAUTION: Check with your doctor before
beginning any diet or exercise program.

1/10/2021
Updated 1/18/2021
Updated 5/25/2021
Updated 7/12/2021
Updated 9/19/2021
Updated 2/9/2022
Updated 3/10/2022