I Don't Know What to Believe Anymore: A Guide for Scientific Claims

Greetings.
I just wanted to make this publication available, to those who have not seen it, and would perhaps be interested in keeping the link handy on their desktop. A new guide to peer review has just been launched to help the public make sense of research claims.
Don't know if it will be useful in your threads, maybe not.


People are bombarded with claims in newspapers and on the internet that are based on "scientific studies."

When faced with a headline that suggests an Alzheimer's drug increases the risk of heart attack or that watching TV is bad for children's mental health, or that pesticides are causing a decline in bee populations, people have to work out what to believe.

Which claims should be taken seriously? Which are 'scares'?




The publication: I Don't Know What to Believe: Making Sense of Science Stories... explains the peer review process -- the system researchers use to assess the validity, significance and originality of papers.

Understanding peer review and asking about the status of claims is important to society because it helps people make decisions.

Science Daily

"Peer review is an important part of the scientific process, and one indicator that can help readers distinguish in the mass of science they hear reported every day between what they can have confidence in and what they should treat with more caution. Furthermore, understanding how peer review works gives an insight into how science itself is done: I Don't Know What to Believe bridges a crucial gap in understanding between scientists and the public."

In a world where unfiltered news and information are everywhere, people are seeking a roadmap to distinguish what is sound, fact-based content.

So, below is the link to the publication, do your homework and be sure to put some peer review, along with your own research, to use in your efforts of your next paper or thread perhaps.

I don't know what to believe anymore PDF

Proper peer review is important however there is peer review and there is "peer review". As the article you linked points out not all journals carry the same reputation for effective peer review. An recent example would be the claims about fossil diatoms being found in a meteorite. The journal (of which the author of the article is an editor) is not well known for the strength of its review process.

Also, even though a paper may be carefully reviewed, it doesn't mean that the conclusions reached are necessarily correct.

Originally posted by Phage
Proper peer review is important however there is peer review and there is "peer review". As the article you linked points out not all journals carry the same reputation for effective peer review.

Roger that, thanks for pointing out what is pointed out in the article that I should have pointed out separately in the thread, to be clear.

An recent example would be the claims about fossil diatoms being found in a meteorite. The journal (of which the author of the article is an editor) is not well known for the strength of its review process.

Good example there of how to not 'jump the gun' before making such claims, possibly because no one in the review process (or anyone else) had enough historical experience of how fossilized phytoplankton or algae would present themselves on a meteorite in the first place? To assume makes a....

Also, even though a paper may be carefully reviewed, it doesn't mean that the conclusions reached are necessarily correct.

Peer reviews are not infallible? No,surely you jest.
One day eggs are good for you, the next, they're as bad as smoking.

Kidding aside, I thought about your encouragement in motivating others (in another thread) to do a little digging themselves in findings confirmations to answers to their questions that you had provided through your own research, i.e. (sunlit mountain tops on the moon).

So even with consideration to some questionable reputation of review processes, at least the author making the claim will have learned much more than having done nothing at all.

Perhaps a peer review of one's peer reviewers would be in order.

Originally posted by Lonewulph
I just wanted to make this publication available, to those who have not seen it

However 'deep' in my knowledge I may be I know one thing now, I don't need to see it to believe it, but thank you.

Originally posted by Phage
Proper peer review is important however there is peer review and there is "peer review". As the article you linked points out not all journals carry the same reputation for effective peer review.

Physicsforums links to a list of journals and if the journal is not on this list, they don't consider it an acceptable source...I think this is it:

ip-science.thomsonreuters.com...

I'm not really sure how good that list is at filtering out the less reputable journals from the more reputable journals, any thoughts on that? I can only say that for maybe 10 journals or so that I thought looked very shady, I didn't find them on this list, and that quite a few well known reputable journals I've looked for are there. So maybe it's a source to consider in evaluating the status of a particular journal.

Another consideration I found when looking up the "appeal to authority" logical fallacy, where the source argued that if the source was an expert in the field and there was also a consensus among experts, that it wasn't necessarily a fallacy. So, in evaluating a new paper, with a new finding, it's not realistic to expect there could be a consensus at that point, whether the finding is correct or not, because it takes some time for the research community to catch up with new papers.

Therefore if the paper is in a field I'm not familiar with, and it's a new finding, I have a tendency to wait and see what a consensus of other experts in that field think before jumping to any conclusions myself. It didn't take long to get a consensus of expert opinions on the faster than light neutrino paper, though it did take a while to isolate the faulty connection.

reply to post by Lonewulph


There were few movies about that topic, it can be from movie, not the real coverage...
This is the real one...

Originally posted by Arbitrageur


Another consideration I found when looking up the "appeal to authority" logical fallacy, where the source argued that if the source was an expert in the field and there was also a consensus among experts, that it wasn't necessarily a fallacy.

that is the exact definition of appeal to authority..

"well,,, this person is smart, they have 37 degrees and badges, and they said this, so it must be true"

"they have 50,000 experts who also have degrees.. and thats more experts following that theory then any other.. so that theory is right"

that is an appeal to authority.. and appeal to majority..

The thing with certain areas of science... Noone living knows exactly how to fundamentally perceive or qualify some events and classifications of quanta.... so the groups of experts and theorizers... are tautologically backing their man made models... it is possible that their models coincide more with the ability of their minds to process information and reality... then the models coincide with reality.. science, or the quest for truth, becomes even more dangerous when they seek stars and flags.. i mean awards and promotions and glory, then the truth comes second, to competition, money, and appearance and who has the most experts on their team, and if their team can more authoritarianaly beat the other team. it becomes a sport, a game, a war.. and no longer a cooperative effort to understand reality... There is only one all encompassing totality of truth, and this is called complete science... science is not complete. If there are still many questions that cannot be answered by current models and theories, I am not saying the current models and theories are wrong or bad... I am saying the creators and workers on said theories, should not be so close minded to the ways of which reality may be, that they cannot fathom, or complete their models and theories, because they emotions and personality dont allow them to consider possibilities that dont coincide with their feelings and emotions and personality and outlook and in look and around look...

People are bombarded with claims in newspapers and on the internet that are based on "scientific studies."

Originally posted by Lonewulph
Peer reviews are not infallible? No,surely you jest.
One day eggs are good for you, the next, they're as bad as smoking.

It isn't just the review that's the issue ... its the whole freaking media.

A study that concludes 'based on reactions of X protien Y may be the result, but this has been a relatively small sample group' becomes ... SCIENTISTS SAY EGGS GIVE YOU CANCER when it hits the dailymail. Or a discursive piece on biomedical ethics in abortion becomes ... SCIENTISTS SAY KILLING BABIES IS A-OKAY.

They also just dig up random research and declare the writer of the documents 'scientists' as if there was some meeting next to the mensa water cooler declaring that this one paper should represent all of scientific research on the planet.

Documentaries are essentially guilty of it also ... I cry a little inside everytime discovery channel uses football pitches or whales stacked on top of one another as a unit of measurement, or implies that basic science supports the physics in an anime cartoon

Is forumula:

science study + grant and grade requirements + small resources + small sample sizes + any major news network = headline made of utter nonsense to the power one thousand squared.

(Otherwise I approve of the general message of investigating and such like for yourself)

The problem even with peer review is that consensus of the time has a big impact on the interpretation of the evidence. Another determining factor in this interpretation is the need for funding of science which can be very costly. The evidence as tested, with all it's parameters listed, is accurate but limited and assumptions as to it's applications can be misapplied. To read a research article you must read the whole article and find what it applies to, most people read just part of it and then use it as evidence to back up their claims. The Consensus and need for funding do steer this interpretation within the scientific community though with big names understanding where their paycheck comes from.

I always try to find the original research to try to see what it says. The organization, Science Daily has pretty good translations and it looks like they are edited occasionally after release to fix any misconceptions. I still often see that there appears articles shortly after with evidence that contradicts some early published article though. This is because different groups with different beliefs can interpret the evidence to fit their beliefs.

Occams Razor is often used to make evidence acceptable to consensus of the time. Sometimes this skews the interpretation so it gets accepted. Since the truth is cut it is no longer truth and deceit allows even more challenges to occur.

So what do I think? I think everyone is just as nutty or even nuttier than I am

Peer review is the way science works.
However there are some taboo subjects that never see the light
of the day in peer review journals.
Evgeny podkletnov and Ning Li did managed to sneak in some papers in
some not so wellknown peer review journals, and it destroyed their careers.
Both are most like in the can somewhere.
So bottom line is if you have somehing that will most likely open a whole can of worms, dont
attempt any peer review. Just go ahd and bring same to the market if you can.