It looks like you're using an Ad Blocker.
Please white-list or disable AboveTopSecret.com in your ad-blocking tool.
Thank you.
Some features of ATS will be disabled while you continue to use an ad-blocker.
Statistics of Scientific Fraud
page: 10
share:
A scientometric study estimating the percentage of fabricated experimental data in biomedical scientific literature somewhere at 5-10 %
An awkward biochemical procedure of graphical data presentation has created a unique trap for unscrupulous researchers. "Absent-minded" fabrication of certain type of this procedure's output very often results in an "impossible" picture that physically can not be based on any real data. Cases seem to be sufficiently frequent to make statistics for quantitative measuring scientific fraud.
The problem of estimating the percentage of fraudulent research in science is actually a political one. Perhaps, like estimating the spread of stimulators' abuse in professional sport, or counting the number of bribe-takers in Moscow governmvent. Everywhere we may find an horde of hypocritical bureaucrats-ideologists always ready to argue that cases of fraud within their jurisdiction are negligibly small. In the field of estimating scientific fraud the situation seems to be the most absurd. The prevailing bureaucratic opinion is that only from 0.004 to 0.1 percent of scientific publications are fraudulent. I think it is meaningless to say that these numbers are wrong. The real problem (IMHO) is to understand whether this number is closer to 1%, to 10% or to 99%. This my work tries to shift an estimate closer to the right on this scale, yet it is too far from absolute clarity in this question.
Methods
Scatchard plot analysis is an extremely popular procedure in many fields of biochemistry, immunology, pharmacology, etc. The meaning of this procedure is that the saturation isotherm (i.e. the amount of substrate (ligand) Bound to some binder is measured as a function of added (or Total) amount of this substrate - see left panel of the figure) is redrawn on a very strange coordinate plane with Bound on X-axe and Bound/Free on Y-axe (where Free=Total-Bound) - see the graph in the center of the figure.
If this is the first time you hear about Scatchard plots, you should have here the right feeling that it is rather difficult to understand how the experimental data should look like on such coordinate plane. Indeed, this graphical transformation (actually, it has some meaning) converts a plain and clear presentation of experimental data into something lying absolutely beyond understanding of most biomedical researchers. One funny consequence is that fabrication of trustworthy Scatchard plot seems to be also a too difficult job for them.
The picture above illustrates a dreadful pitfall in the process of fabrication.
Left panel represents the typical scenario of two "parallel" binding experiments, i.e. two almost identical experiments giving pairs of experimental points with the same Total concentration and different Bound concn. Of course, on the usual coordinate plane it results in pairs of experimental points lying precisely one under another.
On the Scatchard plot both X- and Y-axes are linear functions of Bound signal, so, (perhaps with some effort) you may understand that these pairs should lie approximately on the line drawn through the start of coordinates as it is shown at the central panel of the figure (Free concentration usually may be considered equivalent to Total).
Now let's imagine that you are trying to fabricate a nice-looking Scatchard plot for such experiment directly, without fabricating data on normal coordinate plot first. Obviously, if you are not too cautious, almost certainly you will draw pairs of points lying one under another (right panel of fig.1).
In my view, this peculiarity in the Scatchard plot is quite sufficient to make accusation of data fabrication. It is almost impossible to imagine real dataset that may correspond to such plot, or at least its receiving requires enormous and absolutely meaningless efforts. I also do not see what sort of "honest" error may lead to this appearance.
Yet, a pair of official opinions (just simplest bureaucratic figures of speech from Nature journal and ORI) do not share my point of view at least partially.
Goals
Of course, this is not the most important type of scientific fraud, it may be called an 'absent-minded' data falsification; but it may have interesting applications. After I have found the case of this folly in an article in Nature, I clearly understood that it is really a very widespread type of error and it may provide unique database for quantitative measuring scientific fraud.
Indeed, Scatchard plot analysis is an enormously popular procedure in biomedical sciences; citation index shows over 1000 citations of original Scatchard's article every year. About 3-4 times more authors use his plot without citing anything. So, stupid browsing scientific journals and counting falsified plots may resolve two problems:
1. It makes possible to estimate directly the percentage of fabricated experimental data in biomedical sci. literature simply by dividing the number of falsified plots (i.e. like right panel of fig.1) by the number of correctly drawn plots of parallel experiments (i.e. looking like the central panel of fig.1).
2. The Scatchard's article is dated by 1949, so it may also be possible to estimate the trend of changing this percentage during last 30 years in order to check the popular opinion that quality of science seriously deteriorated itoday in comparison with heydays in sixties.
Results
Apparently, the most convenient way to collect statistics for this study is browsing the Journal of Biol. Chemistry - yearly subscription of this journal contains usually about 200 articles displaying various sorts of Scatchard plots. So, survey of this journal may be quite sufficient for the first goal listed above, though, I guess, estimating the historical trend of fraud may require more thorough study.
So far, I have succeeded to browse the subscription for this journal for 1992. Take look, if interested, at technical data. The result is that I have found 32 correctly drawn "parallel" Scatchard plots and two fabricated plots.
It is not quite correct, but I also may add the mentioned above case in Nature estimating that the number of Scatchard plots I have seen accidentally since I started this project is less than that contained in a half-year subscription for J.Biol.Chem., .
So, the "initial estimate" based on three cases is that the percentage of fabricated plots is somewhere at 5-10%.
This number may be bigger if there are significant additional number of correctly fabricated Scatchard plots. Yet, I don't think there may be such cases in reality; in my view correct falsification requires from the author much more wits than honest conducting the whole experiment, and, therefore, I don't see why he should use data falsification.
Morale
This estimation does not mean that only 5-10% of biomedical researchers fabricate data. Obviously, every research paper usually contains data from several different types of experiments such as Scatchard plot analysis. So, the percentage of articles with fabricated pictures at least for one of them should be several times more - that is about 20-30%. Then, every researcher participate in writing more than one article during his lifetime.
Therefore, the conclusion is that:
MOST LABORATORY RESEARCHERS IN BIOMEDICINE FABRICATE DATA FROM TIME TO TIME.
Dealing with Scatchard plots, I have found at least two other simalar indicators of data fabrication. Unfortunately, they can't serve for numerical estimation; but being substantially more widespread, they certainly support impression that my conclusion above is correct.
For more detailed report read my article Absurd trivial errors in Scatchard plot analysis; in brief, the first other indicator is that the scattering of datapoints around trend line on most Scatchard plots does not comply the usual property of physical measurement that small signals are measured with bigger relative error (CV). Instead, they seem to stress the common notion of a "nice looking curve". It is not a conclusive indicator, but, as I wrote, it just supports an impression.
The other indicator is again a funny one. There is another type of binding experiment called inhibition (or displacement) experiment resulting in, roughly speaking, the same curve but turned upside down. Scatchard transformation can not be applied to such curve directly (like floppy disk can not be read if it is inserted upside down). Or if an obvious modification was used, it physically can't result in a "nice looking curve". Nevertheless very numerous papers present really nice Scatchard plots derived from inhibition curves. I think it is impossible and all these plots were fabricated, but I am not sure. Several times I tried to receive explanations from authors about method of deriving their Scatchard plots, but, of course, there were no responses. So, again, this indicator just supports my impression.
www.orc.ru...
An awkward biochemical procedure of graphical data presentation has created a unique trap for unscrupulous researchers. "Absent-minded" fabrication of certain type of this procedure's output very often results in an "impossible" picture that physically can not be based on any real data. Cases seem to be sufficiently frequent to make statistics for quantitative measuring scientific fraud.
The problem of estimating the percentage of fraudulent research in science is actually a political one. Perhaps, like estimating the spread of stimulators' abuse in professional sport, or counting the number of bribe-takers in Moscow governmvent. Everywhere we may find an horde of hypocritical bureaucrats-ideologists always ready to argue that cases of fraud within their jurisdiction are negligibly small. In the field of estimating scientific fraud the situation seems to be the most absurd. The prevailing bureaucratic opinion is that only from 0.004 to 0.1 percent of scientific publications are fraudulent. I think it is meaningless to say that these numbers are wrong. The real problem (IMHO) is to understand whether this number is closer to 1%, to 10% or to 99%. This my work tries to shift an estimate closer to the right on this scale, yet it is too far from absolute clarity in this question.
Methods
Scatchard plot analysis is an extremely popular procedure in many fields of biochemistry, immunology, pharmacology, etc. The meaning of this procedure is that the saturation isotherm (i.e. the amount of substrate (ligand) Bound to some binder is measured as a function of added (or Total) amount of this substrate - see left panel of the figure) is redrawn on a very strange coordinate plane with Bound on X-axe and Bound/Free on Y-axe (where Free=Total-Bound) - see the graph in the center of the figure.
If this is the first time you hear about Scatchard plots, you should have here the right feeling that it is rather difficult to understand how the experimental data should look like on such coordinate plane. Indeed, this graphical transformation (actually, it has some meaning) converts a plain and clear presentation of experimental data into something lying absolutely beyond understanding of most biomedical researchers. One funny consequence is that fabrication of trustworthy Scatchard plot seems to be also a too difficult job for them.
The picture above illustrates a dreadful pitfall in the process of fabrication.
Left panel represents the typical scenario of two "parallel" binding experiments, i.e. two almost identical experiments giving pairs of experimental points with the same Total concentration and different Bound concn. Of course, on the usual coordinate plane it results in pairs of experimental points lying precisely one under another.
On the Scatchard plot both X- and Y-axes are linear functions of Bound signal, so, (perhaps with some effort) you may understand that these pairs should lie approximately on the line drawn through the start of coordinates as it is shown at the central panel of the figure (Free concentration usually may be considered equivalent to Total).
Now let's imagine that you are trying to fabricate a nice-looking Scatchard plot for such experiment directly, without fabricating data on normal coordinate plot first. Obviously, if you are not too cautious, almost certainly you will draw pairs of points lying one under another (right panel of fig.1).
In my view, this peculiarity in the Scatchard plot is quite sufficient to make accusation of data fabrication. It is almost impossible to imagine real dataset that may correspond to such plot, or at least its receiving requires enormous and absolutely meaningless efforts. I also do not see what sort of "honest" error may lead to this appearance.
Yet, a pair of official opinions (just simplest bureaucratic figures of speech from Nature journal and ORI) do not share my point of view at least partially.
Goals
Of course, this is not the most important type of scientific fraud, it may be called an 'absent-minded' data falsification; but it may have interesting applications. After I have found the case of this folly in an article in Nature, I clearly understood that it is really a very widespread type of error and it may provide unique database for quantitative measuring scientific fraud.
Indeed, Scatchard plot analysis is an enormously popular procedure in biomedical sciences; citation index shows over 1000 citations of original Scatchard's article every year. About 3-4 times more authors use his plot without citing anything. So, stupid browsing scientific journals and counting falsified plots may resolve two problems:
1. It makes possible to estimate directly the percentage of fabricated experimental data in biomedical sci. literature simply by dividing the number of falsified plots (i.e. like right panel of fig.1) by the number of correctly drawn plots of parallel experiments (i.e. looking like the central panel of fig.1).
2. The Scatchard's article is dated by 1949, so it may also be possible to estimate the trend of changing this percentage during last 30 years in order to check the popular opinion that quality of science seriously deteriorated itoday in comparison with heydays in sixties.
Results
Apparently, the most convenient way to collect statistics for this study is browsing the Journal of Biol. Chemistry - yearly subscription of this journal contains usually about 200 articles displaying various sorts of Scatchard plots. So, survey of this journal may be quite sufficient for the first goal listed above, though, I guess, estimating the historical trend of fraud may require more thorough study.
So far, I have succeeded to browse the subscription for this journal for 1992. Take look, if interested, at technical data. The result is that I have found 32 correctly drawn "parallel" Scatchard plots and two fabricated plots.
It is not quite correct, but I also may add the mentioned above case in Nature estimating that the number of Scatchard plots I have seen accidentally since I started this project is less than that contained in a half-year subscription for J.Biol.Chem., .
So, the "initial estimate" based on three cases is that the percentage of fabricated plots is somewhere at 5-10%.
This number may be bigger if there are significant additional number of correctly fabricated Scatchard plots. Yet, I don't think there may be such cases in reality; in my view correct falsification requires from the author much more wits than honest conducting the whole experiment, and, therefore, I don't see why he should use data falsification.
Morale
This estimation does not mean that only 5-10% of biomedical researchers fabricate data. Obviously, every research paper usually contains data from several different types of experiments such as Scatchard plot analysis. So, the percentage of articles with fabricated pictures at least for one of them should be several times more - that is about 20-30%. Then, every researcher participate in writing more than one article during his lifetime.
Therefore, the conclusion is that:
MOST LABORATORY RESEARCHERS IN BIOMEDICINE FABRICATE DATA FROM TIME TO TIME.
Dealing with Scatchard plots, I have found at least two other simalar indicators of data fabrication. Unfortunately, they can't serve for numerical estimation; but being substantially more widespread, they certainly support impression that my conclusion above is correct.
For more detailed report read my article Absurd trivial errors in Scatchard plot analysis; in brief, the first other indicator is that the scattering of datapoints around trend line on most Scatchard plots does not comply the usual property of physical measurement that small signals are measured with bigger relative error (CV). Instead, they seem to stress the common notion of a "nice looking curve". It is not a conclusive indicator, but, as I wrote, it just supports an impression.
The other indicator is again a funny one. There is another type of binding experiment called inhibition (or displacement) experiment resulting in, roughly speaking, the same curve but turned upside down. Scatchard transformation can not be applied to such curve directly (like floppy disk can not be read if it is inserted upside down). Or if an obvious modification was used, it physically can't result in a "nice looking curve". Nevertheless very numerous papers present really nice Scatchard plots derived from inhibition curves. I think it is impossible and all these plots were fabricated, but I am not sure. Several times I tried to receive explanations from authors about method of deriving their Scatchard plots, but, of course, there were no responses. So, again, this indicator just supports my impression.
www.orc.ru...
I'd dispute that 5-10%. The author is making a lot of wild guesses based on his own difficulty in interpreting the charts.
In fact, I would bet this week's paycheck that the originator of the article Never ***EVER*** got a piece published in a professional journal. For the record, I was published in the Texas Journal of Epidemiology back in 1980 (after the piece was rejected for the Journal of the American Medical Association) and can tell you that the review process is just brutal. We had to go back and clarify and redo (and justify) our statistics about four times before they would finally accept it.
Yes, there's fraudulent research. Yes, researchers make up and misinterpret data. Everyone who's ever done research is aware that our field has its share of snake-oil salesmen. That's why new discoveries are hopped on by other research groups and universities run experiments to see if they can duplicate the data (as in the "cold fusion" debacle of a few decades ago.)
There's a lot of "checks and balances" in the system this way. Bogus and trivial research does get done, but the scammers are unmasked far more often than scammers in other areas.
[Edited on 17-9-2002 by Byrd]
In fact, I would bet this week's paycheck that the originator of the article Never ***EVER*** got a piece published in a professional journal. For the record, I was published in the Texas Journal of Epidemiology back in 1980 (after the piece was rejected for the Journal of the American Medical Association) and can tell you that the review process is just brutal. We had to go back and clarify and redo (and justify) our statistics about four times before they would finally accept it.
Yes, there's fraudulent research. Yes, researchers make up and misinterpret data. Everyone who's ever done research is aware that our field has its share of snake-oil salesmen. That's why new discoveries are hopped on by other research groups and universities run experiments to see if they can duplicate the data (as in the "cold fusion" debacle of a few decades ago.)
There's a lot of "checks and balances" in the system this way. Bogus and trivial research does get done, but the scammers are unmasked far more often than scammers in other areas.
[Edited on 17-9-2002 by Byrd]
new topics
-
Liberal Madness and the Constitution of the United States
US Political Madness: 3 hours ago -
New York Governor signs Climate Law that Fines Fossil Fuel Companies
US Political Madness: 10 hours ago
0