This item is a good anecdote to the recent item on the precognition claims recently made by scientists. It suggests that the theoretical methodology was not tight enough and this is actually way more common than even scientists want to suppose. Bias does creep in and the data sets are always a way too small to be very comfortable.
The obvious problem is separation. This type of test shows a small separation and that means a little bit of data error can kick results all over.
It is not decisive such as a simple test I conducted to show efficiency of a product in promoting burn healing. In that case all treated burns sitting next to untreated burns showed major improvement. There were no overlapping signals at all and any such would have had me go back to the drawing board.
Thus a lack of decisive separation means generally that one must increase the sample size by an order of magnitude or two.
At least the paper is suggestive and supports an effort to fund a much larger study. It also provides a protocol that could actually winkle out scientific bias in this work. Trying to predict bias as part of the experimental design must be helpful.
Precognition experiments show that academic standards of evidence are too low
NOVEMBER 22, 2010