If I wanted to see if you could play a musical instrument, I might ask whether you played one, which one it is you play. I could then procure an instrument, give it to you and ask you to play, and within a few seconds I would have a fairly good idea of whether you could play or not. I wouldn’t need to bring along years of experience of being on the jury of New Zealand’s Got Talent. Nor would you need to be able to play the most difficult piece of music written for your instrument. And, unless you were playing something by John Cage, the evaluation of your claimed skill would be pretty straightforward. That is the essence of a qualitative test.
Now even if the ability to dowse is rare, then the person with that rare ability should be able to pass the test consistently. If the dowsers tested were simply guessing where the object was, then they would only pass the test on very rare occasions. That is, this test – like the test of musical ability – gives a yes-or-no answer to the question, “Does this person have the ability to do X?”, where X is playing a musical instrument, or dowsing, or whatever. Some of the details of the test will vary according to what X is. In the dowsing test, blindedness, even double blindedness is necessary to exclude the possibility that the dowser is using information other than that obtained by dowsing to achieve the desired result. As such, blindedness is not a prerequisite for testing musical ability for the simple reason that musical ability is more difficult to fake than to get right. As Randi himself says of his test, “You’re either pregnant or you’re not,” and you can’t really fake either.
One thing that people get hung up on here is the details. Depending on the particular test, it may need to be blinded, double-blinded, may require include a control group, and/or randomisation, but none of these elements of itself necessary. The idea of a gold standard of double blind, placebo controlled, randomly sampled may be a noble goal but we do have to keep some common sense and ask what is it we really need for an investigation to work: That the result be so unlikely to occur by chance when alternative explanations can be eliminated.
Emily Rosa’s test of Therapeutic Touch is a good example. Not only was it not performed double blind, but it was impossible to perform double blind. The point was that Emily’s single blinding was sufficient (with some other precautions) to prevent cheating by the subjects.
As far as I know no blinded testing with a randomised control group has ever been done in emergency medicine, yet this is a field of medicine which is remarkably successful. Every now and then an ER will design and test a new procedure, carefully comparing the types of incoming traumas and the outgoing results with those of other ERs with standard procedures. If the new procedure appears to be successful it is tested in more and more ERs until it becomes the standard.
In education, not only would blinding experimenters and subjects be ludicrous, but it also makes no sense to have an independent control group other than an internal one, i.e. the learners before and after the intervention. From the very nature of science, only one outcome counts, whether you can perform the conceptual manipulation that science has discovered. If you can’t, you can’t fake it, and you either can or you can’t.
Finally, the concept of a qualitative test with a yes-or-no answer should clear up the “spectrum” fallacy where there are two points on a continuum and the fallacy is to assume that every point in between is significantly populated. The qualitative test results in a bimodal distribution with only sparse to empty populations falling in between. The spectrum fallacy might explain the rise in autism diagnoses, where children whose behaviours are simply on the extreme end of the normal range are being lumped in with real illnesses (autism, Aspergers, Tourettes) whose symptoms are far more serious.
