Sham Inquiry Run Riot

"What do you think science is? There's nothing magical about science. It is simply a systematic way for carefully and thoroughly observing nature and using consistent logic to evaluate results. Which part of that exactly do you disagree with? Do you disagree with being thorough? Using careful observation? Being systematic? Or using consistent logic?"

Dr. Steven Novella

Sharon Hill wrote a 2010 thesis, Being Scientifical: Popularity, Purpose and Promotion of Amateur Research and Investigation Groups in the U.S. According to her blog, Doubtful, Ms. Hill holds a B.S. in Geosciences and a Masters degree in education focusing on science and the public. In addition to blogging, she maintains Doubtful News, one of her many undertakings in pursuit of studying her main area of interest, “sham inquiry,” which is passing off activities to look like science that actually are not.

The doubtful Sharon Hill

Hill's thesis contained a study of 1,000 websites of amateur research and investigation groups, each of which conducted activities focused around unexplained events, and a majority of which claimed to use science as part of their approach to investigation and research. She conducted a thorough examination, resulting in a qualified and factual analysis that those of us looking for the truth out there - and what Hill conclusively demonstrated to be its frequent misrepresentation - might find helpful to review.

ARIGs

Each amateur research and investigation group, or ARIG, involved in Hill's study had certain qualifying characteristics. ARIGs examined had primary interests in ghosts, UFOs, mystery animals and similar such reported phenomena. Group activities did not provide a primary income for participants, and groups studied were not part of an academic institution or headed by working scientists. The Mutual UFO Network, for example, fell within the criteria and was included in the study, as was The Atlantic Paranormal Society, known to ghost enthusiasts far and wide as TAPS.

Hill had a hypothesis that group participants untrained in scientific methods would not be able to produce research results up to the standards of the scientific community. She additionally expected to find misuses of scientific jargon and prevalent misunderstandings of scientific concepts. Almost all ARIGs, she suspected, would nonetheless claim to be scientific as portrayed on popular television shows about hunting ghosts, UFOs and the like.

To test the validity of her hypotheses, Hill examined group purposes, goals, activities and results of investigations as published on websites. She also conducted email correspondence with ARIGs that specifically advocated a scientific method. Samples of reports, articles, terminology used by ARIGs and similar information was collected and examined.

Findings

ARIGs frequently stated admirable and ambitious purposes, Hill reported, but often demonstrated confusion and inconsistency in executing their missions. Groups commonly suggested that mainstream science would one day recognize their activities as important contributions. ARIGs likened themselves to pioneers of science, and frequently claimed to measure and substantiate paranormal phenomena by applying scientific methods.

Contents of an April, 2013, email distributed by MUFON,
an organization claiming dedication to scientific study.
The email was titled, "Dr. Greer to share startling
NEW [emphasis theirs] DNA evidence on the
Atacama Humanoid. Have we found ET?"   

In actuality, Hill found, blatantly non-scientific investigation was typical, including a great deal of conjecture and occult methods. ARIGs commonly demonstrated deep misunderstandings of data validity, and the use of non-scientific methods was often preferred if they stood to yield more favorable results.

Groups rarely conducted experiments to test hypotheses, tending instead to stagnate on unproven assumptions and premature conclusions, and the 1,000 ARIGs examined failed to either collectively or individually compile a cohesive knowledge base. Hill observed that UFO and cryptozoological groups maintained databases of reported sightings, but she additionally observed that it was not evident if or how the databases were used in any actual research. She noted that niche magazines and journals published speculation and case studies within the community, but found it to be closed to outside critique. ARIGs were nonetheless observed to publicly proclaim their research as credible that, in actuality, had no established system whatsoever of quality control.

When held accountable for explaining exactly what is scientific about their methods, Hill reported that ARIGs abandoned their platform of conducting scientific investigation. ARIGs provided vague answers and invalid explanations, demonstrated significant misunderstandings of science and withdrew their claims altogether of conducting scientific research. Hill concluded the groups were comfortable appealing to the public's sense of science while evading discussion of relevant issues when confronted by a knowledgeable inquirer.

"The most disturbing finding," Hill wrote in the thesis, "is the emphasis by ARIGs to educate 
the public."

"The most disturbing finding,” Hill wrote in the thesis, “is the emphasis by ARIGs to educate the public. In the process of communicating their work to the public, they most often promote a matter-of-fact paranormal viewpoint and present their methods and conclusions as sound. The public is delivered inaccurate information and a distorted view of science.”

Additional Items of Note

Hill presented a great deal of relevant and professionally documented information. Complete details of her work, such as a list of ARIGs examined, statements obtained, specific research methodology and full results, may be viewed in the thesis and should be directly referenced for accuracy.

A well composed summary of the evolution of views held within modern science was presented in the thesis, as were descriptions of the manners the public typically perceives science, and I recommend their reading. I also highly recommend viewing Hill's writing on the characteristics of pseudoscience, as they will be quickly recognized as virtual standard modes of operation within the UFO community. She additionally noted that few Americans understand scientific concepts. Hill demonstrated how the overall situation adds up to the use of sham inquiry to market products, services and impressions to a public that respects scientists and equates science with legitimacy, yet often fails to accurately differentiate between actual science and its impostors.

It was noted that it was possible that ARIGs could become much more productive than is currently the case. Hill suggested that raising the standards of investigation could be aided by consultations with credentialed scientists. That would require accepting constructive criticism and doing away with chronically held yet unsubstantiated ideas, but could result in taking some steps in the right direction, such as developing an accurate knowledge base.

Hill also suggested that collaborating with skeptical groups could potentially increase the quality of research produced by ARIGs. Moreover, she added that such a collaboration could be a learning experience for all involved.

However, Hill explained that competently addressing long-standing questions requires multi-disciplinary and coordinated efforts. They are not solved by maverick individuals or small groups.

“As long as ARIGs continue to tread the same ground by using dubious methods, making broad, unsupported claims, and remaining a community closed to critique,” Hill wrote, “their results are incomplete and no lasting progress can be achieved.”

Sharon Hill

I emailed Sharon Hill and asked if she would answer a few questions for this post. Following were the questions and her responses:

Would you please comment on how large a role pop culture plays in the process of people forming beliefs, inaccurate as they may typically be?

I would say it plays a primary part. We are immersed in our culture and that tells us what is acceptable or not acceptable. Since we are social animals, most of us strive to stay within those bounds. We get much of our beliefs and way of looking at the world from our parents or caregivers when we are younger and are influenced by our peers more as we get older. But since everyone sees television, movies, the internet these days, we cannot avoid its influence.

How commonly do you think people base their beliefs and conclusions on the ways circumstances make them feel (i.e., emotions), as compared to verifiable and accurate information?

We rarely have time to check the accuracy of information. We assume it's valid and usually that works out pretty well. The value of skepticism comes in when the result of the decision can have some important outcome - buying a product or investment, a health care decision or figuring out what was that weird thing that just happened to me? The consequences can be far reaching. Making calls on such a conclusion based on emotion and not more objective means is dangerous. We are terrible tools, we make mistakes and are biased and subjective. We have to want to get to the best answer and place emotion aside. Emotional responses can often move you down the wrong path. People put far too much reliance on what they feel, remember, perceive. We screw up often. 

In your opinion, to what extent within the UFO community do unsubstantiated claims and misrepresentations of supposedly scientific research influence the beliefs and interpretations of personal experiences of its members?

I can't really say since I'm not involved with the insiders there too much. But just listening to what they say, they seem to be awful certain about some things that are unwarranted, such as government hiding alien technology, recorded sightings, and other taking various dubious concepts at face value when they have no merit in science, such as regression therapy, hypnosis, implants. They bank on these items as being legitimate and fail to account for mundane or uncomfortable answers to these claims (such as people are mistaken or mentally troubled).

Commenting on MUFON, Ms.Hill wrote:

I like that MUFON has a protocol. It may not be great but it's better than nothing and an organized effort is better than a thousand individuals off on their own (like ghost hunters). But I still think the wrong questions are being asked. What, IF ANYTHING, happened? Not, "Is there evidence of a UFO?". The worst issue with UFOs is that they are so fleeting and there is often nothing to go on except the word of the witnesses. It would be good if there was a comprehensive checklist to consider during an investigation, like flight paths, planets visible, potential for remote control vehicles being airborne. It's a very tough thing to investigate. 

What do you think is most important for people to understand when they are trying to identify accurate and credible information?

Eyewitness testimony is THE WORST kind of evidence. Humans are prone to mistakes in observation and memory. There is such limited use of these anecdotes. Anecdotes CAN'T make up a science. It just will be unreliable and get you nowhere. Anecdotes may lead you to a hypothesis that you can then test, so it can be the start of an investigation. But with UFOs and many other paranormal phenomena, that is ALL they have. That's simply not going to cut it. It's just a story - like witches, werewolves and leprechauns. 

Encouraged to add further comments as she might like, Hill wrote:

The main problem I see with amateur investigators (and lots who call themselves professionals) is that they are not trying to find out the best answer. They are instead trying to find evidence to support their prior belief in a phenomena. That's so biased and worthless - what I call Sham Inquiry. It's certainly NOT the right way to move forward into answering these big questions. Yet I suspect many people are happy to stay immersed in the mystery, not actually figure it out. That makes me sad. It shortchanges human endeavors. 

Knowledge Versus Supposition

While I am not a scientist, I particularly appreciate Hill's thesis, the comments she provided for this post and the points she presented. My years in the UFO community most certainly led me to relate to her findings and concerns. I think her thesis, and her scientific examination and professional documentation of the state of ARIGs contained therein, is a valuable contribution to the community.

When people seek sympathy and understanding related to existential mysteries, a need is indeed created for individuals who will provide information and attention. As history has taught us, an inquisitive public will absolutely embrace individuals claiming to be qualified to provide such services and whether or not they are actually able to be helpful. Those seeking assistance are often experiencing states of emotional vulnerability, creating conditions conducive to deception and exploitation, as well as the entertaining and spreading of unsubstantiated speculation.

The misrepresentation of scientific investigation on a wide scale, or sham inquiry run riot, is clearly occurring within ufology and related genres. It should concern anyone sincerely interested in the well being of vulnerable community members and the discerning of reality. The rampant acceptance of myth, superstition and occult investigative techniques chronically hinders any progress that might otherwise be made.

"Are ET Hybrids [sic] among us? Find OUT!" was the title of
the May, 2013, MUFON email containing the above image.

Moreover, self-described witnesses and alleged alien abductees continue to be subjected to endless tales and anecdotes. The resulting effect creates ever changing suppositions, uncertainty and increasing anxiety associated with memories of events fading further into the past with each passing day. To supposedly help them with their mounting confusion and decreasing interest in other aspects of life, they are very frequently encouraged to participate in regressive hypnosis, sustaining what becomes hypnotically induced terror and traumata. Meanwhile, they are often and at the same time discouraged from seeking professional treatment for traumata. This all occurs in the course of what is called investigating events that, in actuality, cannot so much as even be verified to have ever taken place, and all of which is facilitated by individuals and organizations outrageously purporting to conduct scientific investigation and provide quality emotional support.

I present for consideration that if anything of credible substance is ever to be learned of reported alien abduction or related high strangeness, activities that harm witnesses and do not produce verifiable information are obviously not part of the solution. I also suggest that accurately differentiating between what we know to be true and what we suppose to be true is essential. It is unequivocally in the best interests of the search for truth and the welfare of the witness to implement effective quality control systems and hold self-described researchers accountable for sham inquiry.

Speculate all you want. Contemplate all the possibilities you care to imagine. Suppose anything you choose. Just please make an effort to differentiate between what you know and what you suppose, and don't allow people to impose their empty assumptions on you under the guise of scientific research. 

That would be unless, of course, you would rather believe a lie than know the truth. If that's the case, you can always look on the bright side: you're in the current majority. Be careful, though, because the truth might turn out to be a whole lot more interesting than all those commonly held assumptions - but we'll never know if we settle for accepting sham inquiry instead of holding out for accurate and systematic identification of the truth.