“One doesn’t expect to replicate a result which is published in the newspaper rather than a science journal.” (Karen Street, in a comment at Alexandria)
This is how we travel from Carey’s village. First I ride out of the village on the back of her bicycle. Then we crowd into a matatu. These are filled first by fitting in everyone who can possibly squeeze into the seats along the side, and then by fitting everyone possible into the aisle (bent over, because they’re not actually tall enough to stand in). Sometimes someone will hang onto the back, outside. We make our way across the country, by matatu, and by bus, and by train. Once, we are stopped, and taken off the bus, then loaded back on again, once the driver has paid the appropriate bribe for overloading the bus.
Inland changes to coast, the Luhya language for Swahili. Women’s clothes become more colorful, and attitudes toward cameras more jaded. (In Carey’s village, people run to stand in the picture; on the coast, people want money before they’ll let you photograph them.)
The year is 1986, and Carey and I are on our way to see her college roommate, Joanna Mountain, who, like Carey, has chosen to join the Peace Corps after college, and who, like Carey, has been stationed as a teacher in Kenya.
Decades later, when, diagnosed with cancer, I decide it’s time to fulfill a long standing desire to get consumer DNA testing, 23andme is the obvious choice, partly because it’s the one company that combines health related testing with genealogical information, but also in large part because it’s the company that has, as its research director, Joanna Mountain.
If there’s one name I trust in genetics, it’s Joanna Mountain’s, and that for the simplest of reasons: Joanna Mountain is the only person in genetics whose career I’ve actually followed for the past thirty years. I celebrated my twenty-first birthday with my sister and her roommate Joanna in the trailer park section of student housing at Stanford, where Carey, Joanna, and I all got our undergraduate degrees. I still have photos of Joanna from her stint in the Peace Corps after graduation. I’ve seen her once in a very blue moon since then, when visiting Carey, but more often heard from Carey about what Joanna was doing, through her return to Stanford for a Ph.D. in genetics, her postdoctoral research at Berkeley, and beyond. And so I have a good enough idea about Joanna Mountain to trust her intelligence, her character, and her grasp of science. If there’s any single person I’d be willing to be corrected by, on matters of nature, nurture, and genetics, it would be Joanna.
Most of us, in fields in which we aren’t ourselves expert, make our judgments at least partly in this way. We pick people, from among those who know more than us, whom we can trust, and we pick them, in part, based on whom we know, on old school ties, family relationships, or friendships. It was my physicist brother who convinced me not to adopt my friends’ oppositon to nuclear power, because the environmental risks of coal were actually much greater. I consult my mother (who taught at a medical school) about medical matters, so I know just what to ask my doctors and when I might want a second opinion, and she, in turn, uses me for computer tech support. Or we pick, from among the available experts, people who are suggested by our friends, Gary Kleck on guns perhaps if you’re suspicious of gun control, or Paul Krugman on economics if you prefer your criticism of Obama to come from the left. Or you go by which institutions you trust (and so, for instance, Cavalli-Sforza gains points with me, if I’m looking for an expert in genetics, because of his affiliation with Stanford).
But this method of figuring out whose knowledge and expertise to trust has its obvious limitations. One is that there’s a limit to how much I can verify in this way. I don’t have Joanna on my speed dial (she’s my sister’s friend, not mine directly), and, even if I did, she can’t reasonably be expected to answer all my questions. More important, you don’t know Joanna Mountain, and have no reason to accept anything she says based on any testimonial from me. This problem gets worse when we get to experts (however well credentialed) whom we have chosen based on our political beliefs. Ask my fellow bloggers at Alexandria whether they agree on the merits of Paul Krugman.
Fortunately, we have a process for discerning truth in science that gets us beyond such personal testimonials, the process known as peer review. You don’t have to have met Joanna Mountain back when she was teaching in a village near Mombasa to decide whether to believe what she has to say about mitochondrial DNA, or population divergence times, or global sequence diversity of BRCA2. Instead, we have a process, where her articles are submitted to journals like Human Molecular Genetics and Genetical Research, and reviewed by her professional peers before publication, and where her findings are either well replicated by other researchers or not.
But even with peer review, questions still remain about how you decide which scientific research you believe and trust.
The year is 1989, and the papers are full of a new discovery: cold fusion. Cheap and abundant energy is at hand. One of the world’s leading electrochemists, we’re told, has produced this energy in a tabletop experiment. I watch the buzz on Usenet, but I already know that cold fusion is probably doomed. My sister Carey, now back from the Peace Corps and studying for her Ph.D. at Caltech, has passed the word to me that her Caltech roommate, involved in the effort to replicate the finding, is getting negative results. Soon, what I’ve heard privately will be public knowledge.
If there’s anything I learn from the story of Fleischmann and Pons, it’s the difference between research and peer reviewed research. Peer review started when they made their announcement. Or, as Karen Street put it, “One doesn’t expect to replicate a result which is published in the newspaper rather than a science journal.” The first, and simplest, pitfall is understanding that “someone did a study” isn’t the same thing as “a study passed peer review.” Sometimes I’ve seen studies referenced in a blog (or even a newspaper or magazine) that turn out not to have been published in any peer reviewed journal. The fact that someone did a survey or study is meaningless if you can’t confirm that the study used a reasonable methodology.
Suppose, though, that we have, not just, “someone did a study,” but a really, genuinely peer reviewed result? What further do we need to consider?
- Some journals are more reputable than others. For instance, when I was at Stanford studying experimental psychology, the really prestigious journal for social psychologists to get published was the Journal of Personality and Social Psychology. There was at least one other journal with a name that would have sounded quite similar to someone not in the know, that was much less prestigious.
- But even reputable journals can publish results that you may want to doubt. For example, Daryl Bem, who is very highly regarded for his work in self-perception theory, published an article on precognition in the Journal of Personality and Social Psychology. Should you believe this result? Not till it’s very well replicated. Extraordinary claims require extraordinary proof, and precognition counts as an extraordinary claim. Of course, on matters of nature and nurture, part of why we disagree in how we’re guided by research is that we disagree on what is an extraordinary claim in the first place. The Journal of Personality and Social Psychology just recently published an study by Bobbi J. Carothers and Harry T. Reis, which found that, on a variety of measures, men and women are both from Earth, not all that different after all. Some may find this confirmation of common sense; others may see it as something counterintuitive and unlikely, which needs a lot more evidence before they’ll believe it. What are extraordinary claims to you, and what are ordinary ones?
- There are also different degrees or levels of peer review. One study, unreplicated, may well later prove to be wrong. Consider, for instance, the famous study in JAMA, later refuted, that showed an association between the MMR vaccine and autism. In fact, many single published medical studies later prove to be wrong. A whole series of studies reporting the same thing are stronger evidence. Review articles may tell you more than single studies (which is why my mother, when I was considering a choice between endometrial ablation and a hysterectomy before we learned that my chances of cancer were high enough that a hysterectomy would be required, told my to do my Highwire medical literature search with “reviews only” checked). And organizations like the National Academy of Sciences may provide another level of review.
- Some fields may have more robust peer review than others. But which ones? Razib Khan is wary of cultural anthropology, while Echidne of the Snakes is wary of evolutionary psychology. You probably have your own ideas about which of the two you agree with. Which scientific fields do you trust? Are there scientific fields whose peer review process you doubt?
So, especially for lay people with no real background in the science they’re trying to understand, and especially for issues where we already have strong beliefs, knowing when we need to let ourselves be corrected by the actual science can be a tricky process. On some issues, like nature vs. nurture, we may get different impressions of the relative influence of the two from different scientific fields even when peer review is working, simply because one field is studying more the one, and another the other.
What I do know, though is this: Believe peer reviewed research over what hasn’t met the test of peer review. Believe what’s been well replicated over what’s shown only in a single study. And what’s agreed on by the scientists of many fields (such as the theory of evolution) is a more robust finding than something where one field of study may point one way, and another field in a different direction.
(1) You’re a terrific writer. (2) You’ve raised a point I was thinking of earlier today, though I wasn’t thinking particularly about science–when people cite conflicting “experts”, whether on economics or on climate change (okay, that one is about science), how can we possibly have a discussion when we don’t even agree on the basic facts? On economics, I trust the likes of Paul Krugman, Dean Baker, and Ezra Klein; but I agree with you that many (most?) here at Alexandria are not so inclined. So what do we do? Whose “experts” carry more authority? Paul Krugman and Joseph Stiglitz both won Nobel Prizes in Economics, but so did Milton Friedman. And we all know that the “consensus” can be wrong, in science or anything else; sometimes it’s the lone dissenter who turns out to be right. I fear that many of us–by which I mean, “I”–often just find “experts” who confirm our (my) views on an issue. There is no issue in the entire world–political, philosophical, economic, ethical, religious–on which opinions do not differ, and on which “experts” can’t be found in disagreement with each other (e.g. the notoriously tendentious use of “expert witnesses” in trials). People of equal intelligence and of equal goodwill can disagree on the most basic, and the most important, facts (or ideas) about life: is there a God? Are humans inherently good or inherently evil? What is “justice”? Is government a necessary evil or a positive good?
I appreciate your raising, and presenting so well, the issue of “who should we trust?” And I agree that science has an excellent system of “peer review”. But where are the peer-reviewed studies in politics, or in philosophy? In those areas, are we laypersons left to our own devices, and our own prejudices, as to who to trust?
““One doesn’t expect to replicate a result which is published in the newspaper rather than a science journal.” (KS)
.
.
This “appeal to authority” is what people lean on when they either (A) don’t understand the subject matter themselves or (B) are tied to an pre-conceived (ie. biased) agenda.
The presumption that someone who never studies theoretical math or quantum mechanics should take my word for something seems insanely arrogant to me and always has. I’ve had one major talent in my unillustrious life (in mathematics) and that along with a visually photographic memory helped me overcome the effects of a form of dyslexia and ADHD (both undiagnosed until near adulthood). None of that helped me much with my “disassocaitive personality disorder, although that was harnessed by others early on and put to an “effective” use.
But even though I “knew” even when I was young that I “thought differently” especially perceptually, spatially and mathematically than most others, I DID NOT consider my way of thinking “better”….just different. What’s more, I STILL insist that anyone with a curious mind and the ability to think for themselves CAN easily understand virtually any mathematical treatise.
When my Dad was dying of cancer I spoke with various oncologists at Sloan Kettring and to their credit, I was never “shined on” or told simplistic versions of the situation. Everything was explained in detail, often with, “Are you still interested in this,” or “You can verify some of these things here…and here.” In short, they approached their subject the way I approach math or commodities trading….there are no mysteries. There is nothing there that’s “too hard” for most people to understand.
I think most people you meet who aren’t able to explain why they believe what they do are mostly just lazy-thinkers. That doesn’t make them “bad people,” just a little slothful.
As to the questions raised by JS, things like “Is there a God,” and “Is government a necessary evil or a positive good?”….just my view on such things, all anyone can do is to rely upon the best evidence available so far, which would SEEM to make agnosticism (“I DON’T know…need more information”) the most evidence-based answer to the “God” question and history seems to show that governments are not just necessary EVILS, but for the most part monstrous ones…the best of them have been veritable criminal conspiracies.
Even in those simpler times of the late 17th Century, England’s nobles including ints King (King William III) backed Captain William Kidd’s expedition as a pirate hunter, outfitting hiw with a decked out man-o-war, then when Kidd couldn’t find the six ships they went looking for “somewhere in the Indian Ocean,” they threw Kidd under the bus, tarring him as a pirate and hanging him. Ironically enough, on his way to the gallows, Kidd saw Robert Culliford (one of the real pirates he was sent to search for) and found that Culliford had been given a parden in exchange for his perjured testimony against Kidd.
This too ties in with the whole idea of what constitutes “proof” or “evidence.” History gives us all very good reasons to mistrust ALL our institutions and that’s not a bad thing at all. After all, IF you are told to believe in something because “so-and-so says so,” or “that group of esteemed people say that it’s so,” that really is no different at all, then a religious zealot believing something (say Creationism) because his esteemed Pastor and theologians of one sort or another says it’s so.
JMK: Who can argue with your statement that “History gives us all very good reasons to mistrust ALL our institutions”? For that matter, history gives us all very good reasons to mistrust each other. In reality, all we’ve got to guide us in this world is some version of the Reagan Doctrine: “Trust but verify”.
I go with several categories on “Trust but verify”.
1) Trust, and don’t verify: This works best for facts where my source is *way* more unlikely to be wrong than the time it would take for me to independently verify the fact (e.g. physicists’ report about what the speed of light is), or where my source is quite likely to be right and I don’t depend on the fact enough in my daily life for it to be worth my time to verify it (e.g. much of astrophysics), or where the cost of correcting on the occasions when my source turns out to be wrong is smaller than the cost of fact checking my source (e.g. trusting Google Maps).
2) Trust, but verify: A doctor’s diagnosis and recommendations sometimes fall in this category, since, on the one hand, your doctor usually knows more than you and more often than not knows what he’s talking about, but, on the other hand, it’s not a sure thing, and sometimes the costs of trusting and not verifying can be high.
3) Eh, I don’t know whether to trust or not, but I can’t be bothered to verify: Things I’m content to be agnostic about, because I don’t have a good enough reason to take the time to find out who’s right, and no one’s so obviously reliable that I’d call it a “trust, and don’t verify.”
4) Don’t trust, but verify: This source is probably wrong, but it’s worth taking my time to verify it to keep myself honest.
5) Don’t trust, and don’t verify: Perhaps this source is so unlikely to be right that it’s not worth my time checking it (e.g. someone’s selling what looks like obvious snake oil). Perhaps this source is unlikely to be right, and I really wouldn’t care that much anyway if it was (e.g. Trig Trutherism). Or perhaps someone whom I have no particular reason to trust is directing me to a source that’s in a format I don’t care to evaluate (e.g., if you want me to *watch a video* in favor of your political or economic views, you’ll have to clear a higher bar in convincing me that it’s worth my time than if you *point me to an article*, because I’d usually rather read such information than watch it, and if you’re pointing me to a video, there are usually articles competing for my time that I’ll pick instead).
Obviously, YMMV on some of this (I’m sure other people’s mileage varies a lot on the “what format do you like to consume things in” part).
Arguments for doing something like going to war should always be in the “verify” category, not in the “trust and don’t verify because our leaders know more than us” category. In your personal life, managing your money should generally be in the “verify” category, unless you are, not just inexpert, but aware that you have some brain disorder that impairs you enough that you need a conservator to handle your money. Diet and exercise programs should involve a certain amount of “verify,” because a lot of people are selling dubious advice there.
Hi Lynn,
This is a good post, and there’s a lot to comment on. As an (incipient) biologist, I’ve thought a little bit about what we mean by ‘evidence’, and about how the field of statistics (both in practice and theory) has developed in order to address the fundamental philosophical question ‘how do we know what we know’? Statistics is an interesting field, as it’s sort of a mathematical version of what I believe the philosophers call ‘epistemology’, and there’s a surprising amount of statistical debates and tradeoffs that boil down to some rather fundamental philosophical questions about what we mean by evidence, certainty, etc. This is interesting to me from a practical point of view, of course (how to make sure the conclusions of my studies are well supported, my methods of analysis reliable, etc.) But it’s also interesting from a more armchair, philosophical perspective, I think.
Re: Extraordinary claims require extraordinary proof, and precognition counts as an extraordinary claim
This is the sort of, David Hume approach to the extraordinary. I don’t think that it actually holds up well in Hume’s formulation (he was actually writing before modern statistics was developed, and the co-discoverer of evolution, Wallace, neatly demolished his argument against miracles by pointing out that if we took Hume literally, we would never credit that flying fishes and other biological curiosities existed). You can put the principle on a more sound footing, I think, by thinking about it in a Bayesian context, of prior and posterior probability. The likelihood of a certain claim being true, is affected by both the results of a particular study (that support or weaken the claim) *in combination with* our prior expectation of the claim being true.
So, for example, let’s consider we are doing a study on whether 1) a homeopathic preparation of duck liver cures malaria, or 2) a synthetic analogue of a compound from wormwood extract cures malaria (we know now that it does, but let’s suppose we were doing this study 10-20 years ago). Our prior probability of the former is exceedingly low, because we don’t see a mechanistic way that homeopathy could possibly work, and it would overturn the laws of biochemistry, physics, and pharmacology if it did work. Our prior probability of wormwood extracts curing malaria is somewhat higher, because they were traditionally used in China for the purpose, and because we know that many plant extracts have antimicrobial properties. if we were, today, doing the study to see if mildly changing the artemisinin molecules resulted in a better malaria cure, our probability would be higher still, since we already know that artemisinin is effective.
One of the issues, of course, with Bayesian statistics (I don’t use Bayesian methods in practice, because I’m not skilled or knowledgeable enough, but I think the philosophical grounding is strong) is how you address what the prior probability of your claim is. If we are debating whether the Raising of Lazarus really happened, then our ‘evidence’ would be the testimony of St. John, and we would combine that with the prior probability of miracles happening. I’d probably set that prior much higher than Richard Dawkins, for example. Which would put us at an impasse.
I think issues about prior probability come into play when dealing with issues where you have prior reasons to believe that a claim is true or false. I.e. with global warming, we have a priori reasons to think that higher carbon dioxide concentrations should lead to global warming, based on our understanding of physics and chemistry. The null hypothesis, in this case, is quite unlikely.
Speaking of the value of statistics, our good Steve supplies us with an excellent example of the potential of a chart posted on a website with Statistical in its name to influence people’s opinions here.
H. M. Stuart
Alexandria
Good point about Bayesian probability. Bayesian probability does have advantages over the David Humr formulation in that prior probability is a continuum and not a binary.
Some further extended commentary on Hume and Bayesian prior probability by those not previously conversant in it, like me:
Hume on Miracles, Frequencies, and Prior Probabilities (1998)
H. M. Stuart
Alexandria