In an attempt to point out that not every article that makes it into peer review survives the scrutiny of the science community, New Yorker author Jonah Lehrer apparently goes a little further than he intended, and says so here. The Truth Wears Off begins with a number of examples of when the effects described in peer reviewed articles don’t seem to be real, notably in medicine, the life sciences, and psychology. Lehrer gives some examples from physics, as well.
To some, it appears that the effect first seen declines over time. Examples:
• people shown a face and asked to describe it showed a lower ability to recognize the face (verbal overshadowing) two decades ago, but the effect shrank dramatically year after year.
• Anti-psychotic drugs tested in the 1990s appear to be less effective today. Note: the article does not examine whether the schizophrenics in this study are similar to schizophrenics studied a decade ago—this includes severity of and type of symptoms, and any other treatments they may have received.
• In an ESP test from early last century, some initially appeared to show paranormal ability, but further tests failed to substantiate this result.
• A purported correlation between female barn swallows and symmetry in their mates led to a number of studies finding similar results for swallows and other species, but the correlation has since disappeared. Michael Jennions found that a large number of results in ecology and evolutionary biology demonstrate this decline effect.
In an apparent misunderstanding of the process, Lehrer discusses the problem when “rigorously validated findings” can no longer be replicated as a problem with science. Most scientists would assume there is a problem with both the findings and the sloppiness that leads to a large number of poor results.
Lehrer then discusses a few problems in the article, but does not tease out the importance of each:
• Journals and scientists look for results that disagree with the orthodoxy. Scientists are less likely to submit null results to journals and journals are less likely to print them. Once the orthodoxy changes (from symmetry is irrelevant to symmetry is important to female barn swallows), confounding results become interesting. Note: This is considered a real phenomenon, but Lehrer gives little idea as to whether this is a problem with 0.5% or 95% of articles submitted. Climate change skeptics—if results are submitted to peer review which are contrary to scientific orthodoxy on climate change, these results will get prominent play, if they make it through peer review.
• The barn swallow studies were not double blind studies, with different people measuring feather length and assessing behavior. When it came time to round up or down, errors crept into measurements that differed by millimeters. Similarly, published acupuncture results vary by country, in part because the person testing for the effect knows whether acupuncture has been used.
• A number of studies, such as those finding genetic effects on hypertension and schizophrenia, were so badly done that the results are meaningless. One review of 432 such results found the vast majority worthless. Note: This is considered an important problem in some fields of science, notably medicine, and also my field, education. See comments below for what those in the life sciences and medicine think. There appears to be little support for Lehrer’s including physics experiments in his article.
• Lehrer assumes that all the later-refuted results were analyzed statistically in an appropriate way. Note: Statisticians do not, see Andrew Gelman’s comment below.
Are there reasons that explain these results besides the one favored by many, that science is a crapshoot? The person who told me of this article certainly feels that way; he picks and chooses among scientific results, except when he knows scientists are wrong and so goes with other analysis.
Lehrer says, “We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe.” Only science doesn’t prove so much as disprove, and what is left standing gains credibility.
Lehrer does not provide enough information or context so that we can make sense of what he says. He repeats what everyone in science already knows: that research in some fields, and some peer review, is of lower quality, and that while a number of peer review results turn out to be uninteresting, this is much more often true in medicine and some of the life sciences. The one important point I got from the article, that results that no longer appear to be true are still used by some doctors, disappears among the noise.
Not mentioned is that people whose exposure to science comes primarily from articles on medicine see reason to doubt medical science, and many extrapolate to other fields of science. Those who prefer to doubt science will find justification in this article.
Comments from others
Jerry Coyne believes that his field, evolutionary biology, has a problem, in part because so few eyes look at each result.
I tend to agree with Lehrer about studies in my own field of evolutionary biology. Almost no findings are replicated, there’s a premium on publishing positive results, and, unlike some other areas, findings in evolutionary biology don’t necessarily build on each other: workers usually don’t have to repeat other people’s work as a basis for their own. (I’m speaking here mostly of experimental work, not things like studies of transitional fossils.) Ditto for ecology. Yet that doesn’t mean that everything is arbitrary. I’m pretty sure, for instance, that the reason why male interspecific hybrids in Drosophila are sterile while females aren’t (“Haldane’s rule”) reflects genes whose effects on hybrid sterility are recessive. That’s been demonstrated by several workers. And I’m even more sure that humans are more closely related to chimps than to orangutans. Nevertheless, when a single new finding appears, I often find myself wondering if it would stand up if somebody repeated the study, or did it in another species.
But let’s not throw out the baby with the bathwater. In many fields, especially physics, chemistry, and molecular biology, workers regularly repeat the results of others, since progress in their own work demands it. The material basis of heredity, for example, is DNA, a double helix whose sequence of nucleotide bases codes (in a triplet code) for proteins. We’re beginning to learn the intricate ways that genes are regulated in organisms. The material basis of heredity and development is not something we “choose” to believe: it’s something that’s been forced on us by repeated findings of many scientists. This is true for physics and chemistry as well, despite Lehrer’s suggestion that “the law of gravity hasn’t always been perfect at predicting real-world phenomena.”
Lehrer, like Gould in his book The Mismeasure of Man, has done a service by pointing out that scientists are humans after all, and that their drive for reputation—and other nonscientific issues—can affect what they produce or perceive as “truth.” But it’s a mistake to imply that all scientific truth is simply a choice among explanations that aren’t very well supported. We must remember that scientific “truth” means “the best provisional explanation, but one so compelling that you’d have to be a fool not to accept it.” Truth, then, while always provisional, is not necessarily evanescent. To the degree that Lehrer implies otherwise, his article is deeply damaging to science.
Note: most scientists in physics, chemistry, and molecular biology, so far as I know, agree.
David Gorski, an advocate of science-based medicine, says that people in medicine have been talking about a number of these issues for years, however, Lehrer goes too far in generalizing poor medical studies into problems with science.
Jennions’ article was entitled Relationships fade with time: a meta-analysis of temporal trends in publication in ecology and evolution. Reading the article, I was actually struck by how relatively small, at least compared to the impression that Lehrer gave in his article, the decline effect in evolutionary biology was found to be in Jennions’ study. Basically, Jennions examined 44 peer-reviewed meta-analyses and analyzed the relationship between effect size and year of publication; the relationship between effect size and sample size; and the relationship between standardized effect size and sample size. To boil it all down, Jennions et al concluded, “On average, there was a small but significant decline in effect size with year of publication. For the original empirical studies there was also a significant decrease in effect size as sample size increased. However, the effect of year of publication remained even after we controlled for sampling effort.” They concluded that publication bias was the “most parsimonious” explanation for this declining effect.
Personally, I’m not sure why Jennions was so reluctant to talk about such things publicly. You’d think from his responses in Lehrer’s interview that scientists would be coming for him with pitchforks, hot tar, and feathers if he dared to point out that effect sizes reported by investigators in his scientific discipline exhibit small declines over the years due to publication bias and the bandwagon effect. Perhaps it’s because he’s not in medicine; after all, we’ve been speaking of such things publicly for a long time. Indeed, we generally expect that most initially promising results, even in randomized trials, will not ultimately pan out. In any case, those of us in medicine who might not have been willing to talk about such phenomena became more than willing after John Ioannidis published his provocatively titled article Why Most Published Research Findings Are False around the time of his study Contradicted and Initially Stronger Effects in Highly Cited Clinical Research. Physicians and scientists are generally aware of the shortcomings of the biomedical literature. Most, but sadly not all of us, know that early findings that haven’t been replicated yet should be viewed with extreme skepticism and that we can become more confident in results the more they are replicated and built upon, particularly if multiple lines of evidence (basic science, clinical trials, epidemiology) all converge on the same answer. The public, on the other hand, tends not to understand this.
Gorski also discusses the effect of subject popularity on calculations of error rates. Commenters look at the challenges Lehrer presents from physical science, and do not support his conclusions.
It’s always good to run your results by someone who is very good at statistics. Andrew Gelman, statistician, says,
The short story is that if you screen for statistical significance when estimating small effects, you will necessarily overestimate the magnitudes of effects, sometimes by a huge amount. I know that Dave Krantz has thought about this issue for awhile; it came up when Francis Tuerlinckx and I wrote our paper on Type S errors, ten years ago.
My current thinking is that most (almost all?) research studies of the sort described by Lehrer should be accompanied by retrospective power analyses, or informative Bayesian inferences. Either of these approaches–whether classical or Bayesian, the key is that they incorporate real prior information, just as is done in a classical prospective power analysis–would, I think, moderate the tendency to overestimate the magnitude of effects.
Note: I don’t understand statistics, or Gelman’s solutions, but I learned early on that poor statistics is the downfall of many a conjecture.
PZ Myers, biologist
Early in any scientific career, one should learn a couple of general rules: science is never about absolute certainty, and the absence of black & white binary results is not evidence against it; you don’t get to choose what you want to believe, but instead only accept provisionally a result; and when you’ve got a positive result, the proper response is not to claim that you’ve proved something, but instead to focus more tightly, scrutinize more strictly, and test, test, test ever more deeply.
Steven Novella, neurologist, discusses how the naive, the skeptical (scientists mostly fit in this category), and the deniers see science, then says,
Lehrer is ultimately referring to aspects of science that skeptics have been pointing out for years (as a way of discerning science from pseudoscience), but Lehrer takes it to the nihilistic conclusion that it is difficult to prove anything, and that ultimately “we still have to choose what to believe.” Bollocks!
John Horgan sees this as the decline of illusion. He is not a big fan of truthiness.
Lehrer’s reference to physics was checked by Charles Petit. He quotes Lawrence Krauss,
“The physics references are (deposit scatological bovine expletive here) … the neutron data have fallen, reflecting under-estimation of errors, but the lower lifetime doesn’t change anything having to do with the model of the neutron, which is well understood and robust … And as for discrepancies with gravity, the deep borehole stuff is interesting but highly suspect. Moreover, all theories conflict with some experiments, because not all experiments are right.” / LMK
The January 5 NY Times has an article, Journal’s Paper on ESP Expected to Prompt Outrage.
It repeats a number of the points made by people cited above:
Claims that defy almost every law of science are by definition extraordinary and thus require extraordinary evidence. Neglecting to take this into account — as conventional social science analyses do — makes many findings look far more significant than they really are, these experts say. …
Peer review is usually an anonymous process, with authors and reviewers unknown to one another. But all four reviewers of this paper were social psychologists, and all would have known whose work they were checking and would have been responsive to the way it was reasoned.
Perhaps more important, none were topflight statisticians. “The problem was that this paper was treated like any other,” said an editor at the journal, Laura King, a psychologist at the University of Missouri. “And it wasn’t.”
Many statisticians say that conventional social-science techniques for analyzing data make an assumption that is disingenuous and ultimately self-deceiving: that researchers know nothing about the probability of the so-called null hypothesis.
In this case, the null hypothesis would be that ESP does not exist. Refusing to give that hypothesis weight makes no sense, these experts say; if ESP exists, why aren’t people getting rich by reliably predicting the movement of the stock market or the outcome of football games?
Instead, these statisticians prefer a technique called Bayesian analysis, which seeks to determine whether the outcome of a particular experiment “changes the odds that a hypothesis is true,” in the words of Jeffrey N. Rouder, a psychologist at the University of Missouri who, with Richard D. Morey of the University of Groningen in the Netherlands, has also submitted a critique of Dr. Bem’s paper to the journal.
Attempts to replicate the results have failed.
I think most people want science to be straight line plots, when it is often scattergrams. The latest results are just that. They may change. I think it helps if one has followed a field for a while, like climate change, to see how the science has refined itself.
In medicine we face several recurring problems. Most studies are too small to give us the power that we really want and need. Next, research is expensive. It is difficult to fund duplicate studies. Then, how do we apply large scale statistics to individual patients? In what key ways will the patient be similar to the study population and in what ways different.
Still, I always enjoy the skeptics who say science cannot be trusted as they type on computers they dont understand onto the web (which they dont understand either) while talking on their smartphone. Science clearly gets a lot of things right.
Steve
I would question Lehrer’s idea that there is, necessarily, a bias against null results in science. The effect he cites is probably real (i.e. people are less likely to submit or publish null results), but there’s a contrary effect which he doesn’t mention, which probably inflates the number of null results out there.
A study with a lot of error can include either systematic error, or nonsystematic error. Systematic errors may increase the chances of a false positive result (type I error), but nonsystematic errors will, other things being equal, increase the likelihood of type II errors (false negatives, concluding that there is no effect when there really is). Nonsystematic error is, I would bet, a really, really important factor in a lot of studies. It will happen whenever you have an inexperienced undergrad, or a semi-experienced graduate student, taking the measurements, it will happen whenever you have unpredictable weather creating day-to-day variation in your results, it will happen whenever you have a researcher using a method they haven’t had a lot of past experience with, it will happen whenever you have variation in nutrient content, disease pressure, or lots of other natural factors that are hard to control for. And often times, when you try to avoid systematic errors, you may increase the likelihood or random noise.
There’s not really much that can be done about eliminating this kind of ‘noise’, particularly when you’re doing work in the field as opposed to the lab or greenhouse, and when you’re looking at a natural system that is highly complex and affected by lots of different factors, but it probably does make it more difficult to detect effects and differences that might really exist in nature, and thus makes it more likely that we will see null effects, when in reality there is an effect.
In general, I think the Bayesian approach has a lot going for it.
The most difficult, single obstacle a software designer faces is the user that refuses to learn how to use an application, who will sooner rather than later abandon using it because “It doesn’t work!”
If there is a single general point to the discipline of the scientific method, it is to mitigate (100% removal: impossible?) the user bias and error. That’s why repeatability is on the short list of requirements. That’s why it is never a waste of time for a scientist who, upon discovering a valid reason to doubt, revisits decades- or even centuries-long “settled” science.
This from the quoted article sums it up best for science in general, emphasis added: The material basis of heredity and development is not something we “choose” to believe: it’s something that’s been forced on us by repeated findings of many scientists.
My good Karen,
Coming as this post does so closely, so voluminously, and so suggestively on the heels of your post here, I should point out several things.
First, the editors of The New Yorker, not Jonah Lehrer, created the article title “The Truth Wears Off: Is there something wrong with the scientific method?”. Anything extrapolated from that misunderstanding is concomitantly misleading, deliberately or not. In point of fact, Lehrer’s contribution seems little more than a retread of his earlier offering two years ago, “How We Decide”, discussed at some length here, dealing not at all with the scientific method itself but rather with the human psychology of decision making, that is, as our good Lance might, deciding how to decide on climate science.
Second, my comment here stems only from Kuhn, that is, that the one thing that seldom progresses by the scientific method is science itself, instead leaping from one popularly agreed upon, if at the time apparently scientifically derived, consensus of opinion to another finally able to break through the former popular hypothesis’ psychological-ideological-political sclerosis of defenses against any revision. Hence your reflexive poo-flinging about “Holocaust deniers”, and hence our good Lance’s and your remarkably silly valuation of the agreement among scientists, climatologists or otherwise, as if such agreement itself, any more than the agreement among a clique of girls at a junior high lunchroom table that a girl at another table is substandard and non-clique-worthy, constitutes some sort of enhancement of the scientific method, much less even being scientific in any way itself. Of course, it is not, it is purely social, nothing else, and, as the case with Heliobacter pylori, among many others throughout the history of science demonstrates, many identically wrong conclusions added together never manage to sum up to a correct one but rather, to the contrary, usually stand in the way, unfortunately if inescapably, of better and more truly scientific conclusions being given greater voice.
H. M. Stuart
Alexandria
It is not that a bunch of scientists are right, rather that they have all collected similar data or done similar experiments which confirm their hypothesis. Teenage girls function based upon feelings. So do some adults.
Steve
My good Steve,
Actually it is, and only is whether a bunch of scientists are right.
What you are misrepresenting above is my criticism that the citation alone of agreement among scientists is no more meaningful than agreement among a clique of teenaged mean girls.
If one is going to cite science, then one must cite each and every scientist, independently, and how his efforts separately and independently address the issue in question.
To take the shortcut of appealing merely to “agreement among scientists” is to betray faith-based thinking, not scientific thinking.
H. M. Stuart
Alexandria
“If one is going to cite science, then one must cite each and every scientist, independently, and how his efforts separately and independently address the issue in question.”
That is not how it works. If nothing else, it takes too long and would make science grind to a halt. Other than very leading edge science, there is usually a consensus opinion around which people work. There will always be people supporting or disagreeing with that consensus. People will publish papers to support or refute that consensus. Those working specifically within that field should know all of the varied research and opinions. However, those not doing the research dont need to know the work of every dissenter. We need to know the consensus and use that as our current best understanding.
What is really important is that we be open to new findings and adopt as the consensus opinion changes. Fortunately, in practice, this works quite well. When one looks at research comparing hedgehogs and foxes, the foxes generally do better at using and implementing information.
So, on climate change, we know that there is a rough consensus on its existence and cause.. We think that about 10% of scientists disagree with that consensus (estimates vary). We know that those who agree with the consensus still have many questions and are continuing research that may disprove parts of that consensus. At that point, the consensus will change.
The area where I think there is no real consensus, is what to do about climate change. That is less science oriented and more of a political issue.
Steve
Steve, sort of.
In general, consensus can occur when the question is no longer scientifically interesting, or when the shift in thinking is all in one direction and there are relatively few holdouts and they are not explaining themselves well. Sometimes there will be a consensus that Einstein’s model is correct even though alternative explanations exist that have not been disproved (sort of an untested consensus). There may be people who disagree with the consensus, but it is rare for them to contribute to the discussion that begins with peer review. A prominent scientist who was no longer young when atoms were proposed died believing that atoms were a model only, and did not really exist. So far as I know, he never figured out a way to explain himself to others.
Some 2 – 3% of climatologists, and perhaps scientists in general (people in oil exploration excepted) believe that climate change will not be as serious this century as mainstream thinking predicts. (I don’t believe that anyone in the science community, well there is that prominent guy who has had serious dementia for 2 decades, claims climate change isn’t happening or that greenhouse gases are unimportant.) They cannot get their ideas through peer review, either because they don’t submit them, or because they cherry pick the data, use data obtained from misadjusted equipment, do not provide an explanation that sounds coherent to scientists, etc. It is common for them to attack the United Nations, as if somehow the UN gathered the data and hypothesize the explanations, rather than simply sifted through the work done by others to see what stood the test of time.
Sometimes everyone or almost everyone believes something, eg black holes exist, or sea level rise this century will be between one and two meters, and there is no consensus. A prominent physicist told me she was would accept black holes when there was a quantum theory of gravity, but years later told me that there were now enough lines of evidence that she had shifted. Now there is enough evidence on the mechanics of ice sheet dynamics this century that a consensus may be close or exist on sea level rise, and the prediction is the same, 1 – 2 meters.
Ironically, scientists pride themselves on being skeptics. But their definition of skeptic includes a willingness to change one’s mind. Now the word has been appropriated by people who for the most part can’t follow the science, have no desire to follow the science, yet know the science is wrong.
The consensus is likely to change, but on some points (evolution occur, climate change is happening and greenhouse gases are an important cause, nuclear is both safe and necessary for addressing climate change), it is unlikely there will be a change in consensus thinking. Well, in 2100+, nuclear power may no longer be seen as necessary, but it is unlikely to ever be seen as unneeded in 2011.
“If one is going to cite science, then one must cite each and every scientist, independently, and how his efforts separately and independently address the issue in question.”
That is not how it works. If nothing else, it takes too long and would make science grind to a halt.
My good Steve,
“Takes too long”? Is that a scientific parameter?
Anyway, you are correct. That is how how it works stops being science and becomes instead sociological/ideological/political/religious consensus external to the science, which can then promote or retard any further advancement of the science it wants to speak for or against depending on the relative power any given such sociological/ideological/political/religious consensus wields among scientists.
H. M. Stuart
Alexandria
“That is how “how it works” stops being science and becomes instead sociological/ideological/political/religious consensus external to the science, which can then promote or retard any further advancement of the science it wants to speak for or against depending on the relative power any given such sociological/ideological/political/religious consensus wields among scientists.” (HMS)
.
.
That is absolutely and undeniably correct.
Science has become as much a tool of the political class as religion was in the Middle Ages.
To the extent that we have a massive modern Corporatist state, it SHOULDN’T be surprising that much science has been corrupted by its being directly influenced/funded by business and government.
What the CRU email dump showed was the widespread use of “tricks,” one of which was “adding temperature to the data collected since 1998 to “hide the decline” in global temps, another was the spectacle of professional “researchers/scientists” exposing their pettiness and venality, in openly discussing the blackballing of others who weren’t able to replicate their data.
Tim Carney’s works on the subject are well-documented and well worth reading on that score.
So make this work for me guys. I cannot use any drugs, it is not scientific, unless I know every bit of research on that drug? How do we keep science from coming to a complete halt? How do engineers ever use the science that we have learned? At some point, you need to look at all the available research, and then make decisions based upon the best summation of that research.
Maybe you two should describe how it should work. :-)
Steve
“So make this work for me guys. I cannot use any drugs, it is not scientific, unless I know every bit of research on that drug? How do we keep science from coming to a complete halt?” (Steve)
.
.
You’re a physician and you’re not aware of the many abuses of junk science and decisions based on lobbying efforts?!
Ever hear of the Trasvlol disaster?
“A common surgical complication is excessive bleeding. In patients at high risk of bleeding, intravenous drugs are administered ahead of time.
“While lower-cost alternative drugs are available to reduce bleeding complications, pharmaceutical giant Bayer successfully penetrated the market with a drug called Trasylol® that costs about $1,000 per patient.
“If you wonder how this kind of price gouging occurs, large drug companies aggressively promote expensive new drugs to doctors, in some cases paying cash kickbacks so that the more expensive drug is used in place of an alternative of equal efficacy.
“In the case of Trasylol®, the results turned tragic.
“Despite data showing that Trasylol® inflicted severe kidney damage in animals, the FDA approved it for human use in 1993. Low-cost alternative anti-bleeding drugs are less likely to produce this lethal side effect.
“Soon, the same kidney side effects observed in animals were occurring in humans. One surgeon observed that the most common side effect seen in patients given Trasylol® was renal dysfunction. This surgeon then conducted a 20-patient study (not funded by Bayer) and found that 13 of 20 patients given Trasylol® had problems with kidney function after the surgical procedure.
“When the FDA approved Trasylol®, they did note that kidney toxicity was a problem. But Bayer lobbied the FDA hard, and by 1998, the FDA expanded approval of Trasylol® to cover all heart bypass patients.
“Sales of Trasylol® in 2005 hit $300 million, and Bayer envisioned a billion-dollar-per-year blockbuster.10 These kinds of profits provide an enormous war chest to lobby FDA officials to turn a blind eye, even while thousands of surgical patients were dying each year from kidney failure caused by Trasylol®.”
http://www.lef.org/magazine/mag2011/jun2011_The-FDA-Most-Heinous-Drug-Approval_02.htm
When results are bought and paid for they are generally NOT “science.”
They’re often umbrella’d under that term. . .“Junk Science.”
Junk Science has deadly effects in medicine, as with drugs like Trasylol being approved despite numerous warnings and actual science documenting its dangers and its equally true in all other areas where results are bought, paid and expected by business and governmental entities.
H M, I am curious whether you believe that natural laws can be explained, or if the world we live in just random events—sometimes apples fall, and other times they rise for no reason? What differentiates ideas you may accept (apples always fall) vs ideas you may not accept (greenhouse gases added to the atmosphere lead to more heat retention)?
Steve2, your points are important. Science clearly leads to understandings that prove technology (eg, when I first learned about the tunnel effect, I both “disbelieved” that it was possible and knew that this understanding had led to the tunnel diode).
Hector and Franklin, ditto re your points!
H M, I am curious whether you believe that natural laws can be explained, or if the world we live in just random events—sometimes apples fall, and other times they rise for no reason? What differentiates ideas you may accept (apples always fall) vs ideas you may not accept (greenhouse gases added to the atmosphere lead to more heat retention)?
My good Karen,
Please; you have no such curiosity at all. Your statement is merely one more insult mendaciously framed with poorly feigned innocence as an inquiry, as was your post
“Hey, kids, how many ridiculous alternatives to true belief in settled climate science can you list for us to make fun of?”.
You have already made it abundantly clear in your last several posts that the least thing you are interested in is the epistemological yield of climate science, or even of science itself, compared to its utility to you in differentiating yourself in some moral-ideological manner from others.
Please keep doing so. That’s what we’re here for.
H. M. Stuart
Alexandria
I’m not up to addressing the scientific questions raised here. But this discussion does relate directly to how we decide how to use resources, treat patients and other vital life issues.
If a drug is no longer considered, according to later research, nearly as effective as once thought, how, short of an FDA recall, does one stop thousands of clinics from continuing to prescribe it? I suppose eventually the insurance companies refuse to pay for it?
Advice about how to eat for different disease conditions has changed and may continue to, as more research gets done. Who sends the memo to the folks who don’t go to their clinicians for updates?
Etc.
This issue points to a problem with living our lives and organizing our economy, medical practices, dietary practices, agricultural practices, production practices and on and on, according to “the latest findings.” For so long many Americans and Westerners in general have placed a level of faith in science, perhaps due to a misunderstanding of it.
As HM points out, science itself may be the biggest beneficiary of these findings. Without turning the entire population against science (as many believers in, say, young earth creationism have), how do we inform people without turning them off altogether? Especially in the new “college is a waste of time” environment? (And BTW, who issued that memo/meme? It’s showing up everywhere.)
Elizabeth, re medical procedures, it seems to me that insurance companies could be more effective with a limited pool of money if they require evidence for the procedures they fund. I don’t know if this should come through legislation. Of course, one problem is that many in the public want medicine funded even in the absence of justification.
Medicine has more problems than some fields of science with the quality of their research (all that money wasted on Vitamin C), with the quality of their peer review, and with the speed with which preliminary results are shared with the public (and the quality of the explanations).
And look at all the money wasted on whether cell phones cause cancer, a result thought to be unlikely by essentially all scientists, when simple legislation requiring cell phones to not be used by drivers ever, and strict enforcement, would save so many lives.
I think it’s long been established that people fear big scary diseases they might contract at some time in the future more than they think it likely that they will personally have an accident from texting or talking while driving. More lives might be saved by checking to see if car tires have are inflated to the proper air pressure every week than by any number of medical research topics. Who would get funding to build a career by studying tire pressure, and how interesting it is, really?
The problem with requiring evidence for medical procedures and drugs before reimbursing for them is that it might bring the medical world to its knees. Before they shut down the office of technology assessment in DC back in the 80s, they had determined that at least 70% of common medical procedures and prescriptions had not been proven to work to any scientific standard. No wonder they shut that office down.
Evidence-based medicine – the very term should scare people, most of whom thought medicine was already proceeding based on evidence.
H M, No I am curious. That is why I deleted my response and posed a question, to try to understand what you are saying. Also, it would be a lot easier for me to try to understand, then respond to a coherent message about what you believe, than to do the same from an attack on what I said.
Cultural theory of risk says that we see the world through the distorted lens of our own beliefs and then use that distorted image to distinguish between the bad guys (you know, them) and the good guys (that would be us). We remember best what is said by scientists, or people who might have taken science, or who understand it even if they never studied it, when they agree with our preconceptions, and so call our understandings scientific. See for example, culturalcognition.net
I do the same, all humans do. Still, I do not limit my information sources to those who are like-minded; I have sources I depend on to contradict my thinking. I changed my mind on nuclear power in 1995 because I sat down with the information, and found those who agreed with my preconceptions clueless, while those who disagreed with me clearly knew what they are talking about. I do not belong the cultural outlook most accepting that climate change is happening and serious (egalitarian communitarian), yet those who were pro-nuclear whom I read in 1995 said that a MUCH bigger issue than pollution was climate change. I didn’t want that to be true, I already had problems I preferred, and solutions I preferred. But I was uncomfortable ignoring the understanding of people who clearly knew what they were talking about. So I changed my mind, and did the work to understand their concerns. I have changed my mind many times since, as I better understood what people were saying. And the science changes, as people learn more. I have a friend in her 80s who feels that if she doesn’t change her mind on something important once/week, she’s not paying attention. I don’t require quite that pace, but I do require challenge to my thinking.
I spend a lot of time working against groups saying We Good, They Bad, indeed, an upcoming article, which you will hear about when it is posted, speaks to that. I do this in a number of ways with a number of audiences. I am very short of perfect, but find amusing that you hone in on one of the very issues I focus on.
I am interested in the limits of science. I particularly am interested in what scientists, particularly those in the harder sciences (eg, cell biology, physics, climate science) say about the limits of science. I don’t find myself interested in what Kuhn says, certainly not enough to read him. The Wikipedia article basically presents an argument among science historians. Popper, from what I little I know of these people, seems closer to the thinking of scientists, with his emphasis on falsifiability, as that is a crucial aspect of science. The discussion about whether science is composed predominantly of breakthroughs or iteration may be important, but is not important to me at this time. That doesn’t mean the subject isn’t interesting, but I already feel like I’m putting in 25-hour days.
Perhaps you would like to explain your thinking in a post?
HM, I am curious whether you believe that natural laws can be explained, or if the world we live in just random events—sometimes apples fall, and other times they rise for no reason?
H M, No I am curious. That is why I deleted my response and posed a question, to try to understand what you are saying. Also, it would be a lot easier for me to try to understand, then respond to a coherent message about what you believe, than to do the same from an attack on what I said.
My good Karen,
No, you are merely piously disingenuous – no person who is being genuine these days asks another whether he believes apples sometimes do other than fall – although such behavior would in no way preclude your believing every bit of your narration of your virtues, above, and of course I would have it no other way.
I posted the Wikipedia link to Kuhn rather than the Amazon.com link for its greater informativeness. When I read Kuhn, in 1970, he was required reading across the board in the hard sciences like geology and biology as well as in the social sciences. Popper, on the other hand, I encountered only in philosophy of science courses and, unlike Kuhn’s empirical starting points and bases of thinking, Popper’s unfortunately begin and end in stubbornly held tautologies. Kuhn, of course, was a Harvard-trained PhD. in physics; Popper’s doctorate was in psychology.
You are, as you say, interested in the limits of science, but not enough to read on what those outside of science have to say about its virtues and limitations, which fairly well delineates what you want to know and what you want not to know.
Be that as it may, and since you have already challenged me in front of our readership to prove that I do not beat Madame Stuart, perhaps this response to a previous Author will help:
H. M. Stuart
Alexandria
H M, that explains it. At the time you were majoring in geology and biology and reading Kuhn, I was majoring in physics. We did not read Kuhn. Nor a few years later when I got my master’s in electrical engineering did we read Kuhn. And I just checked Ned’s bookstore at UC, Berkeley to see if any Kuhn books are required for the fall—if so, they are not yet listed.
Since there are dramatic differences between the sciences, I’m also going to suggest IPCC”s working group 1 to you. http://www.ipcc.ch/publications_and_data/ar4/wg1/en/contents.html
From reading science in great detail over a long period of time, one begins to understand a little of how scientists think: how they explain to others and lay people what they know (with uncertainties—cause and amount of same, what they don’t know, what they would like to know). Only second to doing the science itself is reading the reports that come out of major groups responsible for explaining what is known…. National Academy of Sciences has also produced a number of reports saying the same.
Over time there is evolution in what scientists think: it took a few years to accept the idea of very rapid climate change. Until this century, it was assumed that it would be hundreds of years before the contribution to sea level rise from ice sheet melt passed the contribution from thermal warming of the oceans. But the basic beliefs are added to incrementally over time rather than being overturned on any regular basis.
My friend in the social scientists says Kuhn is very important. And worldview is clearly of importance to most who study the social sciences. Two friends insist they are scientists because they let their understanding follow from the facts, behavior they felt to be unusual in their fields.
H M, that explains it. At the time you were majoring in geology and biology
My good Karen,
I don’t believe I claimed to have majored in either.
Good luck in convincing anyone you are capable of separating facts from preconceptions and assumptions.
H. M. Stuart
Alexandria
It’s important to remember that climate change is not, and cannot be, a topic that you would address in the same standard, experimental way that you would test a question like ‘how fast would apples fall?’ That’s because we only have one earth, we can’t do a replicated experiment with multiple earths, and we can’t run experiments on the whole earth, nor can we isolate one little section of it and pump up carbon dioxide there and nowhere else. (Well, in truth, we are running a giant experiment on the whole earth at once, but that’s a regrettable reality, and it would be much better not to). However, that doesn’t mean we should conclude that climate change is not real. Because that would be an unsupported conclusion as much as anything else. Rather, it means we need alternative methods to address the question of climate change, and it means that we are never going to be quite as confident in any conclusion that we come to, as we are confident in the law of gravity. But that’s OK.
In general, I think climate change (and many other issues) are one where we really need the Bayesian approach as opposed to the frequentist approach. Bayesian approach is one which doesn’t use the language of ‘null hypothesis’ and ‘research hypothesis’ as much as it talks about prior probabilities and revising them in the light of each new piece of evidence that comes in. What should our prior expectation be, in the case of climate change? Should it be ‘the earth will get warmer’, or ‘the earth will not get warmer’? I would say that our prior expectation, before we even start looking at historical climate data, should be that higher carbon dioxide levels will lead to warmer temperatures. That’s because we know, mechanistically, that carbon dioxide absorbs infrared. Lots of previous, replicated experiments lead us to conclude, with close to a certainity, that carbon dioxide absorbs infrared, and we have machines that work based on that premise all the time. Therefore, it follows that a world with more CO2 should be warmer, unless we postulate some as yet unknown mechanism that will dampen that effect. It’s important, of course, to look at the historical data and see if the earth is in fact getting warmer, but that should be our default expectation that we set out to test. Setting ‘no increase in temperature’ as our null hypothesis, in this case would be a very bad idea.
When we do that, of course, we find that the historical climate data presents a real, but by no means conclusive trend. How you interpret that data is going to come down to what you choose as your prior expectation. If our prior hypothesis was ‘no effect’, then the climate data might not be enough to overturn it; however, if our prior hypothesis was ‘the earth will be getting warmer’, which I argue that it should be, then the data is certainly consistent with that expectation, and in fact should strengthen us in the belief that climate change is real.
Much of our thought process, and what we expect to be likely, comes down to carefully choosing the questions we ask. If we ask people, ‘Do you think the earth is getting warmer due to human-induced greenhouse gas production’, then they might not feel confident in saying yes. Whereas if we pose the question, ‘Do you think it is likely that as-yet unknown and un-thought of mechanisms are going to counterbalance the increase in greenhouse gases, and suppress any increase in temperatures’, then most people would, correctly, conclude that we have little evidence for that, and therefore that increased temperatures are more likely than not.
Hector, yes, it’s not possible to run the climate change experiment on a number of Earths to see if they repeatedly warm for given inputs, and in a predictable fashion. But there is an enormous amount of experimental results and paleoevidence that can sifted to figure out what is happening and what will happen. And scientists can compare temperatures on the moon and Earth, compare Venus’ temperature to what is expected, etc. Greenhouse gases producing global warming has not been in doubt in the lifetime of anyone alive today, but there is a detail or two after that.
Statisticians normally approve of climate work, or say that they really should be using more modern techniques, but the results would be the same. That means that it doesn’t “depend” on the null assumption.
The paleoevidence does produce pretty clear evidence for GHG producing climate change. Also, other forcings.
It’s always useful to check what scientists actually say. See for example http://www.ipcc.ch/publications_and_data/ar4/wg1/en/contents.html to get some sense of how scientists lay out what they understand, and how well, and what they don’t understand.
Well, obviously I believe in global warming. I think my point is, though, that I don’t believe it on the basis of historical climate data alone, nor on data about the temperature of Venus. My primary reason to believe in global warming is that we know that carbon dioxide, and some other gases, absorb infrared radiation, so we should expect that a world with more of them should be warming. Without that mechanistic basis, the case for believing in global warming would be substantially weaker, for the reason I mentioned. Ultimately I think what you set as your prior expectation is critical to how you assess each piece of evidence (on climate in the past, or on more recent data about yearly temperatures) as it comes in.
Hector, you picked one the most vital reasons! But you might be interested in reading IPCC to see why scientists believe what they do. While they at times accept one line of information, they feel more comfortable accepting information from a variety of explorations, a number of lines of evidence. The greenhouse property of some gases is crucial, but if there were no other corroborating evidence, they would feel more reluctant to be so confident.
As I say in the original post, medicine and some of the life sciences produce the least robust results among all the natural sciences.
Scientists in many fields of science, including climate science, tend to be pretty critical. Scientists are amused at the misappropriation of the word “skeptic”, as they tend to feel that their claim to integrity has been donned by those who are not particularly skeptical.
Hi Karen,
I definitely do appreciate your thoughts on this blog!
For full disclosure, I’m a graduate student in plant biology. My only direct interest in climate change, professionally, is that I’m interested in plant photosynthesis and metabolism, and curious to see how plants might deal with higher carbon dioxide levels. I’ve read some in the literature about plant responses to high carbon dioxide, enough to know it’s a complicated subject with a lot of potential constraints to plant responses, and it’s certainly not true what some say, that plants will just ‘soak up’ the excess carbon. Plant abilities to absorb the excess carbon are very limited. Beyond that, as a citizen and a biologist-in-training, I’m of course interested in climate change in general, but I can’t say that this is my area, and I trust the climate scientists to make accurate determinations. In general, I suspect the change in carbon dioxide (for plants, not for animals) is likely to be a more interesting and difficult to predict effect than increased temperature, and I’m very interested to see what the future holds. It’s likely that the increased temperature and increased CO2 may have conflicting effects on a number of plant taxa.
Hector, good luck with your work.
This is the main article I’ve read on high CO2: http://californiaagriculture.ucanr.org/landingpage.cfm?article=ca.v063n02p67&fulltext=yes
I had to skip all the science in order to understand it. Is it still current?
[...] Article in the New Yorker, Is There Something Wrong with the Scientific Method? [...]