Most scientific papers are bullcrap.

This started as a post to complain, because I lost several hours of my life researching the sleep hormone melatonin for a blog post before deciding that I couldn’t trust anything that I read.

But the post got bigger and more uncomfortable, because science has gotten political. You are either pro-science or anti-science, and there isn’t much market for being a nuanced critic of any field, least of all a critic of science as a whole.

But screw it.  I read scientific research for my work, and scan through dozens of papers for each entry in this blog.  And people who read what I write care about science, and are (hopefully) adult enough to grapple with the truth that:

1. Most scientific papers are wrong
2. It’s possible for entire fields of science to run off the rails in major ways, for decades at a time

This sounds depressing.  I roll my eyes *a lot*. But there is hope.

First, a lot of the worst stuff can be filtered out pretty quickly, if you know how.

Second, to answer the first question that pops into people’s heads: Yes, global warming is real.  And it’s possible to distinguish good scientific consensus (as with global warming) from temporary scientific insanity, without resorting to simply polling the scientists.

Third, science actually progresses faster when it lets some of the bad ideas leak out with the good.  At any moment, a fraction of the literature *should* be wrong, because if we waited for perfection nothing would ever get published.^[1]

I want to talk about all of these things, because getting this stuff right matters to me.  But to do this abstract topic justice, I need a guiding metaphor.

So let’s put big ideas aside for a moment, and talk about canals.

Atlas of Mars by Giovanni Schiaparelli, 1888

In 1877, the Italian astronomer Giovanni Schiaparelli published a map of Mars. His map different in a significant way from others at the time:  the planet’s surface was crisscrossed by lines, each rod-straight and up to thousands of miles long. Schiaparelli called these lines “canali”, Italian for “channel”. This was incorrectly translated into English as “canals”.

Mars was little more than an orange blur in telescopes of the day, and Schiaparelli’s maps were met with skepticism.  In the late 1800s, telescopic images could not be photographed; maps were hand-drawn, trusted based on the reputation of the astronomer.  Schiaparelli possessed a stellar reputation, but his claim of “channels” were met with skepticism – natural waterways meander and curl, following rolling topography.  Straightness caused concern. Could Schiaparelli’s optics be somehow damaged, misleading him? Was there something wrong with his telescope?

Yet in subsequent “oppositions” – occasions when Mars is closest to the Earth and visible all night – astronomers stationed at the most powerful telescopes in the world stepped forward.  They too saw Schiaparelli’s canals.

While some astronomers held to their skepticism, acceptance of the canals in the field gradually grew.  Science popularizers such as the gentleman astronomer Percival Lowell spread the notion that not only were the canals real, but they were evidence of intelligence on another planet, as only an advanced civilization could create such unnaturally straight lines.  Support for the idea of life on Mars grew to such strength that the naturalist Alfred Russel Wallace wrote an entire book disputing the notion.  Yet even Wallace was convinced the canals were etched into Mars’s surface, writing that “so many other observers have now seen them that the objection of unreality seems no longer valid.”

Let’s fast-forward to today.  We know with certainty, from data captured by the Mariner 4 probe sent to Mars in 1965, that there are no straight lines that cross the Martian surface.  There are no canals.

Look, ma, no lines!  Image of Martian surface, Mariner 4, 1965.

Scientific consensus was wrong.  Utterly, spectacularly, publicly wrong.  For an entire generation.

How did this happen?

It’s in the nature of science that the first glimpse of anything is through a blurry lens.  For Schiaparelli’s images of Mars, this was literal truth.  Yet in any field, each new observation arises from stretching the edge of what’s visible.  And inevitably, sometimes scientists stretch just a bit too far.

In 1903, a group of astronomers proposed a new interpretation for Schiaparelli’s canals: they were optical illusions.  The scientists suggested that the mind naturally threads connections between blurry but real features of the Martian surface, so the lines existed only in the brains of observers.  At the time, optical illusions were still relatively new to science, yet to these astronomers this radical interpretation seemed far more plausible than canals on Mars.

To test their idea, the researchers constructed discs reproducing features of the Martian landscape, without canals, and asked groups of schoolchildren to draw what they saw from distances of twenty to sixty feet away.

Their results gave astronomers pause.  Children at intermediate distances from the discs – not too close, yet not too far – strangely drew lines between the features even though there were none in the original drawings.  A bit of blur – just the right amount – seemed to be essential to finding the lines.

(Left) A drawing of Mars presented by researchers to schoolchildren, and (right) a reproduction of this drawing from a distance of 34.5 feet.  From “Experiments as to the actuality of the ‘Canals’ observed on Mars.” Monthly Notices of the Royal Astronomical Society 63 (1903): 488-499.

Intriguingly, astronomers began to realize that the canals were more difficult to see when the skies were clear and the wind was still.  As telescopes continued to improve, images of Mars became clearer, and the lines less visible.  By 1907, when a Wall Street Journal editorial ranked as one of the year’s chief events “the proof by the astronomical observations…that conscious, intelligent human life exists upon the planet Mars”, some of Schiaparelli’s most respected supporters began to recant.

Two drawings of the Elysium region of Mars.  The left, based on photographs taken from 1909-1926; The right, based on maps by Schiaparelli from the late 1800s.  Some of the lines turned out to correspond to actual topographical features on Mars, while others were pure fancy.  From Sagan, Carl. “On the nature of the canals of Mars.” Nature 212 (1966): 117-121.

The story of how wrong ideas form and fade in science is ultimately a story about the tools the scientists had access to.

For the illusion of canals to exist, telescopes had to have just the right level of resolving power. Mars just the right brightness in the sky.  Countries in Europe had to be in the middle of their own canal-building frenzy, to supply a plausible interpretation for what a perfectly straight line across a planet could be.

It seems impossible that all of these things could go wrong at once.  Yet any experiment has only one way to go right, and thousands of possible ways to blunder.  With so many routes to delusion, its no wonder that even wary scientists find themselves tired at the end of the path, embracing a phantasm as though it were truth.

Yet the impossibly random sources of these errors is also a boon to those teasing apart what’s real from what’s fake, because artifacts rarely manifest the same way for two different tools.  When we see the same result using different approaches, we can have a high degree of certainty that it’s real.

This is the key:  When science goes bad – say, a belief in the superiority of a specific diet (pick your favorite) – you can generally blame the problem on having access to only one type of tool.^[2]  Having independent scientists agree is not enough, if they are all seeing the world through the same blurry lens.

Which brings me back to global warming.

A map of the earth’s temperature in 2015; color indicated unusually hot (red) or cold (blue) regions.  Image from Goddard Space Flight Center, via Wikipedia

To gain confidence that global warming is real, non-experts cite polls of experts: More than 97% of climate scientists agree that humans are causing the earth to heat.  That’s a shocking level of consensus for a group whose members revel in proving each other wrong.  It’s also far greater consensus than Schiaparelli ever amassed.

Yet reality is not a popularity contest.  And for those of us who know that science is occasionally prone to fits of fancy, a message of “trust the experts” feels frustratingly tenuous.

A more sophisticated approach asks:  What kinds of tools are scientists using to see this reality?  How many separate lines of evidence are there that agree?

For global warming, here is a partial list of data that support the idea that the Earth is undergoing a temperature rise unprecedented in global history:

• Land surface temperatures increases
• Sea temperatures increases
• Arctic sea ice melt
• Glacier melt
• US, European, and Australian climate models ^[3]
• Jet streams moving towards the poles
• Earlier breeding seasons of animals
• Longer growing seasons for farmers
• Plants distributing to higher elevations
• Increase in record hot days, without an increase in record cold
• Antarctic ice core records

I could go on, but you get the idea.  The tools used to measure the date of emergence of the Melbourne butterfly are different than those used to detect the loss of ice in Greenland.  Each of these measurements is like a new kind of telescope.  And while each measurement may be wrong independently, it’s nearly impossible for such different techniques to be wrong in the same way, together.

The story of global warming shows that we can know things, and that’s important.  But we also should remember Schiaparelli, who taught us that the road to knowledge is full of detours. Science is made up of brave and foolish men and women who insist that, through superior intellect and sheer tenacity, they will become the first to see something no other human has seen.  Mostly, they are wrong.  But maybe, just maybe, one of them might find an advanced civilization on the planet next door.

And then all mistakes will be forgiven.

1893 Soap Ad, attributed in text to an unnamed Chicago newspaper. – Contemporary Astronomy – second edition, by Jay M. Pasachoff, published by Saunders College Publishing 1981

1 I made a rough model once assuming that scientific progress followed the Pareto Principle -being 80% correct takes just 20% of the effort of being perfect; if making 100% sure your paper was right took 5 years, being 80% sure took 1 year.  Using that simplistic model, I found science progressed fastest when about 15%-20% of published work was hooey.  At that error rate, people can still publish rapidly, but others won’t invest huge effort reproducing crap research.  Unfortunately, I believe we are way over this target; in some fields we are closer to 85% hooey than 15%.  But that’s a discussion for another time.

2 Nutrition and health have a particularly acute problem, because they are ultimately interested in how long it will be before people die.  This is not only a very limited lens, but it takes an awfully long time to make a measurement.

3 Even the major computer models of planetary climate were each developed separately, so while they each err in their own unique ways, when they agree it is notable.