You know peer review, right?
It’s the way academics check each other’s research papers.
It ensures that only the good ones are published and prevents the bad ones from getting through.
Right?
Wrong.
Peer review does precisely the opposite of what you think it does.
It prevents the good papers from being published, and ensures that only the bad ones get through.
Peer review is suffocating science.
If we want to reverse the stagnation of science over the last 50 years, then we’ve got to get rid of peer review.
Experimental
In his splendid article The rise and fall of peer review, Adam Mastroianni describes peer review as a scientific experiment.
We tend to think that peer review is just the way science works. But science had been working rather well for centuries before peer review became a thing.
When Newton published his laws of gravitation three centuries ago, his work wasn’t peer reviewed.
When Einstein published his laws of gravitation one century ago, his work wasn’t peer reviewed.
At least, Newton and Einstein weren’t peer reviewed the way we do peer review today.
Their peers did review their work: their fellow scientists heard about their ideas, read their papers and made up their own minds as to whether these new theories were revolutionary or rubbish.
The way we do peer review today caught on around 1973, when the journal Nature began to require it.
Yep. 1973. You’ll no doubt recognize the year from my article Why has there been no progress in physics since 1973? That’s a coincidence. Correlation is not causation. Pure coincidence. Right?
Anyway. If peer review is an experiment, shouldn’t we be assessing, after 50 years, whether or not it has been successful?
And if peer review is intended to ensure that only the good papers are published and the bad ones are prevented from getting through, shouldn’t these be the criteria for assessing whether it has been successful?
Does peer review prevent bad papers from being published?
Do the good papers get through?
How it works
Here’s how peer review works.
You do some research.
You’re pretty excited by what you find.
You’d love to write it up in simple, clear language, but you’re not allowed to, because the peer review process requires that you use opaque language that’s full of technical terms.
You’d love to write it up in an engaging way that communicates your excitement, but you’re not allowed to, because the peer review process requires that you do it in the dryest possible way.
OK, so you submit your dry, opaque paper to a journal.
You wait an interminable amount of time, because the peer review process assumes that your research is inconsequential, so there’s really no hurry to get it out there.
A couple of academics, the type who’d rather spend their time criticizing other people’s papers than doing research of their own, get back to you with their objections.
Generally, these objections follow the form: “That research you did. That’s not how I would have done it. I would have done it like this. Do it exactly that way I would have done it, and maybe it’ll pass muster.”
Now you have a choice.
You can redo your research exactly the way these academics would have done it. Assuming, of course, that the academics would have done it the same way as each other.
Or you can submit your paper to another journal and hope you get different academics to get back to you with different objections.
And at that point you have another choice.
You can redo your research exactly the way these other academics would have done it. Assuming, of course, that they would have done it the same way as each other.
Or you can submit your paper to yet another journal and hope you get yet different academics to get back to you with yet different objections.
And so on.
You can see how productive this process is for the advancement of science.
A few bad apples
But this process, however ponderous, prevents bad papers from being published, right?
Wrong.
With millions of academic articles published every year, there’s no shortage of bad science.
Indeed, there seems to be no shortage of fraudulent science.
Some scientists simply make stuff up.
Fraudulent studies don’t just make it into unknown journals.
They make it into respectable journals like Science, Nature and The Lancet.
In other words, they make it past peer review.
The thing is, peer review isn’t designed to detect fraud.
The academics who peer review papers don’t check the data in those papers to makes sure it hasn’t been made up.
How could they? To check the data, they’d have to reproduce the research in its entirety.
Peer review can’t detect fraud.
It can’t detect error.
It can’t even detect sloppiness.
Peer review can’t prevent bad papers from being published.
One funeral at a time
If peer review does no good, at least it does no harm, right?
Wrong.
The physicist Max Planck proposed a principle that’s often paraphrased as: “Science progresses one funeral at a time.”
He knew that scientists are not very good at changing their minds.
They learn theories when they’re young. They invest decades of their lives in those theories. When they’re old, they’re so invested in those old theories that they’re not so interested in any new theories that come along. They don’t have the decades left in them to learn these new theories. So they dismiss the new theories.
The new theories slowly gain ground, not through the old scientists’ changing their minds, but through the old scientists’ dying off.
Science progresses one funeral at at time.
What could be worse than having to wait an entire generation for any new theories to gain ground?
Well, what would be worse would be giving the old scientists the power to prevent any new theories from being published.
And that, of course, is precisely what peer review does.
Peer review requires young scientists to get permission from old scientists to publish anything new.
It’s difficult to overstate what a disaster this is for science.
If you have to get permission from those old scientists – the ones who, according to Planck, never change their minds, the ones who, according to Planck, must die before science can progress – then you can forget about publishing anything new.
The only research you’ll be able to publish is pedantic elobrations of the entrenched theories.
And since, in academia, your career depends on publishing papers, this is the only research you’re incentivized to do.
The chilling effect of peer review is to encourage incremental improvements on tired, old theories, while discouraging bold, new ideas.
Peer review doesn’t just prevent good papers from being published, it prevents good research from being done.
Silencing Einstein
Let’s take a step back.
Which is more important?
Preventing bad papers from being published?
Or letting good papers through?
Well, this is an easy one.
Do you remember all those bad papers that were published in 1905, long before peer review was put in place to save us all from bad science?
No, of course you don’t.
I bet you can’t name a single bad paper published in 1905.
All the mediocre monograms, all the misconceived experiments, all the mistaken theories... all are long forgotten.
Can you think of any good papers that were published in 1905?
I can. Four papers come to mind: one establishing the existence of photons, one establishing the existence of atoms, one introducing the theory of relativity, and one establishing the equivalence of mass and energy. You know: e = m ⋅ c2.
Each of these four papers, published by Albert Einstein in 1905, was truly revolutionary.
None of them was peer reviewed.
If they had been, Einstein’s so-called “peers” would no doubt have prevented their publication, complaining, correctly, that his theories didn’t accord with the well-established laws of motion, matter and light.
So which would have been more important?
Preventing all the long-forgotten bad papers of 1905 from being published?
Or letting Einstein’s papers through?
Here’s a radical idea
OK, so peer review doesn’t work.
It doesn’t prevent bad papers from being published.
It does prevent good papers from getting through.
Worse, it prevents good research from being done.
But what’s the alternative?
We can’t let just anyone publish just anything, surely?
We need some kind of authority to put a stamp of approval on good science, surely?
And we need some kind of authority to keep the bad science from corrupting our impressionable minds, surely?
Well, I’m not so sure.
In practice, this censorship might be preventing today’s Einsteins from publishing their own bold, new ideas, indeed, discouraging them from ever embarking on what might prove revolutionary research.
In practice, that stamp of approval isn’t just giving us confidence in good research, it’s giving us undue confidence in bad research, too: in trivial research, in sloppy research, in erroneous research, in downright fraudulent research.
Which leaves me wondering...
Why, exactly, can’t we just let anyone publish anything?
Open
Since I launched The Last Theory to explore Wolfram Physics, I’ve heard the same criticism of Stephen Wolfram over and over again.
I hear furious complaints that none of the research that’s come out of The Wolfram Physics Project has been peer reviewed.
I could respond by saying that that’s not actually true: Jonathan Gorard, for example, has published papers on the Wolfram model in peer-reviewed journals.
But I’m not going to take the easy way out.
The fact is, Stephen Wolfram has, very deliberately, circumvented the peer review process, publishing many thousands of pages of articles and books, and many hundreds of hours of podcasts and videos, without asking anyone’s permission.
And that’s a good thing.
The fact is, none of these articles, books, podcasts or videos would have passed peer review.
For a start, peer review is designed for scientific papers. Like, you know, on paper. Articles on this new-fangled nonsense called the web? Not on paper, so not allowed. And books? podcasts?? videos??? Definitely not permitted as a way to promulgate science.
Also, peer review is designed for a very specific form of research written up in a very specific way.
It’s very, very important that scientific papers be written in language that’s impenetrable to anyone other than a few dozen academics steeped in the minutiae. If it’s written in language that’s so simple and accessible that anyone can understand, then it’s just not science.
And it’s very, very important that scientific papers be crammed with the equations and derivations of the established, mathematical paradigm. If it’s crammed instead with the algorithms and simulations of a new, computational paradigm, then again, it’s just not science.
Stephen Wolfram was right to put out his ideas on the web.
His articles, books, podcasts and videos have inspired others to pursue his ideas, even if they’re not scientists.
They’ve inspired me to pursue these ideas, and I’m definitely not a scientist.
I love this open way of doing science.
Well, of course I do. I’m the founder of the Open Web Mind. Of course I love all things open.
Wolfram Physics might prove to be a true model of our universe.
And it might not.
Either way, Stephen Wolfram’s putting it out there, unsanctioned, uncensored, unobfuscated, unfiltered, in all his thousands of pages of text and hundreds of hours of talk, has given us all the opportunity to explore that possibility.
Peer review would never have let these ideas see the light of day.
Try not to breathe
I’ll say it again.
Peer review is suffocating science.
If we want to reverse the stagnation of science over the last 50 years, then it’s got to go.