The peer review system, so important to the bolstering the voice of the climate establishment and suppressing dissent, is broken. Not that it was perfect and somehow got wrecked, but that it was never stringent or transparent in the first place. As the force of money, power, and reputations was ramped up, it was an eminently corruptible system, and thus it has become. Seriously, what other profession would call unpublished comments by two unpaid anonymous colleagues “rigorous”?
Dear IRS officer, my tax return was audited by two accounting friends I won’t name, and they say it’s right. OK?
Nigel Calder (a former editor of New Scientist) recently discussed the merits and failings of peer review and pointed at a couple of interesting articles in The Scientist. Not surprisingly, it’s not just climate science where peer review is up-the-creek. Other branches of science are subject to the same petty personality squabbles in a system where no one really gets much benefit from doing a proper honest analysis of their competitor (or compatriots) work.
I Hate Your Paper
Source: The Scientist. The story of how some journals are trying to fix peer review.
Suggestions include ways of allowing authors to carry reviews of a submission from one journal to another, posting reviewer comments alongside the published paper, or running the traditional peer review process at the same time as a public review (bring on the blogs).
…several journals, including Biology Direct, …have decided to eliminate anonymity from the peer review process altogether. “Under the Biology Direct model, everything is transparent, and everything is in the open,” says Eugene Koonin, one of the journal’s editors-in-chief. Authors are responsible for choosing their own reviewers from the journal’s extensive editorial board of more than 200 scientists, and must find three willing reviewers for their manuscript to be considered. This process eliminates “the potential for irresponsibility in the anonymous approach to peer review,” Koonin says, adding that upon acceptance of a paper “the reviews themselves are published alongside the paper for everyone to read.”
I was surprised to hear that BMJ (British Medical Journal) also now have a fully transparent system. ( BMJ is one of the Mr Big publications; it has an impact rating of 14.) The other journals using transparent review apparently haven’t suffered either; their impact ratings have increased.
Problem #2: Peer review is too slow, affecting public health, grants, and credit for ideas
And see my post yesterday for a case study in concerted slowness…
Problem #3: Too many papers to review
Some journals publish the prepress version, allow comments and reviewers to add their thoughts, and then publish in the journal, but others point out that a multitude of prepress versions may be ignored by most people bar the competitors who may submit unfavorable comments.
Peer review isn’t perfect— meet 5 high-impact papers that should have ended up in bigger journals.
The Scientist: Breakthroughs from the Second Tier
Nature has an impact rating of 35, but a paper describing BLAST – (bioinformatics software for searching sequences of DNA or proteins) came out in 1990 in a minor journal (J Mol Biol, rating 3.9) and went on to gain not just dozens of citations, but nearly 30,000.
The article goes on to list 5 papers that each got more than 1000 citations but didn’t make it into big name journals (many were rejected by the big name editors, showing just how often seminal papers can be missed by people who’s specific job is to spot them). The papers are mostly from molecular biology and life science, but the process relates to climate science even more so (no one is saying the debate about genetics is settled).
The Editors, what about the editors?
I don’t know that we have to enforce anonymity or total transparency of reviewers. Ultimately the responsibility for the decision ought to rest with the editors. If other reviewers make nasty baseless comments, surely a dispassionate, well informed editor would see that and get another opinion, or recognize spurious arguments and publish the paper anyway. Editors have been wimping out of their responsibilities by pretending that the decision rests with the reviewers. Convenient excuse, eh?
But the real problem might be that Editors themselves are not free of personal motives or conflicting interests. When most of your subscribers, advertisers and supporters all prefer one scientific theory I guess it’s no surprise that editors prefer to publish stories they can all cheer about. Let’s face it, unfunded skeptics can’t afford to subscribe to many professional journals, so they’re not exactly a top target market for publishers. An impartial journal serves humanity better, but “humanity” is not subscribing — government funded libraries, institutions, departments and grant-seekers are.
Meanwhile, here in the free-for-all unfunded skeptic world, a peer-review system has spontaneously emerged. When people send me contributions I send them out to one or two trusted experts (and I can always use more, so if you don’t mind reading an article very occasionally and want to drop me an email or comment mentioning your specialties, it might come in handy sometime, I do appreciate advice).
PS: I’m awol this weekend, so quiet for a couple of days. Cheers 🙂
PPS: If you are in Australia and want to make a difference at the Election on Sat Aug 21, I know people like Dennis Jensen (Perth), Steve Fielding (Melbourne) and Climate Sceptics (everywhere) would love any help they can get handing out How to Vote cards. These people have really put in a lot of work, and in some cases, taken very real risks to serve us all. This election promises to be very tight. Thanks…
have a nice weekend Joanne – you deserve it.
Totally OT – I’ve just finished reading the Zero Carbon Australia Stationary Energy Plan Synopsis, a proposal to make Australia 100% powered by renewable energy within 10 years.
http://www.energy.unimelb.edu.au/uploads/ZCA2020_Stationary_Energy_Synopsis_v1.pdf
Here’s their reason for the need to spend $370 billion to go totally renewable!
10
Of course!
Having a co-member of your Mutual Adoration Society review your work is called peer review. It is a basic rule because it places you in a situation where you can reciprocate and fuss over their papers.
Complaints of quality of life issues fell sharpley after Colonel Sanders at KFC was put in charge of the henhouse. The papers coming from the chickens were not peer reviewed and had to be discarded.
10
Who needs peer review when NOAA can dream up this doozy
NOAA can dream all it likes but Susan Bohan tells a different story
It seems that NOAA has been fiddling while lakes are burning
10
The problem lays in monopolization. If a single or only few journals or a single scientific society working as a gild that controls the journals keep control over a research branch then the process must degenerate regardless the formal procedures. That cannot be solved without a competition of various scientific “schools” and journals that compete for attention of readers and for the authors who can attract the attention.
10
To Janama 1:
Mankind stroke its limits of grow before the 1st industrial revolution as grow meant overburden for nature in the time. Then people came with substitute for soil and ecosystems exploitation – that means energy based on mineral carbon. It brought enormous growth of productivity and wealth along with liberation of people from drudgery and from feudal owners of the soil and, on top of it, liberation of soil from overburden. Now they want us to stealth the energy from the ecosystems again to detriment of nature and re-establish dominancy of soil owners.
10
I’ve commented on this peer review issue before and think in general an open transparent process where the reviews are published is best. Obviously there can be modifications of this (eg. the reviewer can have the option of hi/her name not being published etc ).
But the reality is the the traditional Journal is under more competiton ( like newspapers ) from the “new media”. This competition will reduce the intellectual/academic snobbery that has existed in some circles.
I have seen people now writing a paper and just putting it up on the web and asking for critical comment — from anyone.
Students are now taught how to correctly reference information they get from the internet , not matter what the source. In areas like IT this has to be done because of the speed of change in the industry — they cannot wait for a year or so to get information published in the traditional way.
So the Journals have to adapt along with their peer review proceedures , whether they like it or not.
10
I have always been an advocate of open review in science – in concept.
At its heart, science is all about ideas and the interpretation of observations – expanding our imaginations to explain the world around us.
This is a process that requires the sharing of ideas, and debate over what observations might mean, based on our previous understanding. So it strikes me that science would be so much better off, if that process was visible and accessible to anybody who was interested.
But of course, that is a simplistic position. Very little “pure” science is done just for the sake of curiosity. There is invariably an element of, “show me the cash”. It always comes back to funding, in an environment where time equals money.
So, as important as the method of peer review would seem, it is only a symptom of a much larger problem – the way the research is funded in the first place.
The current funding models are very efficient at producing what the funding providers want, whether it is a new drug, or a new political system.
Consider: Michael Mann has retained his position at Penn State, not because of his research (which was dodgy to say the least), but because he is extremely good at producing funding grant applications – he is the best marketing rep/account manager that Penn State has – and like any good marketing rep, he will try to tilt the playing field to his own advantage.
The games played in the peer review process were a by-product of vigourously competing for funding to produce research that carried the message that the funding providers wanted. In military terms, the Team were merely using a dubious peer review process to protect their flanks.
10
janama: #1
I have a better idea. Let’s give these “dedicated people” sufficient funding to build a large eco-dome somewhere, and let them experience their preferred lifestyle on their own for a while, but without the benefits of any external economic or infrastructure support.
If they can demonstrate that their ideas are sound, after say, a generation or two, then we can all join in and be much the better for it.
10
Simple post your draft on Internet, and let everyone have a crack at.
Good insight can come from the most unlikely sources.
10
Rereke Whaakaro
The Amish already do that, funny I haven’t noticed the Greens(or AL Gore) clamouring to join them.
10
What has been demonstrated to work in the Open Source community is publish everything and publish often. Drafts should be published ASAP, and comments on the draft should be published too. It’s no great shock that drafts will have errors and these can be fixed such that a new draft is issued. Eventually, all the obvious errors are cleaned up and we can rule a line and call this the first “real” revision. Anyone viewing the paper would see the latest revision, and the latest comments, unless they deliberately step it back to study historical drafts.
I support the idea of anonymous comments because nobody wants the emperor’s wrath when they point out he is naked. Sometimes anon comments are the only way for new ideas to get out there. However, anon comments also tend to be filled with rudeness and general BS. This is where you need a reasonably neutral third-party “editor” to sift the comments, and delete out anything that is too rude or trivial or obvious spam, and clearly mark the anonymous comments as distinct from those with a verified “real name” author (so readers can quickly narrow down the comments that they feel are important to them). Some of this could be automated, some will require a human editor.
Finally, I suggest a system of “commendations” (which cannot be anonymous) where readers can flag an article as “worth reading” and the real names of the people making the commendations are linked to the article in some way. A lot of what the “peer review” system attempts to achieve is merely a way of indexing the most important articles on a given subject by hiding those articles considered of lesser importance — however, every index ranked by “importance” is also a subjective thing! The commendation system with real names allows you to find other readers who have similar opinions on what is worthwhile.
10
Rereke Whaakaro and Binny –
it’s not about changing lifestyle – these guys genuinely believe they can power the whole of Australia with renewable energy.
They have 4 sources of power
1 – wind – 6500 new 7.5MW wind turbines – these babies are huge!! they are being TRIALLED in Germany. To be erected at 22 sites throughout the country.
2 – solar thermal – same system as the new trial solar system with hot salt being TRIALLED in Spain. They hope to produce a 20 – 50MW power station in Spain. These guys are proposing we built 12 sites around the country with each site containing 13 x 217MW solar thermal power stations for a total output of 3.5GW!
3 – Hydro – incorporating the Snowy and Tasmania into the system – they propose no new hydro.
4 – Biomass – these are backup for the solar systems and will consume wheat stubble!!! they intend to generate 15GW from this – considering the whole biomass (bagasse) of The Condong and Broadwater sugar mills produce around 60MW they sure are counting on some pretty huge wheat crops.
The plan would generate 325 terawatt hours of electricity a year, meeting the nation’s entire power demands in the year 2020
The whole this is a joke but their premise for doing it in the first place is even more of a joke!
But guess who has offered his support for the project? Malcolm Turnbull {rolleyes} – guess who is luke warm about it? Martin Ferguson 🙂
10
If I was able to vote in your election I’d do the right thing but I’m a (Scottish) Pommie.
Go Aussies!
10
janama; you raise an interesting point; namely whether the renewable advocates actually believe they will work; if they do they must have incredibly blinkered vision given the great amount of evidence of the failure of renewables over the last 15-20 years in places like california, Spain, italy, Portugal etc; whenever I ask for proof of a nation which is powered by renewables the stock answer is Iceland which, as I’m sure you know, has no wind or solar, the darlings of the greens, but sits on top of a volcano with some hydro to back up; in short Iceland is unique and is not an emblem of the renewable hoax.
I rather tend to think that the motives are more insidious than a genuine belief in renewables [and when you think about it it doesn’t matter whether their belief is genuine, it’s still going to ruin the country]; I think the Turnbulls have a buck at the frontal cortex driving them forward, plus we should not underestimate the influence of the Eastern suburb doctor’s wife/yuppie/urban elitists peer group pressure effect on him; that is who he mixes with and with whom he presumably shares values; of course there is an even more sinister Turnbull conspiracy out there:
http://barnabyisright.com/2010/04/28/rudd-destroys-his-ministers-beliefs/
But I think Binny is on the right track with the reference to the Amish, which I hadn’t thought of; the noble primitive informs much of the green ideology and sensibility; in fact Clive Hamilton, the noble twit, has described in great detail, in Growth Fetish [now endorsed by that amiable hypocrite Dick Smith], a pastoral paradise where the workers can toil to natural rhythms and have a Kelly McGillis in every barn.
10
Off topic but it looks like the “Team” have a problem ( a BIG One )
http://www.climatechangefraud.com/climate-reports/7491-official-satellite-failure-means-decade-of-global-warming-data-doubtful
10
Oh so parts of the Wisconsin area did not reach 612 deg F then. Hmmm
10
How about Orbitgate?
10
The two greatest works in scientific history – Darwin’s Origin of Species and Newton’s Principia were privately published in book form by the authors without any formal peer-review process.
10
omigod! shock, horror…is this credible! well it has been published in a journal!
13 Aug: ABC: AM: Climate change has fish moving south
The CSIRO says about 30 per cent of the species found in south-east Australia have moved to new habitats..
MEREDITH GRIFFITHS: Dr Last and his colleagues at the CSIRO have just published their findings in the journal Global Ecology and Biogeography. It shows about 30 per cent of the fish that live in south-east Australia have moved to new habitats and started to breed..
http://www.abc.net.au/am/content/2010/s2981717.htm
10
Pat,
“PETER LAST: We thought we knew a bit about what was actually happening here until we went underwater about five years ago in fact and found some pretty dramatic shifts in the distribution of some species.”
Yeah, what a typical CSIRO B/S story! The East Aust Current has been bringing down warm water and species from the north for ever and he just went looking 5 years ago.
Thanks, Jo for highlighting this problem. Be interesting to get Mc & Mc’s thoughts on it.
10
yeah – spangled – that was my reaction when I heard it on the radio as I was in town to get a replacement battery for my Van.
I never heard the word PDO once!
10
He may be a hypocrite but at least he is honest enough to be getting out there and saying that population control is the real agenda. Most of the green movement just whisper population control under their breath and then blah blah about the dangers of carbon. What they really don’t like is sharing their space with other people.
10
Re: Janama, #12:
So, that’s a maximum of 48.75 GW when the wind is blowing, and another 3.5 GW when the sun is shining. Better hope that there aren’t too many calm nights.
This proposal is what blows, and the ZCA2020 authors should stuff it where the sun never shines.
10
Tel; by hypocrite I mean Smith now agitates against the very economic behaviour which brought him his wealth and is now the basis of his influence. He does not have the wit to realise that the best social contraceptive [with one exception] is prosperity which comes from relatively cheap energy and private property and business rights. By championing AGW and its ‘solutions’ he is working against that paradigm and solution to his concerns about population increase.
10
cohers,
“Social contraceptive” I like.
The Dick Smiths, Ted Turners, Bill Gates etc all have to repent any perceived transgressions and also because their poppy tops stick out.
They should be more like George Soros.
10
cohenite, Dick Smith also campaigns against consumption while living in a house over 10 times bigger than what I live in, flying round in helicopters all over the place (the least efficient form of transport ever invented), and yeah I doubt he would be too willing to have his own private property rights trampled over — frugality for someone else!
But I still believe that the real basis behind the whole global warming scare is population control. Once you can throttle off the energy supply, you can throttle food, heating and production. Beyond that they are trying to take control of productive farmland by stealth further reinforcing the throttle on food. Now we have bill S510 in the USA which will be the doorway to shutting down small independent food production and probably even blocking people from growing food for themselves. If that turns out to be successful in the USA then we in Australia will get something similar.
Food and energy — the ultimate keys to power.
10
[…] Peer Review Be Fixed? By wormthatturned From Jo Nova: Can Peer Review Be Fixed? “Seriously, what other profession would call unpublished comments by two unpaid anonymous […]
10
Tel,
I read S510 and it scares me to death. This is a strangle-hold on the food supply of the U.S., right down to legal requirements on parents of children with food allergies. I don’t need government to protect me from myself; I need someone to protect me from my government.
I don’t know if this will pass or not. The left is feeling some real heat over the probable losses on November 2nd and is trying to avoid giving any more offense (to use a polite term). After the election it’s anyone’s guess depending on how many seats Conservatives can pick up. And a lame-duck Congress might do a lot of damage just for spite.
Can we get some peer review of Congress?
10
Friends:
I write to make two points.
Firstly, peer review exists solely for the benefit of journal editors.
The editor of a journal needs to decide if the information in a paper is
(a) sufficiently novel and/or
(b) potentially seminal, and/or
(c) advances information and/or understanding
for it to be of use to the journal’s readers.
But a paper that appears to have one of these qualities may contain a major flaw that would only be discernible by an expert in the subject of the paper. Therefore, the editor submits the paper for comment by persons perceived as being experts in a field for them to assess it. These perceived experts are supposed “peers” of the author of the paper.
Thus, the editor seeks information on whether or not the paper contains such blatant flaws that it should not be published.
So, peer review says nothing concerning the worth of a paper.
It merely provides a degree of protection to journal editors.
The worth of a paper is revealed later by the usefulness that the information in the paper subsequently proves to have.
Furthermore, the persons perceived as being experts in a field probably have prejudices that could hinder their assessment of a paper. They have invested their careers in their work so are likely to be favourable to papers which support their own past work while tending to be censorious or hyper-critical of papers which dispute their own work.
For this reason, the then editor of Nature magazine chose to not submit either of Einstein’s relativity papers to peer review but to publish them without any review.
On the other hand, the Wright brothers failed to get their seminal paper on aeronautics published in a peer reviewed journal. Editors always rejected it on the basis of the review comments it received. So, they published it in a journal on bee-keeping.
The value of the Wright brothers’ paper is proven by the existence of the Boeing 747 and not by its failure to gain approval from peer review.
This demonstrates that the value of the information in a paper is what counts, and not the opinions of its reviewers.
The only solution to the problem is for editors of journals to have the courage to publish papers for comment (perhaps together with review comments) when they see the papers have potential great value although those papers failed to obtain supporting review comments.
Secondly, and off-topic, those who have expressed an interest (above) in renewables may want to read my item at
http://scienceandpublicpolicy.org/images/stories/papers/reprint/courtney_2006_lecture.pdf
especially its Section 14 (on pp13-19).
Richard
10
I am not convinced that peer review exists solely for editors. Peer review developed because there are more scientific studies than other scientists can cope with reading on a daily or monthly basis. So someone somewhere has to filter them on behalf of the scientific community.
If every paper, that a scientist from anywhere in the world decides to write up, is published in a free for all – then surely the scientific community won’t have time to review them and build on those papers for further advances or duplicate their work to give it credence.
So we would seem to need some kind of filtering process.
Historically the flaws of peer review were limited to damage to individual scientists careers (not to be sniffed at either). Bad papers were eventually exposed and good papers were eventually supported by further supporting work and the progress was eventually made. Unfair and damaging to individuals and the specific science that the papers referred to – but not to the rest of the world. In other words the passage of time sorted it all out …… sometimes a l-o-n-g time.
What has created our current concern about peer review is that in this case we have a group of scientists asking the world to change how it lives and spend hundreds of billions in making that change, based on their claims. NOW it matters !
I suspect that the most workable changes that could be made to peer review is to create a much larger panel of potential assessors through a public, transparent process ensuring a wide range of views within each science. Then have a process of randomly choosing an appropriate number of experts out of that panel to which the paper is sent for review. The majority view should not be the deciding factor, it’s not a democracy. If a paper received more than, say, two supporting reviews then it should be published.
10
Howard:
At #30 you suggest:
Sorry, but that is not so.
Journals instituted peer review and there was no pressure from others for it.
And peer review does not act as a filter in the manner you suggest: the total number of pages available in journals does that.
Editors select the papers they will publish, and they use peer reviewers to help them with the selection.
There are always more papers submitted than there is room in the journals.
Also, you propose:
But that is completely unworkable because it removes the right of editor to reject papers. Each edition of a journal
(a) has finite size
and
(b) needs an appropriate spread of subjects (from its discipline) to have interest for all its readers.
Richard
10
Sorry Richard but I don’t follow you. You seem to disagree with my early points yet you confirm that journals do filter papers because there are only so many pages available. This is what I said. The total no of pages does not chose which papers to include and which to exclude. Thats silly. The edirtors and peer reviewers make those decisions.
On my proposal – Yes Editors need to be jettisoned from the process. Exactly! There is too much control by Editors. The process should be controlled by the peer panel and the impact of personal prejudices reduced or eliminated. There is too much emphasis on this ‘spread of subject’ thing anyway. The science should come first.
Also the age of ePublishing is upon us and number of pages is now irrelevant. Journals should be moving with the times and making use of new filtering process comment systems etc.
10
Howard:
At #32 you assert:
How do you propose to reduce “the impact of personal prejudices” when editors have been “jettisoned”?
And who is to select these “peer panels”?
Sheesh!
Richard
10
How do you propose to reduce “the impact of personal prejudices” when editors have been “jettisoned”?
When editor’s ability to chose which studies get published and which don’t, that in and of itself takes a huge amount of personal prejudice out of the process. Editors can then stick to what they should be doing.
And who is to select these “peer panels”?
As I posted above “. . to create a much larger panel of potential assessors through a public, transparent process ensuring a wide range of views within each science. Then have a process of randomly choosing an appropriate number of experts out of that panel to which the paper is sent for review.” I see no reason why a reputable Journal cannot develop a transparent process of recruiting a wide ranging panel of accepted leading scientists in each field, encompassing all sides of the major views within that science. It will create wide credibility and a renewed respect for the papers published and for the legitimacy of the scientific views within that science.
Also as I said above the need to be so strict about publishing based on the number of pages available is already becoming moot. Scientific Journals are already experimenting with online formats and the future is clearly in ePublishing, which will mean that the only limitation is quality of work.
10
Howard:
I write to ensure that you do not think I am ignoring you.
In my opinion we have each clearly presented our views that disagree. So, there is no purpose in our continuing this discussion between ourselves because others can now evaluate our different views for themselves.
If you do not agree then please come back to me.
Richard
10
Hey Richard… no sweat. I am just shooting the breeze on this whole peer review thing. It’s cool to disagree. I am just disappointed at the lack of other contributions and ideas. I have no monopoly on good ideas but there aren’t many innovative suggestions popping up here.
10
[…] Nova ponders the future of peer review, and is harsh: Seriously, what other profession would call unpublished comments by two unpaid […]
10
I’m wondering if journals will go the way of the dead tree press and move online, or die. I’m a fan of visualisation and blogs have been using blink comparisons and animations. Print journals can’t do that, online publications can, and removes some of the space restrictions.
As for the current process, I’d like to see more transparency. I’m not sure removing anonymity would be a good idea but I do think review comments should be published.
I’ve also seen lack of funding used as an excuse for not checking or validating papers thoroughly, if at all. If funding is a requirement to perform credible reviews, then who provides that?
10
700 Peer-Reviewed Papers Supporting Skepticism of “Man-Made” Global Warming………
http://www.populartechnology.net/2009/10/peer-reviewed-papers-supporting.html
10