Last time, we examined how scientific publication has ceased to be a good measure of scientific accomplishment because it has now become a target, following Goodhart’s Law. In today’s post, we will continue that examination by turning to the peer review system, and how that system is being gamed by unscrupulous publishers and researchers alike.
In theory, the peer review process is intended to ensure that research papers do not get published unless impartial experts in the field deem them worthy of publication. Peer review is well-intentioned, but flawed in many ways.
First, the best researchers are incredibly busy and naturally more inclined to do their own research than to review someone else’s work. Thus, peer review is often cursory or done by people who have ample time on their hands because they do relatively little research. Ill-informed and contradictory reviews are familiar to everyone who publishes in academic journals.
From the journal’s perspective, academic journals can be a cash cow since authors and reviewers are not paid anything and universities feel obligated to subscribe. In 2012 Harvard University reported that some journals were charging its library $40,000 annually and that its total annual cost of journal subscriptions was an “untenable” $3.5 million. In 2019, after lengthy negotiations, the University of California stopped its Elsevier subscriptions which were costing $10 million annually.
The escalating costs have led to an open-science movement in which journal articles are freely available online, with the publication costs borne by authors (and are typically paid for by their employer or with grant money). There are now several respectable open-access journals, including PLOS (Public Library of Science).
However, as the open-access model gained traction, unscrupulous publishers moved in to exploit authors fighting the publish-or-perish battle. Pretty much anyone who has published an article anywhere regularly receives invitations to submit papers for fast review and publication in a predatory journal.
The journal names sound legitimate; indeed, they are often variations on the names of bona fide journals. The Journal of Economics and Finance is a legitimate journal; the International Journal of Economics and Finance is not. Journal of Banking and Finance is legitimate; Journal of Economics and Banking is not. Advances in Mathematics is legitimate; International Journal of Advances in Mathematics is not.
The invitations often flatter the recipient, give phony journal impact factors, and promise a fast review process. The last claim is real. There is typically little, if any, review. The only criterion for publication is a willingness to pay the open-access charge.
Invitations from predatory journals pay little attention to one’s fields of interests. I have received invitations to write papers on such diverse topics as animal science and veterinary science, osteology and arthrology, properties and prospects of concrete with recycled materials as aggregates or binders. The only thing these topics have in common is that I know nothing about them.
The e-mail invitations are often marred by unusual grammar and flawed interpretations of published work. Here are snippets from one that I received.
Dear Colleague!
We have seen your other research manuscripts which are accessible on the internet. We are so interested to invite you to send any of your other new works to our periodical.
I would like to let you know that our journal “Frontiers in Life Science (HFSP)” is a long lasting BLOCK: N56 peer reviewed periodical printed in Strasbourg, France. Today, as a well BLOCK: N57 scientific periodical, Frontiers in Life Science is abstracted and indexed in Science Citation Index (ISI Thomson Reuters) with the five year impact factor of 3.093. So foremost, we owe our thanks to the authors who have decided to entrust Frontiers in Life Science with their best work, allowing us to benefit from the high quality of their scientific BLOCK: N59. We hope to remain worthy of their trust and to BLOCK: N60 providing a rigorous and respected environment for their publications.
Journal Information
ISSN: 2155-3796
Impact Factor: 1.273
Five-Year-IF: 3.093
Average Impact Factor: 38.732
The BLOCKs seem to be formatting instructions that were ignored. The impact factors are fictitious and, if true, would place the journal among the top journals in the world.
Even more worrisome, this is a journal hijacking, which is another arrow in the quiver of nefarious tricks used by predatory journals. There is a legitimate journal titled Frontiers in Life Science, but this is not it. More than 100 examples have now been documented of bogus journals creating fake websites that mimic the websites of authentic journals. When a duped researcher “submits” a paper to the imposter journal, it is accepted with minimal fuss and the author is directed to send a publication fee to the huckster.
Two fed-up professors, David Mazières (at NYU at the time, now at Stanford) and Eddie Kohler (at UCLA at the time; now at Harvard) wrote a joke paper titled “Get me off your f**king mailing list” (with the F-word fully spelled out). The paper itself consisted of 10 pages of that admonition repeated over and over and over again, along with two figures incorporating the admonition.
They did not submit this irate paper, but Peter Vamplew, an Australian Professor of Information Technology, responded to an invitation from the predatory International Journal of Advanced Computer Technology by submitting the Get Me Off paper, with Mazières and Kohler listed as the authors. Soon after, Vamplew received a congratulatory e-mail informing him that the reviewer had judged the paper “excellent,” subject to minor revisions, and that the paper would be published upon receipt of $150 wired to the journal’s editor. The reviewer’s comments in full:
a. Please Use latest references in order to increase your paper quality.
b. Introduction part is precisely explained.
c. Kindly prepare your camera ready version of paper as per IJACT paper format.
d. Use high resolution images in order to increase the appearance of paper.
Unfortunately, it has become a two-way street. Unscrupulous journals exploit researchers and unscrupulous researchers exploit journals.
In 2014 the esteemed journal Nature published an article titled, “Publishing: The peer-review scam.” It recounted the story of Hyung-In Moon, a South Korean medicinal-plant researcher. He had submitted an article to a journal that invites authors to suggest possible reviewers. In this case, the editor’s suspicions were aroused when it took less than 24 hours for favorable reviews to come in. The editor questioned Moon and he admitted that the reviews took very little time because he had written them himself! It turned out that this was a common practice for Moon. When he suggested reviewers, he used a mix of real and phony names and his own disguised e-mail accounts. A total of 28 papers were subsequently retracted.
One experienced editor, Robert Lindsay, says that he has seen authors recommend not only close friends, but family members and students they are supervising. One brazen researcher recommended herself, using her maiden name. Some technically savvy researchers have hacked into editors’ accounts and sent review invitations to bogus email addresses controlled by the author.
Next: A further examination of the undermining of scientific publication using two examples – SCIgen and citation counts.
You may also wish to read the first in this three-part series:
Publish or Perish – Another Example of Goodhart’s Law. In becoming a target, publication has ceased to be a good measure. Researchers game the system to beat the publish-or-perish culture, which undermines the usefulness of publication and citation counts. (Gary Smith)
Also:
Why It’s So Hard To Reform Peer Review. Here is a historical summary focused specifically on the measures by which professors are promoted and get raises, which illustrates that Goodhart’s and Campbell’s laws are parasites on promotion and tenure metrics. (Robert J. Marks)
"Review" - Google News
June 15, 2021 at 10:02PM
https://ift.tt/3zsASIf
Gaming the System: The Flaws in Peer Review - Walter Bradley Center for Natural and Artificial Intelligence
"Review" - Google News
https://ift.tt/2YqLwiz
https://ift.tt/3c9nRHD
Bagikan Berita Ini
0 Response to "Gaming the System: The Flaws in Peer Review - Walter Bradley Center for Natural and Artificial Intelligence"
Post a Comment