Authors of Submitted Journal Articles Are Asked to Explain Their Papers and It Doesn't Go Very Well
hubie writes:
Asking Authors About Their Own Papers:
I am one of the Editors-in-Chief of the Transactions on Machine Learning Research (TMLR). I reached out to authors of 10 paper submissions to TMLR, originally slated for desk rejection, asking for a meeting to discuss their paper. The author-attendees of the meetings included undergraduate students, master's students, PhD students, faculty, and independent researchers. Most, but not all, of these papers were solo authored.
Of the ten submissions:
- Authors of one paper withdrew their submission.
- Authors of one paper said they were unavailable due to other commitments.
- Authors of one paper scheduled a meeting but did not show up.
- Authors of three papers were unable to answer basic questions about the paper.
- Authors of three papers answered questions about high-level ideas in the paper but had difficulty when asked further questions on technical details.
- Authors of one paper answered all of my questions (although I identified a major flaw in that paper).[...] Out of the ten papers I contacted, the authors of one paper withdrew their paper after the message. The author of another paper said they were too busy with other commitments at that time. I scheduled Zoom meetings with the authors of the remaining eight papers.
[...] The authors of one of the remaining eight papers did not show up for the scheduled meeting. I had meetings with authors of the other seven papers. The author-attendees in the meetings included undergraduate students, master's students, PhD students, faculty, and independent researchers.
[...] Broadly, I asked two types of questions:
(1) Basic questions about the problem setting, notation, and results claimed in the paper; and
(2) More detailed questions about particular technical expressions, theoretical results, and design choices made in the experiments.The authors of three of the papers were unable to answer basic questions about their paper. All three of these papers were solo authored. Two of the authors appeared to have almost no substantive understanding of the contents of their own papers. Another author could not identify where some key results claimed in their abstract were presented or supported.
The authors of the remaining four papers were able to answer basic questions about their problem setup, notation, etc. However, the authors of three of these papers had difficulty when asked deeper questions about technical aspects or design choices.
The authors of one paper were able to answer all my questions about the paper. However, my examination of the paper uncovered a major error in one of the paper's main claims, which the authors subsequently acknowledged
[...] Subsequent to these conversations, we made the following decisions on the ten papers. For the paper where the author was able to answer all questions about the paper, we desk rejected but allowed a resubmission after correcting the error or reducing their claim appropriately, and issuing various clarifications for parts that were not clear. We desk rejected the remaining nine papers without an option to resubmit.
All in all, authors are ultimately responsible for ensuring the accuracy and integrity of the papers they submit under their name. If they cannot explain the basic claims, methods, or technical details of those papers, then this is a serious problem. Journals and conferences should think carefully about the objectives of their review processes and quickly adapt via various initiatives and experiments; we are doing several of these already at TMLR (here, here, and here) and will continue to do so.
Read more of this story at SoylentNews.