Have you ever used PubMed to search for a paper? Do you know how to tell a bad paper from a good one (i.e. critically appraise a paper)? Are not you sometimes afraid that certain papers contain mistakes? Did medical school equip you with enough skills to critically appraise the literature? Is there anyone to help you? Yes, there is help is since October 2013!
I believe that for most medical graduates in third world countries, where medical research is something rare, the answers to all the above questions is no. Understanding published papers, with their complex statistics and ideas was, and still, hard for me. This is not to mention critically appraising these papers and telling a good paper from a bad one. Did you experience this? Are you still experiencing this?
How can an inexperienced person tell a bad paper from a good one? Be this person a medical student, an inexperienced graduate, or even an experienced doctor who rarely does research. How can a person know the mistakes, which a paper contains? I believe that a good percentage of papers contain mistakes, whether scientific or linguistic, whether by mistake or with aim of fraud. The Abstract of the paper is only there on PubMed or in the Journal website. What is more important, the current system of “letters to the editor” does not seem to be effective. How often did you see such important feedback published? More importantly, do you think that many experienced people will take time to go through this lengthy and boring process of sending a letter to the editor?
When someone suggests a book to you, how do you make sure that it is a good book before going ahead and buying it? I believe that the easiest way is to read reviews or simply comments of people who read it. Very beautiful examples are Amazon, Bookreads, or whatever other website that allows users to publish comments and reviews in the books’ pages. Do not you benefit from reading these comments? Is not this a quick, relatively trustworthy, and incredible method that allows you to know how good a book is?
Source of the screenshot: http://www.amazon.com/Master-Boards-USMLE-Step-CK/dp/1609787609
In addition, we can consider the case YouTube videos. Do not you enjoy reading some of the comments that are posted under YouTube videos? Have you ever been curious to know how a magic trick was performed, and then you quickly knew what was going on through a comment that explained exactly what was going on? Have you ever not been impressed by a video explaining how to prepare something, and then was persuaded that the video is a piece of trash after reading some of the posted comments? Do not you like the discussions that break out below political or religious videos? Are not they sometimes more informative than the videos themselves?
Source of the screenshot: http://www.youtube.com/watch?v=ZjA6eJ3R-UQ
Why do not they allow comments, reviews, and subsequent discussions under abstracts of published papers? Is not this a great way to allow inexperienced individuals to assess the quality of published papers? Why not to allow experienced users to expose the defects in published papers? The solution is simple then: Allow comments under published papers Just like Amazon and Goodreads allow comments in each book’s page or like YouTube allows comments under each video. Is not this a small change with great consequences? Did not anyone think of that?
Less than an hour ago, I was searching PubMed when an Ad talked about something called “PubMed Commons.” This was interesting and I immediately googled it. I found a great paper in the website of Stanford University (The world’s third best university according to the World University Rankings). A Professor called Tob Tibshirani wrote it. He is, according to his article below, one of the minds behind what I think would revolutionize research (all respect). I highlighted in red the important aspects of the article.
“PubMed Commons: A system for commenting on articles in PubMed”
The Need for a Comments System
We all read a lot of papers and often have useful things to say about them, but there is no systematic way to do this – lots of journals have commenting systems, but they’re clunky, and, most importantly, they’re scattered across thousands of sites. Journals don’t encourage critical comments from readers, and letters to the editor are difficult to publish and given too little space. If we’re ever going to develop a culture of commenting on the literature, we need to have a simple and centralized way of doing it.
Pat Brown, Mike Eisen and David Lipman
Last year, I approached my Stanford colleague Pat Brown, a founder of PLOS, with the idea of creating a site where scientists could comment on ANY published research article – something like comments on movies at Internet Movie Data Base (IMDB) or comments on books and other products at Amazon. Pat said that he been discussing similar ideas with his PLOS co-founder Michael Eisen, and that they felt strongly that a standalone site would be unlikely to work because it would not get enough traffic. They felt that the best way to develop a successful culture of commenting on science papers would be to make this an option at PubMed. Pat introduced me to David Lipman, the Director of the NCBI (the home of PubMed), who said that the idea has been raised many times in the past, and that he was open to implementing such a system if I could demonstrate broad support in the community.
So I organized a group of 34 team leaders, representing diverse scientific fields. They recruited teams of prominent researchers in their fields – 250 in all, who were committed to the idea. David took the idea to the NIH leadership, who approved the development of a pilot commenting system called PubMed Commons. The team of scientists I assembled agreed to beta test the system during development and to provide feedback on its design and operation.
Who should be able to post comments?
A central issue for PubMed Commons was the question of who should be able to post comments. One would like the system to be inclusive as possible but many scientists would not be interested in posting comments in a system with a high proportion of irrelevant or uninformed comments. NIH also needed a rule for who could post that would be pretty clear cut and not based on e.g. some judgment of the experience or knowledge of the participants. The decision was made that comments could only be posted by authors of papers in PubMed. This would make the situation symmetric in that all people who comment can have their own work commented on. It would also include a large number of potential participants and would meet NIH’s need for something unambiguous. Unfortunately it would leave out many people who could add valuable input, including many graduate students, patient advocates, and science journalists. I’m a little worried about this restriction, as I want to make the system open to as many users as possible. But hopefully that is a pretty wide net, and it may be widened further in the future. And a group commenting feature to be described below could help improve inclusiveness.
Anonymous comments allowed?
One big issue that we have faced was the question of whether anonymous comments should be allowed. After much discussion, the group remained deeply split on this issue. Those wanting anonymous posts were concerned that many scientists, especially junior researchers, would be reluctant to make critical comments. But those opposed to anonymous comments believed that the quality of interchange would be higher if commenters were required to identify themselves. In the end, these differences weren’t really resolved and the decision was to start without anonymous comments and re-evaluate after the system had been fully public for a while. While debating this issue various proposals were put on the table for ways to allow participants to review and essentially sponsor the anonymous post of another participant.
Gary Ward, an active member of the lead user group, was very keen on using PubMed Commons to post comments from a journal club for a class he participates in at the University of Vermont. He proposed that there should be some way for PubMed Commons to accommodate comments posted by a group. David Lipman noted that group comments would also be a way to allow participation by a wider range of commenters: A group could be initiated by a regular PubMed Commons participant (i.e. was an author of a paper indexed in PubMed), giving it a title, short description, and list of participants and then posting comments on their behalf. While a group comment could be submitted by a particular group member, in many cases, they would reflect the consensus of the group and such collective comments could be quite valuable.
PubMed Commons is here!
The NCBI team developed a working version of PubMed Commons earlier this summer and I posted the first comment in the closed pilot on June 17. Since then the user group has noted bugs and made a number of requests for modifications. Jonathan Dugan of PLOS labs pulled together members of the publishing world for strategic advice, and has provided many valuable suggestions about the design of the system. Hilda Bastian, the editor of PubMed Health and a blogger at Scientific American rallied the community’s science bloggers to help get the word out.The current system is pretty simple – after registering you’ll see the PubMed Commons landing page which has all the most recent comments and links for information on how to use the system. When you’re signed in you’ll see below each PubMed record a box for posting comments or replies to existing comments as well as a place to indicate that an existing comment or reply was useful. There are instructions for how to specify simply formatting of a comment and if you cite another PubMed record in your comment, there are links back from that cited paper to your comment.
We believe the system is now ready for a wider range of participants. If you’ve been funded by an NIH Extramural grant (or in the NIH Intramural program), NIH has the information it needs to get you into PubMed Commons automatically. Once you’re a registered participant, you can invite other published scientists to join. NCBI is investigating ways to open Commons up directly and automatically to more groups of published scientists but if new participants invite their colleagues, the network effect could broaden membership and expand participation dramatically.
The system will still be in a closed pilot mode where only registered participants can see the posted comments but NIH leadership will be evaluating the closed pilot with the hope of making all comments visible to all users of PubMed. All comments are covered by Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/ ) and if the decision is to make the system fully public, NCBI will provide an API so that other groups (e.g. publishers or other information resources) can make these comments useful to the community.
PubMed Commons was released for broad use on October 22, 2013
See the PubMed Commons Landing page
I am very excited by this initiative, and hope it can improve the quality of scientific interchange the community.
Rob Tibshirani, Stanford University
N.B. I found this article through this nice blog.
Source of Prof. Tob Tibshirani’s photo: Professor’s page in Stanford University.
Some important links:
Find all PubMed Citations with comments: This search results page contains all PubMed papers that has comments. Today, 8/3/2014, the number of PubMed Papers with citations is 876. This does not reflect the total number of comments as some papers has more than one comment.
How to join PubMed Commons? Can you?
PubMed Commons on Twitter: The description of this account reads, “PubMed is piloting a beta commenting system. Get involved! Follow us and we’ll keep you posted.”
Examples of comments:
Finally, why am I excited?
1) In the past, I depended on the number of citations that a paper has, in order to judge how good it is. But now, I can also more reliably depend on the comments posted under the paper in PubMed. This is important especially for newly published papers that do not yet have many citations. Moreover, this is important for papers about topics that do not bring many citations.
2) In YouTube videos, when a person reads the different comments under a video, and sees, for example, how some users are proving or disproving an argument. Consequently, the person would learn to think in a new way. By seeing how others analyze a point of view, and then how they prove or disprove it, then a person will certainly learn this skill by time. Similarly, I believe that when inexperienced individuals see how experienced researchers criticize the information mentioned in a paper, then they will start learning how to do that. They will know, for example, what to look at when reading a similar paper. Imagine this situation: You read a paper and decide to cite it in your paper. You then remember PubMed Commons and go and read the comments posted under that paper. After reading a few comments, you will get that point of:
“How did I did not notice that when I read the paper? Yes, what the comments say is logical! There are huge mistakes in the study methodology and the results are therefore unreliable. I should not cite it although it has a good number of citations. Probably, the people who cited this paper did not read these comments in the PubMed page of the paper. I wonder if they know about PubMed commons!”
Source of photo: http://pixabay.com/en/woman-sad-crying-thinking-old-71735/
3) Authors will benefit too, not only be embarrassed :). Firstly, this will provide them with a feedback regarding their work so that they can avoid repeating the same mistakes in the future. In addition, I believe that authors will start paying more attention to what they publish. The situation is now not like in the past and many people can in minutes expose to the whole world the mistakes in their lousy work.
4) I expect that experienced people do not have time to go through the lengthy process of sending a “letter to the editor.” I believe that the lag of time between sending the letter and the time for a reply will decrease enthusiasm. In addition, I believe that there is a limit to the number of “letters to the editor” that a journal will publish. Online comments can solve all of this.
To sum up, I am really very excited about the implications of this “small step” on research and researchers worldwide. The scientific community should have thought of allowing comments on papers since a long time ago. Nevertheless, as the maxim says, “Better late than never!”