Monday, August 15, 2005
The Boston Globe reports on the some serious potential problems of peer reviewed medical research, i.e., it doesn't seem to be catching flaws in many studies. The Boston Globe reports,
They are two of the most widely publicized pieces of medical research in recent years: Reports in prestigious journals declared that women who underwent hormone replacement therapy, and people who ingested large amounts of Vitamin E, had relatively low rates of heart disease.
But after research contradicted those studies -- frustrating anyone who had followed their recommendations -- some specialists began looking at whether peer review had failed to identify serious flaws in the research.
But the specialists found that it was almost impossible to discover what had happened in the vetting process, since peer reviewers are unpaid, anonymous, and unaccountable. Moreover, their reviews are kept confidential, making it impossible to know the parameters of the reviews.
Now, after a study that sent reverberations through the medical profession by finding that almost one-third of top research articles have been either contradicted or seriously questioned, some specialists are calling for radical changes in the system.
In advance of a world congress on peer review next month in Chicago, these specialists are suggesting that reviewers drop their anonymity and allow comments to be published. Some are proposing that peer reviewers be paid to ensure a more even quality of review and analysis among all journals.
Dr. Drummond Rennie, who relies on review as deputy editor of JAMA, the Journal of the American Medical Association, said of the process, ''The more we look into it, the harder it is to prove whether it does good or bad."
Rennie has called for greater study of whether peer review improves research, and he has a personal policy of disclosing his name when he reviews articles.
''It would be lovely to start anew and to set up a trial of peer review against no peer review," Rennie said. ''But no journal is willing to risk it."
Rennie's journal published the study, which said that subsequent research had found that almost one-third of the top papers that appeared in top journals over a 13-year period from 1990 to 2003, had been either contradicted or found to have potentially exaggerated results. All the articles had undergone vigorous peer review, leading to questions about whether problems should have been caught by reviewers.