Is peer review all it's cracked up to be?
by AccountingWEB on
Some call it “auditing the auditors.” On a regular basis, professionals cross check each other. That sounds like a good thing, and it can be. But like most things, the process is both fallible and corruptible if the wrong motives are in play or the bar is set too low. That’s why some participants and observers have grown skeptical about the merits of peer review, not just in accounting, but in most fields. Is a report that earns the label “peer-reviewed” really worthy of a Good Housekeeping Seal of Approval?
Those skeptics raise some good questions: Are there partisan special interest groups involved? Is there a political motivation pushing the outcome of the peer review? Is the depth and breadth of the peer review adequate to render a good opinion? Are enough reviewers utilized to do a thorough examination? Can the results of the peer review be duplicated by another reviewer?
Here’s what author John Moore wrote in Nature about the corruptible process of peer review:
"It's been peer-reviewed so it must be right, right? Wrong! Not everything in the peer-reviewed literature is correct. Indeed, some of it is downright bad science. Professional scientists usually know how to rate papers within their own fields of expertise (all too often very narrow ones nowadays). We realize that some journals are more stringent than others in what they will accept, and that peer review standards can unfortunately be too flexible. A lust for profits has arguably led to the appearance of too many journals, and so it can be all too easy to find somewhere that will publish poor-quality work."
It’s true that, in any field, peer-reviewed work is more likely to have fewer errors than work that hasn’t had other eyes pore over it. Still, just the label “peer-reviewed” is no guarantee of quality. A report from the Journal of the American Medical Association estimated that one third of the studies published in peer-reviewed journals cannot hold up under closer scrutiny.
Jack Dini is a Texas accountant who writes a blog called Skeptical CPA, as well as other materials. He wrote:
“As bad as peer review is in the hard sciences, it's worse in the CPA business. About 12 years ago Florida considered making peer review mandatory for CPA firms. As luck would have it, one member of Florida's House was a CPA. His fellow legislators were horrified when he told them large CPA firms had been peer reviewed since about 1977 and that peer review has and would do nothing to protect Floridians from crooked or incompetent CPAs.”
Dini worries that professionals, and more importantly, the public, tend to equate peer review with a “gold standard.” Then, says Dini, people who are politically motivated, or just plain lazy may use an inadequate or non-independent peer review process to push a point of view or secure a recommendation that isn’t earned.
Look at the well-publicized case of Bernard L. Madoff Investment Securities LLC. Many accounting firms inspected the books and did not see the fraud. Neither did representatives of the feeder funds that channeled client money into Madoff’s company. Legal and accounting experts say, those firms may now be legally vulnerable. Former chief accountant for the SEC, Lynn E. Turner seems to doubt that the feeder fund auditors looked at the books at all. “If they didn’t,” said Turner, “then investors will hold the auditor accountable.”
Four years ago, in 2005, CalCPA (the California state society for CPAs) expressed its own skepticism when it decided not to embrace the AICPA policy of mandatory peer review. The CalCPA board made this decision because they said the process did not provide enough transparency and they had unresolved questions about the scope of the peer reviews. In addition, CalCPA believed the policy of exempting small firms from mandatory peer review would result in a disservice to the public. The board concluded by saying they would reconsider their decision this year, in 2009.
Change does seem to be in the air. In 2008 the AICPA upgraded its peer review requirements to increase transparency, understandability, and clarity. Now firms that are undergoing peer review will receive reports designated “pass,” “pass with deficiencies” or “fail.” Peer reviews will not include comment letters, but will include a documentation of findings which will not affect the report but will be retained with it.
New York recently passed sweeping reforms of the accounting profession in their state, designed to ensure that the public’s trust in accountants is justified. One of those reforms is mandatory participation in peer review for all but the smallest CPA firms.
The skepticism about peer review may be well-earned by some bad players. But this doesn’t seem to be a baby that should be thrown out with the bathwater. If used as intended, it can still be a great measure of quality. By taking another, more intense look at peer review, groups like the AICPA and the New York State Society of CPAs may be able to put some muscle back into the process so that it can once again be the “gold standard” it was meant to be.