Is peer review all it's cracked up to be?
Posted by AccountingWEB in on 07/31/2009 - 11:23
- Some call it “auditing the auditors.” On a regular basis, professionals cross check each other. That sounds like a good thing. But is it?
Some call it “auditing the auditors.” On a regular basis, professionals cross check each other. That sounds like a good thing, and it can be. But like most things, the process is both fallible and corruptible if the wrong motives are in play or the bar is set too low. That’s why some participants and observers have grown skeptical about the merits of peer review, not just in accounting, but in most fields. Is a report that earns the label “peer-reviewed” really worthy of a Good Housekeeping Seal of Approval?
Those skeptics raise some good questions: Are there partisan special interest groups involved? Is there a political motivation pushing the outcome of the peer review? Is the depth and breadth of the peer review adequate to render a good opinion? Are enough reviewers utilized to do a thorough examination? Can the results of the peer review be duplicated by another reviewer?
Here’s what author John Moore wrote in Nature about the corruptible process of peer review:
"It's been peer-reviewed so it must be right, right? Wrong! Not everything in the peer-reviewed literature is correct. Indeed, some of it is downright bad science. Professional scientists usually know how to rate papers within their own fields of expertise (all too often very narrow ones nowadays). We realize that some journals are more stringent than others in what they will accept, and that peer review standards can unfortunately be too flexible. A lust for profits has arguably led to the appearance of too many journals, and so it can be all too easy to find somewhere that will publish poor-quality work."
It’s true that, in any field, peer-reviewed work is more likely to have fewer errors than work that hasn’t had other eyes pore over it. Still, just the label “peer-reviewed” is no guarantee of quality. A report from the Journal of the American Medical Association estimated that one third of the studies published in peer-reviewed journals cannot hold up under closer scrutiny.
Jack Dini is a Texas accountant who writes a blog called Skeptical CPA, as well as other materials. He wrote:
“As bad as peer review is in the hard sciences, it's worse in the CPA business. About 12 years ago Florida considered making peer review mandatory for CPA firms. As luck would have it, one member of Florida's House was a CPA. His fellow legislators were horrified when he told them large CPA firms had been peer reviewed since about 1977 and that peer review has and would do nothing to protect Floridians from crooked or incompetent CPAs.”
Dini worries that professionals, and more importantly, the public, tend to equate peer review with a “gold standard.” Then, says Dini, people who are politically motivated, or just plain lazy may use an inadequate or non-independent peer review process to push a point of view or secure a recommendation that isn’t earned.
Look at the well-publicized case of Bernard L. Madoff Investment Securities LLC. Many accounting firms inspected the books and did not see the fraud. Neither did representatives of the feeder funds that channeled client money into Madoff’s company. Legal and accounting experts say, those firms may now be legally vulnerable. Former chief accountant for the SEC, Lynn E. Turner seems to doubt that the feeder fund auditors looked at the books at all. “If they didn’t,” said Turner, “then investors will hold the auditor accountable.”
Four years ago, in 2005, CalCPA (the California state society for CPAs) expressed its own skepticism when it decided not to embrace the AICPA policy of mandatory peer review. The CalCPA board made this decision because they said the process did not provide enough transparency and they had unresolved questions about the scope of the peer reviews. In addition, CalCPA believed the policy of exempting small firms from mandatory peer review would result in a disservice to the public. The board concluded by saying they would reconsider their decision this year, in 2009.
Change does seem to be in the air. In 2008 the AICPA upgraded its peer review requirements to increase transparency, understandability, and clarity. Now firms that are undergoing peer review will receive reports designated “pass,” “pass with deficiencies” or “fail.” Peer reviews will not include comment letters, but will include a documentation of findings which will not affect the report but will be retained with it.
New York recently passed sweeping reforms of the accounting profession in their state, designed to ensure that the public’s trust in accountants is justified. One of those reforms is mandatory participation in peer review for all but the smallest CPA firms.
The skepticism about peer review may be well-earned by some bad players. But this doesn’t seem to be a baby that should be thrown out with the bathwater. If used as intended, it can still be a great measure of quality. By taking another, more intense look at peer review, groups like the AICPA and the New York State Society of CPAs may be able to put some muscle back into the process so that it can once again be the “gold standard” it was meant to be.
- 3032 reads
Tags:
Peer Review
I've been involved with peer reviews since I worked for an international accounting firm many years ago. I performed internal peer reviews for that firm, I've performed peer reviews of other firms within the AICPA peer review program as well as for a state board of accountancy, I've reviewed peer reviews and I oversaw a peer review program for a state society of CPA's and I've been peer reviewed.
Is Peer Review perfect? No. Can it be perfected? No. It's not like a "peer reviewed" article for a professional journal, where you're looking at that one thing (the article) and, in effect, professing an opinion on that one thing. In a peer review of a CPA firm you are looking at the system of internal controls of that firm and, as part of that review, you are reviewing a sample of reports and working papers to provide evidence about that system of internal controls. Like most things when you are sampling, you may miss things. The firm may have issued some improper reports or performed substandard work. You're not reviewing everything. Also, just because the firm has a good system of internal controls in place at the time the peer review was performed does not mean that in the next week or year they may fail to follow their system of internal controls and may issue an improper audit report. They may for some reason lack appropriate independence. The client's audit fee may be too big to lose. The partner's bonus may be based on his total fees earned by the firm and the auditor may not be as skeptical as he should have been about accounting for some "off balance sheet" items as possibly happened in the Enron case.
Has it improved the work performed by CPAs? Yes. Years ago I saw an government's balance sheet that looked like a corporation's balance sheet. I've reviewed reports of specialized industries that indicated the auditor didn't know what they were doing and they were told by the state board to stop doing that type of engagement. I've seen many firms that improved their work based on comments from peer reviewers. Reports have improved. Documentation of work performed has been improved. But can I absolutely say an audit failure has been prevented due to peer reviews? No. It's hard to prove a negative...something that didn't happen. But...Madoff's accounting firm had not been peer reviewed. If it had been, I'm pretty sure the fraud would have been caught and stopped many years ago.
I hope the recent changes by the AICPA in the peer review standards will make the process more transparent...more useable.
peer review
It is my considered opinion that the peer review should be scraped or allow some means of opting out of the mandatory either by paying an additional licensing fee or some other means of handling the process.
My twenty years of mandatory participation have not solved audit failures, nor is there any empirical data that the audit process has been strengthened by having mandatory reviews by people outside of our firms.
In my opinion, this has been yet another "reform" foisted on the majority of the CPA's by the firms so they can protect themselves from the problems such as Enron, Worldcom that they created! I am personally very tired of constantly spending money to save Price, KPMG, etc.
The new review standards are just more of the same. It is a chance for the Melancon and his supporters to pat themselves on the back and say what a great thing we are doing, but addressing the needs and issues that affect small practioners like myself.