How to Write an Effective Referee Report and Improve the Scientific Review Process

Journal of Economic Perspectives—Volume 31, Number 1—Winter 2017—Pages 231–244

he review process for academic journals in economics has grown vastly

more extensive over time. Journals demand more revisions, and papers

have become bloated with numerous robustness checks and extensions

(Ellison 2002b; McAfee 2010; Spiegel 2012; Harvey 2014). For example, Card and

DellaVigna (2013) document that recent published papers are on average three

times longer than in the 1970s—a total which excludes the many robustness checks

now contained in online appendices. Even if the extra resulting revisions do on

average lead to improved papers—a claim that is debatable—the cost is enormous.

We argue that much of the time involved in these revisions is a waste of research

effort.

The leading explanation for the expansion of the review process, as modeled

by Ellison (2002a), is that social norms have evolved toward increased demands for

revisions. We believe that part of the explanation for why the profession has evolved

to this equilibrium is that referees feel the need to demonstrate their intelligence

or industriousness to editors by identifying problems in papers. The result is that in

many cases reviewers inflate minor blemishes to the status of major flaws.

How to Write an Effective Referee Report

and Improve the Scientific Review Process

■ Jonathan B. Berk is A. P. Giannini Professor of Finance, Graduate School of Business,

Stanford University, Stanford, California. Campbell R. Harvey is Professor of Finance at

the Fuqua School of Business, Duke University Durham, North Carolina. David Hirshleifer

is Professor of Finance and Merage Chair in Business Growth, Paul Merage School of Busi-

ness, University of California-Irvine, Irvine, California. Their email addresses are jbberk@

stanford.edu, cam.harvey@duke.edu, and david.h@uci.edu.

†

For supplementary materials such as appendices, datasets, and author disclosure statements, see the

article page at

https://doi.org/10.1257/jep.31.1.231 doi=10.1257/jep.31.1.231

Jonathan B. Berk, Campbell R. Harvey, and

David Hirshleifer

232 Journal of Economic Perspectives

Another cause for concern is the level of disagreement amongst referees. As

Welch (2014) documents in a study of eight prominent journals in economics and

finance, in cases with multiple referees on the same paper, the probability that

one referee recommends inviting a revision rather than rejecting, conditional on

another referee doing so, is only marginally higher than the unconditional proba-

bility. Furthermore, the correlation of referee recommendations at a major finance

conference was only 0.28. This pattern of disagreement suggests a high level of

arbitrariness in the review process.

To identify and highlight what is going right and what is going wrong in the

reviewing process, we wrote to a sample of former editors of the American Economic

Review, the Journal of Political Economy, the Quarterly Journal of Economics, Economet-

rica, the Review of Economic Studies and the Journal of Financial Economics, and asked

them for their thoughts about what might improve the process. We found a rough

consensus that referees for top journals in economics tend to make similar, correct-

able, mistakes. The italicized quotations throughout this paper are drawn from

our correspondence with these editors and our own experience. Their insights are

consistent with our own experiences as editors at the Journal of Finance and the

Review of Financial Studies. Our objective is to highlight these mistakes and provide a

roadmap for how to avoid them.

This article is not a comprehensive “how-to guide” for refereeing. Readers inter-

ested in such a guide can consult Berk, Harvey, and Hirshleifer (2015), or the essays

in this journal by Hamermesh (1992, 1994). Instead our objective is to draw attention

to specific shortcomings in the process that we believe can and should be improved.

We begin by discussing what we see as three central current problems of refereeing

in economics: 1) giving appropriate weight to the importance and innovativeness of a

paper (and not being distracted that such papers may have minor flaws); 2) drawing

a clear and explicit separation between comments that must be dealt with to make

the paper publishable and suggestions that do not affect the publication decision

and are therefore optional; and 3) living up to the implicit contract involved in a

revise-and-resubmit process. We then offer thoughts on some other topics involved in

refereeing: conflicts of interest, cover letters, politeness, and acting ethically.

Importance: The Hardest Decision

Perhaps the hardest part of a referee’s job is assessing the importance of the

paper, which involves judgments about whether a paper is of sufficiently broad

interest and offers a sufficiently original contribution to be appropriate for the

journal to which is has been submitted. Submissions exceed the available space in

all A-level and most B-level journals; there are plenty of “correct” papers that do

Similar findings are reported in a study of 823 submissions to the Journal of Applied Psychology by Gilliland

and Cortina (1997). In an earlier study, Laband (1990) tries to measure the value added by referees by

analyzing the relation between reviewer comments and citations.

Jonathan B. Berk, Campbell R. Harvey, and David Hirshleifer 233

not make a significant enough contribution to existing knowledge to appear in

a top-tier journal. The editor needs to assess the importance of the contribution.

Thus, a referee report should contain a scientifically based argument that explains

the referee’s assessment of the importance of the work and details the consider-

ations that bear upon your judgment. Only an argument that is scientifically based

is useful and informative to an editor in making the decision.

I quickly learned that there were three critical determinants of whether a paper was pub-

lishable in the Journal: (i) whether it addressed a question of sufficiently broad interest;

(ii) whether it made a sufficient leap over existing literature; and (iii) whether the analy-

sis was correct. Since most of the papers I received were “correct”, my decision often boiled

down to considering how well the paper fared according to the first two criteria. … I often

felt that if referees more clearly understood the factors that make a paper publishable, they

would organize the reports accordingly, making them more useful overall.

The importance of a contribution can be undervalued in some cases by referees

and editors. After all, papers that are more ambitious are often more likely to have

loose ends, which gives referees and editors a reason to avoid taking a chance on

them. Indeed, Arrow (1995) pointed out: “I think the publication selection proce-

dure at the major journals has become methodologically more conservative, more

given to preferring small wrinkles in existing analysis to genuinely new ideas. This

conservative tendency also appears in the allocation of grants by government agen-

cies and in faculty appointments and promotions.”

Thus, we offer this advice to referees: Do not dismiss papers that attack larger

issues merely because flaws can be found. The important question that you need to

assess is whether the flaws actually invalidate the contribution. If the flaws do not

rise to this level and you judge the contribution to be important enough to warrant

publication, then you should recommend publication. All papers have flaws, and

no amount of revision removes all uncertainties. There is always need for further

research to provide deeper perspectives. Try to ask yourself the following ques-

tion: Flaws and all, would I have been pleased to have written such a paper? If yes,

that gives a strong hint that it should be strongly considered for publication, flaws

and all.

Separating the Essential and the Suggested: The Highest-Cost Mistake

All papers have weaknesses, and catching a serious problem can be an indi-

cator of referee quality. As a result, referees have a perverse incentive to persuade

the editor that their own intellectual work is of high quality and that they are smart

by pointing to minor blemishes and trumpeting them as serious problems. The

distinction between these two categories may not be obvious to an editor who is

not a specialist in the area of the paper: for discussion of this dynamic, see the

signal-jamming model of Hirshleifer (2015). So by pointing out these supposedly

234 Journal of Economic Perspectives

critical problems and requiring authors to address them, referees believe they are

improving their reputation with the editor.

Some younger referees feel that they need to be overwhelmingly negative about everything

in a paper in the report to the author to prove their own mettle and critical insights.

They don’t appreciate the need to at least demonstrate that they have read the paper and

to provide some kind comments indicating some understanding of what the author is

trying to do.

Often, the ultimate outcome of such signal-jamming is an unwieldy and lengthy

paper. This raises the question of why signal-jamming would be more important

now than in the past. One possible reason is that growth in the profession makes it

hard for an editor to have an independent assessment from personal contact of the

quality of referees—even those at top schools. This increases perceived incentives

to boost reputation by other means. Also, growth in the body of knowledge needed

to evaluate papers makes it harder for editors to see through signal-jamming efforts

(though in our experience, signal-jamming referees are often not nearly as subtle

as they think they are).

Perhaps more importantly, signal-jamming pressure may cause social norms

about standards for publishability to evolve over time toward increased demands

for revisions, in the spirit of the model of Ellison (2002a). When referees engage

in signal-jamming, more ambitious and potentially more important papers tend

to lose in competition with routine papers in the review process because of a stan-

dard that effectively demands, as a prerequisite for publication, the virtual absence

of minor shortcomings. The emphasis on superficial perfection over substantive

importance may have the disastrous side effect of reducing the incentive to engage

in innovative research.

I handled a paper in which the reviewer offered micromanagement of how the paper

should be written, and made a specific demand for how the paper should be reposi-

tioned—in a way that highlighted a paper of the reviewer’s. As for cases in which

reviewers demanded numerous unimportant extensions or robustness checks—this was

so common that specific examples are not especially noteworthy.

One key step to reduce dysfunctional signal-jamming is that a referee report

should divide comments into two clearly demarcated sections: 1) problems that make

the paper unpublishable, which (if revision is invited) must be addressed before the

paper is publishable; and 2) problems that are not essential for the publishability

of the paper, which should labeled as “suggestions.” From our perspective, perhaps

the most common and most crucial mistake most referees make when providing a

revise and resubmit recommendation is not clearly distinguishing between sugges-

tions that are required for publication and ones that are not.

From the referee’s perspective, the key difference between these two catego-

ries is the amount of justification that is required in the report. Statements about

How to Write an Effective Referee Report and Improve the Scientific Review Process 235

problems that must be addressed before publication require rigorous justification;

others do not. More specifically, must-address statements require a scientifically

convincing argument for why these problems currently render the paper unpub-

lishable. Of course, the same requirement may lead to a recommendation that the

paper be rejected. A referee is obligated to provide a scientifically based argument for

why a perceived problem renders the paper unpublishable. The argument needs to

be clear and understandable to the editor (and authors). After all, the goal is for

this argument to be used in the editor’s decision.

It is surprising how many reviewers have a highly refined olfactory sense;

several editors write about handling empirical papers in which a referee comments

on the “smell” of the results. A referee hunch that theoretical or empirical results

are wrong is not a scholarly argument against a paper’s conclusions. Instead, as

referee you must provide scientifically based reasons why you suspect the results are

not correct.

A theory paper is submitted to the Journal and a senior referee recommends rejection

because they “do not believe the proof”. This is insufficient. While it is expected that the

referee should be skeptical, the referee needs to provide a solid foundation as to why the

proof is incorrect. Hunch is not sufficient.

If there is a mistake in the proof, the appropriate recommendation is often

rejection, though in some cases a mistake is fixable. A mistake can also be identi-

fied by finding a counterexample to a claim. However, if the authors demonstrate

(correctly) an error in your counterexample, then their proposition stands. It

is not grounds for rejection to retreat into generalized skepticism. A revise-and-

resubmit for a theoretical paper can also ask the authors to explain how they

reconcile their results with existing findings, or the extent to which their result

may depend on nonstandard assumptions. But remember that it is possible

that the results in the paper are right and it is the previous literature that got

it wrong.

When it comes to empirical papers, a Bayesian should be skeptical of a conclu-

sion that has a low prior probability—more evidence needs to be brought to bear,

and greater validation of the evidence is required, to be persuasive. But a referee

cannot dismiss evidence simply because he or she finds the results to be surprising.

Obviously, surprising results have a greater probability of being wrong, so the

appropriate response is to request confirming evidence. You can ask the authors to

explain how they reconcile counterintuitive or unexpected empirical results with

existing findings. You can also evaluate the robustness checks that the authors have

done and consider whether it might be important to perform a limited number of

other robustness checks. But in making such comments, the explicit line between

what is necessary and what is suggested should be preserved. If the author satis-

fies these additional requirements, then the referee should leave it to subsequent

research to evaluate whether existing preconceptions need to be updated based on

the paper’s results.

236 Journal of Economic Perspectives

An empirical paper is submitted to the journal and the referee recommends rejection

because “the empirical results do not pass the smell test”. While there are plenty of

reasons that the result may not be solid, the referee needs to ask questions or request

specific robustness tests. “Smell” is an insufficient reason for rejection.

The review process for academic journal articles should sharply focus on what

is essential. This focus allows authors greater freedom to write papers in the way

that they think is best. All else equal, scholars should be able to develop their ideas

as they see them. On issues that are not essential for publishability, there is no

presumption that authors and referees need to see eye to eye. As a referee, it is

important to keep in mind that unless comments about secondary issues are care-

fully labeled as such, such comments are inherently coercive. Few authors have the

luxury of risking rejection of a project in a top journal for the sake of skipping the

robustness check mentioned by a referee, or of expressing their ideas in the way

that the author thinks is most effective but the referee does not

The Implicit Bargain in a Revise-and-Resubmit

As a referee, it is important to keep in mind the implicit deal you are making

with the author when you recommend a revise and resubmit: if the author satisfac-

torily addresses the issues that you have raised, you will recommend publication.

When you make a revise-and-resubmit recommendation, you are actually making

three statements: 1) the paper is of sufficient importance in terms of scope and

findings that you believe it is suitable for the journal at hand; 2) there are problems

with the paper that currently make it unpublishable in its current form; and 3) these

problems are correctable.

In short, you are helping the editor provide a road map to publication. Keep in

mind that the editor’s road map will require heavy investment of time by the authors

and usually relies heavily on referee advice. It is also an implicit contract. If the

authors address these requests in a satisfactory way, then the editor will likely accept

the paper. This should be the case even if other good ideas for improvement occur

to the referees in the next round. Avoid putting the editor in the very unfortunate

position of discovering that there are serious problems with the paper that were not

noticed in the first round. It is very important that your own ideas for improvement

be as refined as you can make them in the first round, so that the editor is positioned

to offer a useful and reliable road map.

In a revise-and-resubmit recommendation, there will likely be other problems

with the paper that are not severe enough to render the paper unpublishable. For

such other problems, you do not as a referee need to provide detailed reasons (or

a scientifically based argument) for your opinion. However, if the author chooses

not to address these problems, you cannot use that lack of reaction as a basis for

recommending rejection of the revised draft. In many cases, reasonable people can

disagree about what should and should not go into a given paper. Ultimately, the

Jonathan B. Berk, Campbell R. Harvey, and David Hirshleifer 237

author’s name goes on the paper, not the referee’s name, so the decision on how

best to write the paper is the author’s. Do not hold a submission hostage because, in

your (undoubtedly wise) opinion, the paper could be better written.

I handled a paper where the referee provided a thoughtful report which led to a revise

and resubmit. The author addressed the comments and resubmitted. On the revision,

the referee produced a new, long list of additional comments—each of which could have

been detailed in the initial report.

In making requests of authors, weigh the costs of the request. It is not enough

that a particular request will improve the paper. The benefits must exceed the costs,

so that the improvement has positive net present value. Since the author bears the

costs, it is easy for a referee to make absurd demands thoughtlessly. Don’t.

I receive a positive referee report on a paper that uses 19 years of hand-collected data.

The earliest data was the most time consuming to load because the authors had to

visit the archives and deal with paper documents. The referee insists that the data be

expanded back to 20 years. This 20th year was not a special year but it was clearly going

to be punitive for the authors to collect this extra year of data. In addition, the extra year

would unlikely be influential for the results. This is an example of “make work” and as

Editor it is often very difficult to separate the work that must be done from the frivolous

work. In this case, it was easy and I instructed the authors to ignore the referee’s com-

ment. I never called on that referee again.

By following these suggestions, the revise-and-resubmit process should gener-

ally take just one round before reaching a final decision on publication, and that is

how referees should view the process. Of course, sometimes a second round may be

required for unavoidable reasons. For example, perhaps the authors only partially

addressed the necessary changes specified in the first round. In such cases, referees

and editors should pause and consider the wisdom of going another round rather

than just recommending that the paper be rejected. A related possibility is that in

an intelligent and honest effort by authors to address the comments of the first

round, unforeseen issues have unavoidably arisen. Finally, cases may arise in which

a request for a second revision is based on an issue that should have been apparent

on the first round. In this case, the referee has made a mistake, and the referee

should admit this oversight to the author and to the editor. Such requests present

the editor with a hard choice, and should be rare.

I get a positive recommendation from a referee with a list of items for the authors to take

care of in a revision. I issue a R&R [revise and resubmit] and the paper comes back

with the authors making a serious effort to revise the paper and they address each of the

referee’s comments. The referee declines to provide a report on the resubmission but writes

to me saying that he recommends rejection because of “insufficient incremental contribu-

tion”. The rejection had nothing to do with the author responses. The referee changed his

238 Journal of Economic Perspectives

mind with no reasoning. If there was insufficient incremental contribution, that case

must be made in the initial submission.

In summary, if the author has appropriately satisfied the key requests of a previous

revise-and-resubmit referee report, the paper should then be accepted. Do not

invent a new set of requests—or even worse, reject capriciously.

Other Advice

Thus far we have highlighted the mistakes by referees and editors that impose

the highest costs on authors. In this section, we will briefly cover a few other impor-

tant lessons uncovered in our own experience as editors and in our correspondence

with other editors.

Declining the Invitation

Upon receiving a referee request, decide immediately whether you will be

able to complete the review within the allotted time. If not, respond immediately

with suggestions for alternative reviewers. Often the editor is not a specialist in the

specific area, and does not know exactly who the best match for a submission is, so

thoughtful suggestions for other reviewers are valuable. Also, editors understand

that peak loads can create the occasional need for a declined invitation or a request

for an extended deadline—or at least greatly prefer this to having to deal with a

severely late report, or a systematically derelict referee.

Agreeing to do a report at the requested deadline and then never doing the report

on time and never responding to reminders or queries about the report me or the

editorial office. In many of these cases, the individual did not have time to do the

report but felt compelled to say they would do it and embarrassed to respond. It

is much better for all involved to upfront decline a request immediately then to say

yes when it won’t be feasible to do the report in a timely manner. You lose a lot more

face in my view from going into hiding and not responding or doing reports much

later than promised, than from being upfront and saying you can’t do the report

at all.

Try to form an objective assessment about whether you are a good match for

the submission. If you know little about the relevant literature, it might be wise to

contact the editor to make sure that a mistake has not been made. The editor may

have had a good reason for selecting you, and verifying this can make your job more

useful and focused. For example, in a paper with both theory and empirical work,

the editor might be seeking the views of the empiricist for the empirical sections

and of a theorist for the theory sections.

If you have reviewed the paper for another journal, again immediately alert

the editor. Some editors would prefer getting a fresh view. Let the editor make

How to Write an Effective Referee Report and Improve the Scientific Review Process 239

that decision. Other editors may be satisfied that the referee can assess effectively

whether the paper has improved.

You might also feel that you cannot complete the report anonymously. For

example, you may have discussed the paper and feel your report would be so close

to the discussion it will be obvious to the author that you are the referee. If you

wish to maintain your anonymity, it is reasonable to decline to referee the paper.

If you do not mind the loss of anonymity, you should still alert the editor, since

nonanonymous reviewing can create agency problems. You may also provide advice

to the editor in a letter without a report. Your views will be important even when

they cannot be passed on to the author, though such service is not as useful to the

editor as a formal report. In this case, the same rules apply. If you feel the paper is

not publishable you must provide a scientifically based argument that will allow the

editor to make an informed judgment.

If you are already working on a paper that overlaps substantially with the

submission, you should notify the editor. As discussed further below, this is a poten-

tial conflict of interest and might be a reason to recuse yourself.

Finally, if a paper is obviously far below the bar at the journal where it has been

submitted, a short (one-page) report is perfectly acceptable. It should be straightfor-

ward to provide a scientifically based justification. If it is not, you should reconsider

whether the paper really is that far below the bar. In this case, impress the editor

by returning your report within a week—not by spending time on a lengthy anal-

ysis of what, by hypothesis, is an obvious conclusion that would be shared by the

overwhelming majority of referees. If the editor disagrees with your assessment, the

editor can then turn to another referee without undue delay.

The Cover Letter

The cover letter should contain an assessment of the contribution of the paper

along with very concise reasoning supporting your recommendation. The editor

wants to know the positives as well as the negatives of a paper. The cover letter

should be brief. It should not be a cut and paste of the referee report, which is a

waste of the editor’s time. In addition, your cover letter and recommendation to the

editor should be consistent with your report. Otherwise you impose on the editor

the task of explaining the discrepancy to the author. This will irritate both of them.

I assigned a paper by a well-known author to a particular referee. I had a very low prior

on the paper. The referee submits the report which is amazingly positive using words

like “pathbreaking”, “important insight”, “game-changer”, “will change the way people

think about a problem”, etc. I do notice that the referee has checked off “reject” in the

electronic system—which I assumed must have been a mistake. However, after reading

the cover letter, I understand. The referee thinks the paper is a poor idea and is nowhere

near the hurdle for the journal. I reject the paper and the author is furious and imme-

diately appeals the decision quoting the very positive referee report. Such inconsistencies

between recommendations and the report frustrate both authors and editors—and create

a lot of extra work for the editors.

240 Journal of Economic Perspectives

The ideal cover letter should succinctly provide three types of information. First,

it should include a statement of the broad interest and importance of the paper’s

contribution relative to existing work. Keep in mind that the editor may not be an

expert in this subfield, and it is often hard to figure out the paper’s main point

or line of reasoning. Second, is the analysis convincing? Third, the cover letter

should offer a frank assessment as to whether the paper is publishable as it stands,

or whether the paper is likely to be publishable within one round of revision.

Be decisive. As a referee, you are being asked to make a recommendation:

accept, revise, or reject. The reasons given for the recommendation are at least as

important as its summary value. But it is much easier for the editor to understand

which considerations the referee considers most important when a summary value

is provided. For example, if you are recommending a revise-and-resubmit, you must

give a sound reason, or the editor will likely reject the paper.

It is fine to mention reasons for uncertainty about the recommendation. If the

paper is somewhat outside your area, you might suggest that a second opinion be

sought, and you should provide names of candidate referees, and if possible, what

specific issues the alternative referee can address that you felt were outside your

area of expertise. But a request for a second opinion should not be used as a cover

for indecisiveness or not wanting to spend a lot of time studying details.

Length of the Referee Report

A referee report is not a mind-dump about the paper. Polish your report with an

eye to condensing. It is almost always possible with an extra revision by the reviewer

to make the report both shorter and more incisive.

The main purpose of the report is to help the editor decide whether to publish.

It can also be extremely valuable to have extra material to help the authors improve

the paper, but only when presented in a way that does not interfere with the main

purpose. So if comments about minor details are provided, they must be segregated

into a clearly marked separate section that is easy for the editor to skip. Broad philo-

sophical issues or rhetorical flourishes should be eliminated if they have no bearing

on improving the paper or on determining whether the paper should be published.

Finally, our suggested format of sharply separating comments which affect publish-

ability from those which are just to improve the paper will be very helpful for the

editor in making a decision efficiently and accurately.

Overly long referee reports are a burden on editors. Unless a referee needs to make

extremely technical points, 2–3 pages should be sufficient. Going beyond this raises the

likelihood of coercion/overburdening. As an editor, I view a 10-page referee report as

punitive.

We would not go so far as this editor to endorse a blanket principle that 2–3

pages is the right length for almost all papers, but brevity is valuable. Indeed,

brief reports often contain more important content than lengthy ones. We have

seen reports that are too minimal in both length and thought, but the more

Jonathan B. Berk, Campbell R. Harvey, and David Hirshleifer 241

common problem is lengthy reports that are not accompanied by correspondingly

extensive insight.

A more mechanical point, but one that has a surprisingly large effect on the

efficiency of the review process, is that comments should be numbered. A report

consisting mainly of discursive undifferentiated paragraphs is a burdensome disser-

vice to all involved in the review process. Insightful discussion is a plus, but not

at the expense of a clear bottom line that includes a numbered list of suggested

actions or problems. We suggest separate numbering for category 1 of key problems

and category 2 of suggestions.

Ethics

A referee who has any conflict of interest with the manuscript must alert the

editor—promptly, and before agreeing to accept the assignment. The editor might

decide to find a new referee or might ask the original referee to complete the

report. Conflicts that require alerting the editor include when an author of the

paper is: a past (over the past five years), current, or planned coauthor; a current

colleague; a former student or advisor; a close personal friend or family member;

or a person with a financial relationship with the reviewer. Conflicts can also arise

if you have current research that is competing with the research in the submitted

paper. If the paper contradicts or corrects your research, you need to tell the editor.

If there have been disputes between you and one of the authors in the past, alert

the editor.

Manipulation by referees of recommendations and reports for the purpose of

advancing or holding back publication with the goal of advancing their own work

is unethical. One such manipulation is for a referee to focus on adding citations to

the referee’s own work.

I received a paper that I had a low prior on that cited the work of a potential reviewer.

I assigned this person as the referee and the reviewer enthusiastically recommended

a revise and resubmit. The report made reference to an extraordinary six extra cita-

tions of the reviewer’s work. Essentially, the reviewer wanted to increase the cite

count from two to eight in the paper. After carefully reading the paper, I rejected the

manuscript.

In addition, it is not acceptable to hold up the reviewing process with the goal

of buying time for your own related research agenda.

I assigned a paper to a referee on a hot current topic. I sent many reminders to the ref-

eree because the report was very late. I then notice the referee posts a competing paper

on SSRN. The competing paper cites the paper in review (which was also public on

SSRN). I withdraw the referee request but the referee has successfully delayed the review

process for the original paper. While it seems like the referee was successful, that was not

the case. I alerted the Editors of the other top journals. I never saw the referee’s paper

in print.

242 Journal of Economic Perspectives

Of course, it is inevitable that a referee will sometimes get an idea based

on reading a submitted paper for how to write another paper on the same

broad topic. If the author has not distributed the paper publicly, and if you

have already agreed to review a paper, you cannot write your own paper on

the topic, as you are not in a position to reference the submitter’s prior work

appropriately.

In contrast, if a paper is publicly distributed, you are free to work on the same

topic (just as everyone is), as long as you clearly cite the relevant paper as prior

existing work. If you decide that you will be working on the same topic before

completing your review, you must inform the editor immediately. The editor can

then decide whether to remove you as referee; and whether to invite an addi-

tional, potentially more objective referee. To be ethical, do not advance your

new paper in your work queue for the purpose of pre-empting the submitter’s

existing paper.

Do not seek to game the system by magnifying a paper’s drawbacks or contribu-

tions. In addition to being unethical, such gaming can hurt your reputation. Editors

often catch unethical reviewer behavior.

Finally, if you are aware that the authors have behaved unethically with respect

to the submission—such as submitting essentially the same paper to multiple jour-

nals, or they have committed plagiarism—notify the editor immediately. Focus on

the facts, not on expressions of contempt or outrage.

Courtesy

Be courteous in the report and focus on substance. Avoid ascribing bad intent

to authors (“The authors were trying for a cheap publication,” “The authors were

trying to brush past literature/conflicting findings under the rug …”) and focus on

the substance of the paper. Do not insult the authors, or use overly emotional or

accusatory language.

Reports that are totally negative and nasty in language undercut the critical points

made [in the report]. And one can be critical and tough and still be polite in the write

up. If one really wants to be nasty to the authors, do it in the cover letter to the editor

and not in the report itself.

If there are indications of intellectual dishonesty, state the facts rather than

speculating on intent. If an accusation of such dishonesty is made, leave it for the

cover letter to the editor, who can then decide whether it should be passed along

to the authors.

Moreover, be mindful that referees are subject to the same behavioral biases

as everyone else. It is a mistake to approach a paper looking for evidence that

confirms your pre-existing viewpoint and discount evidence that does not. Try to

avoid forming unduly favorable assessments of work that appear to be written by

well-known authors, members of the referee’s personal social network, or papers

that cite the reviewer.

How to Write an Effective Referee Report and Improve the Scientific Review Process 243

Conclusion

The peer review process that occurs under the auspices of academic journals

is crucial for the advancement of research. It is a central mechanism by which

the profession determines collectively, through a decentralized process, both the

standards for economic research and what constitutes progress in such research.

However, we believe that fundamental change in how researchers review each

other’s journal submissions is needed to improve the integrity, quality, and effi-

ciency of the review process. We believe that such change in refereeing culture

is possible, and that when this is widely recognized, it will happen. Such change

will improve how new research is developed and communicated, and will allow

scholars to reallocate time from navigating the publication process to developing

innovative research.

Refereeing is a hard job. Unfortunately, just like others in the profession, the

three of us are all guilty of making many of the mistakes highlighted in this article.

We hope that by discussing guidelines for referees, and by pointing out some

dysfunctional features of current refereeing practices, we can improve our own

refereeing, and play some small part in changing the culture of the review process

in economics. It may be too ambitious to aim for a world in which nobody makes

the mistakes that we underscore. But to improve, we need to be cognizant of our

failings. A general awareness that certain refereeing practices are barriers to the

advancement of knowledge would be a very large step forward.

■ We thank the editors—Mark Gertler, Gordon Hanson, Enrico Moretti, and Timothy

Taylor—for extremely helpful comments.

References

Arrow, Kenneth. 1995. “Forward.” In Rejected:

Leading Economists Ponder the Publication Process,

edited by George B. Shepherd, p. vii. Sun Lakes,

AZ: Thomas Horton and Daughters.

Berk, Jonathan, Campbell R. Harvey, and David

A. Hirshleifer. 2015. “Preparing a Referee Report:

Guidelines and Perspectives.” Available at SSRN:

http://ssrn.com/abstract=2547191.

Card, David, and Stefano DellaVigna. 2013.

“Nine Facts about Top Journals in Economics.”

Journal of Economic Literature 51(1): 144–61.

Ellison, Glenn. 2000a. “Evolving Standards for

Academic Publishing: A q - r Theory.” Journal of

Political Economy 110(5): 994–1034.

Ellison, Glenn. 2002b. “The Slowdown of the

Economics Publishing Process.” Journal of Political

Economy 110(5): 947–93.

Gilliland, Stephen W., and José M. Cortina.

1997. “Reviewer and Editor Decision Making in the

Journal Review Process.” Personnel Psychology 50(2):

427–52.

Hamermesh, Daniel S. 1992. “The Young

244 Journal of Economic Perspectives

Economist’s Guide to Professional Etiquette.”

Journal of Economic Perspectives 6(1): 162–79.

Hamermesh, Daniel S. 1994. “Facts and Myths

about Refereeing.” Journal of Economic Perspectives

8(1): 153–63.

Harvey, Campbell R. 2014. “Reflections on

Editing the Journal of Finance, 2006–2012.” In

Secrets of Economics Editors, edited by Michael

Szenbert and Lall Ramrattan, 67–81. Cambridge:

MIT Press.

Hirshleifer, David. 2015. “Cosmetic Surgery in

the Academic Review Process.” Review of Financial

Studies 28(3): 637–49.

Laband, David N. 1990. “Is There Value-added

from the Review Process in Economics? Prelimi-

nary Evidence from Authors.” Quarterly Journal of

Economics 105(2): 341–52.

McAfee, R. Preston. 2010. “Edifying Editing.”

American Economist 55(1): 1–8.

Spiegel, Matthew. 2012. “Reviewing Less—

Progressing More.” Review of Financial Studies 25(5):

1331–38.

Welch, Ivo. 2014. “Referee Recomendations.”

Review of Financial Studies 27(9): 2773–2804.