New Data Debunks Old Beliefs: Part 2
Our original New data debunks old beliefs blog post plotted the impact factors of 570 randomly selected journals indexed in the 2014 Journal Citation Reports (Thomson Reuters, 2015), against their publicly stated rejection rates. The goal was to understand the relationship between rejection rates and journal quality. Despite a widespread belief that high rejection rates secure high impact factors, no significant correlation was found. This study was preliminary, to start a discussion, because it suggests that such an entrenched belief may be wrong.
This blog post is going one step further by removing what could be the main reason why we could not find any correlation: varying citation rates across academic fields. It is widely known that articles in some fields typically get more citations than in other fields. Perhaps a correlation would become evident once we removed this variable.
In Figures 1-7 below, we normalized the impact factors by field, thus effectively removing this variable from the results (data accessible here). We have done this by calculating the ranking of each journal within its own Journal Citation Reports category (or field). As an example, a journal that has the 4th largest IF among 200 journals in its category will be (1-(4/200)) or 98% percentile on the Y axis in all the figures below. In other words, this indicates that it’s impact factor is higher than 98% of journals in its category.
Thomson Reuters groups the 200+ categories within 5 major Research areas (see ref 1). We also wanted to know whether the results differed between them and if some showed a correlation between rejection rate and impact factor percentile. The 5 research areas are listed below. Figures 1 through 5 each covers one of the 5 areas.
|o Arts Humanities|
|o Life Sciences Biomedicine|
|o Physical Sciences|
|o Social Sciences|
We notice that the correlation never exceeds 0.05 (Life Sciences and Biomedicine) and is even negative for the Physical Sciences. The correlation for Arts and Humanities isn’t shown as we only obtained two data points.
Figures 6 shows all 5 Research areas combined. We notice that each has a similar spread over both axis, except for the Social Sciences which tend to have higher rejection rates (purple dots). Figure 7 is identical to Figure 6 but shows the overall correlation for all 568 journals in all 5 Research areas, which stands at 0.0187, with a P-value of 0.0011. Hence once we normalize by field, the rejection rate still only explains less than 2% of the overall variation in impact factor. Furthermore, no significant correlation exists at the Research area level either. Our initial conclusion therefore seems to hold, even when removing the field variable: Rejection rates have little influence on the impact factor.
A similar study by Aarssen et al. came to the opposite conclusion (see ref 2): that there is a strong correlation (0.687) between rejection rates and impact factors. However, their study only focused on a single field, namely Ecology while our study covers over 156 Journal Citation Reports categories (about 2/3rds of all categories). They also performed their analysis on only 60 journals, while we obtained the data for 568 journals. Lastly, their small sample size is over a decade old (2004). A more recent study will be required to determine whether the Aarssen et al. study’s conclusions still hold for Ecology today.
Figure 1: Arts and Humanities journals with rejection rates and impact factor percentiles (for sources, see below). Impact factors from Thomson Reuters Journal Citation Reports (2014).
Figure 2: Life Sciences and Biomedicine journals with rejection rates and impact factor percentiles (for sources, see below). Impact factors from Thomson Reuters Journal Citation Reports (2014).
Figure 3: Physical Sciences journals with rejection rates and impact factor percentiles (for sources, see below). Impact factors from Thomson Reuters Journal Citation Reports (2014).
Figure 4: Social Sciences journals with rejection rates and impact factor percentiles (for sources, see below). Impact factors from Thomson Reuters Journal Citation Reports (2014).
Figure 5: Technology journals with rejection rates and impact factor percentiles (for sources, see below). Impact factors from Thomson Reuters Journal Citation Reports (2014).
Figure 6: Journals from all 5 Research areas with rejection rates and impact factor percentiles (for sources, see below). Impact factors from Thomson Reuters Journal Citation Reports (2014).
Figure 7: Journals from all 5 Research areas with rejection rates and impact factor percentiles (for sources, see below). Impact factors from Thomson Reuters Journal Citation Reports (2014).
The sample is identical to the original blog post, except that 2 journals couldn’t be included. One has been removed from the 2014 Journal Citation Reports (The North American Journal of Economics and Finance) while the other journal had an impact factor mentioned on its page but that was from the 2012 JCR. It is no longer listed (although it will be back in the 2016 JCR).
3. Sources for rejection rates (by publisher):
- AAAS: http://www.the-scientist.com/?articles.view/articleNo/23672/title/Is-Peer-Review-Broken-/
- American Psychological Association:http://www.apa.org/pubs/journals/statistics.aspx (2013 data)
- American Medical Association: http://www.the-scientist.com/?articles.view/articleNo/23672/title/Is-Peer-Review-Broken-/
- AmSci: http://www.amsci.com/acceptance-rate/
- Copernicus: http://journals.ametsoc.org/doi/pdf/10.1175/2009BAMS2908.1
- Elife: http://elifesciences.org/elife-news/A-flavour-of-eLife
- Elsevier: http://journalfinder.elsevier.com/#results (this is a journal finder)
- Frontiers: Internal data for 2013 spontaneous submissions (Original research articles)
- Hindawi: All data is from individual journal websites. Has been done for all journals that have an impact factor.
- MDPI: http://blog.mdpi.com/2014/03/20/kudos-to-our-peer-reviewers/
- NPG: Various sources Nature Materials:http://www.nature.com/nmat/journal/v11/n9/full/nmat3424.html;
- Nature: http://academia.stackexchange.com/questions/8755/what-proportion-of-papers-submitted-to-nature-are-actually-sent-for-review;
- Nature Communications: oaspa.org/wp-content/uploads/2012/11/James-Butcher-NPG.pptx;
- Nature Neuroscience:http://www.nature.com/neuro/journal/v11/n5/full/nn0508-521.html;
- Scientific Reports: http://occamstypewriter.org/trading-knowledge/2012/07/09/megajournals/)
- PeerJ: https://peerj.com/about/FAQ/
- Taylor and Francis: http://explore.tandfonline.com/content/beh/ntcn-call-for-editor-in-chief
- The BMJ: http://www.bmj.com/about-bmj/resources-authors/forms-policies-and-checklists/bmj-right-journal-my-research-article
The results are interesting, especially the scatterplots. Considering Figure 7 for example, the regression line doesn’t come close to the most concentrated region of points, as would be expected if the standard model holds, i.e., conditional equal variance normality. I’d wonder if the data supported a mixture of (two or more) regressions, one of which might pick up the upper right portion of the plot where the point density is the greatest. Even if this were so however, it leaves open, for me at least, what a substantive interpretation might be.
It seems to me that the journals with very low rejection rate and with high IF are the journals where all or most of their papers are commissioned and are mainly topical reviews. Of course, such journals should not be even compared with classical journals where ordinary mortals submit their recent research papers.
Great post, but I do not understand why fig 3 has a negative curve with a positive r value. Additionally though correlations are clear there seems to be a clear higher density of journals above 60% impact with high rejection rates. What percentage of the total do these journals represent?
Statistics in Figures are R^2, not r, so they are always positive. I guess they try to show how much (more appropriately, how few) variability in (these) journal’s IFs is explained by a linear association to the rejection rate.
This analysis seems to ignore the feedback loop where authors submit weaker research to lower journals, and vice versa. If impact-based rejection has any effect at all in selecting high quality impactful papers, then a more selective journal (higher reject rate) will print only the papers at the high end of the input submission distribution. That will however feed back such that the submission pool will then change over time.
i.e. Reject more – better papers printed – journal rep increases – get better submissions – no need to reject as many. Rejection rate is therefore self-adjusting to a (large?) extent. If everyone submitted the papers they normally submit to PLoS One to Nature instead, Nature’s rejection rate would skyrocket.
The prediction is not that rejection rate should correlate with impact factor, but that rejection rate should correlate with the _rate of change of impact factor_. Journals that are more selective than average should move up the chart, while journals that are less selective than average will move down the chart.