The benefits to climate science of including early-career scientists as reviewers

Early-career scientists (ECSs) are a large part of the workforce in science. While they produce new scientific knowledge that they share in publications, they are rarely invited to participate in the peer-review process. Barriers to the participation of ECSs as peer reviewers include, among other things, their lack of visibility to editors, inexperience in the review process and lack of confidence in their scientific knowledge. Participation of ECSs in group reviews, e.g. for regional or global assessment reports, provides an opportunity for ECSs to advance their skill set and to contribute to policy-relevant products. Here, we present the outcomes of a group peer review of the First Order Draft of the Intergovernmental Panel on Climate Change (IPCC) Special Report on the Ocean and Cryosphere in a Changing Climate (SROCC). Overall, PhD students spent more time on the review than those further advanced in their careers and provided a similar proportion of substantive comments. After the review, participants reported feeling more confident in their skills, and 86 % were interested in reviewing individually. By soliciting and including ECSs in the peer-review process, the scientific community would not only reduce the burden carried by more established scientists but also permit their successors to develop important professional skills relevant to advancing climate science and influencing policy.

1 Introduction

Acting as peer reviewers is an important opportunity for scientists to recognise the components of strong scientific papers, and it can help to improve the quality of their own work (Silver, 2016; Lerback and Hanson, 2017). While several training opportunities exist for early-career scientists (ECSs; also referred to as early-career researchers – ECRs) to bolster their voice and develop writing skills and leadership (Geffers et al., 2017), only 6 % of journal articles are reviewed by ECSs (Taylor and Francis, 2016).

Global assessment reports rely on reviewers from multiple regions and diverse scientific disciplines to ensure that they are scientifically accurate and are widely understandable. Here, we focus on a report produced by the Intergovernmental Panel on Climate Change (IPCC) that provides an assessment of the scientific, technical and socio-economic literature on the current state of knowledge on climate change (IPCC, 2013a). Ensuring effective climate change adaptation and mitigation requires policymakers to be informed by the scientific community through robust and evidence-based reports reflecting the scientific consensus (Bolin, 2007; Tollefson, 2010; Ding et al., 2011; Lewandowsky et al., 2013). To best achieve this goal, the scientific community requires the inclusion of scientists from heterogeneous backgrounds and experiences (Maibach et al., 2014; Hallegatte et al., 2016), including ECSs, who can provide diverse perspectives. Each IPCC report undergoes a multistage review process by expert and government representatives (IPCC, 2013b). For example, the Working Group I Fifth Assessment Report (WGI AR5) attracted comments from 1089 expert reviewers from 55 countries. While no individual possesses the required expertise to review an entire IPCC report, as a group, ECSs have also proven to be efficient and motivated reviewers, providing added value to this type of manuscript as shown by van der Veer et al. (2014), who pioneered the crowdsourcing and organisation of groups of ECSs to participate in the IPCC report review process. By serving as reviewers, ECSs support the IPCC assessment process and learn more about interdisciplinary endeavours while developing new skills for synthesising their own research into the limited scope of a publishable paper.

Through the Association of Polar Early Career Scientists (APECS), ECSs were recruited to participate in a group review of the First Order Draft (FOD) of the IPCC Special Report on the Ocean and Cryosphere in a Changing Climate (SROCC). APECS is an international and interdisciplinary network for undergraduate and graduate students, postdoctoral researchers (postdocs), early-career professionals, educators, and others interested in polar and alpine regions and the wider cryosphere (Allen et al., 2014). While most ECSs in APECS are PhD students or postdocs, membership also includes master students, undergraduate students, early-career academics and other educators. APECS thus organised a group review of the SROCC which included 75 ECSs from 22 of the countries within the APECS network (Fig. 1a). The pool of reviewers was composed of 38 women and 37 men. Earth sciences were most represented among the reviewers, while other disciplines included biology, geography, social sciences and civil engineering (Fig. 1b).

Recognising that some of the reviewers had neither published a paper nor participated in peer review before, APECS designed a comprehensive training programme, providing ECSs with a rigorous reviewing frame, ensuring that the reviews produced would all be of equal quality. Here, we report the first results of this programme and highlight that the competence of a reviewer is not influenced by career level. We first describe the methods, how the group review was organised and the statistics applied for this study. We then make use of the statistics and of the response to a post-review survey to explore the number of comments by career level. Finally, we conclude with a list of propositions for editors to encourage the inclusion of ECSs in review processes.

2 Methods

2.1 IPCC review process

The drafts of the IPCC reports undergo two rounds of review which have an open call to the scientific community to participate as expert reviewers. Here, for the SROCC, we focus on the first of these, the expert review of the FOD. The expert review process is open, transparent and objective, with wide participation involving independent experts from a range of scientific disciplines, regions and countries. Expert reviewers self-nominate through a registration process, and, after being approved and signing a confidentiality agreement, expert reviewers receive an individually watermarked draft of chapters and a pre-formatted spreadsheet for entering review comments. The individual expert reviewer spreadsheets are submitted to the Technical Support Unit (TSU), which combines and sends all the review comments to report authors for consideration and response. All comments are assigned to an expert reviewer, and the report authors provide a written response to each review comment. For the SROCC, a process was also put in place for group reviews, where multiple reviewers work as a team, with a single spreadsheet being delivered by the group and attributed to the group name (e.g. APECS) and list of all expert reviewers participating in the group review submitted to the Working Group II TSU. The names of expert reviewers, their comments and IPCC author responses to the comments are freely available once the report is released after its approval by member governments (for the SROCC on 25 September 2019).

For the SROCC, comments could be entered under two categories “editorial” and “substantive”. Editorial comments are those related to language, sentence structure and formatting. Substantive comments are those focused on content and structure of the report and chapters, concepts, literature assessed, confidence language, conclusions, gaps, overlaps, and the use of and content within figures and tables. Expert reviewers are encouraged to focus on the latter, as the final report undergoes professional copy editing before publication. Expert reviewers can choose to review the whole report, chapter(s) within the report or section(s) of chapters most relevant to their scientific expertise.

2.2 Organisation and recruitment of the APECS group review

The project was led by 10 council members of APECS from various academic levels: master (1), PhD (4), postdoc (3) and early-career academics (2), representing six different countries (Canada, Chile, France, Germany, Italy and the USA). Each chapter was chaired by two to five of these project leaders, whose roles were (a) selecting the participants, (b) assigning chapter sections to the participants, (c) reviewing and sorting the comments (e.g. finding duplicates), and (d) finalising the global review sent by APECS.

The call for participants, published on the APECS website, requested information about applicants' credentials and motivations for participating in the group review and received 153 applications. Among the applicants, 72 % had already reviewed a scientific document (such as a paper, a proposal or a scientific report). Not all chapters received the same number of applicants (the outline of the report can be found at https://www.ipcc.ch/site/assets/uploads/2018/11/Decision_Outline_SR_Oceans.pdf, last access: 9 April 2020). For instance, the chapter on “Polar Regions” (estimated at 50 pages) was requested by 110 applicants, while the chapter on “Changing Ocean, Marine Ecosystems, and Dependent Communities” (estimated at 65 pages) was only requested by 33 applicants.

As the number of applications from less experienced scientists (PhD students and below) was roughly equal to the number of postdocs and early-career academics, it was decided to review the applications separately, opening the same number of positions for both categories. We used the applications to evaluate (1) motivation, (2) experience and relevance of the application, (3) country of residence, and (4) equal distribution amongst chapters, meaning that applicants who applied to more than one chapter were often assigned to a secondary choice.

Table 1 Number of participants and their countries of origin (including the project leaders); see also Fig. 1a.

The resulting selection of participants in the review process strongly reflects the established network of APECS members and mailing list recipients (Table 1). While gender was not a criterion for selection, 51 % of the participants were women. Only 5 of the 22 countries represented are developing countries or economies in transition. The selected participants included a majority of PhD students, a similar number of postdocs and early-career academics, and only two undergraduate and master students (Table 2).

Table 2 List of academic levels of the participants (including the project leaders).

The horizontal structure of this group review (all participants are ECSs self-organising to realise this review) strongly differs from the previous group review attempts of an IPCC review project, for which a government was organising the review and providing incentives to participants to participate in such review (van der Veer et al., 2014).

2.3 Training of the ECS reviewers

In order to introduce the review process to participants, a guide was created by the project leaders from APECS, validated by the APECS executive director and reviewed by members of the IPCC TSUs from the two working groups (WGs), providing scientific leadership to the SROCC (WGI: The Physical Science Basis; WGII: Impacts, Adaptation and Vulnerability). This guide explained the process including the objectives, timeline, leadership and rules. The guide is available in the Supplement.

Two series of online workshops were organised to train the participants in the review process. To facilitate participation from numerous time zones, two sessions were organised for each workshop series at different times, and recordings were made available for later viewing (session 1: https://vimeo.com/292679338, last access: 9 April 2020, session 2.1: https://vimeo.com/292679417, last access: 9 April 2020; session 2.2: https://vimeo.com/292679451, last access: 9 April 2020). Each online workshop lasted between 1.5 and 2 h.

The first series of workshops was conducted by members of the IPCC TSUs (WGI: Sarah Connors; WGII: Katja Mintenbeck and Elvira Poloczanska), who introduced the IPCC and its processes, discussed the purpose and government-approved structure of the SROCC, and advised participants on what entails a constructive review. The second workshop involved vice-chairs of WGI and WGII and was held after the distribution of the respective chapters and sections to the participants. Greg Flato (Canada, WGI vice-chair, workshop session 1) and Andreas Fischlin (Switzerland, WGII vice-chair, workshop session 2) discussed the use of the IPCC uncertainty language (https://www.ipcc.ch/site/assets/uploads/2018/05/uncertainty-guidance-note.pdf, last access: 9 April 2020), which is a key component of IPCC assessments. Other advice for reviewers was provided by Carlos Mendez (Venezuela, WGII vice-chair) and Jan Fuglestvedt (Norway, WGI vice-chair) for the first and second session, respectively. All documents shared during these workshops (i.e. PowerPoint presentation or minutes) were made available to the participants for later viewing. Furthermore, the guide was amended during the training process to include updated information.

2.4 Organisation of the APECS group review

The review of the FOD of the SROCC was done as a group review, meaning that each participant of the group was only in charge of reviewing a small section of the report and that all the comments were then combined and submitted together under the flag of APECS. Each chapter was distributed by the project leaders to the participants. Depending on the number of participants for each chapter, 10 to 20 pages were assigned to each participant to review. In addition, the entire chapter was made available to participants so that they could provide additional comments outside of the assigned pages and be able to put their assigned section into context. We attempted to assign whole sections as much as possible. We also attempted to balance the workload and in some instances, reviewers were assigned non-contiguous sections to even out the number of pages they were responsible for.

Each selection of pages was given to at least two categories of participants: university students (bachelor, master and PhD students) and postdocs or early-career academics. The initial purpose was to promote interactions amongst participants during the review process, through online exchanges and, in particular, to encourage more experienced ECSs to provide advice. Based on the participants' feedback, it seems that such an exchange between the paired participants did not ultimately occur, and solutions to enhance this mentoring will be applied in the next round of group reviews of the SROCC organised by APECS.

Though participants themselves chose the chapter which they would have to review, a significant number of concerns were raised from participants that felt that the section they were assigned did not correspond to their particular expertise. While this was a valid concern, it was not possible to screen which section would be best suited for each participant and still cover the entire report, as the content of the report was unknown beforehand. As the primary purpose of this report is to inform policymakers, who will not necessarily have scientific expertise concerning the content of the report, having peripheral expertise would not preclude participants' ability to review a given section, and instead we deemed this an asset. In the forthcoming round of the group reviews, applicants will request the specific sections that they want to review rather than chapters, in order to have a more specific application process.

Figure 1 Summary of the results from the APECS group review: (a) geographical distribution of ECSs who participated in the process and the number of participants per country, (b) disciplines of the participants involved in the group review, and (c) comments and time spent per participant by academic stage (the average training time was 3.5 h per participant, regardless of career level).

The participants had a month to review their assigned section. After the deadline, project leaders spent another month compiling the 2155 received comments into one document, removing potential duplicate comments and filtering those interpreted as out of the scope (see Fig. 1c) to ensure the relevance of the comments transferred to the IPCC. The total workload of the participants (obtained through a survey after the review) was less than the project leaders, who spent an estimated 40 h preparing the project, participating in the webinars, reading the different chapters in which they were involved and sorting all the comments. Following discussions with authors of IPCC reports and with members of the IPCC TSU, the 2155 comments were sorted by the project leaders as “editorial”, “substantive” and “unfit for submission”. Editorial comments corresponded to suggestions such as copy editing and reference corrections; substantive comments were those indicating errors, the need for new and more relevant references, and new content. Comments deemed unfit for submission included mostly duplicates from several reviewers and, in some rare cases, inappropriate comments. We acknowledge that the comment sorting includes some level of subjectivity, as a single chair was evaluating each comment's categorisation (in addition to self-evaluation from the participants), and as a result, we only report the total number of comments for each category and do not evaluate the quality of the review in this paper. This procedure resulted in a list of 2012 comments submitted to the IPCC. This sorting and organisation was carried out during the 3 weeks that followed the receipt of comments from the participants, leaving 5 d to combine the selected comments and to upload them to the IPCC website.

3 Results and discussion

3.1 Analyses of the number of APECS review comments

As described above, the comments were sorted by the APECS project leaders and not by the IPCC. Nonetheless, as the same criteria were endorsed and applied by all project leaders; no obvious difference in the distribution of comments from group review participants was evident.

Statistical analyses of the different number of comments were conducted to evaluate the significance of the obtained figures. The average number of comments per person was 31.8±4.6 (errors on the average; sample size n=61). The average and standard deviation of the numbers of comments for PhD, postdocs and early-career academics are 39.9±6.9, 28.9±7.7 and 34.9±9.5 respectively (sample sizes n=26, 21 and 14 respectively). These averages are relatively similar to the range calculated from the overlap of the standard deviations, suggesting that these three figures are not significantly different. However, the comments of only one participant from the undergraduate and master academic level were obtained (the second one only acted as a project leader and did not provide comments); thus these were not included in the analyses.

For the proportion of comments deemed substantive, editorial and unfit for submission, no significant differences between either the number or the proportion were found for each of the three academic levels detailed here. As the distribution of number of comments for each category and academic level were not normal, we performed a Kruskall–Wallis test and obtained χ²=0.080 and a p value = 0.957, confirming that there were no significant differences in the proportion of comment types for each academic level.

In contrast, the average time spent by reviewers from each academic levels was, at least slightly, significantly different (χ²=7.16; p value = 0.067), with groups varying by roughly 1 h, and the average error for each academic level below ±0.6 h.

3.2 Discussion

The participants produced a total of 2155 comments, with a mean of 31.8 comments per participant. Of the 2155 comments collected, 693 were considered substantive; 1319 were considered editorial; and 143 were considered unfit for submission. To explore potential links between the type of comments made by the participants and their academic level, the average numbers of comments (divided by category: substantive, editorial or unfit for submission) were plotted by academic level (PhD, postdoc or early-career academics; see Fig. 1c). PhD students provided the largest number of comments per participant (almost 40), of which 30 % were substantive comments. Postdocs and early-career academics provided a slightly higher percentage of substantive comments with 34 % and 36 % respectively. The number of comments deemed unfit for submission was highest for early-career academics (10 %), followed by PhD students (6 %) and postdocs (4 %), noting that this category included duplicate, thus redundant, comments. Amongst the different academic levels, no significant difference in the total number and category of comments was noticeable between the different career stages (see statistical analyses in Methods).

Compared to the 388 other expert reviewers of the SROCC FOD, the APECS group review (accounting for a single reviewer) provided 2012 comments to the SROCC authors, which was 17 % of the 12 002 total number of comments (https://www.ipcc.ch/srocc/download/, last access: 18 April 2020). The 388 other experts produced on average 26 comments per person, slightly less than the ECS participants (31.8, the difference being above 1 standard deviation), who had a specific set of 10 to 20 pages each to review, rather than an entire chapter. Van der Veer et al. (2014) proposed that a large number of comments does not necessarily guarantee better quality, but as we only self-evaluated the quality of the comments, we are not able to validate this point with our results.

The PhD students provided as many substantive comments as the more experienced participants of the group review (i.e. postdocs and early-career academics); thus the length of the academic career was ruled out as a factor in the ability to effectively produce reviews. A more comprehensive analysis would benefit from the use of several metrics to determine the quality of the review and the link to the length of academic careers. The comparison with other expert reviewers also supports that ECSs are as productive and efficient reviewers as their more senior peers (Schiermeier, 2016). As 20 % of researchers take on 70 % to 90 % of the peer-review burden (Kovanis et al., 2016; Taylor and Francis, 2016), the peer-review process would be more equitably distributed across the scientific community by the solicitation and inclusion of PhD students and other ECSs as peer reviewers. Widening the reviewer pool could also reduce potential conflicts of interest in the review process and increase its quality. Considering the need for increased transparency in the peer-review process of IPCC reports following false accusations raised in reaction to the publication of the Second Assessment Report (SAR/AR2) (Edwards and Schneider, 2001), this latter aspect is particularly relevant for the climate change community.

Following the completion of the group review, participants were surveyed on their experiences. On average, the participants spent 7 h preparing prior to the review. Half of this time was spent reviewing the guide. The other half was spent attending the two training webinars on the review process and answering pending questions. Nearly all (98 %) of the participants stated that the amount of training provided was adequate.

APECS participants spent an average of 7.3 h reviewing their assigned content (10 to 20 pages; Fig. 1c). However, 6 % of the participants spent less than 3 h on the review, and 35 % spent 10 h or more. Overall, participants spent less time than they had anticipated (35 h) on the review. The relatively reduced time commitment might make participating in future IPCC reviews more appealing to ECSs, not only as a group, but also individually. Indeed, based on the survey responses, the time spent to review a chapter of an IPCC report (between 80 and 120 pages) can be estimated at less than 1 week for reviewers of all career stages.

Participants were asked about their willingness to take part in another review as part of a group or individually (Fig. 2a). The majority (92 %) of participants stated that they would take part in another group review, and 86 % reported that they would participate in an individual review, adding more than 60 potential reviewers to the climate change community. Compared to the total number of experts who reviewed the FOD of the SROCC (389, including participants of the APECS group review), this means that including more ECSs in the peer-review process is a significant addition to the pool of expert reviewers. We expect similar results for other scientific fields and hope that other professional societies will consider collaborating in such group reviews in the future.

The most common motivations that ECSs stated for participating in this review process were learning more about the IPCC (59 %) and experience building (53 %; Fig. 2b). These survey responses, combined with the time commitment that reviewers within our group were ready to invest in this project (7 h of preparation and up to 35 h on the review), indicate that ECSs are highly committed to opportunities that involve international-policy-relevant processes. ECSs also value the experience of learning how to review and improve manuscripts. For example, 82 % declared having achieved personal goals by participating in this review.

Prior to the review, one major concern of many participants was a lack of confidence in the value of their feedback to the IPCC, either because they were unsure about the quality of their comments or the sufficiency of their expertise, as the report covers a large ensemble of disciplines that might not fit their breadth of experience. Because the primary audience of IPCC reports are policymakers and stakeholders who may not have scientific backgrounds, IPCC reports should be written unambiguously. Even ECSs who do not have expertise in a particular topic can still provide constructive comments that strengthen the clarity of a report and the discussion of key concepts. Thus, in our group review, participants could choose the chapter they wished to review, but we randomly assigned within the sections this chapter they actually reviewed, regardless of their expertise or desire. The influence of the assignment process could not be assessed during this project.

Figure 2 (a) Participants' answers to the questions: “Would you be willing to participate in another group (left pie chart) and/or individual (right pie chart) review?” and (b) motivations for participating in such a review.

4 Conclusions

This review included a large and diverse group of ECSs, both in terms of disciplines and countries represented. Participating in such a project furthers ECS career development (Moore et al., 2018), particularly in building skills towards connecting science and policy (Petes and Meyer, 2018). It was also an opportunity for ECSs to gather ideas for their future research projects, as knowledge gaps are clearly identified in the reports. The climate change scientific community needs to train more scientists to tackle future challenges (Goswami et al., 2015), and this type of initiative represents a valuable mechanism to prepare future IPCC authors. Influencing the policy-making process requires persistence (Weible et al., 2012); thus, by including ECSs in the review of IPCC reports, the probability that they have an impact on climate-related policy during their careers increases (Evans and Cvitanovic, 2018).

Participating as reviewers has many benefits for ECSs, i.e. developing skills such as time management, responsible authorship, review and publication practices, and getting recognition for critical review skills within the scientific community. For the scientific community, it also increases the reviewer pool, which could alleviate the workload of senior scientists, and hopefully enhances the scientific rigour of journal articles and reports that support policy-making processes.

APECS is a volunteer-driven, professional society that seeks to provide career development opportunities for ECSs. APECS recognises the valuable contributions that ECSs make to furthering international scientific efforts and communicating the results of these studies to policymakers and the public. APECS organised and trained ECSs to review a large policy-relevant scientific report because there are few peer-review training opportunities for ECSs. APECS hopes that this initiative, which demonstrated that ECSs can serve as adequate peer reviewers, will inspire journal editors and report managers to include more ECSs as reviewers.

There are several scientific societies, which support climate change research and ECSs, which could organise their own group review of future IPCC draft reports. While we use the review of the IPCC report as a case study for the climate change community, we expect that similar results would be found for ECSs across scientific disciplines and for journal articles (Ilgen et al., 2016). We note that advice for ECSs to produce robust reviews can easily be found (Schiermeier, 2016; Silver, 2016), so instead, we offer recommendations to editors of journals and managers of assessment reports to include more ECSs in their review processes:

• Proactively seek to include at least one ECS when inviting reviewers for a manuscript. This could be accomplished by requesting that senior reviewers suggest a suitable ECS to review the manuscript.

• Include ECSs as guest editors. This would diversify perspectives of editorial boards.

• Encourage reviewers to review the sections of a manuscript they feel most comfortable with. This may facilitate participation both for new reviewers and senior scientists who may be too busy to review the entire manuscript.

• Keep reviewers blind to other reviewers' comments during the review phase. As recommended in Besançon et al. (2019), this will facilitate independent feedback, especially for ECSs, who may hesitate to disagree with more senior reviewers.

• Consider methods to encourage and train ECSs to act as reviewers. This could be in group reviews or mentoring programmes (De Vries et al., 2009; Walker, 2018), potentially using ECS-based networks (e.g. APECS) as a focal point for such training, in particular for thematic special issues.