The drafts of the IPCC reports undergo two rounds of review which have an open call to the scientific community to participate as expert reviewers. Here, for the SROCC, we focus on the first of these, the expert review of the FOD. The expert review process is open, transparent and objective, with wide participation involving independent experts from a range of scientific disciplines, regions and countries. Expert reviewers self-nominate through a registration process, and, after being approved and signing a confidentiality agreement, expert reviewers receive an individually watermarked draft of chapters and a pre-formatted spreadsheet for entering review comments. The individual expert reviewer spreadsheets are submitted to the Technical Support Unit (TSU), which combines and sends all the review comments to report authors for consideration and response. All comments are assigned to an expert reviewer, and the report authors provide a written response to each review comment. For the SROCC, a process was also put in place for group reviews, where multiple reviewers work as a team, with a single spreadsheet being delivered by the group and attributed to the group name (e.g. APECS) and list of all expert reviewers participating in the group review submitted to the Working Group II TSU. The names of expert reviewers, their comments and IPCC author responses to the comments are freely available once the report is released after its approval by member governments (for the SROCC on 25 September 2019).

For the SROCC, comments could be entered under two categories “editorial” and “substantive”. Editorial comments are those related to language, sentence structure and formatting. Substantive comments are those focused on content and structure of the report and chapters, concepts, literature assessed, confidence language, conclusions, gaps, overlaps, and the use of and content within figures and tables. Expert reviewers are encouraged to focus on the latter, as the final report undergoes professional copy editing before publication. Expert reviewers can choose to review the whole report, chapter(s) within the report or section(s) of chapters most relevant to their scientific expertise.

The project was led by 10 council members of APECS from various academic levels: master (1), PhD (4), postdoc (3) and early-career academics (2), representing six different countries (Canada, Chile, France, Germany, Italy and the USA). Each chapter was chaired by two to five of these project leaders, whose roles were (a) selecting the participants, (b) assigning chapter sections to the participants, (c) reviewing and sorting the comments (e.g. finding duplicates), and (d) finalising the global review sent by APECS.

The call for participants, published on the APECS website, requested information about applicants' credentials and motivations for participating in the group review and received 153 applications. Among the applicants, 72 % had already reviewed a scientific document (such as a paper, a proposal or a scientific report). Not all chapters received the same number of applicants (the outline of the report can be found at https://www.ipcc.ch/site/assets/uploads/2018/11/Decision_Outline_SR_Oceans.pdf, last access: 9 April 2020). For instance, the chapter on “Polar Regions” (estimated at 50 pages) was requested by 110 applicants, while the chapter on “Changing Ocean, Marine Ecosystems, and Dependent Communities” (estimated at 65 pages) was only requested by 33 applicants.

As the number of applications from less experienced scientists (PhD students and below) was roughly equal to the number of postdocs and early-career academics, it was decided to review the applications separately, opening the same number of positions for both categories. We used the applications to evaluate (1) motivation, (2) experience and relevance of the application, (3) country of residence, and (4) equal distribution amongst chapters, meaning that applicants who applied to more than one chapter were often assigned to a secondary choice.

The resulting selection of participants in the review process strongly reflects the established network of APECS members and mailing list recipients (Table 1). While gender was not a criterion for selection, 51 % of the participants were women. Only 5 of the 22 countries represented are developing countries or economies in transition. The selected participants included a majority of PhD students, a similar number of postdocs and early-career academics, and only two undergraduate and master students (Table 2).

The horizontal structure of this group review (all participants are ECSs self-organising to realise this review) strongly differs from the previous group review attempts of an IPCC review project, for which a government was organising the review and providing incentives to participants to participate in such review (van der Veer et al., 2014).

In order to introduce the review process to participants, a guide was created by the project leaders from APECS, validated by the APECS executive director and reviewed by members of the IPCC TSUs from the two working groups (WGs), providing scientific leadership to the SROCC (WGI: The Physical Science Basis; WGII: Impacts, Adaptation and Vulnerability). This guide explained the process including the objectives, timeline, leadership and rules. The guide is available in the Supplement.

Two series of online workshops were organised to train the participants in the review process. To facilitate participation from numerous time zones, two sessions were organised for each workshop series at different times, and recordings were made available for later viewing (session 1: https://vimeo.com/292679338, last access: 9 April 2020, session 2.1: https://vimeo.com/292679417, last access: 9 April 2020; session 2.2: https://vimeo.com/292679451, last access: 9 April 2020). Each online workshop lasted between 1.5 and 2 h.

The first series of workshops was conducted by members of the IPCC TSUs (WGI: Sarah Connors; WGII: Katja Mintenbeck and Elvira Poloczanska), who introduced the IPCC and its processes, discussed the purpose and government-approved structure of the SROCC, and advised participants on what entails a constructive review. The second workshop involved vice-chairs of WGI and WGII and was held after the distribution of the respective chapters and sections to the participants. Greg Flato (Canada, WGI vice-chair, workshop session 1) and Andreas Fischlin (Switzerland, WGII vice-chair, workshop session 2) discussed the use of the IPCC uncertainty language (https://www.ipcc.ch/site/assets/uploads/2018/05/uncertainty-guidance-note.pdf, last access: 9 April 2020), which is a key component of IPCC assessments. Other advice for reviewers was provided by Carlos Mendez (Venezuela, WGII vice-chair) and Jan Fuglestvedt (Norway, WGI vice-chair) for the first and second session, respectively. All documents shared during these workshops (i.e. PowerPoint presentation or minutes) were made available to the participants for later viewing. Furthermore, the guide was amended during the training process to include updated information.

The review of the FOD of the SROCC was done as a group review, meaning that each participant of the group was only in charge of reviewing a small section of the report and that all the comments were then combined and submitted together under the flag of APECS. Each chapter was distributed by the project leaders to the participants. Depending on the number of participants for each chapter, 10 to 20 pages were assigned to each participant to review. In addition, the entire chapter was made available to participants so that they could provide additional comments outside of the assigned pages and be able to put their assigned section into context. We attempted to assign whole sections as much as possible. We also attempted to balance the workload and in some instances, reviewers were assigned non-contiguous sections to even out the number of pages they were responsible for.

Each selection of pages was given to at least two categories of participants: university students (bachelor, master and PhD students) and postdocs or early-career academics. The initial purpose was to promote interactions amongst participants during the review process, through online exchanges and, in particular, to encourage more experienced ECSs to provide advice. Based on the participants' feedback, it seems that such an exchange between the paired participants did not ultimately occur, and solutions to enhance this mentoring will be applied in the next round of group reviews of the SROCC organised by APECS.

Though participants themselves chose the chapter which they would have to review, a significant number of concerns were raised from participants that felt that the section they were assigned did not correspond to their particular expertise. While this was a valid concern, it was not possible to screen which section would be best suited for each participant and still cover the entire report, as the content of the report was unknown beforehand. As the primary purpose of this report is to inform policymakers, who will not necessarily have scientific expertise concerning the content of the report, having peripheral expertise would not preclude participants' ability to review a given section, and instead we deemed this an asset. In the forthcoming round of the group reviews, applicants will request the specific sections that they want to review rather than chapters, in order to have a more specific application process.

The participants had a month to review their assigned section. After the deadline, project leaders spent another month compiling the 2155 received comments into one document, removing potential duplicate comments and filtering those interpreted as out of the scope (see Fig. 1c) to ensure the relevance of the comments transferred to the IPCC. The total workload of the participants (obtained through a survey after the review) was less than the project leaders, who spent an estimated 40 h preparing the project, participating in the webinars, reading the different chapters in which they were involved and sorting all the comments. Following discussions with authors of IPCC reports and with members of the IPCC TSU, the 2155 comments were sorted by the project leaders as “editorial”, “substantive” and “unfit for submission”. Editorial comments corresponded to suggestions such as copy editing and reference corrections; substantive comments were those indicating errors, the need for new and more relevant references, and new content. Comments deemed unfit for submission included mostly duplicates from several reviewers and, in some rare cases, inappropriate comments. We acknowledge that the comment sorting includes some level of subjectivity, as a single chair was evaluating each comment's categorisation (in addition to self-evaluation from the participants), and as a result, we only report the total number of comments for each category and do not evaluate the quality of the review in this paper. This procedure resulted in a list of 2012 comments submitted to the IPCC. This sorting and organisation was carried out during the 3 weeks that followed the receipt of comments from the participants, leaving 5 d to combine the selected comments and to upload them to the IPCC website.

As described above, the comments were sorted by the APECS project leaders and not by the IPCC. Nonetheless, as the same criteria were endorsed and applied by all project leaders; no obvious difference in the distribution of comments from group review participants was evident.

Statistical analyses of the different number of comments were conducted to evaluate the significance of the obtained figures. The average number of comments per person was 31.8±4.6 (errors on the average; sample size n=61). The average and standard deviation of the numbers of comments for PhD, postdocs and early-career academics are 39.9±6.9, 28.9±7.7 and 34.9±9.5 respectively (sample sizes n=26, 21 and 14 respectively). These averages are relatively similar to the range calculated from the overlap of the standard deviations, suggesting that these three figures are not significantly different. However, the comments of only one participant from the undergraduate and master academic level were obtained (the second one only acted as a project leader and did not provide comments); thus these were not included in the analyses.

For the proportion of comments deemed substantive, editorial and unfit for submission, no significant differences between either the number or the proportion were found for each of the three academic levels detailed here. As the distribution of number of comments for each category and academic level were not normal, we performed a Kruskall–Wallis test and obtained χ2=0.080 and a p value = 0.957, confirming that there were no significant differences in the proportion of comment types for each academic level.

In contrast, the average time spent by reviewers from each academic levels was, at least slightly, significantly different (χ2=7.16; p value = 0.067), with groups varying by roughly 1 h, and the average error for each academic level below ±0.6 h.

The participants produced a total of 2155 comments, with a mean of 31.8 comments per participant. Of the 2155 comments collected, 693 were considered substantive; 1319 were considered editorial; and 143 were considered unfit for submission. To explore potential links between the type of comments made by the participants and their academic level, the average numbers of comments (divided by category: substantive, editorial or unfit for submission) were plotted by academic level (PhD, postdoc or early-career academics; see Fig. 1c). PhD students provided the largest number of comments per participant (almost 40), of which 30 % were substantive comments. Postdocs and early-career academics provided a slightly higher percentage of substantive comments with 34 % and 36 % respectively. The number of comments deemed unfit for submission was highest for early-career academics (10 %), followed by PhD students (6 %) and postdocs (4 %), noting that this category included duplicate, thus redundant, comments. Amongst the different academic levels, no significant difference in the total number and category of comments was noticeable between the different career stages (see statistical analyses in Methods).

Compared to the 388 other expert reviewers of the SROCC FOD, the APECS group review (accounting for a single reviewer) provided 2012 comments to the SROCC authors, which was 17 % of the 12 002 total number of comments (https://www.ipcc.ch/srocc/download/, last access: 18 April 2020). The 388 other experts produced on average 26 comments per person, slightly less than the ECS participants (31.8, the difference being above 1 standard deviation), who had a specific set of 10 to 20 pages each to review, rather than an entire chapter. Van der Veer et al. (2014) proposed that a large number of comments does not necessarily guarantee better quality, but as we only self-evaluated the quality of the comments, we are not able to validate this point with our results.

The PhD students provided as many substantive comments as the more experienced participants of the group review (i.e. postdocs and early-career academics); thus the length of the academic career was ruled out as a factor in the ability to effectively produce reviews. A more comprehensive analysis would benefit from the use of several metrics to determine the quality of the review and the link to the length of academic careers. The comparison with other expert reviewers also supports that ECSs are as productive and efficient reviewers as their more senior peers (Schiermeier, 2016). As 20 % of researchers take on 70 % to 90 % of the peer-review burden (Kovanis et al., 2016; Taylor and Francis, 2016), the peer-review process would be more equitably distributed across the scientific community by the solicitation and inclusion of PhD students and other ECSs as peer reviewers. Widening the reviewer pool could also reduce potential conflicts of interest in the review process and increase its quality. Considering the need for increased transparency in the peer-review process of IPCC reports following false accusations raised in reaction to the publication of the Second Assessment Report (SAR/AR2) (Edwards and Schneider, 2001), this latter aspect is particularly relevant for the climate change community.

Following the completion of the group review, participants were surveyed on their experiences. On average, the participants spent 7 h preparing prior to the review. Half of this time was spent reviewing the guide. The other half was spent attending the two training webinars on the review process and answering pending questions. Nearly all (98 %) of the participants stated that the amount of training provided was adequate.

APECS participants spent an average of 7.3 h reviewing their assigned content (10 to 20 pages; Fig. 1c). However, 6 % of the participants spent less than 3 h on the review, and 35 % spent 10 h or more. Overall, participants spent less time than they had anticipated (35 h) on the review. The relatively reduced time commitment might make participating in future IPCC reviews more appealing to ECSs, not only as a group, but also individually. Indeed, based on the survey responses, the time spent to review a chapter of an IPCC report (between 80 and 120 pages) can be estimated at less than 1 week for reviewers of all career stages.

Participants were asked about their willingness to take part in another review as part of a group or individually (Fig. 2a). The majority (92 %) of participants stated that they would take part in another group review, and 86 % reported that they would participate in an individual review, adding more than 60 potential reviewers to the climate change community. Compared to the total number of experts who reviewed the FOD of the SROCC (389, including participants of the APECS group review), this means that including more ECSs in the peer-review process is a significant addition to the pool of expert reviewers. We expect similar results for other scientific fields and hope that other professional societies will consider collaborating in such group reviews in the future.

The most common motivations that ECSs stated for participating in this review process were learning more about the IPCC (59 %) and experience building (53 %; Fig. 2b). These survey responses, combined with the time commitment that reviewers within our group were ready to invest in this project (7 h of preparation and up to 35 h on the review), indicate that ECSs are highly committed to opportunities that involve international-policy-relevant processes. ECSs also value the experience of learning how to review and improve manuscripts. For example, 82 % declared having achieved personal goals by participating in this review.

Prior to the review, one major concern of many participants was a lack of confidence in the value of their feedback to the IPCC, either because they were unsure about the quality of their comments or the sufficiency of their expertise, as the report covers a large ensemble of disciplines that might not fit their breadth of experience. Because the primary audience of IPCC reports are policymakers and stakeholders who may not have scientific backgrounds, IPCC reports should be written unambiguously. Even ECSs who do not have expertise in a particular topic can still provide constructive comments that strengthen the clarity of a report and the discussion of key concepts. Thus, in our group review, participants could choose the chapter they wished to review, but we randomly assigned within the sections this chapter they actually reviewed, regardless of their expertise or desire. The influence of the assignment process could not be assessed during this project.