Evaluating the impact of climate communication activities by scientists: what is known and necessary?

Climate scientists and others are urged to communicate climate science in a way that non-scientific audiences can understand, that makes it more relevant to their lives and experiences, and that inspires them to act. To achieve this, climate scientists undertake a range of climate communication activities to engage people with climate change. With the effort and time spent on climate communication activities comes the need to evaluate the outcomes, impact, and effectiveness of such efforts. Here, we aimed to gain insight into the impact and effectiveness of climate communication efforts by scientists by conducting a systematic literature review. However, our most important finding is that there are hardly any studies in which climate communication activities by scientists are evaluated: we found only seven articles over the past 10 years. We analyze these articles for the role of the scientists, the audiences reached, and the reported outcomes and impact of the activities. We end our study with several recommendations that should be considered when setting up studies on evaluating the impact of climate communication activities by scientists.

1 Introduction

Climate change is one of today's greatest challenges that people around the world face (IPCC, 2022; Schneider, 2011). The consequences of climate change, such as extreme weather events, sea level rise, and impacts on ecosystems and biodiversity are expected to (further) increase in the coming years (IPCC, 2022). The Intergovernmental Panel on Climate Change (IPCC) emphasizes the need for rapid, far-reaching, and unprecedented actions to limit the rise of temperatures above 1.5 °C compared to pre-industrial levels (IPCC, 2018).

For people to take such actions, it is important that they are aware that climate change is happening, that its causes are anthropogenic, and what adaptation and mitigation actions are needed to address climate change (Hassol, 2008). Therefore, climate scientists and others are urged to communicate climate science in a way that non-scientific audiences can understand, that makes it more relevant to their lives and experiences, and that inspires them to act (Corner and Clarke, 2017; Corner et al., 2018; Kumpu, 2022; Dechezleprêtre et al., 2022).

To achieve this, climate communication activities are needed that engage people with climate change (Kumpu, 2022). Engagement with climate change may be defined broadly as individuals' evaluation of and response to climate change, which comprises cognitive, emotional, and behavioral components. That is, engagement involves what people think, feel, and do about climate change (Lorenzoni et al., 2007; Whitmarsh et al., 2013, p. 4). In this study we use a broad description of engagement with climate change, which includes psychological factors that might impact what people think, feel, and do about climate change. These factors include for example attitude towards climate change, motivation to act, perceived capability in taking action (often referred to as self-efficacy based on Bandura's concept), and social norm (see Van der Linden et al., 2015, for a description of several of these factors).

Climate communication activities to engage people with climate change are very diverse. Examples include exhibitions on sustainable eating during festivals (Kluczkovski et al., 2020), informing the public about climate change through media, such as TV broadcasts (Calyx and Low, 2020), and using participatory arts (Burke et al., 2018). With the effort and time spent on this diversity of climate communication activities comes the need to evaluate the impact and effectiveness of such efforts (Grand and Sardo, 2017). However, to our knowledge, there is no comprehensive overview that describes this impact and effectiveness.

Aim of this study

We conducted a literature review where we aimed to answer the following research question: what is known about the impact of climate communication activities by scientists on people's engagement with climate change? While there are other actors who engage in climate communication (e.g., communication professionals, knowledge brokers), we chose to specifically focus on climate communication activities by scientists because research shows that scientists are seen by the public as trusted information producers (Dziminska et al., 2021), and the public believes that scientists should increase their communication efforts (Cologna et al., 2021). On the other hand, scientists are often hesitant to engage in climate communication activities because they feel not sufficiently trained (Rozance et al., 2020) or for fear of hurting their credibility (Kotcher et al., 2017) and potentially being accused of “advocacy”.

Cologna et al. (2021) describe arguments in favor of and against advocacy by scientists. Arguments against advocacy include that advocacy would undermine the credibility of scientists because it contradicts the scientific ideal of neutrality (Lackey, 2007; Nielsen, 2001) and that it negatively influences scientists' ability to conduct science (e.g., due to time constraints; Nelson and Vucetich, 2009). Arguments in favor of advocacy include that it is appropriate when no advocacy could be harmful to society (Douglas, 2009), that scientists as citizens have the responsibility to engage in political and public debates (Lubchenco, 1998, 2017), and that science is never value-free. Therefore, science and advocacy are impossible to differentiate (Elliott and Resnik, 2014; Schmidt, 2015). It is important to carefully consider these different arguments, given the need for strong climate action while maintaining trust and credibility in science and scientists. Insight into the impact of climate communication activities could inform scientists about when and in what way it is effective and appropriate to partake in such activities.

2 Theoretical framework

2.1 Importance of evaluation

Grand and Sardo (2017) describe that it is important to assess the effectiveness and impact of science communication activities. However, high-quality evaluation can do more than assessing. It can promote innovation, change, and be used to critically reflect on the process of engaging the public (Wilkinson and Weitkamp, 2016). Or as Jensen (2015) describes it, evaluation can help scientists understand which aspects of science communication are working, in what way, for which audiences, and why. However, evaluating science communication efforts is not easy. Good evaluation means that clear objectives and appropriate evaluation methods are required. This means that evaluating impact goes beyond counting the number of people involved or impressions based on informal chats (Grand and Sardo, 2017).

Peeters et al. (2022) distinguish three levels of evaluation. The first level, output, focuses on evaluating material results such as the number of people attending an event, sales numbers, or where participants are coming from. The second level, outcomes, focuses on the direct effects of a communication activity on the public, such as whether the public learned something, was inspired, or is motivated to act. The third level, impact, focuses on evaluating the impact of communication activities on society or the effect on a specific target group over a longer period. Evaluating impact can focus on (changes in) societal norms, values, or actions. According to Peeters et al. (2022), evaluating impact could be achieved by evaluating outcomes over a longer period and contextualizing the results in a societal frame. We use these three levels of evaluation to review the climate communication activities that are described in the literature.

2.2 Role of scientists in climate communication

A strong science–policy interface is required to achieve the global climate targets. However, it is yet unclear whether and to what extent (climate) scientists are willing and required to engage in climate communication activities. In addition, scientists can take on different roles in climate communication activities. In science communication three models are often used to describe the interaction between scientists and the public (see Metcalfe, 2019). The deficit model represents a one-way form of communication where scientists inform the public about science and scientific findings. In this model, the communicator/scientist takes on the role of an expert or information source, where the most important goal is to inform the public about science. This may sometimes be an appropriate role, for example in disaster risk communication.

The dialogue model describes science communication, not as a one-way approach but as a dialogue between scientists and the public. This model is characterized by three main features: (1) scientists are willing to engage in a dialogue with the public to help explain the science and its meaning (Wynne, 2006); (2) scientists are willing to consult the public and are open to responses and feedback on their concerns, perceptions, and questions about science; and (3) scientists acknowledge that “the public” may have useful knowledge and ideas that can help in scientific research. In the dialogue model, scientists still take on the role of expert but are now interested and willing to hear about questions and perceptions from the public to get a better understanding of the societal response. This is an appropriate role when for example designing effective risk communication strategies with communities and could also help the scientists to improve their research (Boon et al., 2022).

The third model is the participatory model, which is similar to the dialogue model. However, the participatory model recognizes and acknowledges different publics as equal with scientists and policymakers. The participatory model describes a clearer shift in power from the scientists to the public. In this model, the scientist is no longer “the expert” but someone with an interest in a scientific topic who likes to engage in discussion with others (non-scientists) about this topic. Again, we used these three roles of scientists to review the climate communication activities that are described in the literature.

3 Method

3.1 Literature search

To identify relevant literature for answering our research question, we conducted a literature search based on title, abstract, and keywords with synonyms for the terms: climate communication, climate change, and engagement. For a full list of the used keywords, see Table 1. The databases that we used were Web of Science, Scopus, and PsycInfo. We focused our search on literature of the past 10 years (2012–2022) because we expect that this reflects current developments in research into climate change communication. We selected peer-reviewed articles that were written in English. We imported the found literature into the Mendeley reference manager program. After removing duplicates, a set of 819 documents remained.

Table 1 Search terms used.

Next, we reviewed the documents based on the title and abstract. We used two selection criteria: (1) the study focuses on climate communication activities by scientists, and (2) the study describes the output, outcome, and/or impact of climate communication activities. The measurements used to evaluate the output, outcome, or impact could be either quantitative or qualitative. If these two criteria were not met, the document would be discarded. In some cases, it was difficult to determine, based on the title and abstract whether the climate communication activity involved scientists. In these cases, we decided to include or exclude based on the full text at a later stage. The analysis based on the title and abstract resulted in a remaining set of 66 documents. In four cases, we had no access to the full text. Therefore, we evaluated 62 full texts for our analysis.

3.2 Inclusion of documents

The 62 documents were reviewed by the first author to determine whether they fit the inclusion criteria. After evaluating 35 documents, the third author evaluated 10 documents that were already reviewed by the first author to see whether he came to a similar conclusion about the inclusion of documents. These results matched the conclusions of the first author. Therefore, the first author finished reviewing the remaining documents. After analyzing all documents, a few remaining documents for which there were doubts about inclusion were discussed in the research team, and a decision for inclusion was made. These steps led to the exclusion of 55 documents. The most important reasons for exclusion were the following:

1. The climate communication activities did not involve scientists (e.g., Geiger et al., 2017; Van Swol et al., 2019).

2. There is no evaluation of the impact of climate communication activities (e.g., Oosterman, 2016; Cologna et al., 2021).

3.3 Analysis

The steps described above lead to the inclusion of seven documents (Calyx and Low, 2020; Illingworth and Jack, 2018; Jacobson et al., 2016; Kluczkovski et al., 2020; Luís et al., 2018; Pathak et al., 2021; Peltola et al., 2020). Although we initially aimed to evaluate the impact of climate communication activities by scientists, the final inclusion of seven documents limits our possibilities for analysis. However, we do think that our finding that there is very little research on the impact of climate communication activities by scientists is important. This suggests that evaluating climate communication activities by scientists has over the past 10 years not been a focus of scientific research.

For our analysis we describe the following for each included study: (1) the role of the scientist in the climate communication activity, (2) the reached audiences, and (3) outcome variables that were evaluated and conclusions regarding impact. A description of the included studies can be found in Table 2.

Table 2 Overview of the studies included in the analysis.

4 Results

4.1 Role of the scientist

In four studies (Calyx and Low, 2020; Jacobson et al., 2016; Pathak et al., 2021; Peltola et al., 2020), the role of the scientists was mostly to act as a source of information for the target audience, for example, by answering questions (Jacobson et al., 2016) or providing information on climate change to the public (Calyx and Low, 2020; Pathak et al., 2021; Peltola et al., 2020). In one study, scientists were involved in designing and building an exhibition (Kluczkovski et al., 2020) but not in the actual exhibition itself. In another study (Luís et al., 2018), it is unclear to what extent scientists interacted with the public, but here scientists were involved in designing workshops on adaptation strategies on climate change for the target audience. We see these roles as examples of deficit communication where scientists are mainly involved as experts.

In only one study, scientists were really collaborating with members of the public by writing poetry on climate change and engaging in discussion with participants during workshops (Illingworth and Jack, 2018). Here, the scientists aimed to get a better understanding of the questions and feelings of these participants about climate change and to create a dialogue between scientists and non-scientists. Interestingly, the authors refer to the involved scientists as “experts”. According to Metcalfe (2019), these are examples of a dialogue approach. Furthermore, this is the only study where the participating scientists were asked to reflect on their interaction with the public. None of the other studies describe how scientists experienced the communication activity.

4.2 Output: reached audiences

In terms of output, the audiences that were reached in the different studies are quite diverse both in number and in characteristics. For example, Peltola et al. (2020) were able to reach 230 secondary school students, whereas Illingworth and Jack (2018) specifically aimed to reach people from communities who are usually underserved by science communication, such as refugees. They were able to reach approximately 21–31 people from these communities. On the other hand, Calyx and Low (2020) describe the impact of a TV broadcast where scientists explained climate change to a climate skeptic politician. It is encouraging to see that climate scientists aim to reach diverse audiences because that might mean that different groups within society have access to climate communication activities.

4.3 Outcomes and impact of climate communication activities

All the included studies show positive results regarding outcomes. In three of the seven included studies, a pre- and post-test design was used to measure for example knowledge about climate change (Jacobson et al., 2016; Kluczkovski et al., 2020), behavioral intention to engage with climate change (Luís et al., 2018) and actual behavior change (Kluczkovski et al., 2020). In these studies, significant increases in the measured variables were found. In three other studies, the authors conclude that their activities provided a platform for sharing views, sharing experiences, and creating open discussions about (topics related to) climate change (Illingworth and Jack, 2018; Pathak et al., 2021; Peltola et al., 2020). Calyx and Low (2020) found that a TV broadcast where a panel of scientists discuss climate change made a climate skeptic politician change his mind about climate change.

None of the included studies evaluate the outcomes over a longer period of time or changes in for example social norm of a specific societal group. We therefore conclude that in none of the studies the impact of the climate communication activities is evaluated.

5 Discussion

The goal of this study was to gain insight into what is known about the impact of climate communication activities by scientists and how scientists can make an impact when partaking in climate communication activities for different audiences. However, our most important finding is that there is hardly any research that evaluates the impact of climate communication activities by scientists. This does not necessarily mean that communication activities are not evaluated. Ziegler et al. (2021) suggest that evaluations of science communication (in general) are often used to reflect upon a project or activity within a team to improve future projects, and the results are often only shared with supervisors or funders. Ziegler et al. (2021) also found that many of evaluation reports are used as summative evaluations to determine the “successes” of a project rather than formative evaluations that could help gain a deeper understanding of the development and process of activities.

Of course, we might have missed evaluations because of too stringent search terms (Table 1). We could have missed relevant keywords, although for example a test to extend the search with the keywords “dialogue” and “participatory” only seemed to lead to false positives. Alternatively, evaluations of science communication activities could have been published outside of the peer-reviewed literature, as for example internal reports, blog posts, or conference presentations.

A possible reason for not making evaluation outcomes available is that negative results of evaluations might lead to criticism. This could be especially problematic if this impacts funding opportunities or the way scientists who are involved in such activities are seen by peers and the wider public. However, sharing evaluations of climate communications efforts, both successful and unsuccessful, could stimulate learning and help other scientists get a better understanding of how, for example, specific groups of people could be approached, or what activities (e.g., lectures, hands-on activities, group discussions) might be suitable for achieving specific goals (e.g., improving knowledge, influencing attitude).

The seven included peer-reviewed studies showed positive results regarding the outcomes of climate communication activities by scientists. This is encouraging, for this seems to indicate that climate communication activities by scientists are valuable in providing platforms for discussion and reflection, and they could stimulate the public to become more engaged with climate change. However, care should be taken in concluding from these seven articles that climate communication activities by scientists always have a positive impact, especially considering the suggestion by Ziegler et al. (2021) that positive results are more likely to be shared than negative results. One such an example is the study by Calyx and Low (2020), which was included in this review. This study describes the positive impact of a TV broadcast where a panel of scientists discuss climate change made a climate skeptic politician change his mind about climate change. Calyx and Low state that they wanted to “share a story of change” (p. 3), which made them decide to, among other things, write a paper about the impact of the TV broadcast. It is possible that the paper had not been written if there had been no impact. We thus encourage the publication of evaluations where the effect of the communication activity is non-positive, too.

The use of existing theoretical frameworks or models from fields such as education and communication can help guide the design and evaluation of climate communication activities. One of the included studies (Luís et al., 2018) is an example of this. In this study the theory of planned behavior (Ajzen, 1991, 2001) was used. Using this theory, Luís et al. (2018) selected variables (attitude, subjective norm, and perceived behavioral control) that might impact stakeholders' intention to plan local adaptations to climate change. Then, they determined how these variables could be defined and measured and evaluated whether these variables were impacted by a climate communication activity (in this case two local workshops on adaptation to climate change). Another finding of this review is that in six of the seven studies, scientists take on a role of expert and engage in one-way communication. Either by being a “source of information” (answering questions but not asking about the views or ideas of the public) or by being involved in designing climate communication activities but not being involved in actual interaction with the public. These are examples of a deficit approach to climate communication. This is finding is similar to the findings of Metcalfe (2019), who found that most science communication activities (in Australia) use a deficit approach, despite the often called-for shift in many countries from “communication” to “dialogue” and from “science and society” to “science in society” (Bucchi, 2008). There has been a lot of critique on the deficit approach to science communication, the most important being that using a deficit approach assumes that providing the public with information about a scientific issue will “correct” or “complement” their views, making sure people “believe the right things” (Seethaler et al., 2019). However, human reasoning is complex, and attempts to correct “misconceptions” by merely providing information often backfire (Lewandowsky et al., 2012). A possible reason for the consistent use of a deficit approach could be that scientists are hardly ever trained in science communication (Simis et al., 2016). Therefore, scientists might not be aware of the limitations of using a deficit approach. In our review, we found that in only one study (Illingworth and Jack, 2018) scientists were asked to reflect upon their interaction with the public. However, reflecting on communication experiences could provide insight into the perspectives of the scientists on science communication and whether and which support they might need to become better communicators. We thus call for more interdisciplinary research where scientists and communication professionals collaboratively investigate the impact of communication activities, specifically those using dialogue and participatory models.

Fortunately, evaluation has become a much more important topic in science communication research in the very recent past (e.g., Hillier et al., 2021). Between the finalization of our analysis and the revision of our article, new, relevant articles have come to our attention (e.g., Hiller and Van Meeteren, 2024). While we decided not to open up our analysis again (because an article like this one would then never be finished), we do expect that an update of this search could be very valuable in a few years.

Future research could focus on the role of non-scientists, such as knowledge brokers (Meyer, 2010) and other communicators, in climate science communication. Such future research could also extend our analysis by focusing on the effectiveness of climate scientists in communicating or training to professionals, rather than the general public.

6 Conclusions

Based on the review presented in this paper, our most important conclusion is that more research is needed on evaluating the impact of climate communication activities by scientists. To achieve this, we believe that a learning-friendly environment is necessary, where the focus of studies is not necessarily on “proving successes” but on sharing lessons learned from evaluating climate communication activities with the goal of increasing understanding of what works and what not.

Furthermore, collaboration between researchers and science communicators could help in setting up more systematic approaches to evaluating (science) communication activities (Ziegler et al., 2021). The experiences of science communicators can help researchers get a better idea of the challenges that are faced when evaluating climate communication activities, and the expertise of social scientists might help to overcome such challenges. This collaboration could help in setting up evaluation practices that are methodologically sound and fit the goals of the climate communication activity.

One challenge in evaluating science communication (in general) is the method of evaluation. There are several papers that describe approaches and methods that can be used to evaluate science communication activities (e.g., Grand and Sardo, 2017; Peeters et al., 2022; Ziegler et al., 2021) that might be useful to consider when evaluating climate communication activities by scientists.

We recommend that evaluation of climate communication focuses not only on the impact of these activities on the public but also on the experiences of the involved scientists. These experiences could provide valuable insights into whether the involved scientists need support or training (Baram-Tsabari and Lewenstein, 2017) to help them become better climate communicators.