This manuscript reports the findings of an in vivo impact evaluation of climate myth debunkings on people who arrived at Skeptical Science via a Google search query. Assessing the impact of attempts to debunk climate misinformation in vivo is a worthy research goal, and the research reported in this manuscript appears to be well conducted. Overall, I find this paper to be useful and will be pleased to see it published.

That said, I encourage the authors to consider how they might make the paper easier to read and grasp their key learning.

The Introduction section, for example, is a bit disjointed. Lines 26 to 42 could be restructured to clarify the more “subtle and subversive impacts” of climate misinformation. It isn’t clear why the “evolution of SKS rebuttals” section was included, given that the focus of the paper is an evaluation of current SKS content. Moreover, as a reader, I would like to see additional citations to substantiate statements about relevant prior research.

In the Methods, I would like to see a better explanation of how participants arrived at the research experience. The paper states they arrived via a Google search, but it would help readers understand the situation better if we knew what they were looking for. Is it the case that the Google research took them directly to a specific debunking and we safely presume that they asked Google a direct question related to that debunking? Readers need to better understand the research participant’s information search expectations to understand how they may have reacted to the question asked by the researchers. A minor point: I find the term “modal” (as in “modal pop-up screen”) confusing in this context. Is there an alternative term that can be used instead?

The results section is currently a bit hard to follow. The current flow of information is, essentially, “here is the data” and “this is what that data means.” I realize this is a standard and common way of reporting research findings, but I encourage the authors consider inverting the order of presentation to make easier for readers understand what the data mean. In essence, I’m suggesting the authors lead with a series of statements about the finding (e.g., the debunkings made visitors less likely to believe the climate myths being debunked) and then show the data that supports each claim.

I found the qualitative investigation of the best and worst performing rebuttals to be extremely useful, but I also it difficult to follow. Here too, I feel the detailed explanations made it challenging for readers to grasp the key learnings. Here too, I recommend leading with the key learning and then providing the details that justify the conclusion.

Lastly, in the Discussion section, I was not clear on why the authors concluded that “most visitors (45%) already strongly agreed with climate facts” (because 45% is not “most”), or that “they were looking for information to assist them in responding to climate misinformation.” I’m not saying these conclusions are unjustified, but I am saying I would find the statements more credible if they were better defended.

The paragraph beginning on line 201, which addresses why some of the debunkings performed poorly, would be better placed after the discussion of the overall pattern of findings—that belief in climate myths declined, but so too did belief in climate facts.

Minor points:

Line 26: the phrase “climate perceptions” does not fully convey the intended idea.

Line 224 : The phrase “a decreas in climate facts” does not accurately convey the intended idea.

Conclusion:

This paper is publication-worthy, but I would prefer to see it strengthened before publication. I thank the authors for their important work.

This article presents the results of a survey on the website Skeptical Science and attempts to determine the degree to which the climate myth rebuttals on the website changes the visitors’ viewpoint on the scientific consensus about climate change. This work is evidently very important in today’s world, where climate misinformation is becoming easier to spread while the need to act on the changing climate is all the more urgent. Therefore, I believe that the research in this piece of work to be highly publishable in this journal.

However, I recommend that the authors improve the quality of some aspects of the article to make it easier to follow, and to avoid making assumptions about why some rebuttals seemingly did not perform well, without any qualitative evidence to back this up (see below).

Introduction – It feels like there is possibly too much background information in this section, especially as it is presented without context as to why it is necessary to know before reading the rest of the study. In particular, I do not feel like Section 1.2 (Evolution of SkS Rebuttals) is necessary as it has no relevance to the survey or the discussion of the study. I acknowledge that background information about SkS is essential, but feel like Section 1.1 alone is adequate in preparing the reader for the rest of the article.

A glaring omission when reading the introduction was a specific research aim, or purpose of the study. All the background information was presented without knowing the context behind it – why do we need to know what has been described in Section 1-1.1? What hypothesis or research question are you seeking to answer by doing this study?

Methods – I feel like a lot of trust is being placed on the assumption that all visitors that completed the second survey read the webpage and took in what was being said. For example, some people could have quickly scrolled to the bottom of the webpage and filled in the survey without reading the content of the website. Has some filter been applied where responses completed below a minimum time are removed from analysis? If not, I feel this should be considered. On a related note, I can’t seem to find Table S1 in Supplementary Material (where there is supposedly a distribution of times) – only Appendices.  

Results – It would be interesting to know whether the “level” of the rebuttal read by the visitors had any affect on their change in perception of climate facts (though I concede this is probably impossible to find out in retrospect). However, my main concern with this section comes from the analysis of individual rebuttals (from Fig. 5) and why the “worst-performing” ones are not believed. Most of the arguments here as to how the rebuttals could be improved come from the authors, and are not the result of the survey itself. Therefore, I find it confusing to make comments like “rebuttal x fails to take in to account y and z” when there is no qualitative evidence from the survey that this is why visitors did not believe the rebuttal. While I think the comments made about these individual rebuttals in this way are useful and the reasoning is sound, this more speculative analysis would be better placed in the discussion section.

Discussion – I take issue with the statement on line 182 that suggests that 45% is “most visitors” – it is less than half! I do however agree that more visitors to the site agreed with climate facts than didn’t, so this sentence should be rephrased for clarity. I do not, however, agree that the implication (line 184) of this means that most visitors are “looking for information to assist them in responding to climate misinformation” – again this is not asked in the survey. These visitors could have been looking for detailed, accurate scientific details about certain aspects of climate change from a reliable source for individual research purposes, or just general interest. The wording around this implication could therefore be loosened to accept that other reasons exist as to why such people would visit the site.