I am happy to see an article addressing the use of virtual out crop models and virtual field trips in geoscience courses. Thought they have been gaining in popularity, the recent experiences with the pandemic have really placed them front and center. The more people we have looking at this kind of strategy, the better informed we will be in all facets of their use, developing, implementing, and evaluating efficacy.

That said, it is my opinion that this paper would need to add to or change its scope prior to acceptance for publication. As stated earlier, using various types of virtual field trips has been gaining in popularity, increasing with more sophisticated technologies, for the past couple of decades. There have been numerous publications addressing many of the problems addressed in this paper. The following represents a list of the present authors’ conclusions and previous literature that support them.

1. VFTs enable a larger volume of data at varying scales to be integrated and developed.

(Arrowsmith et al., 2005; Atchison & Feig, 2011; Hurst, 1998; Çaliskan, 2011)

2. VFTs are logistically easier to plan and deliver.

(Hurst, 1998; Peat et al., 2005)

3. VFTs are financially inclusive

(Fletcher et al., 2002; Jacobson et al., 2009; Litherland et al., 2012; Stainfield et al., 2000; Ramasundaram et al. 2005)

4. VFTs are time efficient

(Ramasundaram, et al., 2005)

5. VFTs have a lower carbon emission

6. VFTs are inclusive to those with restricted physical access

(Atchison & Feig, 2011; Atchison, 2011; Gilley et al., 2015; Stainfield et al., 2000; Çaliskan, 2011)

7. VFTs are flexible, inclusive to those with other time commitments

(Hurst, 1998)

8. VFTs allow individuals to work at their own speed

(Arrowsmith et al., 2005; Bentley, 2014; Fletcher et al., 2002; Li et al., 2003)

9. VFT can serve to prepare/orientate a class for a real excursion

(Bentley, 2014; Litherland et al., 2012; Peat et al., 2005; Çaliskan, 2011)

10. VFTs can be geographically independent

(Stainfield et al., 2000)

Need to incorporate teamwork

(Arrowsmith et al., 2005; Atchison & Feig, 2011; Lukes, 2014; Stumpf et al., 2008)

Reflecting on this indicates that this paper really does not add to the knowledge base. There does not appear, in my reading of the manuscript, to be anything novel to report. This does not take into consideration that many of the conclusions of the article were not even addressed in the paper. For instance “VFTs have lower carbon emmissions.” It seems intuitive here, but there is no measurement or even mention of this in the manuscript at all, nor do they site any other resource that makes such an assertion. VFTs are time efficient and VFTs can serve to orient a class for a real excursion. Again, the manuscript does not entertain these ideas, nor provide any data to support them. They are intuitive, sure, but one thing science has shown us is that intuition oftentimes is not coherent with reality. So, it seems to me that without actual data, or reference to published data someplace else, These concluding statements are inappropraite for this paper as currently written.

The authors use course standard course evaluations plus one made specifically for the reported activities. There is quite some literature on how standardized course/teaching evaluations are unreliable for a number of reasons (Boring, et al. 2016; Esaarey & Valdes, 2020; Spooren et al., 2013). Part of the reason is that though they show that the virtual field trips scored higher than the actual field trip, yet all students said they would reather be on the actual field trip. It could be that because ALL classes were on line at the time, and the field trip course offered a lot of variety and therefore compared to other courese on line, this one scores higher. We just do not know because there is no way to standardize the data.

Where this manuscript could go: In my mind, this manuscript needs to do one of two things to allow it to add to the knowledge base. First would be to focus on the building part, or focus on the evaluation part.

For the building part, there isa lot of technical information already about “how to” create and implement. This is great for people who would like to try it on their own. What it should have, however, is more about mathiing the building to insdtructional outcomes. For instance, “We wanted students to learning X, Y, and Z. There fore we incorporated strategies 1, 2 and 3.” This part does not appear in the manuscript as written. There are some learning goals written, “To study extensional tectonic and associated arid rift basins…etc.”, but this does not outline what the authors want the student s to learn. Without this part, evaluation of the VFT is merely opinion. Students self reported they liked it and said they learned something. However, because we do not know what we do not know, the sense of unserstanding is a terrible predictor of actual understanding (Kuorikoski & Ylikoski, 2015). Without measuring actual knowledge gains, there is no way to know how effective the VFT is. The manuscript was explicit saying that this type of measurement was out of the scope of the investigation. My suggestion is to use the literature to guide what the course learning objectives could be (Mogk & Goodwin, 2012), and then mathch them up with the different aspects of the VFT. This would ultimately give the authors a structure that others could follow, and ultimately give a rubric for doing the much needed evaluation of VFTs in general. The literature here is very sparse to non-existent. This would be a truly important contribution to the knowledge base.

On  side note, I am assuming that the authros received etics approavel to use student derived data for the manyuscript. It would be good to mention this explicitly in the methodology or data collection section.

References

Arrowsmith, C., Counihan, A., & McGreevy, D. (2005). Development of a multi-scaled virtual field trip for the teaching and learning of geospatial science. International Journal of Education & Development using Information & Communication Technology 1(3), 42-56.

Atchison, C., & Feig, A. (2011). Theoretical perspectives on constructing experience through alternative field-based learning environments for students with mobility impairmants. In A. Feig & A. Stokes (Eds.), Qualitative inquiry in geoscience education research (pp. 11-22). Geological Society of America.

Atchison, C. L. (2011). The Significance of Access: Students with Mobility Impairments Constructing Geoscience Knowledge Through Field-Based Learning Experiences [3476983, The Ohio State University]. ProQuest Dissertations & Theses (PQDT). United States -- Ohio. http://search.proquest.com/docview/898368486?accountid=14214http://libsfx.syr.edu:9003/syracuse?url_ver=Z39.88-2004&rft_val_fmt=info:ofi/fmt:kev:mtx:dissertation&genre=dissertations+&+theses&sid=ProQ:ProQuest+Dissertations+&+Theses+(PQDT)&atitle=&title=The+Significance+of+Access:+Students+with+Mobility+Impairments+Constructing+Geoscience+Knowledge+Through+Field-Based+Learning+Experiences&issn=&date=2011-01-01&volume=&issue=&spage=&au=Atchison,+Christopher+Lawrence&isbn=9781124910000&jtitle=&btitle=

Bentley, C. (2014). Telling the story of the canadian rockies via google earth and gigapan 2014 Geological Society of America Annual Meeting and Exposition, Vancouver, BC.

Boring, A., Ottoboni, Kelli, & Stark, P. (2016). Student evaluations of teaching (mostly) do not measure teaching effectiveness. ScienceOpen Research. Vol. 0(0):1-11. DOI: 10.14293/S2199-1006.1.SOR-EDU.AETBZC.v1

Çaliskan, O. (2011). Virtual field trips in education of earth and environmental sciences. Procedia-Social and Behavioral Sciences, 15, 3239-3243.

Esarey, J., & Valdes, N. (2020). Unbiased, reliable, and valid student evaluations can still be unfair. Assessment & Evaluation in Higher Education, 45(8), 1106-1120. https://doi.org/10.1080/02602938.2020.1724875

Fletcher, S., France, D., Moore, K., & Robinson, G. (2002). Fieldwork education and technology: A GEES perspective. Planet, 7(1), 17-19.

Gilley, B., Atchison, C., Feig, A., & Stokes, A. (2015). Impact of inclusive field trips. Nature Geoscience, 8, 579-580.

Hurst, S. D. (1998). Use of “virtual” field trips in teaching introductory geology. Computers and Geosciences, 24(7), 653-658.

Jacobson, A. R., Militello, R., & Baveye, P. C. (2009). Development of computer-assisted virtual field trips to support multidisciplinary learning. Computers & Education, 52(3), 571-580.

Kuorikoski, J., & Ylikoski, p. (2015). External representations and scientific understanding. Synthese, 192(12), 3817-38371-38329. (Academia.edu)

Li, S., & Liu, Q. (2003). Interactive groundwater (IGW): An innovative digital laboratory for groundwater education and research. Computer Applications in Engineering Education, 11(4), 179-202.

Litherland, K., & Stott, T. A. (2012). Virtual field sites: Losses and gains in authenticity with semantic technologies. Technology, Pedagogy and Education, 21(2), 213-230.

Lukes, L. (2014). A new take on the field trip: A low-tech, inquiry-based virtual field experience. The Science Teacher, 8(1), 24.

Mogk, D. W., & Goodwin, C. (2012). Learning in the field: Synthesis of research on thinking and learning in the geosciences. In K. A. Kastens & C. A. Manduca (Eds.), Earth and Mind II: A Synthesis of Research on Thinking and Learning in the Geosciences (Vol. 2, pp. 131-164). Geological Society of America.Peat, M., & Taylor, C. (2005). Virtual biology: How well can it replace authentic activities. CAL-Laborate, 13, 21-24.

Ramasundaram, V., Grunwald, S., Mangeot, A., Comerford, N. B., & Bliss, C. (2005). Development of an environmental virtual field laboratory. Computers & Education, 45(1), 21-34.

Spooren, P., Brockx, B., & Mortelmans, D. (2013). On the Validity of Student Evaluation of Teaching: The State of the Art. Review of Educational Research, 83(4), 598-642. https://doi.org/10.3102/0034654313496870

Stainfield, J., Fisher, P., Ford, B., & Solem, M. (2000). International virtual field trips: A new direction? Journal of Geography in Higher Education, 24(2), 255-262.

Stumpf, R. J., Douglass, J., & Dorn, R. I. (2008). Learning desert geomorphology virtually versus in the field. Journal of Geography in Higher Education, 32(3), 378-399.

The authors seek to highlight the pros and cons of virtual fieldtrips over traditional fieldtrips using student evaluation data from two fieldtrips. Clearly, this is a timely case study given the forced move to more online delivery during the pandemic, which is likely to be maintained in either a blended or replacement form more rapidly that the assessment of pedagogic benefits can be assessed.

The manuscript has a simple scope, which also reveals how complicated a true assessment of benefits/disbenefits of different (combinations of) delivery modes geological fieldtrips can take. As a case study, the contribution has value, but I felt that there was too much crucial aspects considered ‘beyond the scope’, and that there is an opportunity to really critique the data collection methods and to present recommendations for future studies. In general, there is a need for more citation to source literature on different delivery.

Some specific comments/issues:

• There is no discussion of the flaws in post-course evaluations. These issues are well established in the literature (albeit largely ignored by universities in the UK). The big problem is that they do not in any real way assess deep learning – whether knowledge gained or skills developed. When subjects are asked immediately after delivery (although the timeframe is not stated here) the responses are strongly positive, but what of two or three years later? What about evaluation during the course when the subjects will more clearly benefit from changes? Some studies have shown evaluations of learning strengthen with time – although still with the issue of self-reporting.
• The relative pedagogic benefits of traditional, or VFT, or blended delivery is a thorny issue given the number of different parameters to consider, but you have a chance to raise and discuss this – which I have not seen elsewhere. It is crucial to have future delivery options underpinned by rigorous assessment of benefits that are not self-reported. What are the learning outcomes and skills that need to be assessed in both virtual and traditional environments? What forum is better to develop 3d thinking, or understanding of scales, or spatial relationships, and how do we assess this (and will this vary by subject – probably given different learning styles). These are things that you haven’t set out to understand. In real time during a pandemic this in-depth assessment is not feasible, but how are we going to do it in the future? But on reflection your insights and recommendations are crucial to an informed evidence-based debate on future field learning, and before it is assessed from a financial perspective.
• One thing I noticed is that you say that the questionnaires/evaluation documents are anonymous, but then seem to identify two students as having poor wifi and this being the reason for lower scores. You need to be really careful here – has an ethic approval process been completed? If it has, you need to state it.

In summary, although this is an interesting case study on the perceived pros and cons of virtual and traditional fieldtrips from participants, there are inherent flaws in the evaluation data. This is not surprising because a strong evidence base on this topic needs to consider a huge number of different parameters. However, to be a novel contribution, I recommend that the authors are clearer on the flaws in the evaluation approach used, and to recommend the key skills/learnings that need to be the focus of future investigations (e.g. 3D thinking, scales, time spend thinking deeply, challenging siutations, etc.) to inform development of future fieldtrips.

It is such an crucial and pertinent topic for our science, and a highly complex issues to realise a strong pedagogic evidence-base. But this complexity needs to be teased out here. If the authors can bring this out in a discussion section, then it could be an influential contribution that triggers future research.

Minor typographic errors and additional comments ont he annotated manuscript.

This manuscript presents a case study of virtual fieldtrips (VFTs) to highlight the workflow and assessment of VFTs as an alternative to traditional in-person field trips. The authors document development and delivery considerations using modern techniques and a novel visualization platform and report the effectiveness via rigid course assessment and open-ended response from student course evaluations. The manuscript is relevant to the Special Issue it has been submitted to and is timely as it reports on methods and assessment of alternative educational resources/approaches globally adapted by geoscience educators due to the COVID-19 pandemic.

Although the manuscript has its strengths in highlighting the potential benefits of implementing VFTs with Virtual Outcrops to enhance student learning, there are a few key considerations that should be addressed in order add substantial knowledge to the existing literature and maximize impact:

1. The authors report assessment and effectiveness of VFTs, however much of the research aims and conclusions (i.e., benefits and limitations of VFTs, etc.) have been documented by previous studies. In my opinion, this does not reduce the novelty of this investigation. Instead, I think the authors have an opportunity to highlight the differences between their study and the existing literature – the use of photorealistic 3D virtual outcrop (VO) models integrated with various datasets in a novel visualization platform for VFTs (as opposed to previous VFTs reported using largely static forms of media and/or limited resolution digital globes, such as Google Earth). Establishing this difference (advances in VFT methods) and citing foundational VFT research in education would contribute to substantial knowledge gains and progress of methods in geoscience education.
2. Evaluation methods – assessment of the efficacy of a particular method in pedagogical research is challenging. There is no ‘silver bullet’ or perfect means of assessing how well a particular teaching method worked or how well students learned. The authors use surveys of student experience through standard university and specialized questionnaire with both quantitative responses and open-ended free responses. In my opinion, it would be ideal to perform deeper investigations into learning outcomes (specified as outside the scope of this study), but I do believe the methods used by the authors offer new insight into student perception of VFTs using VO. However, the authors need to further elaborate on the limitations of their methods and situate their findings relative to previous work (i.e., how do these results compare to previous VFT investigations using different forms of media and similar evaluation methods?)
3. Discussion and Conclusions - many of the conclusions and discussion points (i.e., advantages and disadvantages) are not fully justified from the evaluations performed or main body of the text (at least they are not fully expressed and clear from the data). I think the data may show sufficient evidence in this study, but it will need to be made more clear within the text and should also be situated in the current literature (as mentioned above). Related to this point – the conclusion section ‘fizzles off’ and would greatly benefit from a stronger statement that brings this study into a broader context and/or suggests future steps.

The written English and style are clear and the structure of the paper follows a reasonable flow, though there are several minor grammatical errors/inconsistencies (see attached document and points below). There are also a few areas where the authors could consider minor reorganization to improve clarity and potentially reduce text to be more concise and expand potential impact for the readership.

• End of introduction (2 paragraphs) feels a bit misplaced and tagged onto the introduction as an afterthought. I think the paragraphs provide important context and should be included/adapted earlier in the introduction to help readers understand the different types of VFTs and modes of delivery. This would better contextualize the scope of this study.
• Section 3.1 and 3.2 - Sections are a bit drawn out. Perhaps consider as supplementary material or appendix and keeping only the most important info concisely here.
• Section 5.1 - This section is a bit hard to follow and doesn’t add much. I wonder if it is worth considering shortening this and focusing on the platforms/activities and modes of delivery and contrasting with how it was/would have been presented in traditional format.
• Several small grammatical errors/typos – I tried to highlight most of the areas where there may be typographic errors and where some punctuation may be required in the attached PDF.

Overall, I believe this manuscript has a lot of potential and is not far from being a strong contribution. Hopefully the authors find these comments helpful towards the revision of their manuscript and help to achieve the research aims.

A considerable amount of work has clearly gone into this project. All that effort also took place during a stressful time, rife with uncertainty. I hope my criticisms and feedback do not come across as diminishing that effort. I hope that this work can eventually be published and recognized in some fashion. 

My opinion is that this paper requires additional work or significant modification of scope to be accepted. My overall conclusions, as well as many of my specific concerns, are consistent with the previous reviews by Glenn Dolphin (https://doi.org/10.5194/gc-2021-37-RC1) and David M. Hodgson (https://doi.org/10.5194/gc-2021-37-RC2). I won’t belabor the points they’ve already made, and well referenced. Suffice to say, that I agree with the points they make.

I do want to expand on some of those points, and bring up a couple more points not previously mentioned. I also want to add the questions that I had in reading this, questions that I felt were unanswered or could be directions to reframe or modify the scope. These questions should not necessarily be taken as “required,” but hopefully they are useful in guiding next steps.

My overarching concern about this work is the scope. The diversity of different types of virtual field trips, how they’re made, how they are presented, and the variety of goals (many of which are mentioned) make blanket conclusions about VFTs seem akin to trying to make conclusions about the value of photographs in teaching. Simply narrowing the conclusions could make this a strong contribution.

Specific Comments

Defining virtual field trips

While examples of different types and classifications are given, this work stops short of assessing the “pros and cons” of the different VFTs, as it is “beyond the scope of this study.” However, as previously mentioned by other reviewers, the conclusions are applied to the entire scope of virtual field trips. This seems an issue of overstated conclusions however, and not an inherent flaw in the work.

As no definition of virtual field trips is, as of yet, universally accepted, I don’t think that this paper requires a global definition. However, it would be helpful to see a definitive “local” definition of how this paper is using the term. “Digital alternative” to field work is mentioned, but this is not clearly stated or expanded on. If that is the working definition here, the paper would benefit from a discussion on why they are using this definition and the potential limitations. For example, if the goal is to compare virtual field trips with in-person field trips or lectures, more distinction seems required. I would argue that many of the “activities” listed in figure 9 are not necessarily “virtual field trips” but rather sophisticated multimedia lectures. Without more clearly differentiating between multimedia lectures and virtual field trips, it is difficult to assess the value of virtual field trips. That is not to say that multimedia lectures are bad, or even inferior. Just that the comparison requires distinction. Is a video of a location a virtual field trip? Where does “lecture” end and virtual field trip begin? To be fair, this is not an entirely settled argument as it pertains to in-person field trips either. Is a road-side lecture a field trip? Or is it just plein-air lecturing? The argument for in-person field trip vs lecture is usually just a matter of whether a student left campus, but that distinction is not so simple with virtual field trips.

Virtual Field Trip Activities

I would love to see a much more detailed description of the activities and how they were designed. The paper mentions the following activities:

• separate projects
• group activities
• exploration play mapping exercise
• field development exercise
• major student exercise dealing with exploration in salt basins
• new assessed exploration exercise
• group exercise
• independent work in LIME
• Atlas of sedimentary systems

None of these activities are described in detail. In my opinion, that is the most interesting part of this work. I would love to know more about them. How did you design the activities? What factors did you design for? Were you able to test-run them? How did students complete them? How was the group work facilitated? Were they working collaboratively in LIME or Google Earth? Were they assessed individually or as a group? How effective was the VFT in teaching outcrop interpretation, for example? Or in facilitating correlation of well log data with outcrop data? How is the “atlas” assessed? Is it given a grade? Could that be matched with evaluations as a measure of actual performance? What else is the student grade based on? The details of how those exercises were constructed and delivered, along with the analysis of how well each approach worked, would be extremely interesting and valuable. I think this would be a possible direction to reframe this work.

Learning Objectives

The learning objectives mentioned are almost exclusively in the form of “To study X”. Setting aside the student learning implications of objectives framed in that fashion, I think they are compromising the ability to measure or assess success. Both Dolphin and Hodgson noted the issues with using student evaluation of teaching (SET) data (well referenced in Dolphin). I would echo that here, and add that without objectives that clearly state a measurable result (, it may be impossible to gather more reliable data. I think much of the challenge in data analysis for this paper stems from the way the learning objectives have been articulated. While the data you have, with its associated caveats, is interesting for discussion, I don’t think it presents compelling evidence in support of the stated conclusions.

Clarity

There is a lot of information contained in both the body of the work and in the tables. However, it's not clear how much of it relates to the narrative. For example, Table 1 shows that a great deal of content went into this project, but there is little discussion of this in the text. Was that enough material? Could it have been less? Were students overwhelmed with the amount of content? What content would you change in future iterations? Without additional discussion it seems extraneous.

Section 5.1 and 5.2 seem extraneous to the topic of “construction, delivery, and implications” of virtual field trips as well. As the paper's conclusions are intended to be widely applicable, general information on the type of information being taught is certainly useful (sequence stratigraphy, fluvial architecture, outcrop interpretation, etc), but the detailed geologic content and location information detracts from the overall narrative.

There is a lot in this work that is relevant, timely, and valuable. I hope that this feedback, and the feedback from other reviewers, is useful. I’m excited to see where this goes next.