Review comments in italics. Responses in bold.

I think the concept presented here is very interesting and potentially very valuable in teaching (and removing barriers for those who may not even be able to attend conventional field courses).

Response: We are grateful to Sebastian Mutz for taking the time to provide helpful comments on our manuscript.

I do have some concerns about the sample size (and some of the larger p-values), however. The authors do acknowledge that problem and I think the authors mostly treat this problem very well and with commendable transparency. That said, as someone who is interested in implementing such an approach, I would want to see more thorough testing of it (as part of future studies at least). I do not think this is absolutely necessary for a pilot study, but I do think the manuscript would benefit from (a) adopting a more careful tone in the discussion of results (/toning down the claims a little), and (b) placing a little more emphasis on encouraging the testing of this type of approach.

Response: We are delighted to hear of more people interesting in adopting this approach and warmly encourage Sebastian in doing so! We are aware of the limitations that a small sample size places on our study – a problem noted by all reviewers – and this is somewhat inevitable due to the small cohort sizes of geology at UGent. We try to treat our results cautiously, but we are happy to revise the text following this and other reviewers’ suggestions to avoid underplaying the uncertainty in our results. We are also very happy to encourage/recommend further work in this area and will revise the text to make sure we do this.

I would also recommend the inclusion of skill-based tests of progress, because confidence does not necessarily reflect skill.

Response: We agree, certainly, that evaluating how a resource like the Rock Garden influences the progress of skills acquisition is an interesting and worthwhile study. However, our primary aim here was to assess how students’ confidence in applying skills that we knew they already. From the outset, we knew (from their work) that the students do have these practical skills, acquired through previous courses (though skills can always be further developed and honed). Our aim here was to understand whether using the Rock Garden as a teaching aid would help students to become independent field scientists by increasing their confidence in working in the field (their self-efficacy). Moreover, as noted in one of the other reviews, we think it may be difficult to get ethical clearance and student support for including more granular marks data, not to mention mitigating the necessarily subjective nature of marking practical field work. We also feel sure that drawing meaningful conclusions on the impact of the Rock Garden on specific field skills attainment would require larger cohorts than we typically have.

Original text in italic emphasis. Responses in bold emphasis.

This is a well written piece exploring an alternative to fieldclasses through the development of a campus ‘Rock Garden’. An inability to travel to fieldclass locations is an issue that affected many education establishments globally during the pandemic, prompting a wider thinking on fieldwork in both undergraduate and postgraduate degree programmes within the Geosciences, but also within Environmental Sciences more broadly. It also highlighted the opportunity to revise how we think about fieldwork and potential improvements we can enact to enable greater inclusivity and equality of access to such opportunities.  The concept and idea presented here is interesting and valuable for others thinking about similar local resource developments.

Response: We would like to thank Heather Sangster for her thoughtful and detailed review.

The main challenge with this manuscript is the small sample size (n=14/13), which inevitably introduces high uncertainty in relation to the confidence that can be placed on the derived statistics. I think the authors would benefit from a larger sample size, and it might have been worth waiting for another year or two so that this could have been achieved.

It is also important to think about student cohorts and changes in outcome relative to other modules students have taken, to demonstrate whether changes might be a function of improved skills/learning, rather than different student cohort characteristics, particularly when the sample size is small. 

Response: We recognise that this is a small cohort study and that this brings statistical problems, and we try to carefully handle the statistical interpretations in view of this. There have been two practical constraints on this project, however. Firstly, we are limited by small student cohorts in geology at UGent. Secondly, two of the authors, including the lead author, have now left the university as they were on fixed-term PhD/post-doc contracts. Whilst not ideal, we thought it preferable to disseminate the results of the project thus far, whilst acknowledging that further work in this area is needed.

It is important to note that there is a difference in confidence and skills, you identify the students have increased confidence, but do not demonstrate any notable shift in skills/learning, based on the marks, this is worthy of further reflection, even if this was something you try and capture in the future, or flag so others can consider this carefully.

Response: We agree that the gain in student confidence/self-efficacy without similarly notable increase in skills, a point also raised by other reviewers, is worthy of further investigation. We are happy to add a little further reflection in this manuscript, in particular pointing to Lundmark et al. (2020) who similarly found an increase in self-efficacy was not matched by an increase in assessed work, but we do not want to overstretch our interpretations about why this may be. We think that a detailed consideration is best left for future work, but are happy to offer some initial discussion here, and certainly to couch our work better within the context of other published literature on this subject (see response to RC3 for more details).

Fieldwork scores (though not the overall scores) did slightly increase in the Rock Garden cohort, but by a very small margin from 14.48 to 14.71 out of 20 (Table 4; Figure 7) that is statistically negligible. We see a few possibilities for further reflection and research in this area, including:

1. That students are not actually learning any more than they were before as the Rock Garden has substituted into, rather than extended, an existing course, they are just doing it in stages and maybe having a better/less stressful learning experience;
2. That we (staff) mark inconsistently across academic year groups but consistently within year groups (i.e. students may be doing better but we’re not adequately recording it), as suggested by RC2 (John Waldron);
3. That we have not sufficiently adapted teaching under the new model to make it work most effectively for the students;
4. That the small increase in field skills abilities we see is real, but that our sample size is too small to properly assess this.

We are happy to add a short paragraph to the discussion on this point, but really feel that this should be left as an open question to encourage future work.

It would have been interesting to see if the same results (post-) would have been achieved without the Rock Garden exercise – is there a difference in endpoint confidence by including the fieldclass?

Response: We agree that this is a fascinating question to address. Unfortunately, we do not have a suite of data from before the Rock Garden with which to test this, partly due to building the Rock Garden during the height of the COVID-19 pandemic to mitigate its impacts on student learning. Ideally, we would have collected such data in the years leading up to the Rock Garden initiative, similar to the approach of Waldron et al. (2016), and we would encourage others who are considering running a project like this to try to collect such baseline data before constructing their projects.

A very interesting paper.

Minor points

Line 28, I would personally reorder 37 and 60 respectively, I think it might read more easily.

Response: Agreed. We will do this, though we will also rephrase this whole sentence in light of the recently updated guidelines from the Geological Society as highlighted in RC3.

L40-45, you can also include the inequalities caused for those with caring responsibility where residential not possible, and therefore these individuals are prohibited/discouraged from undertaking degrees/qualifications in the Environmental Sciences more broadly.

Response: A good point, and thank you for the reminder. We will do this.

L126 Insert to: This allows students ‘to’ work

Response: We will do this.

L232 The sentence lacks clarity and needs to be revised, whilst the Rock Garden exercise saw the most rapid learning response, the real world experience still had a higher score, this needs to be clearly articulated.

Response: We will rephrase the sentence to make this clear.

L225-240 I would expect students to gain confidence after undertaking an exercise and with more experience, therefore some of these findings are not a surprise.

Response: We agree that the result (students gain confidence from training) is not particularly surprising and will add a sentence to this effect.

Review comments in italics. Response in bold.

My sincere apologies for the very late arrival of this review.

This is a good manuscript that describes an outdoor "garden" for use in geoscience training and education, and in outreach.  The installation follows a developing tradition of outdoor geoscience teaching facilities located within universities that help to offset the many challenges involved in getting students into the field under more realistic circumstances. The authors highlight what we have found to be the principal advantage of our installation: it helps to bridge the gap for students between theoretical, classroom-based learning, and the valuable, but challenging, experience of distant real-world field locations where barriers of safety, unfamiliarity, accessibility, and cost must be overcome. The challenges of Covid-19 have added new urgency to these problems.

I have no problem with the scientific results presented, so my comments really come down to some questions that came up, comparing this study with our own (Waldron et al 2016), and a number of points where the wording could be improved for clarity. 

Response: We thank John Waldron for his thoughtful review.

Overall questions, comments.

I'm interested in the results of your attempt to use actual student grades. In our study (Waldron et al. 2016) we didn't attempt this, for two reasons. First, our ethical guidelines made it very difficult to access student grades without informed consent; our perception was that our students were sufficiently nervous about their own grades that we would have many more students opting out if we asked for permission to access their grades. Secondly, and perhaps more seriously, having taught in the field we are aware that instructors instinctively adjust their grading (in areas that require subjective judgement calls in field situations) so as to take into account difficulties of weather, access, etc., which students encounter, and which vary from year to year. We were so skeptical that anything would show up in the marks that, given the challenges of gaining student consent, we didn't even bother to try this. I'm impressed that you tried, but not surprised that nothing significant showed up. (Note - this doesn't require anything to be changed in the paper - I'm just adding the comment for interest.)

Response: We nearly didn’t include this aspect within our study for the reasons outlined by the reviewer. However, it was considered to be in-line with the university’s general ethical protocol and we considered it interesting to evaluate. We also agree that instructors instinctively adjust their grading dependent on various conditions where such marking is necessarily subjective, and we think this may be one reason for seeing a limited cognitive response to the Rock Garden exercise (see also our response to RC1).

Second, I'm curious about the geology implied by the cross-section shown. This would seem to imply that the older succession was folded, that the fold was truncated by a planar unconformity, and that the fold was then reactivated after the Eocene, folding the unconformity. A folded unconformity is quite an advanced concept for students who are just at the level of learning to measure strikes and dips. Was this a planned complexity, and did you consider setting the students the simpler problem of a fold truncated by a planar unconformity?

Response: The geology is slightly more complex than we initially planned – settling of the rocks after positioning them slightly shifted some of the dips slightly and produced outcrops from which two episodes of folding could be inferred. We agree that this is perhaps too big an ask of students in an introductory exercise and are considering adjustments. We’d certainly recommend a simpler structure to others who look to build similar rock gardens.

Technical points for the authors

Line number: given first for each comment.

25: American could refer to all of N and S America. Specify USA if that's what you mean.

Response: Yes, we will clarify this.

50: The sentence beginning "We identified.." seems to be referring to some institution: maybe Ghent? This should be specified here; don't make the reader wait until line 57.

Response: Agreed. We will clarify this.

51: Multi-word noun phrases like "more skills provision" can be hard to read. I recommend "more provision of field skills": much clearer with very little extra length.

Response: We will rephrase this.

52: "brought in in response" - maybe "introduced" instead of "brought in". Also, "the global pandemic" is rather unspecific. Yes, I know what's meant now, but the paper still needs to be readable in 15 years time, so either specify the years or specify COVID-19.

Response: Agreed. We will change to “introduced” and specify COVID-19.

52 to 56: This is a long and convoluted sentence, especially "a valuable and viable alternative to bring..." where the reader expects "alternative to" to be followed by an actual alternative. Also "and which we also developed and used simpler versions of..." is quite clumsy.  I'd suggest breaking the sentence after the first long list of citations. Then continue "These are a valuable and viable alternative to traditional field courses, bringing specific field areas to students at home or in the classroom (Bond etc..)". Then in a third sentence explain what you (Ghent) did along these lines, noting that it was simpler than some of the previous versions, and if possible citing some information about your virtual field trips...  (I'm assuming here that the initiative described by "which we also developed..." is a separate one from the Rock Garden, and was perhaps not particularly successful, but the description is frustratingly minimal. Either flesh it out with more details or just delete from "and which we..." onwards.

 Response: We will rephrase and split this sentence for clarity.

107: outcrop sites (singular)

Response: Agreed. We will change this.

128: Needs a comma after public. As written it implies that only some of the public have free access and others do not.

Response: Agreed. We will do this.

135: Consider whether the phrase "Due to the limited size and number of outcrops.." really explains why there is no single correct answer. In my experience almost every real field area has more than one viable solution, unless perhaps there is 100% exposure, so the lack of a single solution is definitely not a difference from "real-world" mapping!

Response: We agree, and consider this a strength of the Rock Garden exercise as it introduces students to the concept that there are likely several viable solutions to most (if not all) geological problems. We will rephrase this to make this point more clearly.

160: The question asked here tests a hypothesis very close to that of Waldron et al (2016); I realize we are generously cited elsewhere, but I feel that this question was an innovation of ours; to deal with the difficulty of assessing changes in actual student performance, we assessed student perception or their own performance (metacognition).

Response: We agree that Waldron et al. is relevant to this hypothesis and our testing of it using a metacognition approach, and we are very happy to cite the study here. We would note, however, that we use a similar approach to ask a slightly different question of the students, which is how confident they feel in applying their skills. This is related, but not identical, to how well students think that a course has prepared them to conduct work which is the question asked in Waldron et al. (2016).  

194: The hyphenated word post-Rock breaks the flow because the sense is post-(RockGarden). Options are to write "post-Rock-Garden" or "post-installation", or use the more wordy (but more understandable) "data collected after the Rock Garden exercise".

Response: We used this phrasing to maintain consistency with the tables and figures. We understand the reviewer’s comment but would prefer to keep this as-is, because this is used here and elsewhere alongside the “pre-“ and “mid” prefixes to describe other questionnaires. To aid clarity, we can use the longer form phrase “data collected [before/after] the [Rock Garden/real-world] exercise” throughout.

199: Hyphenate small-cohort because it's used as an adjective.

Response: We will do this.

222: and elsewhere: "Reject the null hypothesis.." The null hypothesis is never actually stated in those cases where it is rejected. The text only mentions the alternative hypothesis that was upheld by the rejection. I recommend explicitly adding the null hypotheses somewhere — perhaps in the list , lines 203 to 205, where the null hypotheses could be put in parentheses following each hypothesis to be tested.

Response: Thank you for pointing this out. We will do this, and in response to RC3 we will rephrase the hypothesis section to better couch the hypotheses in the broader literature.

230: These nested parenthesis are code-like and difficult to read. Consider using one-word shorthands for the different survey questions instead of [b] and [c].

Response: Yes, re-reading the manuscript this seems unnecessarily convoluted. We will clarify this.

267: "less" should be "fewer" unless you are faithfully translating from similarly incorrect Dutch

Response: Correct, this was a mistranslation. We will change to “fewer”.

305: (and elsewhere). In British English this would be practising; the noun has a c and the verb an s.  The form used here follows the US convention of using -ice for both the noun and the verb. I mention it here because you may wish to check the journal style guidelines and be consistent.

Response: The first author’s autocorrect has been corrected. We will change this in the text here and elsewhere.

Review comments in italics. Responses in bold.

General comments / scientific issues:

This is a very well written and presented paper that evaluates the impact of an on-campus rock garden on students’ confidence in field skills. This is a novel approach and I really like that the study is framed in the context of addressing barriers to access and inclusion, and also in response to travel restrictions imposed during the pandemic. The following comments are for consideration alongside those made by the other reviewers, which I broadly support and see no need to repeat.

Response: We thank Alison Stokes for this thoughtful and considered review. Regarding points raised by other reviewers that Stokes supports, we hope that she finds our responses to these to be satisfactory.

The authors have clearly gone to some effect to generate a dataset but I query whether this is ‘research’. Acknowledging that research and evaluation exist along a continuum, I would expect a research article to be more rigorously framed with respect to the wider literature (beyond geoscience education) and a guiding theoretical / conceptual framework, and to present more robust findings and conclusions. This is a preliminary evaluation of a curriculum innovation which has been carefully designed and is well reported, and which provides a fantastic springboard for a more in-depth research study – but the sample size is problematic and I echo the other reviewers’ comments about confidence in the findings (or lack of). I would recommend that the title of the paper is tweaked accordingly as there is clearly some uncertainty around the apparent gains in confidence.

Response: We agree that the small sample size is a clear limitation of this study, but note that we are constrained by small cohorts in geology at UGent. We are happy to adjust the title of the manuscript to better reflect the inherent uncertainty from the small sample size, and suggest “The Rock Garden: campus-based geological field skills training may improve student confidence in real-world field work”.

I’m interested to know the bases for the hypotheses. What are the assumptions that are being made here, and how are these informed by the literature? Some consideration / discussion of the wider literature around confidence gains / self-efficacy would really help to strengthen this. There is evidence to show that students’ confidence in being able to accomplish a task is influenced by how they feel, and that during fieldwork students’ feelings / emotions can be influenced by a whole range of factors such as weather, time of day, the end-goal (are they being assessed etc.) – worth looking into literature around the affective domain, and also the link between confidence and learning gains. 

Response: We think this is an excellent suggestion and are happy to add some broader discussion of the origin of the hypotheses, as well as how students’ self-perceived confidence fits within broader theory of Bloom’s taxonomy and the affective domain.

Field work is especially interesting in the context of Bloom’s taxonomy because it has the capacity to target all three domains; cognitive, affective, and pyschomotor. Our work intersects primarily with the affective domain in that we target a greater understanding of how students’ self-perceived confidence in their abilities (their self-efficacy) changes over a field course.

There is generally considered to be a positive link between students’ confidence in their abilities (self-efficacy) and achievement for a given task (e.g. Boyle et al., 2007; Stokes & Boyle, 2009; McConnell & van der Hoeven Kraft, 2011; Wiggen & McDonnell, 2017). There also appears to be a general positive affective response in geoscience students from field courses (e.g. Boyle et al. 2007; Streule & Craig, 2016; Waldron et al., 2016), and summarised neatly in the title of Boyle et al. (2007): “Field work is good”. However, McConnell & van der Hoeven Kraft (2011) noted that whilst the affective domain is important to student learning, it has been under-studied in the geosciences and indeed is hard to study.

We address this topic by evaluating changing self-efficacy over the duration of a field course including with a component that uses a controlled and familiar environment, the Rock Garden, in training. We think that the familiarity of this environment was an important factor in helping students grow in confidence when applying their skills.

We would like to see this tested in future research comparing the impact of smaller training exercises on campus to similar exercises conducted off campus in the field, for example, echoing the conclusions of works such as Boyle et al. (2007) and McConnell & van der Hoeven Kraft (2011). We think that further research is particularly important in the context of a recent study using a digital field work tool which found, similarly to our work, that the increase in self-efficacy promoted by the new teaching aid did not in fact bring about an improved performance in assessment (Lundmark et al. 2020) in contrast to general theory and previous work. As we noted above in response to RC1, however, we are wary of overstretching this interpretation from our study due to the small cohort size.

As well as generally expanding this body of research, including with larger cohort studies, we would be particularly interested in follow-up research into the longevity of a positive affective response from the Rock Garden exercise – i.e. was this a more state-like (short term) or more trait-like (long term) change in confidence? – and into the potential link between positive affective and cognitive responses. Overall, we think that our small study adds to the growing body of literature in this area, but we would echo the calls of previous studies that this is a topic that deserves much further research.

I’d also recommend placing the findings in a broader context by bringing some of the literature into the discussion. Some interesting examples of other, similar initiatives are introduced in section 2.1 - has there been any similar evaluation of impacts on student learning? If so, what did they find out and how do your findings contribute to the evolving body of knowledge about the potential for rock gardens to support / enhance student learning? 

Response: We are happy to expand the manuscript discussion to better frame our results within the broader ‘Rock Garden’ literature. As far as we are aware, this is the first investigation of the impact of a Rock Garden-type initiative on students’ own perceptions of their confidence in applying field skills. Two other studies have clearly addressed questions of self-efficacy in this context: Benison (2005) and Waldron et al. (2016). Benison (2005) asked students how the campus-based field resource helped them to improve their skills in several specific areas. Waldron et al. (2016) asked the related, but slightly different, question of “How useful [the course] was … in preparing [students] for the following [course]” (Ibid., p. 225) with sub-questions targeting specific field skills. Benison (2005) questioned students only once, and after the course was completed, and found that students were generally supportive of using the campus field resource and that they thought it had helped them to improve their skills. Waldron et al. (2016) report a difference across cohorts, where the first cohort were taught before the University of Alberta Geoscience Garden was built and the second cohort were taught after it was built. Waldron et al. report a post-installation shift in attitudes to being better prepared after the Geoscience Garden was built and used in teaching. Our work qualitatively supports the findings of both Benison (2005) and Waldron et al. (2016) that students find a campus-based field resource helpful training in general, and that it is beneficial for subsequent field work (Waldron et al. 2016). Our work provides some additional nuance as to when self-efficacy gains may be made (i.e. the strongest self-efficacy gain may occur during the on-campus training; our Table 3 & Fig. 6).

We are wary of going too far in this discussion, given our small sample size and the warnings about this imparted by all reviewers, but we agree that it would improve the manuscript to add further discussion on this point.

Overall, this is a nice study that provides some preliminary evaluation of on-campus rock gardens as a learning tool – on this basis it will be a good contribution to the literature. However, there needs to be more rigorous follow-up investigation to support any substantial claims about impact.

Response: We are grateful for this thoughtful review and agree that our study should be seen as providing a springboard to further research on the impacts of artificial field courses on students’ skills and self-efficacy acquisition.

Technical corrections:

L23-24: note that the Geological Society of London has updated its accreditation requirements and no longer specifies a number of field days that students must complete (see https://www.geolsoc.org.uk/~/media/shared/documents/education%20and%20careers/University%20Accreditation/2023%20documents/Introduction%20and%20Guide%202023.pdf?la=en).

Response: We thank the reviewer for pointing out the updated accreditation requirements, which we are very pleased to see. We will amend the text accordingly.

References cited in the response:

Benison, K. C., 2005. Artificial Outcrops Give Real Experience in Interpreting a Geologic History: The CMUland Group Project for Historical Geology Courses, Journal of Geoscience Education, 53, p. 501–507, doi: 10.5408/1089-9995-53.5.501.

Boyle, A., Maguire, S., Martin, A., Milsom, C., Nash, R. Rawlinson, S., Turner, A., Wurthmann, S., & Conchie, S., 2007. Fieldwork is Good: the Student Perception and the Affective Domain. Journal of Geoscience Education, 31(2), p. 299-317 doi: 10.1080/03098260601063628

Lundmark, A. M., Augland, L. E., & Jørgensen, S. V., 2020. Digital fieldwork with Fieldmove - how do digital tools influence geoscience students’ learning experience in the field? Journal of Geoscience Education, 44 (3), p. 427-440. doi: 10.1080/03098265.2020.1712685.

McConnell, D. A., & van der Hoeven Kraft, K. J., 2011. Affective Domain and Student Learning in the Geosciences. Journal of Geoscience Education, 59(3), p. 106-110, doi: 10.5408/1.3604828.

Stokes, A., & Boyle, A. P. 2009. The undergraduate geoscience fieldwork experience: Influencing factors and implications for learning in Whitmeyer, S. J., Mogk, D. W., & Pyle, E. J. (eds.), Geological Society of Field Geology Education: Historical Perspectives and Modern Approaches, GSA Special Papers, doi: 10.1130/2009.2461(23)

Streule, M. J. & Craig, L. E. 2016. Social Learning Theories—An Important Design Consideration for Geoscience Fieldwork, Journal of Geoscience Education, 64:2, 101-107, DOI: 10.5408/15-119.1

Waldron, J. W. F., Locock, A. J., & Pujadas-Botey, A., 2016. Building an Outdoor Classroom for Field Geology: The Geoscience Garden. Journal of Geoscience Education, 64(3), p. 215-230, doi: 10.5408/15-133.1.

Wiggen J., & McDonnell, D., 2017. Geoscience Videos and Their Role in Supporting Student Learning. Journal of College Science Teaching, 46(6), p. 44-49.