Flash Flood!: a SeriousGeoGames activity combining science festivals, video games, and virtual reality with research data for communicating flood risk and geomorphology

The risk of flooding around the world is large and increasing, yet in many areas there is still a difficulty in engaging the public with their own flood risk. Geomorphology is a science that is linked to flooding and can exacerbate risks, but awareness of the science with the public is low and declining within academia. To increase awareness, it is important to engage the public directly with the science and those who are working to reduce flood risks – this starts by inspiring people to seek out further information through positive experiences of the science and researchers. Here, a new design model is presented to engage the public with specific research projects by using useful components offered by the popular mediums of games, virtual reality (VR), and science festivals to allow the public to get “hands on” with research data and models: SeriousGeoGames. A SeriousGeoGames activity, Flash Flood!, was developed around real geomorphology survey data to help engage the public with a flood-risk-related research project by placing them in a river valley as it undergoes a geomorphically active flooding from an intense rainfall event. Flash Flood! was exhibited at two science-focussed events, and formal evaluation was captured using a short questionnaire, finding that the majority of audience had a positive interaction (95.1 %) and wanted to know more about flooding (68.0 %) and geomorphology (60.1 %). It is hoped these interactions will increase the likelihood that future engagements with relevant agencies will be more fruitful, especially when it matters most.

1 Introduction

Flooding is a first-order risk around the world, and the UK is no exception. The UK's Environment Agency estimates that 5.2 million homes are at risk of flooding yet less than 10 % of those consider themselves at risk (Curtin, 2017). Curtin (2017) goes on to compare this to a YouGov poll (Smith, 2017) suggesting that more than 11 % of the UK's 27.2 million households (Office for National Statistics, 2017) have made a plan in case of a zombie apocalypse. It is astonishing that the public seems better prepared for an entirely fictional risk than they are for something that poses real risk, but this is the situation practitioners find themselves in.

Geomorphology is the science of how planetary surfaces form and change. Geomorphic processes can increase the impact of flood events through erosion of the channel and banks, including scouring around infrastructure such as bridges and the transport of material that can make flood waters more damaging. The cleaning up of deposited material, sometimes contaminated, increases the post-event cost. Geomorphic processes also contribute to the likelihood of flooding with erosion and deposition altering a river channel's capacity to hold water or even changing the course of the river itself. Presently, geomorphology is not considered an important component of flood forecasting and is considered a minor source of uncertainty (Flack et al., 2019), yet some evidence suggests that flood-related geomorphology is likely to be exacerbated by climate change due to the non-linear relationship between river discharges and sediment yields (Coulthard et al., 2012). Geomorphology is a key part of many pressing environmental issues, such as flooding (Lane et al., 2007; Slater, 2016), soil erosion (García-Ruiz et al., 2015), sand mining (Bendixen et al., 2019), and the transport of plastic pollution (Hurley et al., 2018), all of which are of great interest to the public and media; however, the term itself as a distinct discipline is declining within academia and is virtually unheard of by the public, in curricula, and in media reporting of geomorphic events (Clarke et al., 2017).

With climate change due to increase the risk of flooding and the geomorphic impacts of flooding, it is unfortunate that practitioners already find themselves playing catch-up in the communication of even present-day risks (Curtin, 2017). Clarke et al. (2017) asserts that the responsibility is with geomorphologists, and by extension flood management practitioners, to effectively communicate these risks.

This paper presents a case study of the Flash Flood! application, an interactive virtual reality (VR) activity designed to highlight the geomorphic risk posed by flooding from intense rainfall, more commonly known as flash flooding. VR generally uses two screens held within a headset (head-mounted display; HMD) so that each eye can only see one screen, with each showing a 3-D scene at a different angle to produce the illusion of depth and immersing the user in a different and artificial environment. The rest of Sect. 1 highlights the proposed SeriousGeoGames model of combining elements of VR and video gaming with elements from research projects, such as field data or numerical modelling codes. In Sect. 2, the specific research context for Flash Flood! is described, followed by a description of the development of the application in Sect. 3. Section 4 details the evaluation methods and the events where the application was tested. The results of the evaluation are shown in Sect. 5 and discussed in Sect. 6, before conclusions are presented in Sect. 7.

1.1 The SeriousGeoGames model

The SeriousGeoGames Lab was established in 2014 to explore the use of games, and gaming technology, in enhancing the research, teaching, and communication of geosciences. The first SeriousGeoGames activity produced was Humber in a Box (Fig. 1), a novel dynamic merging of a research-grade hydraulic model, CAESAR-Lisflood (Coulthard et al., 2013), with a software package used by games developers to create games and virtual environments (known as a gaming engine), Unity 3D. Participants viewed a 3-D model of the Humber estuary, UK, on top of a box in a museum-style space, while tidal flows were calculated using the CAESAR-Lisflood code and animated within Unity 3D. Participants could then simulate past and future scenarios by altering the base sea level, giving them an idea of future flood risk with rising sea levels. The scene was viewed using immersive VR via an Oculus Rift Developer Kit 2 HMD model.

Figure 1 Screenshot from Humber in a Box showing left- and right-eye views. Inside the virtual reality headset these views appear as one, giving the illusion of a 3-D space. The scene on the table is a visualisation of the CAESAR-Lisflood model simulating tides in the Humber estuary, UK.

Humber in a Box proved a popular exhibit at events and festivals across the UK, and the anecdotal experiences of what worked well provided a framework for a simple model to design future SeriousGeoGames activities from. The SeriousGeoGames model is one of design choices and considers that they will be predominantly used within a science festival setting where interactions may be short, a few minutes at most, and the turnover of users is high. They should look and feel like video games, even if they do not qualify as games themselves. They should exploit VR as a medium of interaction immersing people into new environments. Crucially, they should provide people a first-hand interaction with elements of the ongoing research, such as incorporating field data or numerical modelling codes.

A successful SeriousGeoGames activity will achieve two objectives:

1. to create a positive experience for the participant with scientists and the research topic (create fun)

2. and to increase interest for the participant in the research topic (create curiosity).

To use an analogy borrowed from religious evangelism, the purpose is to “plant a seed” with the participant that might “germinate” with future interactions with science, scientists, or relevant practitioners in the future. Whether the positive interaction does in fact plant this seed is a matter of trust and something exhibitors will never be able to view come to light.

It is important to emphasise that the SeriousGeoGames model has been constructed through design choices and anecdotal experiences of previous activities and events. It incorporates three key elements – science festivals, video games, and virtual reality – that can help to achieve the two objectives.

1.2 Science festivals

The science festival is a common feature of public engagement with research landscape. The vibrant UK Science Festivals Network boasts 50 festival members, who in 2018 ran 4018 events, featuring 10 941 scientists and achieved 1 225 779 face-to-face interactions (Woolman, 2019). The US scene is also growing, with the Science Festival Alliance growing from just four member festivals in 2009 to around two dozen in 2012 (Durant, 2013), and in 2017 47 member festivals shared science and research with over 2 million members of the public (Science Festival Alliance, 2018).

Traditionally, a science festival will be focussed on a central exhibition space, populated by stands and exhibits, focussing on interactive demonstrations highlighting either basic science principles or more bespoke demonstrations for research projects. Science festivals also usually feature talks and panels by scientists on contemporary issues and workshops that take people into more detail. Many festivals encourage more creative methods of engaging audiences, including café crawls, story-telling events, improvised comedy, orchestral performances, and films (Durant, 2013).

The goal of a science festival is usually to celebrate science and research (often that performed or funded by the organisers) and to engage non-specialists (Bultitude, 2014). As such, they have become a core method used to engage the public with the latest research (Jensen and Buckley, 2014). The true power of science festivals is their ability to bring the public and scientists together, and the most successful engagements emerge from the conversations engendered (Jensen and Buckley, 2014; Wiehe, 2014).

Science festivals could be described as niche in their nature, appealing to a small sub-set of the population. According to a 2011 MORI (Market and Opinion Research International) poll, only 3 % of the UK population attended a science festival in the previous year (Jensen and Buckley, 2014), and this remained at 3 % for the latest poll in 2014 (Castell et al., 2014). A criticism of science festivals is that they only attract those who are already “science interested” and who tend to be well-educated, meaning that there is little socio-economic diversity across the attendees (Bultitude, 2014). However, evaluations of events that have targeted under-represented groups have seen the same success by facilitating interactions between scientists and the public (Jensen and Buckley, 2014).

1.3 Video games

Video gaming is big business, with retail sales of video games accounting for 51.3 % of the UK's entertainment retail market (including music, video, and games), and is worth GBP 3.84 billion (Entertainment Retailers Association, 2018). It is forecast that there are 2.3 billion people using video games worldwide, with a global market of USD 137.9 billion (Wijman, 2018). The popularity of video games has not gone unnoticed by educators, with dedicated educational versions available of popular games such as Minecraft, Roblox, Assassin's Creed, and SimCity, and the educational games market is expected to reach USD 17 billion by 2023 (Adkins, 2018).

Video games are powerful tools for engaging people with research as they provide a first-hand experience that can inspire an emotional response (Mendler De Suarez et al., 2012; Squire, 2003; Wu and Lee, 2015). In addition, games are fundamentally fun (Wu and Lee, 2015), and as such they are naturally engaging and motivating for the user (Ryan et al., 2006). Video games are popular, with 28 % of UK households (BARB, 2019) and 36 % of US households (Entertainment Software Association, 2018) owning a gaming console. These figures do not count personal computers (PCs), smartphones, or tablets that are used for gaming, which increases the figure to 64 % in the US (Entertainment Software Association, 2018).

The flexibility and complexity that can be afforded by video games has made them an attractive tool for engaging people with complex issues such as climate change (Porter and Córdoba, 2009; Reason, 2007; Warburton, 2003). This has led to the development of “serious games”, games where learning is a core objective without losing sight of the entertainment element (Abt, 1987; Charsky, 2010; Crookall, 2010), and there are several studies showing that serious games have been effective in delivering the intended learning outcomes (Amory et al., 1999; Bellotti et al., 2013; Betz, 1995; Chin et al., 2009; Coleman et al., 1973; Connolly et al., 2012; Gosen and Washbush, 2004; Hobbs et al., 2018, 2019; Lane and Yi, 2017; Mani et al., 2016; Mitchell and Savill-Smith, 2004; Vogel et al., 2006; Wilson et al., 2009). Serious games can be used to create virtual analogues of real world places or physical phenomena for public engagement, such as volcanism (Hobbs et al., 2018, 2019; Mani et al., 2016).

1.4 Virtual reality

Virtual reality can be used to refer to any computer-based simulation featuring a virtual world (e.g. Markowitz et al., 2018; Merchant et al., 2014; Mikropoulos and Natsis, 2011); however it is used here to refer specifically to “immersive” VR where a user will typically use a HMD to view the virtual world. It is currently regarded as an emerging technology, but VR has been around since the 1960s (Sutherland et al., 2003) and has seen various phases of development, particularly in education (e.g. Bricken and Byrne, 1993). It has only been recently, with the development of HMDs such as Oculus Rift, HTC VIVE, and PlayStation VR, that the technology has enabled mainstream use of VR.

VR simulations often share features with video games and thus share many of the same learning advantages, such as being engaging and motivating (Abulrub et al., 2011; Psotka, 2013). However, the immersion and presence (the feeling of physically being in the virtual world) produces experiences that are highly engaging, allowing the user to focus more on the learning outcomes (Bricken and Byrne, 1993; Markowitz et al., 2018; Salzman et al., 1999). Furthermore, users consider the virtual environment as real (Blascovich and Bailenson, 2011) and can develop a strong attachment and internalisation toward them (Clark, 1997; Weisberg and Newcombe, 2017). A particular advantage of VR is that it can allow users to feel closer to otherwise abstract or distant ideas (Trope and Liberman, 2010); for example in Markowitz et al. (2018) users were shown first hand (via VR HMD) the impacts of ocean acidification and reported an increased knowledge gain and interest in the subject as a consequence.

VR is not without its limitations. Cost remains a considerable barrier to its uptake and use, with popular HMDs costing several hundred GBP (for example, Oculus Rift S costs ∼ GBP 400, and VIVE Pro costs ∼ GBP 800) and requiring a gaming specification PC to run. The use of VR can also induce a nausea or dizziness (sometimes called cybersickness), similar to motion sickness, and can also cause headaches and eyestrain (Rebenitsch and Owen, 2016). In one test, seated participants using the Oculus Rift HMD for less than 15 min reported a 22 % occurrence of cybersickness (Munafo et al., 2017).

2 Flooding from Intense Rainfall

2.1 The research context

Flash Flood! was conceived as an engagement activity to support the Flooding from Intense Rainfall (FFIR) research programme, funded by the Natural Environment Research Council UK (NERC). The FFIR programme described itself as “A five year NERC funded programme aiming to reduce the risk of damage and loss of life caused by surface water and flash floods” (Flooding from Intense Rainfall, 2019). The programme, based in and focussed on the UK, brought together experts from several universities, environmental consultancies, the Met Office, the Environment Agency, and the British Geological Survey to better understand the role intense and localised rainfall events had on both rural and urban flooding, with a strong focus on the end-to-end forecasting on events (Dance et al., 2019; Flack et al., 2019). Thunderstorms, driven by strong convection in summer months, form and dissipate rapidly and can be highly localised covering just a 1–3 km wide area. Despite a good understanding and being able to forecast the conditions in which they form, it is presently not possible to provide accurate forecasts of when and where the storms themselves will form.

The focus of the simulation would be on a sub-section of the programme concerning the modelling of the geomorphic impacts of flash flooding. For most flood events in the UK, changes to the river bed, channel, and surrounding flood plain through the processes of erosion, deposition, and transport (i.e. geomorphic activity) are negligible to resulting flooding. This is reflected in the current flood forecasting situation in the UK where geomorphic activity is considered as a source of uncertainty that influences model results to a much lesser extent than other sources, such as the rainfall input (Flack et al., 2019). Despite being rare, there have been recent high-profile examples of extreme geomorphically-active flood events including in Boscastle (2004), Cockermouth (2009), Glenridding (2015), and Coverack (2017). Because of the risk to life and property, it is important there is an awareness of these extreme events and how and when they occur.

The geomorphic activity induced by flash flooding can make the flooding even more devastating to communities who can find their properties inundated with mud and debris as well as water. Transported material in flood water increases its power and ability to erode, making it able to destroy and wash away infrastructure, such as bridges. It can also have a profound effect on the river valleys themselves, with some floods inducing so much geomorphic change that they fundamentally change the behaviour of the river for several years, sometimes decades. These flood events have been referred to previously as threshold events (Bull, 1979; Chappell, 1983; Fryirs, 2016; Milan, 2012; Schumm, 1979).

Threshold events relate to a concept in geomorphology science called river sensitivity. This concept was described by Kirstie Fryirs as “lost”, but of increasing significance for landscapes under a changing climate, in her Gordon Warwick Award winner's address to the British Society for Geomorphology in 2015 and subsequent paper (Fryirs, 2016). The concept can be summarised by the equation below:

$$\begin{array}{}\text{(1)}& \mathrm{River}\mathrm{sensitivity}={\displaystyle \frac{\mathrm{Recurrence}\mathrm{of}\mathrm{threshold}\mathrm{events}}{\mathrm{Time}\mathrm{required}\mathrm{to}\mathrm{recover}}}\end{array}$$

(adapted from Fryirs, 2016).

The equation assumes that every river has a stable behaviour, with it displaying consistent responses to similar events. This stability is maintained by mature vegetation cover and a paucity of sediment that can be moved by the river. However, there exists a threshold magnitude of flood event that will disturb this stability by removing the vegetation cover, exposing sediment, and transporting it elsewhere in the channel. After the event, the channel begins recovery (or relaxation) through a period of enhanced dynamism in the geomorphology until new vegetation has matured and sediment sources exhausted. The balance between how often these events occur and how long it takes a river channel to recover is the river's sensitivity. During the threshold event and the river's recovery the amount of sediment delivered downstream in the system is greatly increased and this in turn may influence the flood risk in those areas (Lane et al., 2007; Slater, 2016). Predictions of climate change for the UK suggest flood events will become more likely and more extreme (Dankers and Feyen, 2008; Ekström et al., 2005; Feyen et al., 2012; Fowler and Ekström, 2009; Pall et al., 2011; Prudhomme et al., 2003) disrupting the balance determining river sensitivity; the impacts of this on rivers and future flood risk is not known but is likely to be negative and increase future flood risk.

2.2 The research data

The case study at the heart of Flash Flood! is the 2007 flood event in the upland valley of Thinhope Burn, northern England, as detailed by Milan (2012). The event was a FFIR event that could be described as a threshold event for the system. During a 6 h period a highly localised yet intense convective storm precipitated 82 mm of rainfall on the upper catchment (Met Office, 2003), resulting in a flash flood. Those who witnessed the event described a wall of water and the sound of boulders crashing along the river bed (Milan, 2012). The valley floor was fundamentally changed by the event with large geomorphic changes, including the straightening and widening of the main channel, stripping out of flood plain vegetation, deposition of material in the channel and on the flood plain (see Fig. 2), and increased mobility of material subsequently (Milan, 2012).

Figure 2 Google Earth images showing the reach section surveyed and used for Flash Flood!. Panel (b) is from before the flood in 2006 (Google Earth, 2019a) and (a) from after the flood in 2007 (Google Earth, 2019b). The flood has cut meanders resulting in a straighter channel, stripped-out vegetation, and deposited loose sediment on the flood plain (the lighter colour in b).

The usefulness of this case study for the development of Flash Flood! was the availability of ground survey data of the stable river valley just 3 years prior to the flood and repeat surveys afterwards, which were used by Milan (2012) and provided for this work. To have detailed surveys shortly before a geomorphically active event such as this is rare and cannot be planned for, so it provided an exciting opportunity. This survey was captured in the summer of 2003 using a backpack global-positioning-satellite (GPS) system across a 500 m reach section. Although similar surveys were available for after the flood, it was decided to recapture the same 500 m in more detail using a terrestrial laser scanner (TLS) in the summer of 2014. The recovery period after extreme events varies widely between different areas, depending on factors like local vegetation, soil, or climate, but can take decades. Although this survey was conducted 7 years after the flood, the channel had still yet to recover and largely reflected the immediate post-flood environment.

To give an indication of the height of the peak flood extent, simple modelling was performed within the CAESAR-Lisflood software (Coulthard et al., 2013), using elevations derived from the 2003 GPS survey and the estimated peak discharges from Bain et al. (2010) to drive the model hydraulics.

3 Development

The Flash Flood! application was designed by the SeriousGeoGames Lab and developed by indie-game (independent-game) developer BetaJester Ltd using the Unity 3D gaming engine. There have been two iterations of the VR-based software, with the second being optimised based on the experiences exhibiting the original version.

3.1 The original Flash Flood!

The original Flash Flood! was developed in 2015. The 3-D environment was built using the popular gaming engine Unity 3D. The flood scenes before and after were constructed from the digital elevation models (DEMs) using the data described in Sect. 2.2, with each converted into a point cloud. A sample of each point cloud was extracted, converted to a mesh, and imported into Unity 3D. The scenes were populated using textured renders and 3-D objects (known as assets), with the scene being more heavily populated with trees than in real life to help blur edges and create a more interesting 3-D environment for participants to explore.

The exhibit used an Alienware X51 R3 (Intel Core i5 6400 CPU @2.71 Ghz with 16 GB RAM and an NVIDIA GeForce GTX 970), which was labelled as “Oculus-ready”, with the consumer model Oculus Rift HMD. The application was optimised to a lower standard than the equipment specification afforded to allow a desktop-only version of the software to be released. For example, the graphics were kept simple (see Fig. 3), and the representation of water kept to an animated plain that was angled down in the direction of the river and would rise and fall, giving the impression of rising and falling water levels as it intersected the landscape. The public participants explored the scene using the two joysticks on an XBOX controller and needed to use no other buttons or d-pads (directional pads).

Figure 3 Screenshot from the original Flash Flood!.

The participant began the simulation within the river valley, viewing it from a first-person perspective. They were free to explore the whole scene with movement restricted at the edges by hills or invisible barriers. The flood animation timeline did not begin automatically and only started when a crew member pressed the P button on the keyboard.

The simulation moved along a 6 h timeline that took 30 s h⁻¹ time steps, for a total of 3 min. It began at 15:00 BST (British summer time) and the following on-screen prompts described the scene at each step.

•  

  15:00: “Clouds begin to gather”;

•  

  16:00: “A storm is brewing”;

•  

  17:00: “The storm intensifies”;

•  

  18:00: “Intense rainfall falls on the uplands of the river”;

•  

  19:00: “Rain water from the uplands swells the river level. A flash flood is coming!”;

•  

  20:00: “The flood has reached its peak”;

•  

  21:00: “The flood has receded leaving a scene of devastation”.

At 19:00 the eponymous flash flood wave passed through the scene; this was produced using two shapes, a box and wedge (as the flood toe), textured in the same way as the water, to give an impression of the “wall of water” described by witnesses (Milan, 2012). Throughout the timeline the water turned increasingly brown to represent the debris within the water. As the simulation transitioned between 20:00 and 21:00 the flood scene before was switched to the flood scene after. Most of the changes were obscured under the height of the water, as this was the peak of the flood, but it still required a removal and repositioning of the participant within the scene (a process known as respawning), resulting in some sudden, unrealistic changes.

The limitations of time and funding meant that there was no sound incorporated into the original version, and narration was provided via a one-to-one interaction with a crew member – usually a scientist within a relevant research area or a science communication generalist. This had the advantage of being able to tailor the message based on the crew member's research field and the age and responsiveness of the participant.

3.2 Flash Flood! Vol.2

In 2018, an opportunity arose to redevelop the original Flash Flood!. Where the original had been limited in its graphics and representation of river flow due to the release of a desktop-only version, there were no such limitations for Flash Flood! Vol.2. Instead, the new development was optimised for a new set of equipment using the Alienware 17R5 Oculus-Ready laptops (Intel i7-8750H @ 2.20 GHz with 8 GB RAM and an NVIDIA GeForce GTX 1070), with an aim of achieving the look and feel of a AAA game (games produced by large gaming companies intended for the global commercial market). This was partly in response to an increasing number of anecdotal comments on the basic level of the original graphics and participants becoming more accustomed to evermore sophisticated VR experiences. Photo-realistic assets were used for textures and 3-D objects, and the scene was made wooded like the original to make a more interesting scene (see Fig. 4). The transitions at the edges of the scene were significantly improved by removing the hills and replacing these with an extended landscape (that could not be explored) and hiding the edges using stone bridges. The basic horizontal plain of water was replaced by the more sophisticated River Auto Material (R.A.M by NatureManufacture) asset, with customisation from the developers for the representation of the flash flood showing a rapidly rising water level with debris in the form of rocks and logs. Flash Flood! Vol.2 uses the same data and flood timeline as the original version.

Figure 4 Screenshot from Flash Flood! Vol.2.

From the point of view of an exhibitor the main limitation of the original version was the staffing resource required due to the one-to-one narration provided by the operator; this interaction was exhausting and a single operator could manage around four or five demos before requiring a rest during busy periods. This means each set-up required a minimum of two operators rotating regularly, and an extra operator for every two sets to allow for breaks and control of the crowd. This limited the number of demonstrations that could be achieved and the size of exhibits that could be supported. To overcome this limitation, Flash Flood! Vol.2 uses a soundtrack with narration. The user chooses between two narrators, Chris (voiced by Chris Skinner) and Jess (voiced by Jess Moloney). As video gaming is often perceived as a male space with women and girls feeling excluded or discriminated against (for example, Delamere and Shaw, 2008), it was decided the choice of narrator would default to Jess so that participants would encounter a female scientist first. The two narrations follow slightly different scripts with Chris's being more general and Jess's drawing more on Moloney's research into dating past flood events (Moloney et al., 2018). The choice of a single male and female voice was a starting point and allows for an increased representation of voices with future developments.

3.3 Ancillary developments

The two iterations of VR software are not the only developments relating to Flash Flood! nor should the achievement of the two objectives be limited to the time and space within the science festival hall. The activity was promoted and supported by the SeriousGeoGames social media accounts (Facebook and Twitter) and website. At times this was enhanced by support from the University of Hull Marketing and Communication team, plus other colleagues at the University of Hull, other universities (particularly Reading and Newcastle), and NERC.

To make the application more accessible, a desktop-only version was made available via SourceForge that could be controlled using a mouse and keyboard. This was free to download and would operate on any reasonably modern Windows machine. However, several schools reported they wished to use the software but were unable to due to networking restrictions on school machines, and in response two 360 video versions were produced and made available via YouTube: a narrated version (Flash Flood! 360) and a non-narrated version (Flash Flood! Classroom). These videos allowed head tracking but not the freedom to explore the scenes. To support both the desktop and video versions, a manual was produced and articles aimed at students and teachers published (Skinner, 2018; Skinner and Milan, 2018).

To support the original version of Flash Flood!, a handout was produced. The handout included brief descriptions of the flood event, links to the SeriousGeoGames website and social media accounts, and an activity that could be done alongside the simulation. The intention was to mimic the taking of field notes performed by geomorphologists, before and after the flood, particularly for use with the desktop and YouTube versions of Flash Flood! outside of events (it was also available as a PDF download). At events the handout was given out along with a “I survived the Flash Flood!” badge and was also free to be taken from the table. It was used to engage members of the public either waiting for a turn or accompanying a participant by getting the participant to describe what they were seeing so it could be written into the field notes section.

4 Evaluation

The different versions of Flash Flood! have been demonstrated at events since its debut at the Hull Science Festival (SciFest) in March 2016, several years before any evaluation activity beyond informal conversation with participants and head counts was conducted. The experience of exhibiting has provided a wealth of anecdotal information valuable for designing new activities but is potentially biased (Jensen, 2015) and not suitable for formal evaluation (Neresini and Bucchi, 2011). Previously, evaluation at events has been eschewed as it was perceived to intrude on the experience of the participants and potentially impede on the success of the objectives, especially when the activity is just one exhibit of many as part of a larger science festival. Summative evaluation, conducted after participation with activities, can reduce the intrusion on interactions – an example would be autonomous methods for participants to leave feedback, such as graffiti walls and feedback cards (Grand and Sardo, 2017). Autonomous methods have been tried alongside Flash Flood! previously, for example at the 2018 Hull SciFest.

The formal evaluation of Flash Flood! was conducted using Flash Flood! Vol.2 during two events. The first event was Scarborough Science and Engineering Week (SSEW) 2019 held 8–10 October 2019 at Scarborough Spa, Scarborough, UK. SSEW was targeted at schools in the local area, with two days (8 and 9 October 2019) for secondary-school and college pupils (ages 11–18) and a day for primary-school pupils (ages 5–11). In total 1361 secondary-school pupils and 1191 primary-school students were booked to attend. The second event was the Open Day for the British Geological Survey (BGS) held at their campus in Keyworth, UK, on 12 October 2019. This was a single-day, ticketed event, aimed at families where all 1800 free tickets were taken up. The potential overall audience from bookings was 4352 people, although it was expected that participant numbers would be much lower than this.

The evaluation for both events used the same questionnaire (see Fig. 5). Questionnaires are not best suited for busy science festival settings but are an effective way of gathering quantitative information (Grand and Sardo, 2017; Wiehe, 2014). In an attempt to reduce this impact, the questionnaire was designed and hosted via the Formstack app on iPads, displayed in stands; participants filled and submitted the form on the iPad rather than using paper surveys. The questionnaire was designed to assess Flash Flood! Vol.2 versus the two objectives in Sect. 1.1, which can be summarised as creating fun and curiosity. Participants were orally referred to the questionnaires by the exhibit crew after finishing their turn on Flash Flood! Vol.2. Completion was voluntary, and participants were not observed whilst completing it. At SSEW, up to four VR stations running Flash Flood! Vol.2 were operating at once along with two iPad evaluation stations, and at BGS Open Day there were up two VR stations and one iPad evaluation station.

Figure 5 Flow diagram showing the questionnaire design. All respondents are offered all questions on the left-hand side, whilst questions on the right-hand side were only shown under indicated conditions. All questions in green boxes had to be answered to allow the form to submit.

The results of the questionnaire were assessed at the event level and for SSEW divided into Days 1, 2, and 3. Through aggregating Days 1 and 2 together, it was possible to compare the audience of secondary-school and college pupils with primary-school pupils. Differences were assessed for statistical significance using the Mann–Whitney U test (with a threshold of p<0.05 as significant) as per Hobbs et al. (2019).

At both events a large (3 m wide and 2 m high) canvas banner advertising Flash Flood! was on display featuring the following text.

Flash Flood!

Geomorphology: The science of how landscapes change

Try our Virtual Reality demo to see how floods can change river valleys

Climate change is predicted to increase flooding, erosion, and changes to our rivers

Flash Flood! has been built using data from a real river and is based on a real flood

The space set up for both events is shown in Fig. 6. Whilst the BGS Open Day was a traditional tabletop activity and banner set up, SSEW featured some more design elements, like event fencing, a static drone display, and an immersive forest soundscape within the fencing.

Figure 6 Exhibit set-up for the Scarborough Science and Engineering Week (a) and the British Geological Survey Open Day (b). The iPad and stand for the evaluation station at the British Geological Survey Open Day is just off shot to the right of the image.

The ancillary developments designed to support the exhibit include the SeriousGeoGames website (hosted on WordPress) and YouTube channel. Both WordPress and YouTube provide detailed analytics of views, audience, sources, and other useful information that can be broken down by date. These analytic data were used to evaluate whether the online content, and the Flash Flood! handout that signposted participants to it, was useful for achieving the two objectives during the NERC UnEarthed event in 2017. This was done by comparing views of the content during a 17 d period covering the event plus the week prior and the week following (10–26 November 2017), allowing the capture of views driven by the promotion of the event, the event itself, and the immediate post-event period.

5 Results

This section details the results of the evaluation of Flash Flood!, beginning with the informal, anecdotal information garnered from years of exhibiting with different versions of the application (Sect. 5.1). Section 5.2 and 5.3 detail the formal evaluation of Flash Flood! Vol.2 over two events, for the two objectives, creating fun (Sect. 5.2) and creating curiosity (Sect. 5.3). In Sect. 5.4, an analysis of the ancillary developments is provided.

5.1 Anecdotal information

Even without a formal evaluation useful lessons had been learned such as it being obvious that participants enjoyed the activity. Some words were often used in informal conversations to describe their experiences, such “epic” and “sick” (meant positively) and particularly “weird” describing the uncanny experience of immersion in a virtual world that is exciting yet out of the ordinary. Other comments included variations of “it's like Minecraft” that have evolved into “it's like Fortnite”, referencing two popular video games. Flash Flood! has been highlighted in the feedback obtained by events, usually via comment walls. At the NERC Into the blue event in 2016 comments under the section about “Things I loved about Into the blue” included “flash flood”, and under the section about “Things I learned at Into the blue” was “Rivers are fantastic!”. Into the blue also ran a public vote for most popular stand, for which Flash Flood! was awarded joint third place out of 40 exhibits and events.

Not all feedback has been positive, and there have been a few negative comments received during exhibits. Mostly these are to do with issues relating to VR; for example it makes them feel dizzy or nauseous, or simply they did not like it. Other comments have been around dissatisfaction with the graphics of the game or wanting more game-like objectives. On this latter point, “What am I supposed to do?” was a common form of question at the start of demonstrations.

In conversation, it was often commonly asked of participants what they might like to see included in Flash Flood!. Common suggestions included better graphics, being able to explore a wider space, or wildlife such as sheep, wolves, bears, or dinosaurs. Others would like more game-like elements, for example something to shoot, such as zombies (see Curtin, 2017). With Flash Flood! Vol.2, where there were usually more VR stations available to do multiple simultaneous demos, several have commented that they would like to have them linked and be able to explore the scene together with their friends.

Flash Flood! Vol.2 was first used at the two day Hull SciFest 2018 as one activity within a wider “Earth Arcade” space of several activities (see https://seriousgeo.games/eartharcade/, last access: 27 February 2020). The event consisted of shows, workshops, and a Discovery Zone of 45 exhibits, of which the Earth Arcade was one. A total of 3039 members of the public visited the Discovery Zone, but there are no data on how many visited the Earth Arcade. An informal evaluation was conducted for the whole Earth Arcade using a post-it board, with four questions:

1. What did you enjoy?

2. What did you learn?

3. What will you do?

4. What would you like to see?

In total, 69 responses were posted on the board, of which 42 related to Flash Flood! directly, featuring identifying terms like “virtual reality” or referred to the Earth Arcade space as a whole. Of these 35 were posted under the first question, and all were positive. Nine of the responses identified particular features of Flash Flood! that they enjoyed. Only one negative comment was posted, under the fourth question, stating “I liked it mostly apart from the graphics”. The results of this evaluation are potentially biased due to the positive framing of the questions.

5.2 First objective: creating fun

The ability of Flash Flood! Vol.2 to create fun was evaluated using questionnaires at two events in October 2019. The first question asked participants was to “Tell us about your experience of Flash Flood!”, and the results can be seen in Fig. 7. A total of 344 responses were collected over the two events (8 % of the potential audience) with 79.9 % (275 of 344 responses) stating they enjoyed it a lot and a further 15.1 % stating they enjoyed it a little (52 of 344 responses), meaning 95.1 % (327/344 responses) enjoyed it in some form.

Figure 7 Charts showing the questionnaire responses to the question “Tell us about your experience of Flash Flood!?” from Scarborough Science and Engineering Week (8–10 October 2019) and the British Geological Survey Open Day (12 October 2019).

This level of enjoyment only varied slightly, with the participants of the BGS Open Day reporting to have enjoyed it the most of the 4 d (98.3 %, 56 of 57 responses). The second day of SSEW saw the lowest levels of enjoyment (92.9 %, 78 of 84 responses). Over the 3 d of SSEW, the primary-school pupils on Day 3 were more likely to say they enjoyed it a lot (84.8 %, 106 of 125 responses) than the secondary-school pupils (74.5 %, 121 of 162 responses), whilst participants at the BGS Open Day reported similar levels to Day 3 (84.2 %, 48 of 57 responses). Neither the differences between the secondary-school pupils and primary-school children (p=0.09) nor the differences between the audiences at SSEW and the BGS Open Day (p=0.25) were significant.

Those who reported they enjoyed the activity were prompted to volunteer a free-text answer to the question “What did you enjoy about Flash Flood!?”, which received 210 answers. Answers were analysed and binned into categories: general (for example, “I enjoyed everything”), content (for example, “I enjoyed learning about the flood”), technology (for example, “I liked it looked real”), and miscellaneous (answers not falling into the above or that did not make sense). Overall, the technology proved most popular (38.1 %, 80 of 210 responses), followed by general (33.8 %, 71 of 210 responses) and then the content (25.2 %, 53 of 210 responses); however, for the BGS Open Day content proved most popular (45.2 %, 14 of 31 responses), followed by general next (29.0%, 9 of 31 responses) and then technology (25.8 %, 8 of 31 responses).

Eight responses were provided for the question “What did you not enjoy about Flash Flood!?” of which more than half referred to the technology (five of eight responses), such as “bad graphics”, “Made me dizzy”, or “It hurt my eyes”. One response was “Chris” which could either refer to Chris Skinner's voice over or himself as he was acting as a crew member for this event.

5.3 Second objective: creating curiosity

The evaluation of whether Flash Flood! Vol.2 created curiosity was conducted through two questions: “Do you want to know more about flooding than before trying Flash Flood!?” and “Do you want to know more about geomorphology than before trying Flash Flood!?”. A total of 68.0 % (234 of 344 responses) of respondents stated they did wish to learn more about flooding, and 60.1 % (207 of 344 responses) wished to learn more about geomorphology. A breakdown of the data for the events and days is shown in Fig. 8. Between the events, the level of curiosity regarding flooding was similar, with 67.9 % (195 of 287 responses) at SSEW and 68.4 % (39 of 57 responses) at the BGS Open Day wanting to know more, yet regarding geomorphology more participants at the BGS Open Day wanted to know more (64.9 %, 37 of 57 responses) than at SSEW (59.2 %, 170 of 287 responses), but neither were significant (p=0.48 and 0.25). The primary-school pupils were more likely to want to know more about flooding (68.8 %, 86 of 125 responses) than the secondary-school pupils (67.3 %, 109 of 162 responses), and they were more likely to want to know about geomorphology (62.4 % to 56.8 %); these differences were not significant (p=0.41 and 0.21).

Figure 8 Levels of respondents responding yes to questions asking if they would like to know more about the research topics in Flash Flood!. Data are split between Scarborough Science and Engineering Week 2019 (SSEW) and the British Geological Survey Open Day 2019 (BGS Open Day) and further into the 3 d of SSEW.

If participants answered yes to either of the questions, they were then offered opportunity to volunteer a free-text response to “Is there anything in particular you would like to know more about?”. The responses have been binned into the categories – general, content, technology, and miscellaneous as in Sect. 5.2 – with the majority of responses (55.9 %, 52 of 93) falling in miscellaneous with responses like “No” or “Not really”. Overall, 28.0 % (26 of 93 responses) wanted to know more about elements of the content, and 11.8 % (11 of 93 responses) wanted to know more about the elements of the technology. At SSEW, 25.3 % (21 of 83 responses) wanted to know more about the content, and 13.3 % (11 of 83 responses) wanted to know more about the technology, whilst at the BGS Open Day 50 % (5 of 10 responses) wanted to know more about the content, and no one (0 of 10 responses) wanted to know more about the technology.

All participants were offered the opportunity to enter a free-text response to the question “Is there anything else you would like to see in Flash Flood!?”, which got 83 responses, 42.2 % (35 of 83 responses) relating to the technology and 14.5 % (12 of 83 responses) to the content. A common theme was for extra features associated with video games, such as challenges, a larger map, better graphics, or multiplayer modes. At the BGS Open Day more participants wanted to have extra features relating to the content (41.7 %, 5 of 12 responses) than the technology (33.3 %, 4 of 12 responses).

5.4 Ancillary developments

To support the activity at events, ancillary activities were produced, mainly online. These include the SeriousGeoGames website and videos on the SeriousGeoGames YouTube channel. This section analyses the potential of these for assisting in achieving the two objectives. Figure 9 shows the growth in views for the website, YouTube channel, the individual 360 Flash Flood! videos, plus the aggregated views of all Flash Flood! videos (three in total: two 360 videos and a demo for the original version). The YouTube channel has more views than the website but only since February 2019; before this both the website and YouTube channel were on similar levels of views and growing at around 200 views a month.

Figure 9 Cumulative views for SeriousGeoGames online content, including the SeriousGeoGames website and YouTube channel, and cumulative views for the videos related to Flash Flood! on the SeriousGeoGames YouTube channel.

The growth in the aggregated views for all these videos is also shown in Fig. 9. As a share of overall views on the SeriousGeoGames channel, the Flash Flood! videos has gradually been increasing and currently accounts for around 48.3 % of the total views. The Flash Flood! Classroom version has gained in popularity with over 3000 views in 2019 and 3515 in total (as of 24 October 2019). A total of 2940 (83.6 %) have come from YouTube searches, with the top five search terms being “360 flood”, “Flood VR”, “VR Flood”, “360 video flood”, and “flood 360”.

The analytics provided by YouTube Studio provide the opportunity to assess whether exhibiting acts drive people towards the YouTube versions after the event. The NERC UnEarthed Science Showcase took place on 17–19 November 2017 and attracted over 5250 visitors, and one exhibit featured both Flash Flood! VR and Humber in a Box. The Flash Flood! handout was used to support the activity, referring people to the Flash Flood! 360 video. For the 17 d period covering the event plus the week prior and the week following (10–26 November 2017), the video received 88 views (35 direct: straight to URL, YouTube search, or channel page), an increase from 41 (6 direct) during the 17 d period 23 October to 9 November 2017. This reduced down again to 69 views (36 direct) for the 17 d period 27 November to 13 December 2017.

6 Discussion

6.1 Objectives

The SeriousGeoGames activity Flash Flood! has been a success at meeting the first objective – to create a positive experience for the user with scientists and the research topic. Most interactions have been positive, and when users have provided feedback, this has also been overwhelmingly positive. During the two events where formal evaluations were collected, 95.1 % of respondents said that either enjoyed it a little or enjoyed it a lot, with 79.9 % enjoying it a lot.

The success of the second objective – to increase interest for the user in the research topic – was also assessed via questionnaire at two events, and Flash Flood! was shown to be able to meet this objective, with 68.0 % of respondents wanting to know more about flooding and 60.1 % wanting to know more about geomorphology. The level of curiosity generated for geomorphology is lower and likely reflects that it does not feature as prominently within the exhibit; there is a small description on the banner but little mention within the simulation itself (an extra optional response of “I don't know what geomorphology is” might have proven revealing for this question).

6.2 Comparison between school and family audiences

The formal evaluation was conducted at two different events. At SSEW the audience were groups from local schools accompanied by teachers, whilst at the BGS Open Day the audience was self-selecting, having chosen to book a ticket and attend the event; consequently, there were more adults at the BGS Open Day. The audience at the BGS Open Day was more likely to report having enjoyed the activity and were more likely to want to know more about both flooding and geomorphology. When asked what they enjoyed, the BGS Open Day audience was more likely to say something relating to the content over the technology and likewise when asked what they would like to know more about and what they would like adding to the activity. In contrast, at SSEW the majority of responses wanted technology-related features adding to the activity. The nature of the BGS Open Day means that those electing to attend are likely to already have an interest in science (Bultitude, 2014), so the content will more likely be in line with their pre-existing interests. None of the differences between the audiences were statistically significant.

6.3 Comparison between primary- and secondary-school audiences

The SSEW event segregated its audience by having 2 d attended by secondary-school pupils followed by a single day attended by primary-school children. Although differences between the two age groups were observed, none of them were statistically significant. Over all factors, the primary-school pupils were more positive, with a slightly higher overall proportion enjoying the activity but a greater proportion reporting they enjoyed it a lot. Both secondary- and primary-school pupils reported similar levels of wanting to know more about flooding after trying Flash Flood!, although this was slightly higher with primary-school pupils. Primary-school children were more likely to want to know about geomorphology than secondary-school children. Although primary-school pupils do respond more positively to the activity, secondary-school pupils also respond positively in the majority, suggesting the activity is effective for engaging both age ranges.

6.4 Ancillary developments

To support the Flash Flood! activities, there is online information via the SeriousGeoGames website and YouTube channel. During the NERC UnEarthed event of November 2017, a handout was used referring participants to the Flash Flood! 360 video on YouTube, and this did result in an increase in views from 41 for a period before the event to 88 for the period before, during, and following the event. Of the 88 views, 35 were direct, meaning they came from typing in the URL, from YouTube searches, or selecting the video from the SeriousGeoGames YouTube channel, whilst 47 views came from using links, including 15 views on Twitter and 11 on preventionweb.net (https://www.preventionweb.net/english/multimedia/v.php?id=53908&cid=183, last access: 27 February 2020). Even if it is (wrongly) assumed that all 47 of the increased views came from participants at the event, this would represent just 0.009 % of the 5250 attendees suggesting that the exhibit and handouts are not successful in driving traffic to the online content.

The Flash Flood! Classroom version was produced in response to discussions with teachers at events for use in schools and has been supported by articles targeting this use (Skinner, 2018; Skinner and Milan, 2018). This video has seen increased growth in 2019, with over 3000 views where 90.7 % are from YouTube searches. However, only 0.6 % of these searches used the term “flash flood classroom version”, suggesting that the increase in views is a result of the video showing up in search results for more generic searches rather than being used in schools. The majority of views come from the US (38.5 %), with the UK share of audience too small to be shown by YouTube's analytics, suggesting that views are not likely to be a result of the UK-focussed articles.

The results from the ancillary developments are disappointing and do not suggest that they are effective at supporting the exhibition activity of Flash Flood!. There is little evidence of it being used within classrooms too. However, the increase in views for Flash Flood! Classroom via generic search terms indicates that a new audience can be found through optimising the use of search terms and presents an attractive area of future development.

6.5 Limitations

A major limitation of this study was the potential data that were not gathered, such as demographics of the individual participants. These would provide additional granularity to the analysis yet would add complexity to the questionnaires and impede further on participants' time and enjoyment of the events. Another limitation experienced at the BGS Open Day was that some family groups completed the questionnaire form out together, with potentially a single response covering the experiences of several participants. This could be mitigated by including a question about who is completing the form and on whose behalf; groups may not wish to complete forms individually as they would rather spend time interacting with other activities.

6.6 Reflections

A major development between the original Flash Flood! and the Flash Flood! Vol.2 that was used for the formal evaluation was the inclusion of a voice-over track. This helped to engage more participants at one time, as it no longer required a one-to-one interaction with a crew member. It also reduced the resources needed to crew exhibits, as it reduced the level of fatigue within the crew. However, it also limited the conversations between participants and crews, which are where the most positive science engagements occur (Jensen and Buckley, 2014; Wiehe, 2014). For events like SSEW, with large school groups in attendance, where the volume of participants makes such interactions difficult, Flash Flood! Vol.2 seemed particularly suited. At family-oriented events like the BGS Open Day, interactions are more relaxed, and the activity could benefit from follow-on interactions providing additional information on flooding, geomorphology, and how the 3-D scene was constructed (akin to the debrief of Crookall, 2010). In this, Flash Flood! Vol.2 shows potential for use in facilitating more in-depth interactions between the public and scientists at appropriate events.

The next steps for developing SeriousGeoGames activities, including Flash Flood!, would be to broaden the objectives to include learning objectives and/or to drive behavioural changes. For example, an application could teach people about specific elements of flood risk and encourage them to make flood plans or sign up to flood warning services, or an application about plastic pollution could teach people about the hidden sources of plastic and encourage them to use less of these. However, Flash Flood! has been designed for short-term interactions in busy event spaces and would likely need to be adapted and expanded to meet such objectives. The video game elements in Flash Flood! are the least developed and present the area of greatest opportunity going forward. At present it cannot be classified as a game, as it lacks objectives for participants to achieve or challenges to be completed, yet it stills creates fun and curiosity. However, some comments were received stating disappointment that there was little do other than exploring the limited game world and observing the flood. If the narrow objectives of Flash Flood! were expanded to include defined learning objectives, possibly within the workshop or a classroom environment, developing more gaming features would be the obvious way to achieve this.

7 Conclusion

The SeriousGeoGames design model seeks to build activities for festival-like events that allow the public to interact directly with elements of research, such as field observations and numerical models. The activities should look and feel like a video game and be experienced via virtual reality. The objectives are to create fun and curiosity about the subject matter for the participant. Through the Flash Flood! activity, a virtual reality simulation showing geomorphically active flooding from an intense rainfall event based on a real event, the SeriousGeoGames model was shown to be successful, with most participants reporting to have enjoyed the activity and the majority reporting to wanting to know more about the subject matter of flooding and geomorphology. This remains true for several audience types, including groups across all school age ranges and also family audiences. Ancillary developments online offered little support to the exhibition of the activity, with minimal traffic relating to events, but it could offer a new audience for the activities outside of events.