Evaluating participants' experience of extended interaction with cutting-edge physics research through the PRiSE “research in schools” programme

Physics in schools is distinctly different from, and struggles to capture the excitement of, university research-level work. Initiatives where students engage in independent research linked to cutting-edge physics within their school over several months might help mitigate this, potentially facilitating the uptake of science in higher education. However, how such initiatives are best supported remains unclear and understudied. This paper evaluates a provision framework, Physics Research in School Environments (PRiSE), using survey data from participating 14–18-year-old students and their teachers to understand their experience of the programme. The results show that PRiSE appears to provide much more positive experiences than typical university outreach initiatives due to the nature of the opportunities afforded over several months, which schools would not be able to provide without external input. The intensive support offered is deemed necessary, with all elements appearing equally important. Based on additional feedback from independent researchers and engagement professionals, we also suggest the framework could be adopted at other institutions and applied to their own areas of scientific research, something which has already started to occur.

1 Introduction

Research, policy, and practice all agree that participation in science, technology, engineering and mathematics (STEM) needs to be increased and widened (e.g. Campaign for Science and Engineering, 2014), with these issues being particularly acute for the subject of physics (e.g. Murphy and Whitelegg, 2006; IOP, 2014). Physics as a field has become within society strongly aligned with intelligence/cleverness, masculinity and whiteness, all of which can dissuade school students (even those highly enthusiastic about the subject) from pursuing it further and thereby showing inequitable effects on those from under-represented backgrounds (Archer et al., 2020). Some of these issues arise from practices in school-level physics education. Debarring and gatekeeping of physics based on attainment (disproportionately so compared to other subjects) simply feeds the alignment of physics with cleverness and can make even high-attaining students' confidence in the subject precarious. Teachers and the school environment often (even unconsciously) reinforce stereotypes about physics and physicists that are patterned by biases. Curriculum practices in physics often teach oversimplifications at younger ages which are later completely reconceptualised without being presented as refinements to a model, making students perceive the simpler versions as “lies”. Furthermore the general deferment of “interesting” physics in the curriculum produces a disconnect between “school physics” and “real physics”, i.e. the cutting-edge research undertaken by professional physicists, making continued participation in physics education something of a “test of endurance”. These concerns are further reflected in results from national surveys. While 20 % of 16–18-year-old physics school students in the UK aspire towards a physics degree and 80 % aspire towards STEM more broadly (Wellcome Trust, 2017), only 9.7 % and 59.3 % actually go on to study either physics or STEM respectively (McWhinnie, 2012). These constitute odds ratios for aspirations vs. destinations of 2.3 and 2.7, both of which are considerable. All of these issues raised cultivate and contribute to reproducing inequitable, and low overall, patterns of participation within physics (Archer et al., 2020).

Davenport et al. (2020) suggest that for STEM subjects in general an intervention approach that sustains and supports science identity is most appropriate for students in late secondary/high-school education in the context of their educational journey. However, in the case of physics specifically, Archer et al. (2020) comment that existing interventions based on simply enthusing, inspiring and informing students about physics will not significantly change uptake or diversity in post-compulsory physics. While they advocate for widespread changes in science education policy and practice, both at school and university levels, they note that if interventions are also used they need to fundamentally address the problematic processes and practices present within both physics teaching and physics as a field generally.

The stark differences between school, university, and professional science practices have long been noted – while research is one of the main activities of professional scientists, it is quite removed from how science is taught in schools, with some arguing that science education is not “authentic” in this respect (e.g. Hodson, 1998; Braund and Reiss, 2006). Indeed, Yeoman et al. (2017) report that school students are largely unaware of what research actually is, finding a disconnection between “research as information gathering” and the “research question”, and in general have little opportunity to set their own research questions within their school environment. Independent research projects, which provide extended opportunities for students to lead and tackle open-ended scientific investigations (not simply literature reviews or essays, e.g. Conner, 2009; Corlu, 2014), may be one way of exposing students to “real science”. These align with established international pedagogical initiatives such as “inquiry-based science” (e.g. Minner et al., 2010), “problem-based learning” (e.g. Gallagher et al., 1995), and “authentic science” (e.g. Braund and Reiss, 2006). A survey of such projects across 12 countries (Australia, Ireland, Israel, Netherlands, New Zealand, Qatar, Singapore, Spain, Taiwan, Turkey, UK, and USA), however, found them to be rare globally and only sometimes supported by mentors from university/industry (Bennett et al., 2016, 2018). This review found considerable variability in the nature of independent research projects, such as their focus, delivery/provision models, external support, and funding/costs. It was noted that such programmes place demands on time and money beyond standard provisions for all stakeholders, on the skills required by teachers and other adults involved, and on the supporting infrastructure. For successful projects Dunlop et al. (2019) recommend that students should be given the freedom to devise a research question, have ownership over their own data analysis and decision-making, and be given access to experts in their project work. Broadly there are two distinct formats of independent research projects.

• Those associated with dedicated out-of-school events of only a few weeks' duration such as internships/apprenticeships (e.g. Nuffield Research Placements in the UK, Cilauro and Paull, 2019; Raising Interest in Science and Engineering in the USA, Stanford Office of STEM Outreach, 2020), summer schools/camps (e.g. the International Astronomical Youth Camp run across Europe and parts of Africa, Dalgleish and Veitch-Michaelis, 2019), or science competitions/fairs (particularly prevalent in the USA, e.g. Yasar and Baker, 2003).

• Those undertaken within school itself over the course of several months to a year, either in class or supplemented with time in after-school clubs (e.g. an after-school mechatronics project in Taiwan, Hong et al., 2013, class-based biology project in Singapore, Chin and Chia, 2010, or various CarboSchools climate change projects between research institutes and schools across seven European countries, Dijkstra and Goedhart, 2011).

We do not discuss the former here as they are necessarily limited in reach, catering only to heavily bought-in individuals (i.e. typically one to three students; Paull and Xu, 2017) from any given school. Most independent research projects based within schools are not linked to current cutting-edge and novel scientific research topics or questions. However, relatively recently so-called “research in schools” projects have emerged, which do provide students with experiences of genuine contemporary STEM research within their own school environment over several months. While several citizen science projects have also run within schools and aim to help participants learn about current science and to experience the scientific research process, these are typically secondary aims since they primarily concern a single (or small number) of well-defined science questions which will be assisted through developed citizen science protocols (Bonney et al., 2009, 2016; Shah and Martinez, 2016). This contrasts with independent research projects and thus also “research in schools”, where positively affecting the participants is the primary concern and the projects are necessarily open-ended. Nonetheless, the different approaches can have some overlap, and indeed some projects denoted as citizen science, such as the “curriculum-based” projects based in the USA described by Bonney et al. (2016), might perhaps be better framed as “research in schools”. “Research in schools” programmes appear at present to be most prevalent in the UK, and we are aware of three featuring projects in the physical sciences (outside of that at Queen Mary University of London, QMUL, which forms the subject of this paper).

HiSPARC (High School Project on Astrophysics Research with Cosmics) is a scintillator–photomultiplier cosmic ray detector project originating in the Netherlands at Radboud University in 2004, which has subsequently been adopted by other Dutch (Amsterdam, Eindhoven, Groningen, Leiden, Nikhef, Twente, Utrecht) as well as UK (Bath, Bristol, Birmingham, Sussex) and Danish (Aarhus) universities (Colle et al., 2007; van Dam et al., 2020; HiSPARC, 2018). Many of these universities operate a tiered membership scheme for schools: “Gold” enables schools to buy their own detector (GBP 5,500 in the UK); “Silver” is a detector rental scheme (GBP 300 p.a. plus an installation fee) with the contract specifying if they do not participate the detector will be collected with an additional fee; and “Bronze” membership (GBP 200 p.a.) gives schools access to HiSPARC data but not their own detector. Schools signing up for “Silver” or “Bronze” membership are contractually obliged to generate funding to upgrade to “Gold”. While the “Gold” membership fee covers the costs of the detector and installation, the other memberships are to ensure that schools make a commitment to working with the university (Jaap Velthuis and Maria Pavlidou, personal communication, 2016; National HE STEM SW, 2012). It is not possible to compare how these schools go about project work and how much support they are given by participating universities, which may vary by institution, as at the time of writing HiSPARC has not published any reviews of their processes or evaluation.

IRIS (Institute for Research in Schools) is a UK charity formally launched in March 2016 (IRIS, 2020), building on the previous CERN@School project conceived in 2007 (Whyntie et al., 2016; Parker et al., 2019). While IRIS' projects cover all the sciences, current physics projects include the aforementioned CERN@School, Higgs Hunters (Barr et al., 2018), LUCID (Furnell et al., 2019; Hatfield et al., 2019), and Webb Cosmic Mining (in preparation for the James Webb Space Telescope). They have rapidly expanded across the UK since formation, having worked in some capacity with over 230 schools as of 2020. Publications have provided technical details of their projects and case studies of some students' successes within them, including a few examples of resulting peer-reviewed scientific work; however, the exact provision/delivery model implemented and precisely how project work is supported are not fully explored in the available literature. IRIS aims to develop “teacher scientists”, teachers that identify as both science teachers and research-active scientists (Rushton and Reiss, 2019), which suggests a teacher-driven model. While some researchers/academics have designed or consulted on some IRIS projects, they appear in general to have little involvement in supporting students or teachers (Oliver Moore, personal communication, 2020), with IRIS itself being the main point of contact for schools. With a recent change of staff at IRIS in late 2019 has come a reformulation of how they classify their projects. “Seed” projects are for new schools, are the most straightforward, and receive the most support from IRIS, though it is not clear what form that takes. “Sprout” projects are more advanced, seeing students carrying out more complex activity to assist scientists with their research questions, though how this collaboration operates is not specified. “Grow” projects are where students have proposed their own research questions, either independently or using IRIS resources, with IRIS merely providing advice in producing posters, talks, or papers as well as opportunities to present.

ORBYTS (Original Research By Young Twinkle Scientists), based at University College London, was piloted from January 2016 and is nominally based around the Twinkle mission but has expanded into other research areas since (Sousa-Silva et al., 2018; ORBYTS, 2019). A select group of students (with an imposed limit of four to six) from each school undertakes fortnightly meetings with early career researchers (either PhD students or post-docs) throughout their project aiming to achieve, where possible, publishable scientific results (McKemmish et al., 2017; Chubb et al., 2018; Holdship et al., 2019). Teachers, while present, are not typically actively involved in these sessions, and students tend to do little independent work outside of the sessions (William Dunn, personal communication, 2018). The content of the projects changes each year to align with the researchers' current focus, with them typically working with only one school per year each. PhD researchers are paid for their (preparation, travel, and session) time with funds from independent schools, who pay not only for their school, but also for enabling an additional school from a lower socio-economic background to take part.

It is clear that there is currently a lack of published details and evaluation on provision within the emerging area of “research in schools”. This paper therefore explores these aspects applied specifically to the “research in schools” programme of QMUL's School of Physics and Astronomy. This was piloted between 2014 and 2016, as detailed in Archer (2017), and is now known as Physics Research in School Environments (PRiSE, 2020). Section 2 introduces PRiSE's framework, which is then evaluated in Sect. 4 in terms of participating students' and teachers' experience. We also briefly discuss how the PRiSE approach has been received by the university sector in Sect. 5.

2 PRiSE framework

Physics Research in School Environments (PRiSE) is a collection of physics-based “research in schools” projects (see Table 1) brought together under a coherent provision framework. The programme aims to equip 14–18-year-old school students (particularly those from disadvantaged backgrounds) with the ability, confidence, and skills to increase/sustain their aspirations towards physics or more broadly STEM, ultimately enabling them to realise these at higher education and thus contributing to increased uptake and diversity of physics and to some extent STEM (cf. Archer et al., 2020). Through working with teachers, PRiSE also aims to develop their professional practice and build long-term university–school relationships that raise the profile of science and mitigate biases/stereotypes associated with physics within these schools, generally making them environments which nurture and enhance all students' science capital (cf. IOP, 2014). Our rationale for these particular outcomes is left to the Supplement. This section summarises the PRiSE framework, discussing the ethos of the programme and the roles played by the schools and university as well as outlining the various activity stages, interventions, and resources which it consists of. In-depth practical details aimed at practitioners looking to replicate the framework are given in the Supplement.

Table 1 A summary of the existing PRiSE projects at QMUL.

2.1 Approach

PRiSE takes the “research in schools” approach to schools engagement, whereby students are given the opportunity to lead and tackle open-ended scientific investigations in areas of current research. Therefore, the PRiSE projects were developed to transform current scientific research methods, making them accessible and pertinent to school students so that they could experience, explore, and undertake scientific research themselves.

2.1.1 Ethical considerations

Since the programme intends to influence school students and teachers, a number of ethical considerations have been taken into account, following the BERA (2018) guidance for educational research, with regard to safeguarding and to ensure that no harm results. Firstly, to ensure equality of access to the programme, we do not charge schools to be involved (cf. Harrison and Shallcross, 2010; Jardine-Wright, 2012) and try to provide them with all the physical resources they need for their project, thereby removing potential barriers to entry for less resourced schools. Our targeting takes into account several school-level metrics (type of school, students on free school meals, indices of multiple deprivation, gender balance, etc.) to ensure diversity. We aim for the programme to be equitable with all schools being offered the same interventions/opportunities, taking into account and being flexible to their specific needs where necessary. We work with as many schools as we have capacity to do so each year and do not withhold interventions from any students for the purpose of having control groups. The projects are optional and presented as an opportunity that students can take advantage of which will be supported by their teacher and the university; therefore, students are not pressured into being involved. Students and schools can drop out at any point within the programme with no penalty, which does occur in a minority of schools.

2.1.2 Role of teachers

The involvement of teachers at all stages is of paramount importance in terms of delivery and safeguarding. They have helped shape the design of the programme, inform how we update it each year, and serve as our liaison to schools and the students involved. It is the teachers that decide who projects are offered to within their school, with us simply advising that the projects should be suitable for all A-Level (16–18-year-old) students as well as high-ability GCSE (14–16-year-old) students (further contextual information on the UK education system is given in Appendix A). These recommendations were made based on the basic background knowledge required to meaningfully engage with the research. Invariably teachers choose to involve older age groups, with 79±1 % of PRiSE students being aged 16–18 (and so far only one student below our recommended ages has been involved, being 13–14). Unfortunately, we do not have any specific information on exactly how teachers go about selecting students. However, the average number of students per school each year is around 12, which compared to the national average class size in A-Level physics of 16 (RAE et al., 2015) indicates teachers involve a significant majority (or in many cases the entirety) of their cohorts in PRiSE. We allow teachers to determine how best to integrate the projects within their school though provide advice on this. We also aim, through our resources and communications, to equip teachers to manage the day-to-day aspects of the projects without overly burdening them – their role is chiefly one of encouraging their students to persist, providing what advice they can, and then communicating with the university.

Teachers' involvement at all stages also presents opportunities to them for continuing professional development, helping them nurture and cultivate STEM aspirations among students throughout their school (i.e. not just PRiSE students). This is implemented informally and integrated within the programme in the form of both bespoke resources and ongoing dialogues between teachers and researchers. These are aimed at enhancing teachers' knowledge about the underlying science and how they link to curriculum-based topics where appropriate, their skills and confidence surrounding current research topics and methods, and their pedagogy in mentoring independent project work.

2.1.3 Role of the university

It was recognised that teachers in general likely will not have the skills or experience in research to manage projects without expert assistance (Shah and Martinez, 2016; Bennett et al., 2016, 2018). Therefore, PRiSE was designed to be supported by active researchers equipped with the necessary expertise to draw upon in offering bespoke, tailored guidance to the students and teachers. Well-defined roles within the university have been established for each of the PRiSE projects to provide this support.

• Outreach Officer. Manages the entire programme, including university–school relationships, communications, intervention/event co-ordination, programme finances, and evaluation.

• Project Lead. Visible figurehead for the project to schools, typically an academic member of staff.

• Researcher. Providing advice and guidance to students and teachers throughout the programme. Can be delegated to or shared with early career researchers.

Since the primary focus of PRiSE is (unlike typical citizen science) on the participants rather than the research, researchers should not consider students contributing to novel research to be their rationale for being involved. Our position is that it is rather unreasonable to expect investigations that are motivated by school students themselves (an established element of good practice in independent research projects, e.g. Dunlop et al., 2019) to be able to make meaningful contributions to the physics research as a matter of course. We note that in some exceptional cases PRiSE students' work has arrived at promising preliminary results, though these have required significant follow-up work by professional researchers to transform the results into publishable research (e.g. Archer et al., 2018) and thus should not be considered the archetype. Instead, researchers are enticed by the possibility of societal impact underpinned by their research. This is something which is increasingly called upon from funders (e.g. National Coordinating Centre for Public Engagement, 2020) and is notoriously difficult for areas of “blue skies” research such as physics. Furthermore, significant contribution to a coordinated departmental outreach programme can be used as criteria for academic promotions (cf. Hillier et al., 2019). Physics researchers though are largely unmotivated in delivering curriculum content as part of their engagement work, valuing instead aspects relating to their research and role as a researcher (Thorley, 2016). PRiSE thus also aligns with this direction. Ultimately, researcher buy-in is vital to the delivery of protracted research-based engagement programmes such as PRiSE.

It is clear from Table 1 that the topics and activities of current PRiSE projects vary considerably. This suggests that a wide range of fields and project ideas might be able to adopt the PRiSE framework. How projects have been developed has also varied (further explored in the Supplement), though we have adopted a pragmatic approach in taking advantage of opportunities (grant funding, internships, etc.) and adapting existing materials where possible, since creating a project from scratch is a significant undertaking far beyond what most academics (unfortunately) have the capacity to do (cf. Thorley, 2016).

2.2 Structure

PRiSE runs from the start of the UK academic year to just before the spring/Easter break, which teachers had informed us during the pilot stage is manageable and largely fits around exams/other activities for most (but not necessarily all) schools (Archer, 2017). The structure has evolved naturally from the pilot to that shown in Fig. 1.

Figure 1 Graphic summarising the PRiSE framework including a timeline of the different project activity stages (rectangles), interventions and stakeholders' roles within them (rounded rectangles), and resources provided (document shapes). Arrows indicate over what dates interventions (identified by colour) typically occur.

2.2.1 Activity stages

Students work in research groups of typically five people and they are advised to try and work on the project on average for 1–2 h a week. The bulk of this is done outside of regular physics lessons, though some schools integrate the projects within their timetabled “science clubs” or required extra-curricular blocks, whereas other teachers arrange a regular slot for students to work on the projects or leave it up to the students to arrange (though this latter approach often proves unsuccessful). PRiSE projects are split up into three activity stages (see Fig. 1).

• Prescribed work. Given that independent research in STEM is probably unfamiliar to the students, rather than expecting them to be able to come up with their own avenues of investigation in an unfamiliar research topic straight away, we instead give them an initial prescribed stage of research.

• Independent project. Groups are encouraged to set their own research questions and undertake different projects in the topic area when ready. This enables every group to explore something different so that students gain a sense of independence and ownership of their work.

• Writing up. Near the end of the project students produce either a scientific poster or talk to be presented at an annual conference.

2.2.2 Interventions

Several different interventions form the structure and support behind the activity stages.

• Assignment. The opportunity is advertised to schools via existing teacher networks and teachers apply to participate in the following academic year. Schools are assigned a project before the summer break.

• Kick-off. Typically on-campus event featuring an introductory science talk, outline of how the project will work, and a hands-on workshop.

• Visit. Researchers visit schools to mentor students (and their teachers) on their project work in a student-driven intervention.

• Webinars. Drop-in online sessions similar to school visits but allowing students and teachers to gain further support.

• Ad hoc. Further asynchronous support via email as required.

• Comments. Students are offered the opportunity to receive comments on their draft slides or posters.

• Conference. Students present the results of their projects as oral or poster presentations at a special conference at the university, attended by researchers as well as the students' teachers, peers, and family.

Stakeholders' roles within these interventions are given in Fig. 1, with photos depicting some of them displayed in Fig. 2. All stages of the programme and the processes involved are communicated to teachers via email to pass on to their students.

Figure 2 Photos of various stages of the PRiSE framework: students participating in an on-campus kick-off workshop (a), students interacting during the poster session at a conference (b), a group of students display their prizes won at a conference along with their teacher (c), and a group presents a talk at a conference (d).

2.2.3 Resources

To enable the students to take part in PRiSE, the students and teachers are also provided with numerous resources. While some projects require specific equipment, data and software, here we discuss more common types of resources across the different projects as shown in Fig. 1.

• Project poster/flyer. Given to teachers to help advertise projects in their school.

• Project guide. Each project has a student guide covering an introduction to the research field, background physics/theory, an explanation of the equipment/data, discussion of analysis techniques, details of the initial prescribed activity, suggested research questions/methods for independent research, and links to other sources of information. Teachers are provided with the same guide but with extra guidance.

• Project webpage. These showcase anonymised examples of good-quality talks/posters that previous students have produced and provide any links or videos relevant to the project.

• “How to” guide. General articles applicable to most research projects, such as producing scientific talks and posters.

2.3 Scalability

PRiSE's framework attempts to find a balance between the (necessarily competing) reach and significance of the interactions. For example, an academic staff member acting as both project lead and researcher within PRiSE can support four schools' participation (∼50 students), taking around 8 h over the course of 6 months (cf. Fig. 1). Using PhD students or post-docs in the researcher role(s) makes even more efficient use of time. In contrast, under the ORBYTS model each early career researcher can support only one school (four to six students) with 10 h of their time. As noted in the introduction, mentorship from active researchers throughout does not appear prevalent in other “research in schools” programmes at present.

Programmes of repeat interventions with schools will necessarily have a smaller reach than various one-off events. However, PRiSE's efficiency in researcher time is reflected in its reach as shown in Table 2, demonstrating the model now serves around 30 schools per year having reached a total of 67 schools and over 1300 students with the direct involvement of 88 teachers as of 2020. We note that a minority of schools do not complete the full programme and others do not return for subsequent years; however, we do not provide a full analysis of the types of schools involved and their retention within the programme in this paper. Comparing the number of schools to other physical science “research in schools” projects/programmes, University of Oxford researchers have interacted directly with only 14 students from five schools through their Higgs Hunters IRIS project which commenced in 2016 (Oliver Moore, personal communication, 2020, though we note information regarding other IRIS projects is not available), ORBYTS (2019) reports collaborating with 17 schools since 2016, and HiSPARC (2018) discloses 140 schools since 2004 across their network of 13 Dutch, Danish, and UK universities (van Dam et al., 2020), i.e. around 11 per institution. Therefore, the reach through the PRiSE framework by a single university department is considerable compared to the rest of the sector.

Table 2 Number of schools, students, and teachers involved in PRiSE by academic year as the programme has grown.

3 Methods

To determine the perceived value of PRiSE's approach with its key stakeholders, namely participating students and teachers as well as those across the wider university sector, we have maintained regular collection of evaluative data (cf. Rogers, 2014, and references therein) via various surveys which we detail here. These data underpin our understanding of PRiSE and have been collected securely to protect all participants, in compliance with GDPR and in line with the BERA (2018) guidelines for educational research.

3.1 Instruments and participants

We gathered feedback from participating students and teachers via paper questionnaires handed out at our student conferences each year. The only exception to this was in 2020, where online forms were used due to the COVID-19 pandemic causing that year's conference to be postponed. The questionnaire method was chosen so as to gather data from as wide a range of students and teachers as possible, respecting the limited time/resources of all involved (both on the school and university sides). For ethics considerations all feedback was anonymous, with students and teachers only indicating their school (pseudonyms are used here to protect anonymity) and which project they were involved with. Students were not asked to provide details of any protected characteristics (such as gender or race) or sensitive information (such as socio-economic background). Both students and teachers were informed via an ethics statement on the form that the information was being collected for the purpose of evaluating and improving the programme and that they could leave any question they felt uncomfortable answering blank (this functionality was also implemented on the online form for consistency).

The open and closed questions concerning participants' experience of the programme, which varied slightly year to year, are given in Appendix B. While we attempted to collect responses from all participants in attendance, invariably only a fraction did so, yielding results from 153 students and 45 teachers across 37 schools. A breakdown of the number of respondents and their schools per year is given in Table 3, where the numbers of participants and schools in attendance at our conferences are also indicated. We do not have reliable information on how many students, teachers, and schools would have successfully completed the programme in 2020 due to the COVID-19 disruption. Students and teachers did not always answer all of the questions asked, and hence we indicate the number of responses for each question considered throughout. There is no indication that the respondents differed in any substantive way from the wider cohorts participating in the programme. While ideally one would also gather feedback from schools which dropped out during the year, a similar formal feedback process has not been viable bar in a few cases where only the teachers responded.

Table 3 Response rates to questionnaires at PRiSE student conferences.

Feedback from the university sector came from a session at the 2019 Interact symposium that presented the challenges to STEM outreach practice highlighted by recent educational research and the need for deeper programmes of engagement with young people and then summarised the PRiSE approach as one possible example (Archer, 2019). Throughout this workshop an anonymous interactive online survey was used for interactivity and to collect data presented in this paper. The survey included both closed and open questions as listed in Appendix C. Attendees were fairly evenly split between UK university researchers and engagement professionals (gauged in-person by attendees raising their hands when asked), with 19 people participating in the survey and only 7 not doing so. Participants were allocated a unique number by the online survey itself, which did not distinguish between researchers and engagement professionals.

3.2 Analysis

Both qualitative and quantitative approaches were utilised in data analysis, as the open- and closed-ended questions present in the questionnaires produced different types of data.

For all quantitative data, standard (i.e. 68 %) confidence intervals are presented throughout. For proportions/probabilities these are determined through the Clopper and Pearson (1934) method, a conservative estimate based on the exact expression for the binomial distribution, and therefore represent the expected variance due to counting statistics only. Several statistical hypothesis tests are used with effect sizes and two-tailed p values being quoted, with the required significance level being α=0.05. In general we opt to use nonparametric tests as these are more conservative and suffer from fewer assumptions (e.g. normality, interval scaling) than their parametric equivalents such as t tests (Hollander and Wolfe, 1999; Gibbons and Chakraborti, 2011). The Wilcoxon signed-rank test is used to compare single samples to a hypothetical value, testing whether differences in the data are symmetric about zero in rank. When comparing unpaired samples a Wilcoxon rank-sum test is used, which tests whether one sample is stochastically greater than the other (often interpreted as a difference in medians). Finally, for proportions we use a binomial test, an exact test based on the binomial distribution of whether a sample proportion is different from a hypothesised value (Howell, 2007). For ease of reference, further details about the quantitative analyses are incorporated into the relevant sections of the findings.

Qualitative data were analysed using thematic analysis (Braun and Clarke, 2006). Instead of using a priori codes, the themes were allowed to emerge naturally from the data using a grounded theory approach (Robson, 2011; Silverman, 2010) as follows.

1. Familiarisation. Responses are read and initial thoughts noted.

2. Induction. Initial codes are generated based on review of the data.

3. Thematic review. Codes are used to generate themes and identify associated data.

4. Application. Codes are reviewed through application to the full data set.

5. Analysis. Thematic overview of the data is confirmed, with examples chosen from the data to highlight the themes.

4 Feedback from participants

In this section we use the feedback from participating students and teachers to evaluate the provision offered within the PRiSE framework, specifically assessing their experience and the level of support offered.

4.1 Experience

Firstly from 2016 onwards we asked both students (n=150) and teachers (n=42) “Have you been happy with the research project overall?”, giving options on a five-point Likert scale, which we coded to the values 1–5. This scale and the results are displayed in Fig. 3, revealing that 91±3 % of students and 95±5 % of teachers rated their experience as positive (scores of 4–5), with only three students giving a negative reaction (scores of 2). Teachers tended to rank this question somewhat higher (their mean score was 4.50±0.09, where uncertainties refer to the standard error in the mean) than students (mean of 4.17±0.05), with p=0.002 in a Wilcoxon rank-sum test. The PHwP project scored slightly higher (average of 4.59±0.11, $p=\mathrm{8}\times {\mathrm{10}}^{-\mathrm{4}}$) than the overall results with students, whereas ATLAS scored slightly lower with both students (3.92±0.09, p=0.012) and teachers (3.80±0.20, p=0.017) than their respective means. No obvious trends were present by school.

While suggestive of extremely positive experiences with PRiSE, one also needs to compare these distributions against the typical responses of students and teachers for schools' STEM engagement programmes. We use the results of Vennix et al. (2017) as such a benchmark, which surveyed 729 high-school students and 35 teachers about 12 different STEM outreach activities in the USA and Netherlands. This comparison reveals that PRiSE seems to be perceived considerably more positively than usual by both students (benchmark average 3.66±0.01, $p=\mathrm{1}\times {\mathrm{10}}^{-\mathrm{15}}$ in a one-sample Wilcoxon signed-rank test) and teachers (benchmark average 3.84±0.08, $p=\mathrm{1}\times {\mathrm{10}}^{-\mathrm{7}}$).

Figure 3 Distribution of students' (blue) and teachers' (green) overall happiness with their PRiSE projects. Error bars denote standard (1σ) Clopper and Pearson (1934) intervals.

Secondly, students (n=135) were asked for adjectives describing their experience of the projects overall. They were free to use any words they wanted and were not given a pre-selected list. Teachers (n=38) were similarly asked to indicate observations of their students' experience also. Since 2016 this has resulted in 88 unique adjectives, with both students and teachers typically writing two to three words each. We present the results as the word cloud in Fig. 4, where students and teachers have been given equal prominence by normalising their counts by their respective totals. We have indicated by colour from which group(s) the words originated, generally showing a lot of agreement between students' thoughts and teachers' observations. The most cited adjectives were (in descending order) interesting, challenging, exciting, inspiring, and fun, similarly to those from the pilot (Archer, 2017), with the top two adjectives being significantly greater than the subsequent ones. While in the pilot stage only positive adjectives were expressed, since then a few negative experiences have been conveyed, such as time-consuming, frustrating, and stressful. These constitute a small minority of experiences though (6±1 %), and in most cases the same students also listed positive adjectives, apart from only four individuals.

Figure 4 Word cloud of students' experiences. Colours indicate words identified by students (blue), teachers (green), or both (cyan). Students and teachers have been given equal total weight.

Following on from these quantitative analyses, we qualitatively explore the potential reasons behind the results. The most common themes that emerged from students' (n=110) responses to open questions about their experience were that they feel they learnt a lot (62 responses).

We have learnt so many new things relating to the magnetosphere and waves and we have developed new skills. (Student 3, Xavier's Institute for Higher Learning, MUSICS 2016)

I learned so much! I would recommend it to all the younger kids at my school! (Student 111, St Trinians, SCREAM 2019)

I definitely learnt many new and interesting things and it helped me to develop my understanding of particle physics while aiding my A-Level knowledge. (Student 140, Jedi Academy, ATLAS 2020)

They found the projects' content and methods interesting (47 responses).

It brought my interests in programming, physics, maths and space together. (Student 100, Octavian Country Day School, PHwP 2019)

It has been very interesting to work with actual data and plan our own research project. (Student 116, St Trinians, SCREAM 2019)

It was very interesting to learn more in regards to astrophysics and the MUSICS project was a very safe space to do so. We got lots of support and it was fun. (Student 119, Pokémon Technical Institute, MUSICS 2020)

Coming into the Planet Hunting With Python project, my interests were mainly focused on the physics side of understanding brightness-curves and finding equations to solve for planetary parameters. However, in this project, my eyes were opened to the many uses of coding to analyse data, and it was a wonderful experience to learn about such an interesting area through a combination of theory and practical coding. (Student 120, Octavian Country Day School, PHwP 2020)

They enjoyed the style of working in and with research which differs from their regular school experience (33 responses).

It was quite interesting to broaden our views and experience high level education. (Student 90, Hill Valley High School, ATLAS 2019)

I enjoyed the opportunity to do science instead of just learning it. (Student 101, Octavian Country Day School, PHwP 2019)

It was very nice to work with friends and work together to produce something. (Student 106, Boston Bay College, PHwP 2019)

It was very fun to do our own research and I appreciated that help was always available even though it is very independent. It also shows how challenging research can actually be but also how rewarding it is once you start making progress. (Student 143, Sunnydale High School, MUSICS 2020)

Neutral or negative experiences tended to be due to students finding the projects' content or open-ended way of working difficult or confusing (seven responses).

I did not understand most of the project or what I was supposed to do. (Student 147, Jedi Academy, ATLAS 2020)

The vast majority of students seemed to ultimately enjoy this challenge though.

The project gave us a lot of freedom and challenged us to think in different ways. (Student 122, Jedi Academy, ATLAS 2020)

[It] made me more willing to take a go at challenges and what I deem hard. (Student 130, Bending State College, PHwP 2020)

Teachers agreed that the learning curve involved with the projects was advantageous for students.

When the students got the hang of it they really enjoyed it. (Teacher 16, Tree Hill High School, MUSICS 2018)

The students found it hard to identify what to do a project on and would have liked guidance on that, but I felt this was a good experience. They have developed grappling with open ended and difficult material. (Teacher 23, Smeltings, ATLAS 2019)

Teachers' feedback on their experience (n=34) tended to praise how the projects allow their students to access and explore beyond the curriculum (29 responses).

A great framework to explore physics beyond the syllabus but still accessible. (Teacher 26, Octavian Country Day School, PHwP 2019)

Excellent project that is open-ended allowing students to take it where they want. (Teacher 27, Hogwarts, SCREAM 2019)

This year I had an extremely motivated, enthusiastic and well-organised group of 7 students who fully immersed themselves into the project and quickly took it in a direction outside my own understanding of this area of science. This is exactly the experience I wanted them to have, and they were able to discover some genuinely novel processes that had not been observed before – the hallmark of great scientific research! (Teacher 44, Sunnydale High School, MUSICS 2020)

Therefore, both quantitative and qualitative data suggest students and teachers had much more positive and rewarding experiences participating in PRiSE projects than is typical for schools engagement programmes from universities in general.

4.2 Support and resources

We originally asked students whether they felt they had received adequate support, finding overall positive results on a five-point Likert scale (Archer, 2017). However, students' qualitative responses explaining their answers often revealed a conflation of the support provided by Queen Mary with that offered by their teacher. Therefore, from 2019 onwards we explicitly separated these two aspects. Students (n=68) were asked “Do you feel that support from your teacher was provided/available during the project?”, which yielded the following results: Strongly Agree (30), Agree (34), Neither Agree or Disagree (3), and Disagree (1). The average response is 4.37±0.08, which is considerably greater than the benchmark on teacher support reported by Vennix et al. (2017) of 3.60±0.03 ($p=\mathrm{4}\times {\mathrm{10}}^{-\mathrm{10}}$). Students' comments explaining their ratings (n=56) revealed that teachers provided them with advice, encouragement, and enthusiasm (49 responses).

My teacher has been very supportive and has helped us when we didn't understand something as well as encouraging us to taking a more innovative approach. (Student 124, Quirm College for Young Ladies, MUSICS 2020)

If we had a question, teachers were probably not useful. But if we did not know what to do or we were stuck, here teachers were really useful and that was what we needed. (Student 145, Sunnydale High School, MUSICS 2020)

They arranged regular sessions for students to meet and visits or calls from the university when required (seven responses).

Our teacher arranged a Skype call with a professor from QMUL when we needed to ask questions about how certain parts of the data were calculated. (Student 126, Harbor School, ATLAS 2020)

[Our] teacher would often ask us about it and hold meetings to catch up with us on our progress. (Student 143, Sunnydale High School, MUSICS 2020)

[Our] teacher answered some of our questions and organised a weekly meet-up where students could ask each other questions and work together. (Student 148, Jedi Academy, ATLAS 2020)

Neutral or negative responses tended to be explained by their teacher lacking specific knowledge about the research in response to students' queries (two responses).

Although they were always ready to help, sometimes they didn't know the answers to our questions. (Student 122, Jedi Academy, ATLAS 2020)

[They] didn't understand the content of the project. (Student 139, Jedi Academy, ATLAS 2020)

This is something we do not expect of teachers (cf. Shah and Martinez, 2016; Bennett et al., 2016, 2018), hence why support from the university is also offered.

Teachers' (n=18) responses on a yes/no scale (chosen due to expected small number statistics) on whether they felt able to support their students were also highly positive, with only two negative responses, a significant majority (p=0.001 in a two-tailed binomial test). Bear in mind, however, that these responses were in light of the support provided from the university, something which a few teachers referenced in explaining their answers.

My own experience with research was handy but I felt that without this the students would still have been supported (Teacher 2, Hogwarts, SCREAM 2016)

only through Martin's support (Teacher 4, Sweet Valley High School, MUSICS 2016)

the teacher version of the handout was useful, but otherwise I could only give generic advice (Teacher 16, Tree Hill High School, MUSICS 2018)

Several teachers raised pressure on their time, with one teacher using this to justify their negative response.

sheer time pressure – big limiting factor (Teacher 13, St Trinians, SCREAM 2017)

Another said this somewhat limited their ability to support students.

not as much as I would have liked (lack of available time) (Teacher 3, Xavier's Institute for Higher Learning, MUSICS 2016)

However, most said it was manageable.

The autonomous group work, with very little input from me, was great to see (Teacher 1, Hogwarts, SCREAM 2015)

Students were quite self-sufficient so if I made suggestions they were able to do the leg work (Teacher 15, Spence Academy for Young Ladies, MUSICS 2018)

Teachers' ability and confidence in supporting the projects were other themes that emerged. Even with the teacher-specific resources provided, some felt they did not have the specific knowledge or skills to support the projects.

[Unable to support due to a] lack of knowledge of Python (Teacher 19, Boston Bay College, PHwP 2018)

Other teachers reflected that, similarly to their students, they too had experienced a learning curve through their involvement.

[I found it] difficult at first (Teacher 8, Coal Hill School, MUSICS 2017)

They ultimately became more determined and confident with time and in subsequent years.

First time we've done this – I will do better next time (Teacher 17, Sunnydale High School, MUSICS 2018)

Second year that I ran it I feel more confident (Teacher 21, Hogwarts, SCREAM 2018)

The final theme raised was that for successful participation teachers believed the students needed the external motivation to come from the university rather than having project delivery being solely teacher-driven.

Dr Archer was a great external lead to have. If I had been pushing them myself they would have taken it less seriously (Teacher 17, Sunnydale High School, MUSICS 2018)

Therefore, the comments from both students and teachers indicate that teachers alone would likely not have been able to successfully support these research projects in their schools without both the resources and external motivation/mentoring provided as part of the PRiSE framework.

We now consider the specific elements of support shown in Fig. 1. From 2019 onwards we investigated participants' thoughts on each of these various aspects offered. Students (n=68) and teachers (n=23) were asked to rate the usefulness of these as either “unimportant”, “helpful”, “essential”, or “unsure”. This was chosen over a five-point Likert scale due to an expected low number of responses, particularly from teachers. Any unsure or blank responses are neglected, yielding 326 (out of a potential 408) student and 156 (out of 161) teacher responses. We divide these responses into negatives (“unimportant”) and positives (“helpful” or “essential”), though we acknowledge some may consider the “helpful” response to be neutral, and thus our analysis takes both interpretations into account. The results are displayed in Fig. 5 for the individual elements as well as overall results obtained from totalling all responses. Both students and teachers overall rated the elements positively – coding the responses to values of 1 (negative) to 3 (essential), the overall means were 2.62±0.04 for teachers and 2.23±0.03 for students. The majority of teachers tended to give “essential” ratings to most aspects, and while these majorities are not statistically significant in a two-tailed binomial test, the average value for each element was greater than 2 to high confidence (p<0.002 in one-sample Wilcoxon signed-rank tests). Students, on the other hand, mostly rated each element as “helpful” as well as stating slightly more negative responses than teachers, though again all elements' mean scores (apart from the kick-off workshop at 2.15±0.08) were significantly greater than 2 (p<0.023). While there are some variations in scoring amongst the different support elements, such as students and teachers respectively rating researcher visits and communications (the latter of which includes webinars and ad hoc emails) as the most essential, these differences to each group's overall results are slight and not statistically significant. One interpretation of this might be that most respondents answered unreflectively, ticking the same boxes for each item. However, no students and only three teachers gave the same answer in every category. This therefore suggests that all of the elements of support provided as part of PRiSE are almost equally important and necessary. This has been further elaborated on in teacher feedback.

It is very well set up and open ended and the support received is magnificent. (Teacher 33, Boston Bay College, PHwP 2020)

Martin's guide to help the students was a very good balance of useful guidance and allowing them to find their own way through. (Teacher 38, Royal Dominion College, MUSICS 2020)

Truly excellent support from the Queen Mary team! They have visited us multiple times and have been so generous with their time. Students have learnt a great deal from them! (Teacher 44, Smallville High School, PHwP 2020)

Figure 5 Usefulness of support provided to teachers (T, n=23) and students (S, n=68). Results are divided (black lines and associated error bars) into negative (red) and positive responses, with the latter subdivided (grey lines and error bars) into “essential” (blue) and “helpful” (yellow) elements. Error bars denote standard (1σ) Clopper and Pearson (1934) intervals.

5 Feedback from the university sector

Based on the highly positive results from participants, we think there is potential for the PRiSE framework to spread beyond QMUL and be applied to other institutions' own areas of physics (and perhaps even STEM more generally) research. We therefore wanted to assess how it is perceived by those from the university sector with interests and/or expertise in schools engagement. Feedback from our partner organisations seemed promising, for example with the South East Physics Network including PRiSE in their public engagement strategy (SEPnet, 2017), several of their member institutions expressing interest in adopting it, and the University of Surrey having already developed a pilot project. However, for a slightly more balanced perspective we collected evaluative data via an anonymous interactive online survey (see Appendix C) from independent researchers and public engagement professionals during a session at Interact 2019 (Archer, 2019), a day-long UK symposium concerning public engagement in the physical sciences. This workshop contained an overview of recent educational research and then introduced the PRiSE framework to attendees.

The respondents, while heavily bought into schools engagement, tended to only undertake one-off activities (as detailed in Appendix C). After presenting the PRiSE framework to them, when asked on a five-point Likert scale whether they (n=19) would now consider deeper approaches to outreach/engagement with schools, the results were Strongly Agree (5), Agree (9), Neither Agree or Disagree (5), and no responses in the two negative options. Coding these to a 1–5 scale yields a mean of 4.0±0.2, i.e. greater than neutral ($p=\mathrm{1}\times {\mathrm{10}}^{-\mathrm{4}}$ in a one-sample Wilcoxon signed-rank test).

In an open-ended question, participants were also asked to identify the main thing they had taken away from the session. This yielded 17 responses. Through coding these answers three main themes emerged, with all responses covering only one theme each. The first theme is about what types of schools engagement are possible (eight responses), with almost all thinking that deeper programmes like PRiSE are achievable:

Outreach does not have to just be workshop/talk based. It can be an interactive research based activity that supports research activities within HE (Participant 24)

students are probably far more capable than schools and researchers might expect (Participant 16)

Only one person claimed that such approaches are not practical.

Would need a huge amount of time to set up something good - even with input from other people! (Participant 3)

The second theme (five responses) concerns practical aspects towards delivering deeper programmes based on PRiSE's approach:

lots of practical and multifaceted suggestions people in a variety of contexts can take and adapt for themselves (Participant 13)

try and use existing resources available rather than reinventing the wheel (Participant 15)

The final theme (five responses) concerned potential impacts from deeper programmes of schools engagement, which is beyond the scope of this paper. These results suggest researchers and engagement professionals also see the value in PRiSE's approach and may be receptive to adopting the framework, though evidence of action following these immediate attitudes is really needed. We also acknowledge that this was a rather small survey from a group already highly bought-in to schools engagement, and thus results would likely be less positive from a wider and more representative sample of all researchers. These are avenues which could be explored further in the future to gain a better perspective on whether the PRiSE framework could realistically be rolled out further.

6 Conclusions

“Research in schools” programmes, where school students and their teachers get to experience and interact with cutting-edge science through independent projects over several months, may have some role to play in addressing current issues around participation and equity in science (physics in particular) at university. However, how best to go about delivering such projects for schools remains unclear. This paper evaluates the provision framework of the Physics Research in School Environments (PRiSE) programme. These 6-month-long projects mentored by expert researchers include a suite of activity stages, interventions and resources to enable a wide range of students, teachers, and schools to be involved.

Feedback from participating students and teachers upon completion has been significantly more positive than benchmark results on schools engagement programmes with STEM in general. This is because students and teachers have found the research projects of great interest and have relished the challenge of working differently to their regular school experience. This suggests that “research in schools” projects are of greater value to schools than more common forms of outreach activities. Since these experiences can uniquely be provided by active researchers compared to other possible STEM engagement providers, we strongly urge universities consider “research in schools” approaches to engagement. Participants find the numerous elements of supporting interventions and resources provided in PRiSE's framework, uncommon in general with other schemes, as equally valued and necessary for their participation. We therefore recommend that providers of independent research projects offer schools detailed project guides, mentorship from experts (in-person, live-online, and asynchronously), online resources (such as “how to” articles, examples of previous students' work, and videos), and finally the opportunity to present their work to peers, teachers, and family. We note that even with all the support provided there is some attrition within the programme, which is to be expected for any protracted engagement programme. While this has not been explored here, it is assessed in a companion paper (Archer, 2021). Currently though we have few data on the experience of students and teachers that have dropped out of the programme, which is something that could be explored in the future. We have also gathered data on the impact of the PRiSE initiative, which is assessed elsewhere (Archer and DeWitt, 2021).

The PRiSE model attempts to make efficient use of researchers' time, enabling more schools to be supported per institution than other current formats. We have presented data from independent researchers and public engagement professionals showing they seem receptive to the PRiSE framework. Indeed, it is already slowly beginning to spread to other institutions. This potential expansion might allow an assessment of how generally applicable the framework is outside of its current London location and what other affordances might be required in these contexts. While PRiSE has so far only concerned areas of physics research, “research in schools” in general already spans all the sciences (e.g. Bennett et al., 2016, 2018; IRIS, 2020). We therefore see no reason why PRiSE's approach could not also be broadened to other STEM areas, particularly areas of research based on data and/or analysis. We therefore encourage researchers, and the public engagement professionals who facilitate their activity, to consider adopting this way of working and hope this paper can inform this practice. In such cases, it is recommended that PRiSE projects be embedded as core schools engagement activities within research groups. We would be happy to support groups in developing, delivering, and evaluating pilot PRiSE projects around their own research, thereby making use of the learning that has arisen from the programme over the last 6 years.

Appendix A: Information about UK/English schools

To those unfamiliar with the UK/English education system, we provide some further notes here. The curriculum is broken down into Key Stages of duration 2–4 years, with those for secondary schools displayed in Table A1. The final two Key Stages culminate in GCSE and A-Level examinations respectively, with the latter being optional as education post-16 is not compulsory. Our recommendations to teachers about which year groups should be involved with PRiSE are also highlighted in the table.

Table A1 Summary of the stages of secondary education in the English system.

Appendix B: Participant evaluation questions

Here we list the questions posed in questionnaires that are considered within this paper, giving details of what phrasing was used, how participants could respond, and which years the question was posed. Follow-on questions are indicated by indentation and a down-right arrow. (Students responded to the following.

Table B1 The questions asked of students.

Table B2 The questions asked of teachers.

Appendix C: University sector questionnaire

The following questions were posed to university researchers and engagement professionals via an online interactive survey during a session at the 2019 Interact symposium (Archer, 2019).

Table C1 The questions asked of university researchers and engagement professionals.

For context on these participants, Fig. C1 shows the distribution of the types of activity they undertake, where they could select from the following.

• Stall/stand. Drop-in activities for schools at STEM or careers fairs

• Talk. A typically one-/two-lesson slot featuring a predominantly one-way interaction

• Workshop. A typically one-/two-lesson slot with mostly two-way interaction and often hands-on activities for students

• Masterclass/taster. Half-day or day-long activities which may be comprised of talks and/or workshops

• Summer school. Several-day to week-long activities often involving some project work as well as talks and/or workshops

• Extended programme. Multiple interventions with the same group of students over a protracted period of time

Unsurprisingly, one-off activities such as talks and workshops scored the highest, whereas more the protracted engagements, summer schools and extended programmes were significantly (p<0.0019) less common. The attendees were also asked what they hoped to achieve (i.e. the aims or intended impacts) through their school engagements via an open question. Performing a thematic analysis of the qualitative results, it was possible to categorise the majority of answers into the following.

• Change school students' aspirations (nine people), with the word “inspire” often used

• Enhance students' awareness or understanding of STEM (six people), often in the context of primary research

• Tackle societal biases in STEM (four people), most often gender

Note that some responses covered more than one of these aims. Other stated motivations outside of these themes included “access to a student population for [research] studies”, to “build relationships”, and to deliver “meaningful content”. The three themes are in general agreement with those determined by Thorley (2016) in a larger survey of UK physicists.

Figure C1 Bar chart of types of school outreach activities undertaken by university researchers and engagement professionals (n=17). Error bars denote the standard (1σ) Clopper and Pearson (1934) interval.