School students from all backgrounds can do physics research: On the accessibility and equity of the PRiSE approach to independent research projects

Societal biases are a major issue in school students’ access to and interaction with science. Schools engagement programmes in science from universities, like independent research projects, which could try and tackle these problems are, however, often inequitable. We evaluate these concerns applied to one such programme, ‘Physics Research in School Environments’ (PRiSE), which features projects in space science, astronomy, and particle physics. Comparing the schools involved with PRiSE to those of other similar schemes and UK national statistics, we find that PRiSE has engaged a much more diverse 5 set of school students and significantly more disadvantaged groups than is typical. While drop-off occurs within the protracted programme, we find no evidence of systematic biases present. The majority of schools that complete projects return for multiple years of the programme, with this repeated buy-in from schools again being unpatterned by typical societal inequalities. Therefore, school students’ ability to succeed at independent research projects appears independent of background within the PRiSE framework. Qualitative feedback from teachers show that the diversity and equity of the programme, which they attribute to 10 the level of support offered through PRiSE, is valued and they have highlighted further ways of making the projects potentially even more accessible. Researcher-involvement, uncommon in many other programmes, along with teacher engagement and communication are found to be key elements to success in independent research projects overall.

As of March 2020, 67 schools have been involved in PRiSE. A full list of (anonymised) schools is given in Appendix A. Figure 1 demonstrates that these schools (blue) have been fairly broadly spread across Greater London rather than being focused solely around Queen Mary (red). Schools targeting has been limited to London to enable researchers to build relationships with the schools via their in-person interactions throughout the 6 month projects. Most schools have participated directly with Queen Mary, though we note that some have been involved as a partnership of local schools (there have been 8 partnerships across 60 22 schools, listed in Appendix A). Such partnerships could provide an additional support network to students and teachers as well as making interventions more efficient for researchers. However, we have found these partnerships to have been somewhat hit-or-miss so far within PRiSE -while kick-off events with all partner schools present have typically worked, following this the schools have not always worked with their partners on the projects. Further investigation is required to understand what makes these partnerships work. https://doi.org/10.5194/gc-2020-37 Preprint. Discussion started: 6 August 2020 c Author(s) 2020. CC BY 4.0 License.  Figure 2. Distribution of school categories involved in IRIS (red), HiSPARC (yellow) and PRiSE (blue) projects compared to UK national statistics (grey). Error bars denote standard Clopper and Pearson (1934) intervals.
independent schools. Some others have agreed to the principle and tried to implement partnerships but failed to do so, though seem willing to put further effort in to establish these partnerships ready for the next academic year.
Beyond school type and admissions policy we look at several other metrics for the backgrounds of the schools' students.
We detail in Appendix C how we have combined various datasets in order to assess these. For PRiSE schools these methods 95 results in two different metric values for each school -one covering the school's full catchment area (the area from which they draw students) and another purely pertaining to the school's local census area. For schools outside of London we only have access to the local data. In Figure 3 distributions of the gathered metrics are shown in two formats. Top panels display boxplots depicting quantiles of the metric. Bottom panels depict kernel density estimates of the continuous probability distributions, where Gaussian kernels of optimal bandwidth from the Silverman (1986) rule have been applied to each dataset. Standard 100 confidence intervals are estimated by bootstrapping 1,000 random realisations of the data (Efron and Tibshirani, 1993), taking quantiles of their resulting kernel density estimates (with the bandwidth fixed from before).
The first metric we consider is the percentage of students eligible for free school meals, an often-used widening participation criterion in the UK which can be found in Edubase. Free school meals are a statutory benefit available to school-aged children from families who receive other income-assessed government benefits and can be used as a proxy of the economic status 105 of a school's students. Figure 3a shows that both HiSPARC and IRIS schools feature considerably lower free schools meals percentages than the national statistics in terms of the location (e.g. median), scale (e.g. interquartile range), and shape (e.g. tail heaviness) of their distributions. In contrast, the kernel density estimate for PRiSE schools appear to tend towards higher percentages and be somewhat broader. We perform Wilcoxon rank-sum tests, which test whether one sample is stochastically greater than the other (often interpreted as a difference in medians) since it is more conservative and suffers from fewer 110 assumptions (e.g. normality, interval-scaling) than two-sample t-tests (Hollander and Wolfe, 1999;Gibbons and Chakraborti, 2011). These tests, however, reveal that in terms of quantiles PRiSE schools are merely consistent with the national distribution (an absolute +3% and relative 1.26× difference in medians, p = 0.708), as is also evident from the boxplots. This means that  Figure 3. Distributions of a) students on free school meals, b) higher education participation rate, c) index of multiple deprivation score, and d) ethnic minorities. Boxplots in the top panels have whiskers covering 10-90%, boxes spanning 25-75%, and bands depicting the standard confidence interval in the median. Lower panels show kernel density estimates along with bootstrapped standard confidence intervals. Dashed lines indicate only local data is used, which does not cover the schools' full catchment areas.

115
The second metric considered is the higher education participation rate (Figure 3b), which measures how likely young people are to go on to higher education (e.g. university) based on where they live (Office for Students, 2018). All the programmes considered involve students from areas with greater participation in higher education than is representative of the entire country, but they are largely similar to one another. This is not surprising for PRiSE given it is limited to London, since it has been noted that young people across London are generally more likely to access higher education than those elsewhere in the UK (Office 120 for Students, 2018). However, the fact that PRiSE's results are similar to the two national programmes is perhaps surprising as HiSPARC lists no London schools and only 17 ± 3% of IRIS schools are in the Greater London area. Therefore one might expect these two programmes to have markedly lower participation rates in higher education than PRiSE purely due to this fact, which is not the case. Given in-person interactions between researchers and schools is a critical part of the PRiSE model , the geographical reach of the Queen Mary programme will always be limited to London and thus it is school's catchment area, these scores are considerably higher than the national statistics (the median is +9.40 and 1.54× larger, p = 7 × 10 −4 ). This difference, however, disappears when using the local proxy yielding simply a representative distribution (+1.50 and 1.09×, p = 0.371). In contrast, IRIS schools again clearly favour fewer disadvantaged students than PRiSE (−6.56 and 0.65×, p = 4 × 10 −4 ) and thus also the national statistics ( −5.07 and 0.71×, p = 2 × 10 −6 ), whereas these differences is simply due to the fact that London is the most ethnically diverse region in the country though. The distributions for IRIS and HiSPARC are very similar to one another and while their distributions' location parameters are just less than the overall national statistic, they both feature greater ethnic diversity than compared to the full distribution across all census areas (in the case of HiSPARC this is not strictly statistically significant at +4% absolute and 1.79× relative, p = 0.056, likely due to small number statistics).

145
In terms of gender balance, a similar school/area-level approach would not capture the known bias present in physics where only 22% of A-Level physics students are female (Institute of Physics, 2018). Therefore, at kick-off meetings the percentage of young women or girls involved in PRiSE at each school were observed (accurate to the nearest 10%). The median of these is 40% though there has been considerable variation in the gender balance amongst schools. We have worked with 11 girls' schools (compared to just 8 boys' schools) to help address gender balance across the programme. How this variation on a 150 school-by-school basis compares to each schools' A-Level cohorts or across all schools nationally is unknown, as this data is not publicly available. IRIS, HiSPARC and ORBYTS have not yet reported on the gender variations in their programmes.
To summarise, PRiSE has to date engaged with a much more diverse set of school students and significantly more underrepresented groups than other similar schemes. In the case of HiSPARC such biases might be explained by the cost of participation. IRIS's programme, however, is free to schools and perhaps the issue lies in their targeting of and engagement with 155 schools. In contrast, PRiSE reflects national statistics in most measures, and in some cases features higher proportions of underserved groups. Some work, however, needs to be done for PRiSE to be more representative in terms of schools' categories and admissions policies, which is currently being addressed. Future work could expand the evaluation of participation to go beyond school/area-level metrics and investigate individual student's characteristics, however this would require funding to commission such research by social scientists along with the necessary ethical approval.

Retention
It is natural to expect some drop-off in participation of schools for a long-term and primarily extra-curricular programme. We employ the branch of statistics known as survival analysis (Miller, 1997) to quantify the retention of schools throughout the PRiSE programme. In particular we calculate both survival (the probability that a school is still involved with the programme at a certain stage) and hazard (the probability a school still involved at a certain stage will drop out at the next stage) functions.

Across interventions
First we assess the retention of schools across the intervention stages of the programme. This is done for academic years 2017/18 and 2018/19 only, since before this our data collecting was insufficient to track schools' retention throughout the different stages of the programme and in 2019/2020 the programme was disrupted by the COVID-19 pandemic just before the comments stage with the conference having to be postponed. In many cases in our results an assumption has been made about 170 when schools may have dropped out, because in 68 ± 12% of schools which drop out after the kick-off stage the teachers do not inform us and simply no longer reply to our continuing emails (with little difference in this percentage across the different project stages). In these cases we assume that schools dropped out at the earliest point the teacher became unresponsive. While all intervention stages are offered to all schools, it should be noted that not all schools actively engaged in project 190 work take advantage of them: 7 ± 4% don't attend a kick-off, 39 ± 9% do not schedule a researcher visit, and 47 ± 10% don't solicit comments on their work. Schools which don't solicit a researcher visit are much more likely to subsequently drop out (41 ± 14%) compared to those which do (12 ± 8%). This highlights the importance of researcher-involvement in the success of programmes, despite independent research projects in schools often not being supported by external mentors in general (Bennett et al., 2016(Bennett et al., , 2018. The other two intervention stages do not appear to be critical for schools ' retention. Firstly,in 195 the case of the kick-off, this suggests the resources provided are sufficient to still undertake project work. Secondly, while comments on draft presentations are valued by students and teachers , students are still able to produce work that can be presented at a conference without them. In addition to equality in access, it is also important that programmes be equitable so that everyone has a fair chance of succeeding. We find no biases in schools' ability to successfully complete a year by school category (χ 2 (4) = 3.21, p = 0.524), 200 free schools meals percentage (+4% absolute and 1.33× relative differences in medians of schools which do and do not complete a year, p = 0.865), higher education participation rate (+2% and 1.04×, p = 0.677), indices of multiple deprivation (−4.30 and 0.86×, p = 0.219), or ethnic diversity (−3% and 0.93×, p = 0.844). We do not perform this test for gender variation since the data is less reliable, as previously mentioned. Therefore, the typical societal barriers to STEM do not appear to affect students' ability to succeed within the PRiSE framework. 205 We note that as well as schools dropping out, even within those schools which complete a year there is typically some reduction in the number of students that persist with project work. As our reporting only recorded total numbers of students by Key Stage (see Table B1 for more information) at events rather than on a school-by-school basis, we cannot calculate student retention rates for schools which received on campus kick-off events, since these involved multiple schools some of which subsequently dropped out. Neglecting those schools, we find that between 2015-2019 the overall retention rate from kick-off 210 to conference was 56 ± 3% (n = 321), though we note individual schools' rates varied widely with an interquartile range of 39-88% and 7 schools (out of the 27 considered) retained all students throughout the programme. Similar to with participation, future work (subject to funding and ethics approval) could investigate the retention within PRiSE at the student-level and whether this is also equitable.

Across years 215
We have seen considerable repeated buy-in from many schools over multiple years with PRiSE (see Appendix A for the data), thus we also investigate retention across years. Figure 5 shows the overall results in black where only schools which began work on projects are included. We note that because the programme has been carefully grown since its inception, not all schools started at the same time and this is why the survival function is not strictly decreasing, e.g. only 6 schools could have been

245
Here we present qualitative data from teachers relating to the issues of accessibility, diversity, equity, and retention.

Method
We prefer to take a holistic approach in investigating teachers' thoughts about these issues, using evaluative data from a variety of methods throughout the programme. Formal feedback from teachers has been gathered from teachers via paper questionnaires handed out at our student conferences each year (in 2020 due to the COVID-19 pandemic an online form was 250 instead used). All data used here were in response to open questions and further details of this questionnaire can be found in . In addition to this formal feedback, we also use data obtained more informally. These include comments made by teachers in-person throughout the programme, e.g. during researcher visits or our conference, and those passed on via email. Where possible explicit consent has been obtained to use these comments, though in general by participating in PRiSE teachers are aware that they are entering a relationship of mutual trust and that any information passed on by them will be used 255 with integrity (cf. BERA, 2018). These informal comments are recorded at the time and then analysed along with and in the same way as the formal feedback. The anonymity of the teachers and their schools are protected, quoting only the school's pseudonym as well as the project and year the feedback related to along with the method the data was obtained.
All qualitative data were analysed using thematic analysis (Braun and Clarke, 2006), with the themes being allowed to emerge from the data via grounded theory (Robson, 2011; Silverman, 2010) as follows:

Results and discussion
Several teachers from a variety of different schools have raised that they value the diversity present across PRiSE, particularly that students from different schools and backgrounds are able to interact as equals at the student conferences.
"They love the competition with independent schools" (Teacher, Rushmore Academy, MUSICS 2018, in-person) "Giving the students the opportunity to meet with other schools and academic staff in person is a major highlight" 270 (Teacher, Tree Hill High School, PHwP 2020, questionnaire) One teacher expanded on this, contrasting the diversity present to other schemes and positing that the support provided through PRiSE enabled this difference "At [PRiSE student conference] Cosmic Con, pupils from a diverse range of state and independent schools have the opportunity to share their experiences and discuss their findings with each other, widening their perspective 275 from the natural micro-habitat of the school environment to the wider community around them. Groups that have worked on the same project naturally gravitate into discussion. The key to increasing collaboration and empathy for those in different schools is to put pupils in the same place, at the same time, with some interest or experience in common to talk about. There are other schools conferences around, but QMUL has such close links with and provides such support for schools, that this adds to the diversity of Cosmic Con." (Teacher, Octavian Country Day The equality of all students at PRiSE conferences is further reflected in that prizes awarded to students, judged by researchers, have been well distributed amongst different types of schools with no obvious biases (though due to the small numbers we do not perform a thorough statistical analysis). This has, however, had a prodigious effect on students from disadvantaged backgrounds 285 "The students were buzzing on their journey back, they kept saying 'I can't believe our little comprehensive won'." that all their students bar one had ceased project work. While we try and assure teachers upon initiating the projects that we anticipate some drop-off and that there is no pressure from us to remain involved, more may be required in this area.

325
From the minority of teachers that do inform us of their school dropping out, typically via email, reasons have included mock exams getting in the way, students losing interest, realising the amount of work involved, difficulty balancing the project with other activities / their normal school workload, giving up due to uncertainty in how to progress, and not feeling like they've made enough progress to continue. Similar themes have been expressed from teachers in-person concerning some (but not all) of their students dropping out. We feel that many of these issues might have been mitigated through more and/or earlier 330 communication from the school to us, as we have been able to assist with similar struggles at more communicative schools.
Despite a clear support process being laid out at kick-off workshops (see  a number of teachers from schools that successfully completed projects realised in hindsight that they should have taken advantage of the opportunities from the university earlier in the programme than they did This is again something that could be stressed in teacher guidance upon engaging with the programme to better set expectations 340 and good practice, as many new teachers to the programme may not be used to such a reactive way of working rather than the typical 'push' model from teachers to students. Overall, we get the impression that retention within PRiSE can often come down to the individual teacher -those that are communicative and properly engage with the programme and its expectations from the outset are far more likely to see their students succeed. This is backed up by evidence from several schools, where a change of teacher has either led to increased 345 engagement with the programme (e.g. Rydell High School), success at previously unsuccessful schools (e.g. Hill Valley High School), or unfortunately previously successful schools dropping out of the programme (e.g. Prufrock Preparatory School).
One teacher who changed schools (from Hogwarts to Prufrock Preparatory School) bringing PRiSE projects with them also raised with us in-person that the culture within the schools can play a role in students' engagement with the programme (and extra-curricular activities in general). Both of these are challenging issues to address as we aim to increase the retention 350 and equity of the programme. While it is clear that more detailed qualitative research is required in this area, perhaps giving clearer information and expectations upon signup as well as providing further guidance on ways of successfully integrating and nurturing project work within schools, as highlighted by other teachers, these issues might be somewhat mitigated.

Conclusions
Societal inequalities in access to and engagement with science are prevalent even in secondary/high schools (Hamlyn et al.,355 2020). While university engagement programmes, like independent research projects, could address these issues, at present few such schemes specifically target traditionally under-represented groups and in general globally students' participation in such projects are inequitable (Bennett et al., 2016(Bennett et al., , 2018. In this paper we have evaluated the accessibility, diversity, and equity of the 'Physics Research in School Environments' (PRiSE) programme of independent research projects . implemented to improve diversity in these regards.

The schools involved in
Survival analysis has been used to explore the retention of schools within the programme. This was firstly done across the different intervention stages of PRiSE within each academic year. We find a fairly consistent drop-off rate throughout, with no significant differences between the different projects or the years considered. While little research into the retention of schools within protracted programmes of engagement currently exist, the rates exhibited by PRiSE are at least similar to another 370 programme (Hope-Stone Research, 2018). The analysis has highlighted the importance of PRiSE's researcher-involvement in the schools' success. This is despite independent research projects in general often not being supported by external mentors (Bennett et al., 2016(Bennett et al., , 2018. No biases in schools' retention appear present by school category, socio-economic background, or race. This suggests that school students' ability to succeed at independent research projects is independent of background within the PRiSE framework. PRiSE has seen repeated buy-in over multiple years from numerous schools. Hence we also 375 looked at the retention of schools across multiple years, again finding no real differences in the backgrounds of schools which return and those which do not. Indeed, the only predictor for multiple years of participation is whether the school engaged with the programme through to completion for at least one year. Our interpretation is that success within PRiSE often comes down to the individual teacher, with poor communication (cf. Sousa-Silva et al., 2018) or not fully engaging with the programme and its expectations serving as key barriers in schools' participation.

380
Qualitative feedback from teachers have shown that they value the diversity within the programme, seeing the ability of students from different schools and backgrounds to interact as equals at PRiSE conferences as a positive aspect. They also attribute this equity to the exceptionally high level of support provided by PRiSE to the schools involved. The need for slight modifications to make some of the projects more accessible to students of all abilities has been raised. These concerns might also be addressed by prompting teachers to facilitate students' group dynamics and potentially incorporating peer mentoring 385 from previous years' PRiSE students, both of which have been reported as successful in some cases but with mixed results at other schools. More teacher guidance, co-created with teachers themselves, on the expectations within the programme as well as good practice in incorporating and nurturing project work within schools could be provided to help with retention in new schools.
Our analysis has been limited to the London geographic area, so it is not yet clear that the PRiSE framework of independent 390 research projects would necessarily be as accessible or equitable in different parts of the UK or in different countries. With the adoption of this approach to engagement at other universities, however, we hope to be able to investigate this in the future.
Further, only school-level metrics have been considered here and more detailed analysis at the individual student level and their characteristics could be considered in future, which would require funding to commission such research by social scientists along with the necessary ethical approval. Finally, in-depth qualitative research into the reasons behind schools dropping out 395 of PRiSE, both within the six-month programme and between different years, would be beneficial in understanding what the current barriers to prolonged participation are and how these could be best addressed in the future.

Appendix A: PRiSE schools
Below is a table with information about all the schools which have been involved in PRiSE. To protect the anonymity of students and teachers, pseudonyms (taken from https://annex.fandom.com/wiki/List_of_fictional_schools) have been used. The This paper uses the context of the UK/English education system to assess the diversity of schools engaged within the PRiSE (and other) programme(s). To those unfamiliar with this system, we provide some further notes here. Schools are classified by the Department for Education (2018) into the following main categories: -Academies are schools that are state funded and free to students but are not run by the local authority. They have much more independence than most other schools including the power to direct their own curriculum. Academies are 415 established by sponsors from business, faith or voluntary groups in partnership with the Department for Education working with the community.
-Colleges are post-16 education establishments not part of a secondary school.
-Free Schools are a type of academy set up by teachers, parents, existing schools, educational charities, universities, or community groups.

420
-Independent Schools are funded by the fees paid by the parents of pupils, contributions from supporting bodies and investments. They are not funded or run by central government or a Local Authority. They can set their own curriculum.
-Local Authority (LA) Maintained Schools are wholly owned and maintained by Local Authorities and follow the national curriculum.
Other less common categories are not considered in this paper due to small number statistics. In addition to the school category,

425
UK schools can also be classified by their admissions policy. Selective (or grammar) schools enrol pupils based on ability whereas non-selective (or comprehensive) schools are not able to do this. While some independent schools are selective, not all of them are. A-Level 17-18 Table B1. Summary of the stages in the English education system applicable to PRiSE.

Appendix C: Method for gathering metrics on UK schools
In this paper we look at several metrics to assess the backgrounds of the schools' pupils. School categories are listed in Edubase 430 (Department for Education, 2018), as are the percentage of students on free school meals (though this is typically not listed for colleges, independent schools, and some academies). Other relevant metrics are not included in Edubase but are tied to census lower and middle layer super output areas (LSOAs and MSOAs respectively Office of National Statistics, 2011), which ideally necessitates knowledge of a school's catchment area. While such information is available for state schools in the Greater London area through the London Schools Atlas (Greater London Authority, 2014), which lists for each school the resident 435 LSOAs/MSOAs of their student base, unfortunately there is no equivalent publicly accessible data covering the entire UK.
Here we detail how school metrics across their entire catchment areas are calculated. Higher education participation rates for each school are determined using the number of entrants to higher education and cohort populations across each school's MSOAs from POLAR4 data (Office for Students, 2018). Similarly, index of multiple deprivation scores (Ministry of Housing, Communities & Local Government, 2015) are averaged over each school's LSOAs. For protected characteristics such 440 as gender/sex and race/ethnicity, we do not collect this data from students for ethics reasons. In the latter case, while it has been observed by session leaders at interventions that a diversity of ethnicities have been involved, we opt to quantify this through the ethnic diversity of the areas from which students are drawn. Census data (Office of National Statistics, 2011) on ethnic groups is used to calculate the percentage of people from ethnic minorities (i.e. non-white groups) across each school's LSOAs.

445
To still enable some comparison between PRiSE and the national programmes, for which we do not have access to information on schools' catchment areas, we rely on using the metrics pertaining only to the LSOA/MSOA within which the school resides. These results are indicated by dashed lines in Figure 3 and have also been computed for PRiSE schools to ensure like-for-like comparisons. Across the Greater London area we can check the reliability of these local proxies, finding they do correlate but not particularly strongly (the correlation coefficients are R = 0.64, 0.65, and 0.85 respectively).

450
-Unresponsive: schools that did not respond to our communications from that point on -Dropped out: schools that communicated their dropping out the programme at that stage  Author contributions. MOA conceived the programme and its evaluation, performed the analysis, and wrote the paper.
grateful for funding from the Ogden Trust. This programme has been supported by a QMUL Centre for Public Engagement Large Award, and STFC Public Engagement Small Award ST/N005457/1.