The development of a pre-enrolment screening tool to inform targeted support services in the first year in health sciences

Pacific students are from a minority group in New Zealand who like minority groups in other countries find transition to university difficult and success elusive. Nowhere is that successful transition more important at the University of Otago than in Health Sciences First Year (HSFY) which is a competitive entry pathway to health professional courses. Retention for Pacific students is similar to other students but poor academic achievement remains. Tinto argues that students’ pre-entry attributes are major contributors to success in first year of university study. The aim of this paper is to describe the development of an instrument which predicts the performance of prospective students in the first year at university. The purpose of the instrument is to inform the development of tailored interventions aligned with students’ needs. The instrument also provides an early proxy for student engagement and a benchmark for evaluating ongoing interventions. Please cite this article as: Kokaua, J., Sopoaga, F., Zaharic T., & Van der Meer, J. (2014). The development of a pre-enrolment screening tool to inform targeted support services in the first year in health sciences. The International Journal of the First Year in Higher Education, 5(1). 55-66. doi: 10.5204/intjfyhe.v5i1.205 This article has been peer reviewed and accepted for publication in Int J FYHE. Please see the Editorial Policies under the ‘About’ section of the Journal website for further information. © Copyright of articles is retained by author/s. As an open access journal, articles are free to use, with proper attribution, in educational and other non-commercial settings. ISSN: 1838-2959 The development of a pre-enrolment screening tool to inform targeted support services... 56 | The International Journal of the First Year in Higher Education, 5(1) March, 2014 Introduction Pacific peoples are a migrant minority group in New Zealand who are disproportionately represented in poor health and education outcomes (Ministry of Health, 2012). They comprise 7% of the total population and are characterised by a predominantly young demographic structure with low socio-economic status. It is predicted that the number of Pacific peoples aged 15-24 years will increase by 14% by 2023, one of the highest rates of increase among ethnic groups in New Zealand (Statistics New Zealand, 2009). The New Zealand government has prioritised increasing Pacific participation and academic achievement in higher education in response to poor academic outcomes and in line with international trends to increase the number of students from minority groups and improve their levels of academic achievement (Minstry of Education, 2011; Smart, 2006). Furthermore, poor academic performance has a major impact on continuing in the Health Sciences First Year (HSFY) programme as it is a competitive entry pathway to health professional courses (medicine, dentistry, pharmacy, etc.). As a consequence, Pacific people are underrepresented in the health professional workforce. To increase Pacific peoples’ representation in the workforce, supported pathways from secondary school to health professional training in tertiary institutions are required. Alignment between entry criteria into a tertiary institution and requisite knowledge to undertake a given course of study can vary greatly between institutions and teaching programs, especially in the first year of enrolment. Reasons for this misalignment can vary, but are often based around two factors: an institution's desire to provide open opportunities; and/or the pool of students the institution wishes to enrol may not be appropriately prepared for tertiary education for reasons beyond the students' control. Providing targeted opportunities for under-represented groups is likely to provide greater access to tertiary education. However, these opportunities raise questions first around whether these students are being set up to fail, and second the responsibility of the institution to provide appropriate support for less prepared students. Entrance into the HSFY program at the University of Otago is based on the general entry criteria for first year students. Ideally, students will have prior preparation in Chemistry, Biology, Mathematics, English and Physics. We have previously reported (Sopoaga, Zaharic, Kokaua, Ekeroma, Murray, & Van der Meer, 2013) that Pacific students are more likely to be less prepared for HSFY than their non-Pacific classmates. The consequence of this under-preparedness has been a historical under-representation of Pacific students who succeed in admission into a health professional training program (Pacific Islands Research and Student Support Unit, 2013). A response to improve this situation has been to develop a tailored support program for Pacific students. These are facilitated through a series of ten peer mentoring sessions (Sopoaga & Van der Meer, 2011). As part of the wider support programme, we have recognised the need to assess students’ academic preparedness for HSFY to better tailor the support programme to their needs.


Introduction
Pacific peoples are a migrant minority group in New Zealand who are disproportionately represented in poor health and education outcomes (Ministry of Health, 2012).They comprise 7% of the total population and are characterised by a predominantly young demographic structure with low socio-economic status.It is predicted that the number of Pacific peoples aged 15-24 years will increase by 14% by 2023, one of the highest rates of increase among ethnic groups in New Zealand (Statistics New Zealand, 2009).The New Zealand government has prioritised increasing Pacific participation and academic achievement in higher education in response to poor academic outcomes and in line with international trends to increase the number of students from minority groups and improve their levels of academic achievement (Minstry of Education, 2011;Smart, 2006).Furthermore, poor academic performance has a major impact on continuing in the Health Sciences First Year (HSFY) programme as it is a competitive entry pathway to health professional courses (medicine, dentistry, pharmacy, etc.).As a consequence, Pacific people are underrepresented in the health professional workforce.To increase Pacific peoples' representation in the workforce, supported pathways from secondary school to health professional training in tertiary institutions are required.
Alignment between entry criteria into a tertiary institution and requisite knowledge to undertake a given course of study can vary greatly between institutions and teaching programs, especially in the first year of enrolment.Reasons for this misalignment can vary, but are often based around two factors: an institution's desire to provide open opportunities; and/or the pool of students the institution wishes to enrol may not be appropriately prepared for tertiary education for reasons beyond the students' control.Providing targeted opportunities for under-represented groups is likely to provide greater access to tertiary education.However, these opportunities raise questions first around whether these students are being set up to fail, and second the responsibility of the institution to provide appropriate support for less prepared students.
Entrance into the HSFY program at the University of Otago is based on the general entry criteria for first year students.Ideally, students will have prior preparation in Chemistry, Biology, Mathematics, English and Physics.We have previously reported (Sopoaga, Zaharic, Kokaua, Ekeroma, Murray, & Van der Meer, 2013) that Pacific students are more likely to be less prepared for HSFY than their non-Pacific classmates.The consequence of this under-preparedness has been a historical under-representation of Pacific students who succeed in admission into a health professional training program (Pacific Islands Research and Student Support Unit, 2013).
A response to improve this situation has been to develop a tailored support program for Pacific students.These are facilitated through a series of ten peer mentoring sessions (Sopoaga & Van der Meer, 2011).As part of the wider support programme, we have recognised the need to assess students' academic preparedness for HSFY to better tailor the support programme to their needs.

Studies on predicting academic performance
A number of studies have looked at the factors that influence academic performance.Tinto (1975) argued that pre-entry attributes, alongside integration with their chosen institution and social connectedness-both within their institution and beyond-were the most important influences on students ability to achieve.Other international studies have also found previous school or other academic attainment as the most significant predictor of university performance (Barry & Chapman, 2007;Shulruf, Hattie, & Tumen, 2008).In addition, some studies reported that other factors such as the education achievement of parents, learning discipline, and the ability to set learning tasks also contributed to increased student performance (Lemmens, du Plessis, & Maree, 2011;Torenbeek, Jansen, & Suhre, 2013).
A prospective study of, psychosocial, cognitive and demographic predictors by McKenzie and Schweitzer (2001) found the strongest predictors of academic performance were: students' successful transition into their chosen institution; self-efficacy with study techniques; and external employment responsibilities were the strongest predictors.They also concluded that identifying factors that influence academic performance can improve the targeting of interventions and support services in first year at university.A student survey tool was developed by Van Zyl, Gravett, and de Bruin (2012) to investigate whether there were pre-entry attributes associated with success in higher education.The researchers looked at 33 attributes of pre-entry in six general domains and found that the study reinforced the importance of pre-entry attributes to student success, and the need for institutions to identify students that might struggle with engagement in their first year.Another approach used a structural additive model of academic success to create an individualised summary index of poor academic success.Risk factors contributed a positive value to the index while protective factors subtracted from the index (Lucio, Rapp-Paglicci, & Rowe, 2011).This study used an extensive mix of school administrative and survey data and found that being held back a grade, and poor behaviour at secondary school had a significantly negative contribution to later success in tertiary education.Alternatively, academic selfefficacy prior to university study and having had some music tuition contributed positively to the index.Caison (2005), in a retrospective study, used administrative data to investigate whether retention of students was influenced by race, parental education, external work, high school GPA and first semester GPA.The results suggested that identifying key risk factors could lead to more effective interventions to improve retention.Furthermore, routinely collected administrative data can be useful in developing tools to enhance the provision of support services.Several studies of retention or students' success have reported on the use of the identified risk factors or suggested indices in assessing interventions to improve those outcomes (Thomas, 2011).Few have reported on the application or utility of the results to student programs.

The aim of this paper
The aim of this paper is to present the development of a pre-entry instrument to assist in tailoring support for Pacific students in the HSFY at the University of Otago.The instrument uses routinely collected data and incorporates historical patterns of pre-enrolment background and academic results in HSFY to predict the academic results of prospective students.
The instrument has been used to target the delivery of support services for Pacific students in 2013.The intention is to refine the support program for each student to provide them with the best opportunity to overcome any shortcomings in the academic preparation for some or retain and improve the positive expectation of achievement for other students.A logistic regression analysis of the binary indicator for retrospective students passing all Semester 1 papers yielded predictive scores representing the probability that a student had passed.In addition the models produced predicted probabilities for new data and have been used to produce a score for each 2013 HSFY student.Using the predicted probability score three Early Academic Preparation Indicator (EAPI) categories were created.After a sensitivity analysis, the categories were set to:

An indicator of three levels of academic preparedness
• High, if a student has a probability of 85% or higher of passing all four Semester 1 papers; b.Students with a probability of 50% or higher and less than 85% of passing all four Semester 1 papers.

Ethical use of data
The Associate Dean Pacific, Health Sciences, is the head of the team conducting this work and has designated responsibility for monitoring and improving the academic performance of Pacific students at the University of Otago.One of the functions of the team is to produce and disseminate evidence-based information on best practice for Pacific students in Health Sciences.The data for this project was used in adherence with the appropriate use of student data as explained to students' upon admission to the University (University of Otago, 2014) and in accordance with the NZ privacy regulations ("The Privacy Act s6," 1993).

Statistical analysis
All data were analysed using SAS version 9.3.Logistic regressions are used to calculate the predicted probabilities while a receiver operator characteristic curve was used to determine the global performance of the predictive scores against a range of successful outcome measures for HSFY students.The area under a receiver operator characteristic curve is interpreted as the probability that a randomly chosen person observed with each outcome of interest (e.g. with passes in all Semester 2 papers) will have a higher score than that of a randomly selected person without that outcome.Sensitivity, specificity, positive predictive values, negative predictive values were calculated to check the performance of the three level EAPI groups.Logistic regressions are also used to show the effects of EAPI groups and on binary measures such as the proportion of students who passed all or no papers in Semester 1, while generalised linear regressions were used to show the effects of EAPI rating on average grades for students in HSFY.

Predicted probability scores
The first step was to perform two separate logistic regressions of multiple variables reported in an administrative dataset upon a single key outcome.The main outcome was defined as a pass in all of the Semester 1 HSFY papers and subsequent performance in Semester 2. As NCEA school results are only available for school leavers from New Zealand schools, an alternate model was run for students who had not recently attended a New Zealand school.The odds ratios for the logistic regressions are reported in Table 2.A series of additional logistic regression analyses were performed with the resulting predicted probability scores against a number of other binary HSFY outcomes of success.In addition to passing all four Semester 1 papers, these regressions were performed on whether students completed Semester 1, passed all remaining Semester 2 papers and whether students completed the full first year.Table 3 show that the models correctly identified 20-31% of students who had achieved each HSFY outcome, and 27-35% of those who missed the HSFY outcomes.The area under each receiver operator characteristic curve for the series of analyses were at least 76% which is interpreted as a good result for global performance of the predicted scores to predict each respective outcome (SAS Institute Inc, 1999).

A three level categorisation rating academic preparation by students, 2007-2012
Prior to 2013, 306 students who identified with Pacific ethnicity had enrolled in HSFY.Pacific students in HSFY were twice as likely to be in the low EAPI group (students with the lowest probabilities of passing all four Semester 1 papers, as non-Pacific).
Conversely, Pacific students were half as likely to be in the high EAPI category.
Figure 1 shows the proportion of students that passed all Semester 1 and 2 papers between 2007 and 2012 in each of the EAPI categories by ethnicity.Although proportions for Pacific students passing all Further analyses show the EAPI categories have positive predictive values of 70% or more and negative predictive values of 65% or more for a range of HSFY outcomes (data not shown).However, for students in the high category, the model shows a better ability to predict outcomes with

HSFY Pacific students in 2013
One of the main features of the EAPI categories is that they are derived from data that is available, in New Zealand at least, prior to students arriving at University.Compared with the average 51 Pacific students enrolled in HSFY between 2007 and 2012, there were 67 students in 2013.
We used the EAPI model to assign each 2013 student an EAPI score, and compared their results to the 2007-12 cohort (Table 4).The most distinctive result is the effect of the EAPI groups.The proportions who passed all four Semester 1 papers increases clearly from the Low to High EAPI categories while the proportion who passed no papers decreased.Taking into account any effect from the 2013 cohort, the odds that a High category student passed all Semester 1 papers were 8 times those for a student in the Low category (p<0.0001),Moderate category students were more than twice those for Low category students (p=0.002).Conversely, the odds that a student in the High category passed no papers fell to less than 20% of the odds for a student from the Low category (p<0.0001).Likewise, the odds for Moderate category students were a quarter of those for those in the Low category (p<0.0001).Also taking into account for any effect from the 2013 cohort, the mean score for High category students was 58% higher than that of Low category students (p=0.0004) while those in the Moderate category were 37% higher (p=0.006).
The second result was the differences observed for 2013 students compared with past years.Students in the High and Moderate categories in 2013 appear less likely to fail all papers in Semester 1 than students in the same categories from past years.A higher proportion of High and Low category students also passed all four Semester 1 papers than in past years.
Overall, the mean score for 2013 students were 7% higher than past year's students.
Although coefficients of regressions confirm this pattern, when taking into account the effect of EAPI, the difference between current and past years students were not significant (p>0.05).

Discussion
The Early Academic Preparatory Indicator is a useful tool for assessing students' academic preparation as a component of their successful transition into first year in HSFY at the University of Otago.The EAPI is useful for informing the planning and the provision of academic and general support prior to the enrolment process in the academic year.Many publications have relied upon administering a brief diagnostic instrument at the start of the academic year to assess a student's academic preparation (Barry & Chapman, 2007) or relied upon past school academic achievement.While not all encompassing, the EAPI provides a useful summary of students' background beyond school performance, particularly for students who have yet to start university.
The EAPI has the ability to predict more than first-year outcomes.Even in a comparatively small cohort of 114 Pacific students, the proportions of students who passed none, some, or all papers, improved from Low to High EAPI categories.
Similarly, average marks increased by 11 with each EAPI category.
As Tinto's (1975) and other international studies of academic retention or performance (Levy & Murray, 2005;Shulruf, Hattie, & Tumen, 2008) have shown, previous academic success has the greatest influence on Semester 1 pass rates in higher education.Our results are consistent with these findings and are reflected in the EAPI.In particular, passes in Science and Mathematics have also been found to be a proxy for students learning ability (Lemmens, du Plessis, & Maree, 2011;Torenbeek, Jansen, & Suhre, 2013) with Mathematics contributing to success in science-related courses (Caison, 2005).Corresponding with results from other studies, our proxy indicator for academic socialisation-accommodation in a residential college-also had a significant influence for both school leavers and other students.Due to no restriction on entry criteria for HSFY, the age range for students is usually over represented in the 17 to 25 age range.However, consistent with other studies, gender was significant for school leavers as was ethnicity even after controlling for all the other covariates included in each model.
While many studies and interventions are focused on 'at risk' or 'minority' students (Lemmens, du Plessis, & Maree, 2011;Levy & Murray, 2005), few report on whether students feel marginalised by identifying them by how likely they are to pass.We recognise that there is a risk in labelling students as such and how this might impact on their overall performance.From our experience, students appreciate that prior academic preparedness will either increase or lessen their ability to pass the HSFY course.Most are receptive to receiving any additional support that might assist them to achieve their goals.We intend to use the EAPI model to better target student support, and as part of the evaluation process, to examine whether there is a stigmatising effect through its use.
The EAPI rating as an instrument has provided an early indicator for Health Sciences students' academic preparation.While the indicator has been applied to the academic performance of Pacific students in HSFY, it is not an ethnic-specific instrument (as seen in Figure 1)-it can report differences in academic preparation between ethnic groups.The methodology is transferable to other courses and universities in New Zealand.Other courses would require a different mix of factors depending on the specific requirements of each institution (Dickson, Fleet, & Watt, 2000).In terms of application beyond New Zealand, the measure would depend further upon the availability of national or other relevant data.

Conclusion
The Early Academic Preparation Indicator has provided a useful pre-entry instrument for assessing students' academic preparation prior to entering University.It has facilitated the development and delivery of informed targeted interventions aligned with student needs.It provides a very good starting point and an early proxy for developing effective approaches to encourage ongoing student engagement during the year.The EAPI methodology may be useful for other institutions in the development of targeted support, especially for under-represented minority groups.

Figure 1 :
Figure 1: Proportions of students that passed all Semester 1 and Semester 2 papers by EAPI; 2007-2012

Table 1 : Variables included in models for school leavers and non-school leavers Variables included in models for school leavers and non-school leavers Male
An indicator for biological gender Pacific Pacific student indicator College An indicator of whether a student is living in a residence college Auckland Indicator for students from Auckland, a major city in New Zealand where 70% of Pacific peoples reside.Auckland is the furthermost NZ city from Otago Variables

included in models for school leavers only School
SES A three level socio-economic indicator, a truncated version of the indicator developed by New Zealand's Ministry of Education to represent the proportion of students of low socio-economic status in the catchment area of a school):1 represents low socio-economic status and 3 represents high.

included in models for non-school leavers Foundation
An indicator if the student attended the University of Otago's Foundation program (which is a preparatory year before entering HSFY) International An indicator of whether a student came from outside New Zealand Admission A three level indicator of admission type.The latter is equal to: 1 if the student is admitted with NCEA level 3 or equivalent passes, 2 if admitted with an additional qualification; and 3 for other discretionary admission types