Testing the Effect of UNIV1000 on Retention in a Regional University in the US

Student retention is a major concern for many universities. We use observational data from a regional university located in Alabama to test whether taking a first-year seminar improves student retention rates. Using a linear probit model, we find that taking a first-year seminar course is negatively correlated with retention rates, after controlling for several confounding effects. We perform survival analysis and find that the students who take first year seminar courses have a better survival rate for retention than those that do not take the course. We also find that other macro and micro economic factors are equally important in improving student retention rates, such as labor market opportunities and competition from similar universities.


Introduction
Many universities in the US are struggling to find ways of fixing the problem of student retention. One of the most popular methods adopted by universities and colleges across the country is to adopt first year seminar programs (FYS programs hereafter) or first year experience courses. Retention rates vary across institutions and depend on various factors, such as access to financial resources and social support at the time of starting college.
Over 90 percent of institutions offer FYS courses in the US as of now, with 80 percent mandating freshmen to take these courses within the first year of school (Tobolowsky et al., 2005;Tobolowski, 2006). There are usually between 20 to 30 sections of such courses offered every semester, and can usually focus entirely on academic performance, or social integration, or a combination of both. These courses are also taught by staff or faculty from various departments.
We use data from a regional university located in an urban setting in Alabama. Our data spans from Fall 2012 to Fall 2015. The university implemented FYS courses called UNIV in Fall 2013, allowing us to capture the impact of taking UNIV on retention rates. We also have individual level information regarding the students, such as age, sex, race, and their residence status. Controlling for observables, we find that taking UNIV is negatively associated with retention rates at the 5 percent level. However, using survival analysis and non-parametric methods, we find that those who have taken UNIV have a higher survival rate of retention. We investigate reasons behind the negative association between UNIV and retention and speculate that better labor market characteristics and stronger competition from other competing universities might be responsible for the negative correlation.
The rest of the paper is organized as follows: Section 2 gives a brief background of studies that have looked at this question from different contexts. Section 3 describes our data. We present our estimation techniques in section 4. Section 5 presents the results and we conclude in section 6.

Background
First-year seminars (FYS hereafter) have been used as an instrument to boost student retention rates, persistence rates, and academic performance. In 2012, almost 90 percent of all higher education institutions in the US offered FYS program. This is an increase from 68. 5 percent in 1988(National Resource Center, 2017. The main outline of attributes offered by a typical FYS course ranges from helping students with study techniques, skills to better schedule time and study sessions, note taking sessions, to developing skills to maintain a social life in college by developing hobbies or engaging in clubs. The five key components that help FYS achieve success include peer mentor, academic advisors, interactions with professors/students, designing an agenda, and joining organizations on campus (Jaijairam, 2016). Fiddler (1991) use data from 1973 to 1988, comparing sophomore return rates for participants and non-participants in a freshman orientation program. He finds that participants in freshman orientation programs had significantly higher return rates for 11 out of 16 of the years used in the study, even after controlling for sex, academic ability, race, course load, and motivation. Strumpf and Hunt (1993) look for the impact of a freshman orientation course on retention rates and academic performance on first-time first year students. 240 students indicated interest in enrolling in the course, some of whom were randomly assigned to an experimental group while others were in the control group. Their results indicate that orientation courses improved retention rates amongst those that were allowed to enroll in the course. Hoff et al. (1996) study the impact of Dalton College Studies-101 (a first-year seminar course offered at Dalton) on student academic performance and retention rates. Their analysis finds that students enrolled in the seminar course had better retention rates, attempted and completed more credit hours, and showed a higher rate of completion of 90 quarter hours than students not enrolled in the course. Stark et al. (2001) follow the progress of eight cohorts of students exposed to freshman seminar courses between 1986 and 1993 and compare them to those who did not take the course. Their results indicate that the students taking the first-year seminar courses had 6 to 28 percentage point better retention rates than students not taking the course. Moreover, the students taking first year seminar course had 14 pp to 32 pp better outcome on graduation rates relative to those that did not take the course. They also found that students enrolling in first year seminar courses had better GPA, satisfaction with college experience, improved interaction with faculty, and more participation in extracurricular activities.  study 1700 students attending a first-year seminar during academic years 1991 to 1994 at a medium-sized, public Midwestern University. They match these students to others who do not enroll in the seminar and find significant results suggesting that graduation rates for those attending the seminar is better than students not enrolled in the seminar. Furthermore, they find that the seminar benefits students from lower and middle high school deciles more than those from uppermost high school deciles. Land (2007) find similar results in his study of first semester college students enrolled at a public research university in Fall 1998. The students were matched by gender, race, SAT score, high school GPA, and intended program of study. His paper finds that students who took the elective course had better graduation rates, better GPA, and had better academic experience compared to those who did not take the elective course. Similar findings were obtained by analyzing data gathered from a southeastern public community college. Newman (2016) finds evidence suggesting that FYS course employed at a southeastern community college improves student retention rate and students taking the course do significantly better in terms of academic performance relative to those who do not take the course (Newman, 2016).
Using matching score method, Clark and Cundiff (2011) find weak evidence suggesting that first year seminars can improve retention rates but do not find strong evidence that it improves academic performance. Their paper also deals with the issue of self-selection into first year seminar programs and how that might create bias in interpreting some of their results. In our paper, this should not be a big concern since all students after Fall 2013 are required to take the course.
Analyzing data from first year seminar students from a research extensive public university, Hendel finds no evidence suggesting that first year seminars improve retention. Instead, the evidence in that paper suggests that high school performance was a better predictor of freshman retention (Hendel, 2007). The impact of first year seminar courses on academic performance and persistence is mixed. Some studies find that enrollment in such programs improves academic performance, retention, and persistence (Jenkins-Guarnieri et al., 2015;Scnhell & Doetkott, 2003) while others report no effect (Miller & Lesik, 2015;Hendel, 2007).

Data
The data were obtained from two sources. First, most of the student level information such as sex, race, registration for the academic year, was gathered from the university Registrar's office. Second, the UNIV office on campus provided us with information regarding whether the student has taken the FYS course or not. The data spans from Fall 2012 to Fall 2015. UNIV was offered beginning in Fall 2013. Therefore, students before Fall 2013 did not take UNIV course, providing a us with the opportunity to compare retention rates between those that have taken the FYS course and those that did not take the course. jel.ccsenet.org Journal of Education and Learning Vol. 9, No. 5;2020 200 UNIV is offered as a 2 to 4 credit hour course over a 12-week period. This course has approximately 35 sections and each section has around 20 students. The course focuses on teaching students' strategies to succeed in college such as time and stress management, note-taking skills, test-taking skills, studying tactics, etc. All incoming freshmen and transferring students with less than 2.2 GPA were required to take this course within the first year of starting at the university. There were few exceptions for some students based on performance and at the discretion of higher administration. Our dataset lists a few individuals who did not take UNIV when it was offered. This might be due to several factors such as communication error between advisors and students during the initial years of UNIV. However, students not complying with FYS requirements were not systematically different than those that did comply, suggesting a plausible randomness in non-compliance. Table 1 presents the characteristics of the students in our sample, separated by whether they took UNIV or not. The table suggests a larger proportion of female students overall. The average age of students is 23 for those who did not take UNIV and 21 for those who took it. Both cohorts are predominantly female and in-state students. Table 1 also illustrates that the proportion of female and instate students have increased after UNIV was offered. We control for differences in sex, race, and age in our specifications to control for the fact that the student population taking FYS might be different than those that were not exposed to the course, and that these differences are responsible for the results. Note. * GPA information is missing for a few observations in our dataset.

a) Parametric Analysis
We use the following model to estimate the association between taking a first-year seminar course (UNIV) and student retention: We estimate the above model using probit and logit regression methods. represents whether the student i from time t registered for the next academic year or not. If the student was registered for the next academic year, we treat this student as being retained. UNIV is a binary variable that equals 1 if the student i at time t took the first-year seminar course, 0 otherwise. X is a vector of individual controls such as sex (male or female), race (white or black or other), high school GPA, institutional GPA, age and composite ACT scores. Our main coefficient of interest is , which provides the correlation estimate between taking the first-year seminar course, UNIV, and retention. A positive point estimate would indicate a positive correlation between UNIV and retention.

b) Survival Analysis
The previous section assumed a parametric approach to establish a baseline. However, in this section we analyze the impact of first-year seminar courses on retention rates using survival analysis. There are a few benefits of using survival analysis. Since retention can happen in both groups (takers and non-takers of UNIV), we might be interested in the time to failure. Secondly, the data is right-censored and survival analysis can be a useful tool to deal with this issue. Thirdly, we employ a semi-parametric and non-parametric estimation method.
We first estimate a simple survival function using non parametric techniques, differentiating between the UNIV takers and non-takers. Secondly, we use Cox proportional hazard model to add additional regressors. Since we have information of when a student took the test and whether they registered for following semesters or not, our data is a discrete time event.
The two main components of interest are the hazard function and the survival function. Hazard function gives jel.ccsenet.org Journal of Education and Learning Vol. 9, No. 5;2020 the conditional probability that failure will occur in time period j, given that it hasn't happened in earlier periods.
Assume that failure happens at time t, which is random, and has the distribution function F(t). In our analysis, the failure event is non-retention. The hazard function is The survival function is the estimated probability that the failure event does not happen at time j. Therefore, mathematically, survival function can be written as We have two groups: UNIV and non-UNIV. The event is whether or not the student i is retained or not. The data also reveals that 42 percent of students from the non-UNIV group are not retained, compared to 40 percent of students from UNIV group. The survival analysis framework, thus, allows the analysis of the time to the event (non-retention).

Results
Our results for the parametric regression analysis are listed in Table 2. Columns 1 and 2 report the results for the Probit model while columns 3 and 4 report the results for Logit model. It should not be surprising to the reader that the results are very similar. In particular, we see that after controlling for other factors that might affect retention, taking first year seminar courses is associated with a reduction in the retention rate. However, the coefficients are statistically significant only at the 5 percent level. It is worth mentioning that we are not claiming that taking UNIV reduces retention, thus we are not stating that there is a negative causal effect between those variables. In order to obtain a causal effect, the UNIV variable should be truly exogenous, that is, uncorrelated with the omitted factors in our specification. However, omitted factors such as outside job opportunities and increase acceptance rates at the flagship Alabama universities are correlated with UNIV. More specifically, job opportunities improved due to a reduction in unemployment in the period under consideration, and improving job opportunities affect retention rates. We look at unemployment rate in Alabama as a proxy for job opportunities. A declining unemployment rate would indicate an improving labor market. Indeed, between 2011 and 2015, unemployment declined from an annual rate of 9.6 percent to 6.1 percent (Note 1). Moreover, the increase in acceptance rates at the largest public universities in the state also has an impact on retention in the smaller regional university under study. Our specification suffers from omitted variable bias (OVB) and thus the coefficient for UNIV is not an unbiased estimator of the true population parameter.  Vol. 9, No. 5;2020 Next, we use the Cox Proportional Hazard Model and add additional covariates that might influence our dependent variable (time to non-retention). The results are listed in Table 3. Our main coefficient of interest is UNIV, which captures the differential impact of UNIV course on retention rate. We add additional regressors such as age, sex, whether the student is white or black (reference group is other racial background), high school gpa (hgpa), math score, ACT composite score, English score. The difference between Model 1 and Model 2 is that we introduce a dummy variable for students who are older than 25 instead of our previous continuous variable for age. The results suggest that UNIV course lowers the hazard rate for non-retention by approximately 38 percent. The hazard rate is between 0.63 and 0.65 that of someone who has not taken UNIV. We add an interaction between being white and UNIV to test the hypothesis whether the UNIV course helped non-Hispanic white students more or less than students from other racial background. We see no statistically significant results on the interaction term. Moreover, the coefficient for univ is still significant, and the hazard ratio is around 30 percent lower than someone who has not taken the course. Model 4 introduces an interaction term between being female and univ. Similar to that of Model 3, we see no significant effect suggesting differentiating impact of UNIV on male versus female students. We get similar results regarding the impact of high school GPA on retention rate as the logit and probit model. This suggests that students with higher high school GPA are more likely to be retained. The hazard ratio is almost half for every unit increase in high school GPA of a student.

Conclusion
This paper analyzes the association between offering first year seminar courses on student retention rates. We utilize data from a small regional university in Alabama to test our hypothesis. Parametric estimations such as logit or probit models suffer from omitted variable bias and might lead to inconclusive results. To tackle this, we employ a survival analysis method and find that students taking UNIV have a better survival rate for retention. We also find that other macro variables such as labor market opportunities and competition from similar universities in the region might play a crucial role in student retention.
An important aspect of our paper is that we illustrate why it is important to review the context in which the data was obtained. Since students were not randomly assigned to UNIV courses, we cannot establish a causal interpretation to our results. Causal inference would require random assignment of students to UNIV courses, which might be difficult to implement due to legal barriers. We also illustrate the impact of omitted variable biases on regression results. The parametric analysis portrays a negative relationship between students taking UNIV and retention. Our results suggest that UNIV alone might not be capable of improving retention. Universities need to consider macroeconomic factors such as employment opportunities in the economy and competition from rival universities as complementary to first year seminar courses to improve retention rates.