The Impact of High School Exit Exams on ESL Learners in British Columbia

This essay explores the validity of including ELL students in British Columbia provincial high school exit exams. Data and a sample exam from the BC Ministry of education are used to scrutinize the practice of using BC provincial exams for high school exit decisions regarding ELLs. A comparison of failure rates of ELLs and mainstream students provides evidence for the inappropriateness of the BC provincial exams for ELLs. Ministry statistics reveal ELL failure rates ranging from twice to four times those of all students. An analysis of a sample exam found on the ministry website reveals confounding of content and language in the exam such that it is impossible to determine whether the exam is measuring language proficiency or content area knowledge. This exam shortcoming casts serious doubt on the legitimacy of using the provincial exam as an assessment for ELL learners. Possible negative consequences of including ELLs in these exams such as increased likelihood of school disappearance are discussed. Suggestions including providing alternative forms of assessment for ELLs are provided to facilitate managing the challenge of assessing ELLs attempting to obtain a high school diploma in BC.


Introduction
North American schools are becoming increasingly diverse (Goldenberg, 2006) which means that policy makers ignore at their peril the challenges that diverse learners face as they learn to cope with our educational systems.These demographic patterns are the same here in Canada.In fact, Statistics Canada reports that, by 2031, one quarter of Canadians will be immigrants from another country (Statistics Canada, 2010).
Commenting on the American context, Menken (2008) makes a strong case that the era of No Child Left Behind and increased accountability has caused English language learners disproportionate academic hardship.In particular, she focuses on the devastating consequences that including ELL students in high-stakes standardized tests have had across the United States.She concluded that the cultural assumptions, linguistic complexity of these tests and the lack of sufficient accommodations explain why English language learners usually do not perform as well as native speakers (p.96).The British Columbia high school provincial exams are unfair for English language learners (ELLs) because they were not constructed to assess ELLs yet high stakes decisions for ELLs are being made based on these exams.

Background of High School Provincial Exams in BC
Standardized testing is not a new phenomenon in British Columbia.Steffenhagen, (2008) reports that, according to Jerry Mussio, (formerly of the Ministry of Education and Canada's representative to PISA), provincial exams have a very long history in BC.In fact, provincial examinations of reading, writing and math were first introduced in 1876 and were regularly administered from 1925 until 1973.At that point, then Education Minister Eileen Daily cancelled standardized testing due to worries over the use of the tests.As part of Socred government's efforts to "restore confidence" in public schools, Grade 12 exams were re-introduced in 1984.And they have remained in place ever since (Steffenhagen, 2008).At present, the debate in the province over the fairness and validity of these tests continues.
In recent decades, debates over the appropriate uses of standardized test results have intensified as conceptions of test validity have changed quite dramatically.Messick's (1989) introduction of the notion of consequential validity and the realization that validity is not an inherent trait of a test (Koretz, 2008) have compelled scholars and test designers to rethink previously unquestioned assumptions about testing.Based on some of these insights, the issue to be addressed here becomes not about the validity of BC provincial exams per se but rather the validity of their use with ESL students and their possibly deleterious impact on these learners' school trajectories.
To earn a "Dogwood" diploma in British Columbia, all students must take provincial examinations that presumably cover the curriculum of grade 10 English, Science, and Math.They must also write grade 11 Social Studies (or BC First Nations Studies 12) and grade 12 English.The exam requirement as a prerequisite to earning a high school diploma makes them potentially high-stakes tests.That is, if students cannot pass these required tests, they will jeopardize their chances of obtaining the valued "dogwood" diploma.
In addition to containing content and language that ELLs could not realistically be expected to know, these exams do not attempt to allow ELLs to display what they actually do know.Essentially these learners are bringing funds of knowledge (Moll, Amanti, & Neff, 1992) that are being ignored and undervalued by these tests.Corson (1998) points out "often culturally different children come to school knowing very different kinds of things from other children.When they meet the assessment methods that teachers use, they are sometimes asked to display knowledge about 'x, y' when what they really know about is 'y, z'" (p.169).That is, they are being punished for ignorance of things that they would not ordinarily be expected to know in their own cultural context.Shohamy (2000) notes that, in the past, test takers did not really have any rights.
It was understood that there was specific knowledge that the test taker should have, but that the body of knowledge was defined by those who wrote the tests and that the test taker was expected to comply with their decision (p.154).
Based on the foregoing discussion, it could be argued that for these ELL BC provincial exam test takers not a lot has changed.One indicator of the inappropriateness of these tests for ESL students, they are only mentioned in the handbook of administration procedures to explicitly disallow them from using translating dictionaries.This reference to ELLs reveals two policies for testing these students.They are included in the exam yet they receive no accommodations (e.g., first-language versions or bilingual dictionaries) that might allow them to perform at their optimal level.Some might argue that it is appropriate to include ELLs in the provincial exams because they would question whether in fact these high school exit exams could be classified as high-stakes tests.Mathison (2004) defines high-stakes as "tests that have serious consequences attached to the results -these consequences may be for students, teachers, principals, schools and even states" (p.122).This definition would seem to be appropriate for the BC provincial exams because if test takers do not pass the tests, they will jeopardize their chances of receiving their high school diploma.

Previous Research into the Consequences of High School Exit Exams for ELL Learners
The social costs of testing have received increasing attention from the scholarly community.Researchers are becoming progressively more aware of the negative social impact large-scale high-stakes tests can have on the lives of learners -particularly those who are most vulnerable -when the results of these tests are used to make decisions that unfairly limit the life choices of these learners.Shohamy (2007) asserts that "[w]e need to examine the ramifications of tests, their uses, misuses, ethicality, power, biases, and the discrimination and language realities they create for certain groups and for nations, and we need to use a critical language testing perspective [emphasis in original]" (p. 144-145).This is especially true with a test that serves to identify a learner as a high school graduate or not.
Research into the use of high school exit exams with minority learners has accumulated some surprising findings which have led to some pessimistic conclusions.Darling-Hammond (2010) surveyed the research into high-stakes tests and minority students.She noticed a disturbing relationship between grade 9 retention rates and levels of performance on the grade 10 standardized tests.She reported that "many of the steepest increases in test scores occurred in schools with the highest retention and dropout rates" (p.94).She cited a study by Wheelock which found that "some schools improved their scores and reaped [state] rewards by keeping students out of the testing pool or out of school entirely" (p.94-95).Thus, it appears that schools raised their test scores but removing the weakest learners from the testing pool.This issue will be revisited in the discussion of test consequences for BC ELLs.
Those in favor of continued testing of ELLs with the provincial exam might protest that the case of the BC provincial exam is somewhat different than many of the high-stakes state exams in the US.They might point out that BC test takers' fate do not entirely depend on the results of the exam.In fact, in British Columbia, exam performance and grades in related class are weighted at 20% exam/ 80% course grade for those in grades 10-11 and 40% exam/ 60% course grade for those in grade 12. On the surface, this appears to be fairer for test takers except for the fact that those groups who fail the test at a higher rate than the mainstream population still stand a greater chance of not graduating, even if their grades are comparable.This potential disparity in results calls to mind Mathison's (2004) observations that "the danger is that standardized test results will be weighed more heavily than they ought to be, that decisions once made cannot or will not be reversed, and that other compelling information may be ignored" (p.122).In the case of BC, the provincial exams are not the only factor taken into account when deciding whether or not a student qualifies to graduate but this "snapshot" of ELL performance using a tool not designed for that purpose can have serious consequences for those who perform well enough in their coursework but do not pass the exam.This is particularly the case for the grade 12 Language Arts exam.

Examination of Failure Rates and Exam Content
2.1 ELL Failure Rates on British Columbia Provincial Exams over the Past 5 Years Shohamy (2000) was one early observer to raise the alarm about the effect that unfair testing practices could have on disadvantaged and vulnerable learners.She investigated the implementation of tests of Arabic and English in the Israeli context and concluded that "…test takers are the true victims of tests.They suffer badly from tests in this unequal power relationship between the test as an organization and the demands put on test takers" (p.154).Similar patterns of the particularly damaging effects of using high-stakes tests with minority or disadvantaged learners have been reported in the American context as well.In her review of related research Mathison (2004) noted that: …there are clear patterns illustrating who is most likely to be subjected to these tests.High-stakes testing is disproportionately found in states with higher percentages of people of color and living in poverty…of the eighteen states that require passing a standardized test to graduate, eleven are in the south, that is, states with substantial minority populations (Mathison, 2004, p. 122).
The issue here is the fairness of using the BC provincial exams to decide whether ELLs can graduate from high school in British Columbia.Two appropriate pieces of evidence to determine the validity of this practice would appear to be ELL test takers' performance on the exams relative to their peers and the language and content of tests themselves.The following chart, taken from the BC Ministry of Education website, illustrates ELL performance for the examinable subjects over the past five years.* Data obtained from http://www.bced.gov.bc.ca/reports/pdfs/exams/req/prov.pdf** This is considered to be the intermediate level between "Applications of Math" which is geared more toward college and trades preparation and "Principles of Math" which is aimed toward preparing students for university-level study of math.Table 1 demonstrates that over the past five years across all required subject areas examined by BC provincial exams, ELL students have consistently had failure rates that are approximately double those of all students.These numbers in table 1 were calculated by subtracting the number of students who were reported to have passed the exam from the total number of students who took the exam during each academic year.The results for grade 12 English are a particularly dramatic example of the discrepancy in test scores (see figure 1).From the year 2006/2007 through to 2010/2011, ELL student failure rates were four times those of all students.Even for mathematics, a subject that is thought to require less English proficiency than others, ELL failure rates are approximately double those of all students.These statistics indicate that the practice of including ESL students in these exams must be called into question.
Either ELL students are systematically receiving drastically inferior instruction or the test is not a valid measurement of their abilities.If it were, their failure rates would be equivalent to those designated as non-ELL.Given that ELL students consistently have almost double the number of failures (and almost quadruple in English 12) on average, one has to question the fairness of ELL students' inclusion in these assessments.

Bias as an Explanation for Discrepancies in BC Provincial Exam Failure Rates
One possible explanation for the large discrepancies in failure rates is test bias.At present, there are no published results of DIF analyses for the provincial exams that examine the performance of ELLs as compared with English speakers.Therefore, currently, the only option for identifying possibly biased items or content is to analyze exams to discover any content that might potentially cause ESL learners to perform differently than other test takers.
An analysis of a sample grade 12 English exam provided on the Ministry of Education website demonstrated several specific test features that reveal the potential for these exams to be unsuitable for ESL learners for at least two reasons.First, the exams assume a great deal of cultural knowledge that ELLs could not be expected to have.The four year limit on an ESL designation in BC seems insufficient to acquire the nuanced and sophisticated cultural knowledge assumed by these exams.The grade 12 English (not literature, which is optional) exam found on the ministry website provides an illustration of the complex cultural knowledge required to perform well on the exam.One question worth 17% contains a reading passage about Andy Warhol's first visit to Canada.To answer the questions, ELL learners must know about who Andy is and why it is ironic that Andy's sculptures were considered taxable goods by Canadian customs officials.A second difficulty with the passage is that it is filled with colloquial words and expressions like "nada," "deadpan comedy," "a total dud" and "drop 80 works" among numerous others.Certainly the native speakers for whom test was designed should know these expressions but it is questionable whether designated ESL learners who were in the country for four years or less should.
These test features demonstrate that these tests do not attempt to separate the confounded factors of content knowledge and language proficiency for ELL learners.However, for these tests to be a considered the valid measure of content knowledge for ELL students that they purport to be -otherwise they should not be used with these learners -separating language and content knowledge is exactly what the exam must do (Solano-Flores & Trumbull, 2003).
McNamara and Roever ( 2006) make an excellent point that traditional psychometric approaches to test validation have serious limitations in terms of their ability to comment on the social dimensions and consequences of language tests.They note that "…researchers need to go beyond the test booklet to understand fairness and bias in its social context.After all, DIF analyses only detect bias, but the factors that cause advantages and disadvantages for groups of test takers and bias their educational opportunities lie in the larger social context" (p.128).The ironic fact is that the social dimension of tests is what really matters the most.Tests treating learners differently is worthy of attention but tests that forever alter the course of test takers' lives demand close scrutiny.

Consequences of Failure for ELLs in British Columbia
At this stage it is difficult to ascertain what the direct consequences of failing these exams are for ELLs in British Columbia.However, based on some relevant indicators some speculations can be made.Looking at the consequences of failure of exit exams in other contexts might provide some insight into what may be happening here.
Heubert and Hauser (1999) observe that "ELLs are particularly vulnerable to high-stakes decisions based on test results; [for instance] tests used to determine high school graduation…" (cited in Menken, 2008, p. 97).Menken (2008) also highlights that [w]hile passing the Regents is challenging for anyone learning the English language, it is particularly so for older students who arrive in the United States during high school and students with limited literacy skills either in their native language or in English (p. 112).
This situation is similar to what many ELLs in BC face.Older students come during their secondary years and they are unable to learn enough of the requisite academic language to pass the exams in the short time frame they have.As a result, they do not perform well on exams and their graduation status is placed in jeopardy.
The most drastic consequence of high stakes tests noted by several scholars was the increased risk of students dropping out of school due to failure of a high-stakes high school exit exam.Indeed, it has been found that "…graduation tests…are having substantial impact on the numbers of students who are dropping out of school" (Mathison, 2004, p. 124).Darling-Hammond (2010) reported on the results of a large-scale summary of research conducted by The National Research Council.She noted their conclusion …that low-performing students who are held back because of their test scores do less well academically and are far likelier to drop out than comparably achieving students who are moved ahead in school with grade retention increasing the odds of dropping out by as much as 250%" (p.76).
This statistic certainly gives cause for alarm.It is even more disturbing that schools may even be encouraging low-performing students to leave.Menken (2008) mentions her great concern "that because of the challenges that Regent exams pose for ELLs and their high-stakes consequences, the exams act to push these students to leave school, either of their own volition or with their school's encouragement" (p.108).Certainly the original aim of these tests was to compel teachers to do more for at-risk students, but the outcome in some cases seems to be compelling them to do less.
This relationship between failure of high-stakes and dropout rates in other contexts may also exist here in BC.
The disappearance rates of English-as-an-additional-language (EAL) learners from schools across Canada (Toronto, 53%) (Radwanksi, 1987), (Calgary, 73%) (Watt & Roessingh, 2001) and in the Lower Mainland (60%, Gunderson & Murphy Odo, 2009) are disturbingly high especially when they are compared to the official provincial high school dropout rate of 6.2% (StatsCan, 2010).Of particular relevance here is the data from the Lower Mainland of BC.Gunderson (2007) conducted a large-scale study of immigrant students in the Vancouver area and observed a dramatic "disappearance" rate of ESL 54% of students from the district who could not be accounted for.It does not seem unreasonable to ask whether students might be disappearing from BC schools for similar reasons.
Mathison (2004) made a startling observation that "students who have by other indicators done reasonably well in school are disproportionately influenced by poor test scores to drop out of school" (p.127).That is, the students who often show the most promise at school are those who are the most negative affected by these tests.At this stage, it is difficult to determine whether this pattern is repeated in Canada.Gunderson's (2007) investigation of ELL student achievement in Vancouver-area schools revealed that between grades 8 and 12, Canadian students had an average GPA of between 2 and 2.5.Likewise, ELL learners had an average GPA of between 2 and 2.5 though their GPA in 11 and 12 dipped below 2 in English and social studies.This gives some indication that, in terms of class grades in the examinable subjects, ESL students are not performing considerably differently than Canadian students.If that is the case, their failure rates on the provincial exams being twice as high as Canadians coupled with the high disappearance rates should be some cause for concern.
The potential connection between dropout rates and exit exams is disquieting but there are other more insidious consequences as well.Other examples of the deleterious effects that high-stakes tests have been found to have on ESL students include "… ELLs attending high school for more years, attending more classes per day, and attending more after-school and tutoring programs than native English speakers" (Menken, 2008, p. 105-106).
Thus, it appears that in their struggle to persevere ELLs are having to work much harder than their mainstream peers.These observations overlook the fact that after-school tutoring is the luxury of those who are not occupied with part-time jobs to help the family make ends meet.In this way, those ELLs who are disadvantaged in terms of social and economic capital arguably suffer even more disproportionately when compared to ELLs who have access to more resources.
Of course the struggles that ELL students face with these tests are complex.They are compounded by the fact that the label ELL itself, though at times useful, can also mask the widely diverse needs and access to economic and social capital that its members have.On the one hand, many ELLs struggle to acquire the language they need to do well at school but there are often vast differences in the amount and kinds of support to which they have access.In his study of ELLs in the Vancouver area, Gunderson (2007) noted that while many ELLs struggled in school there were also those from affluent backgrounds who were able to manage with these exams because they are able to hire after-school tutors to help them.This phenomenon underscores the wide diversity within the ESL label.
A final potentially dangerous effect of these high-stakes tests is on ELLs' identities.The core of who they are.Foucault argues that tests can shape our identity in powerful ways.Tests have the power to tell us who we are and we often believe them.McNamara and Roever (2006) contend that "…tests become technologies of subjectivity.They act as mechanisms both for the definition of subject positions and for the recognition of subjects.Tests create the identities they measure " (p. 196).This observation is particularly important when considering high school exit tests.If tests create the identities they measure then failing a test that purports to measure the identity of high school educated English user sends the message that the test taker is not suited for secondary -or post-secondary -education.When ESL learners are defined, and come to define themselves, as deficient there must certainly be negative psychological effects.Based on the confluence of findings in her research, Menken (2008) reported that there are.She states "a primary theme that repeatedly emerges in interview data with administrators, teachers and students is that high-stakes tests place tremendous pressure on ELLs and failing the exams negatively impacts their self esteem" (p. 113-114).This crippling stress and lowered self esteem are very likely the main contributors to many ELLs' decision to leave school without completing.

Addressing These Issues with BC Provincial Exams
Opponents might attempt to argue that if the system is so unfair it is the responsibility of the test takers or their advocates to do something to change it.However, as Shohamy (2000) explains, speaking out against flawed and unfair testing practices …requires a lot of courage on the part of test takers to battle powerful testing organizations, especially in a high-stake test where results are so detrimental.This explains why individual test takers rarely appeal.There is a feeling on the part of individuals that there is just too much to lose (p.155).
Additionally, many immigrants are not knowledgeable of the North American education system so they do not know how or where to go to express their displeasure about the current situation (Louie, 2001).Similarly, they may also lack the social or political capital to actually effect some change if they do complain (Louie, 2001).
At present, the best hope appears to be for concerned stakeholders and community members to engage in public debate about the fairness of these assessments for vulnerable or disadvantaged groups.Solutions have been proposed to solve similar problems in other contexts.Reviewing some of these ideas may provide useful guidance for how these issues might be addressed locally.
Probably the most practicable adjustment to make these tests more valid for ESL learners is to not include ELL learners in this assessment and to design a separate test that more accurately reflects their abilities (Solano-Flores and Truball, 2003).This more resource intensive option could also be an adaptation to the existing test to better meet the needs of ESL students.Possible test versions could be a locally-developed assessment that takes ESL learners' culture and language into account or a dual-language test.Of course, the feasibility of this practice is questionable because, to be equitable, developers would need dual-language assessments for all language groups.If those responsible for using the tests lack resources to design exams that accurately measure ELL students' capabilities, they should be prepared to conduct research to identify appropriate accommodations (e.g., the use of dictionaries or more time) to ensure as fair an assessment as possible.Access to suitable accommodations may help ELLs better demonstrate their knowledge while allowing the testing process to remain economically feasible.
Another option would be to rely on alternative assessments.These might be based on some type of portfolio assessment that allows students to demonstrate their areas of strength.A final option might be to weight the results of provincial exams and class grades differently for ESL or other potentially-disadvantaged groups.As was noted above, it appears that ESL students do not necessarily achieve vastly differently than their mainstream peers in terms of the grades that they receive in their classes.Providing a greater weighting for their class grade may give students who do poorly on the test a chance to offset their test score with their better class grade.This would allow them to pass and graduate.
More research into the consequences of these tests for these learners is needed.In particular, the ultimate fate of ESL learners who have difficulty passing these tests needs to be learned to ensure that they are not falling through the cracks and that the promise of the better life they came to Canada to pursue does not turn out to be empty.Policymakers may cling to the misguided notion that including everyone in these tests is somehow fair.However, we should all bear in mind the old adage that fairness is not treating everyone the same, it is giving everyone what they need to thrive.We need to learn what these students need to thrive and give them that.

Figure 1 .
Figure 1.Comparison of percentage of failures on English 12 exam during 2006 through 2011

Table 1 .
Percentages of Provincial Exam Failures for all Students and ESL Students*