Assessment of Grammatical Competence Based on Authentic Texts

This study illustrates how alternative assessment could be made observing both recognition and production skills of the participants who were English language students at the ELT department, University of Cukurova. In an English Contextual Grammar course, news articles were used as lesson material during the first term of the 2009-2010 academic year, and a midterm and a progress test were administered following a rather unconventional set of procedures. The study displays such procedures aiming to show how an effective and truly learner-centered approach to assessment of grammatical competence can be embraced yielding an alternative type to the conventional assessment methods in the field of English language teaching (ELT) through the use of target language news articles; it also reflects students’ perceptions about the alternative method presented in this study.


Introduction
Grammar teaching in any language educational program is essential.The question is thus not whether it should be taught or not, but rather how it should be taught.A relatively new view embraced by some educationists is the use of authentic texts drawn from everyday life representing the language employed by the society.Studies of such kind do exist in literature, yet lack an important aspect of the whole process, that is how assessment could be carried out without endangering validity, reliability and applicability.Therefore, with this study, we aimed to illustrate how alternative assessment could be made observing both recognition and production skills of the participants who were English language students at the ELT department, University of Cukurova.In an English Contextual Grammar course, news articles were used as lesson material during the first term of the 2009-2010 academic year, and a midterm and a progress test were administered following a rather unconventional set of procedures.This study comprises two main parts.In the first part, it tries to depict how an effective and truly learner-centered approach to assessment of grammatical competence can be embraced yielding an alternative type to the conventional methods in the field of English language teaching (ELT) through the use of target language news articles, and in the second part, the study illustrates perceptions of participants' regarding their previous grammar assessment and their current assessment models.

Grammar in Second Language Learning 2.1 Grammar Teaching Models
Current research in ELT focuses on grammatical instruction as well as on its importance in foreign language education.In this field, some research focuses on conscious attention to form or noticing (Rutherford, 1987(Rutherford, , 1988;;Schmidt, 1990;Tomlin and Villa, 1994), grammatical instruction and accuracy building (Long, 1988;Ellis, 1990Ellis, , 2002;;Norris & Ortega, 2000); others deal with focus on form vs. focus on forms (Long, 1991).Whereas focus-on-form involves teachers' attempts to draw learners' attention to grammatical forms in the discourse, focus-on-forms involves grammatical points in isolation (DeKeyser, 1998;Doughty & Varela, 1998;Long (2000).Within a pedagogic view, focus-on-form may be effective as long as the instructional design is learner-centered (Long, 2000).In line with this latest research, there appears alternative ways of treating grammar as interactional feedback (ex.Nassaji & Swain, 2000), textual enhancement (Schmidt, 1990;Sharwood-Smith, 1993), task-based instruction (ex. Ellis, 2003), to name some models related to our research.

Grammar Assessment and Authenticity
In line with recent research, grammar instruction has recently been playing a pivotal role in teaching and learning languages.The place of grammar in the curriculum and its nature in teaching are two rather debated issues in the field of language learning and teaching.Integration of form and meaning is becoming increasingly important in current research in the face of a growing desire for using authentic materials.It is essential that teachers, apart from sticking to the syllabus, incorporate authentic sources into their teaching of courses (Ellis 1994).Making use of authentic materials is not only conducive to learning but also, in a sense, is a must.Given that there are sharp contrasts regarding vocabulary and grammatical features in grammar materials and the language used in the real world, the need to study authentic materials in grammar becomes more evident.Authentic materials such as TV broadcasts, newspapers or native speaker dialogues are all real language, created with not a purpose of language teaching in mind.Across the world regarding this field, we hear calls for the use of "authentic" tasks, materials, and activities in language programs and courses.While authenticity has many different interpretations, its wide appeal seems to be based on the concern felt by teachers that students pass courses by merely studying made-up English, and yet fail to communicate outside the classroom (Widdowson 1996).However, the use of authentic materials is a substantial challenge for most curricula and many teachers since they often include vocabulary and grammar that may be too difficult for learners.Regarding advantages of authentic materials, thus we can state that by using such materials, learners are presented with actual everyday language, just as it appears in real life.Convinced by authenticity in an EFL environment, we have designed an individual based grammar course including assessment consisting of purely authentic materials retrieved from news articles by participants themselves, who were taught a course titled Contextual Grammar, 3 hours a week, 13 weeks in total, throughout the semester.Each week, two hours were allocated for class lecture and discussion and one hour for assessment.Students were informed that they would be assessed for each week's topic following the class and discussion hours, and that an average score would be obtained taking the mean of the 12 weeks' assessment scores, and that the final assessment would comprise all units covered during lectures.The process of individualized grammar instruction is presented in detail below (Ozkan, Bada & Genc 2007): 1 All the grammar points to be dealt with during the semester were decided on referring to the course book, Rhetorical Grammar by Martha Kolln (2003).
Based on these points, a syllabus was prepared and delivered to participants, who, in line with their preferences, brought texts retrieved from American and British dailies such as New York Times, Washington Post, The Independent, and the Guardian.The teacher, first, started the class by putting a statement on the board referring to that day's topic without any grammar rule, and later asked participants to find and underline in their texts similar patterns to the point indicated on the board.2 As a following step, the participants, in groups, discussed among themselves their underlined texts; by doing so, they were able to detect common characteristics of such texts.3 Then, from each group, a student came to the board and wrote sample sentences emerging from members' preferences; this way, all participants were able to observe targeted grammatical points expressed in different language with a different context and topics.4 As a final step, the participants in groups produced oral and written language including the focal grammatical point of that class session.While participants went through the steps above, the teacher provided assistance in terms of jargon terms and culturally-loaded vocabulary concerning the grammar point(s) dealt with in the news articles.To our belief, the delivery of this authentic-material-based course necessitated an alternative type of assessment which was fairly valid, reliable, practical, and authentic with positive backwash effect.Regarding validity, this type of assessment had got content validity for all the students were familiar and had covered the items throughout the semester.The assessment covered adequate samples of topics taught.Additionally, it had face validity since the items were representative of the grammar tasks in class, and included construct validity because it put theory into practice.Reliability was attained by times of repetitions of assessment utilizing different authentic news sources similar to those covered in class.By reassessing or redrawing, we could reinforce learning rather than trapping students.A valid and reliable test is useless if it is not practical (Bachman 1990).For this, we preferred a quick and an easy layout of assessment as well as an efficient use of time and ease in administering.Regarding authenticity, we preferred using the interactional ability approach.Here, students' language ability was engaged in the assessment task since positive 'backwash effect' could only be attained when and if assessment procedures reflected skills and abilities taught in the course (Bachman 1990).
To reflect what was done in class to the assessment process, we integrated recognition and production types of items into one sit-down assessment session.Here, there was a need to ensure that assessment had reflected the accurate usage of authentic language.In real life, people do not isolate grammar from other skills.To turn language tests into more authentic activities, for this assessment, we used an original text, which displayed not only what students knew about the language, but also how they used it.That is, this type of assessment included items to assess recognition as well as production skills.Below, is presented a sample of such an assessment: In Part A (recognition) on the sheet, students were expected to identify each phrase within the text, and in B (production), were supposed to form statements, each including a targeted phrase.Scoring criteria required production of grammatically and socially appropriate sentences, and scoring itself was out of 100.Since five types of phrases (noun, verb, adverb, adjective and prepositional) were aimed at in this sample assessment, students were expected to recognize (Part A) and produce (Part B) structures including targeted phrases.Contribution of both parts (A and B) to the overall score was even: 50% each.Throughout the assessment process, no points were deducted when students wrote inappropriate structures.However, misused items were YDI-112 Contextual Grammar 02.04.2010Assessment 1 -Fall, 2009-2010 Name: A) Read the text below and underline the phrase structures stating the type of the phrase underneath each underlined phrase.Take the sentence below as an example: John and his dog are very good friends.NP VP [VP+NP] Acupuncture 'can ease fear of dentists' By Jane Kirby, PA Tuesday, 30 March 2010 The Independent People who are terrified of the dentist could be helped by acupuncture, research published today suggests.Five minutes of acupuncture treatment in the top of the head cuts anxiety levels by more than half, the small study on 20 patients found.Sixteen men and four women with an average age of 40 took part in the research, published in the journal Acupuncture in Medicine.All had suffered from fear of the dentist for between two and 30 years.On previous dental visits, three patients had needed general anesthetic to cope with their fears while six others had required sedatives.In 14 cases, treatment had to be cancelled because the patient could not go through with it.The patients came from eight different dental practices in the UK and received the acupuncture from their dentists, who are all members of the British Dental Acupuncture Society.They had acupuncture needles inserted into their heads at acupuncture points GV20 and EX6, which have been reported to aid relaxation.Using a well-known anxiety reporting scheme, the Beck Anxiety Inventory (BAI), the patients' levels of distress were measured.Typically, scores fell from 26.5 to 11.5 after acupuncture and all 20 patients were able to undergo treatment.Some one in 20 members of the public have extreme anxiety when visiting the dentist while another 20% to 30% report moderate dental anxiety, the researchers said.The authors, from Sheffield and Denmark, said more studies were needed but concluded: "Acupuncture prior to dental treatment has a beneficial effect on the level of anxiety in patients with dental anxiety and may offer a simple and inexpensive method of treatment."B) B. Similar to those underlined structures, now write five sentences, and in each sentence underline only one phrase; for instance: in sentence 1, underline an NP; in sentence 2, a VP; in sentence 3, a PP; in sentence 4, an ADVP, and in sentence 5, an ADJP.Sentence 1: Sentence 2: Sentence 3: Sentence 4: Sentence 5: highlighted on the assessment sheets, and were later discussed with students.In order to find out how this model was perceived by participants, a two-item open-ended questionnaire, besides a semi-structured interview, was given to the participants asking to highlight their views on a) previous grammar assessment, and b) the alternative grammar assessment dealt with in this study.

Participants
The participants in this study were 125 freshman students at the ELT Department of University of Cukurova, who were all asked to analyze assessment in their contextual grammar course.The participants, ranging from 18-19 years of age, had a fairly standard level of English due to a fairly standard university entrance exam they took prior to their placement in their present department.

Data Collection Tools
The data of this study was gathered through two techniques: an open-ended written questionnaire, and a semi-structured interview.3.2.1 Open-ended Questionnaire With the two-item essay-type questionnaire, we aimed to highlight some perceptions and beliefs of language learners regarding assessment of grammar.While Item 1 sought the participants' beliefs and perceptions in their previous grammar learning and assessment experiences, Item 2 was designed to detect perceived characteristics of learners' current assessment type within the contextual grammar course.The following questions were asked both in the questionnaire and the interview: 1 How would you define your previous grammar assessment types in terms of advantages and disadvantages? 2 How would you define your current assessment type in the Contextual Grammar course in terms of advantages and disadvantages?3.2.2Interview The interview, with a total duration of ≈470 mins, was performed in a one-to-one fashion within two weeks following the administering of the questionnaire.Of the total 125 participants, 118 were each interviewed for approximately 4 minutes.Seven were unable to attend the interviews due to some personal reasons.The data gathered from the interview was recorded and transcribed, each item having been coded and similar codes having been aggregated into themes.During the interview, the following issues were taken into consideration.
 One question was asked at a time.
 The interviewer verified unclear responses. Students were asked open-ended questions. Leading questions were avoided, and unbiased questions were preferred. Follow-ups and probes were used.

Data Analysis and Discussion
In this mixed-method-designed study, content analysis was carried out.Each response sheet was thoroughly examined, and emerging themes in the questionnaire and the interview were categorized forming convergence topics.As a first step, for each re-emerging theme (time-of-mention=ToM), a frequency test was run utilizing the Statistical Package for Social Sciences (SPSS) computer program.At a second stage, a chi-square test was run in order to identify any observable significant difference in the dispersion of overall themes for each item.
The common themes emerging from the analyzed data were presented in tables 1 and 2, Table 1 reflecting previous experience of participants, and Table 2 participants' current experience.

Insert Table 1 Here
The first item in the questionnaire given at the end of the semester aimed to detect the participants' perceptions regarding assessment in their previous grammar courses.For this item, six themes emerged in the participants' responses.As can be observed from Table I, rule-based questions emerged as most prominent (31.5 %) in both the open-ended questionnaire and the interview.Most participants believed that they had to memorize the rules before exams to be successful, a process which led to frustration.Other themes emerged with differing percentages: the theme multiple-choice format (19.4 %), memorization (15.4 %), no production (15.2 %), not meaningful (11.4 %), and easy (7.0 %).The participants voiced potential reasons leading them to be exposed to multiple choice formats mostly.They stated that this may have been associated with the standard examination to enter university.Since the entrance exam in Turkey is totally based on a multiple choice test, "grammar teachers may have been the product of this system."However, in reality, that they "had always wanted to produce language in class as well as in exams."Otherwise, they "would find classes monotonous and devoid of meaning."Without a meaningful unit in grammar classes, some of the participants believed that both classes and assessment were rather easy and did not create any challenge for them.Below, are presented some excerpts from the participants' verbatim: We had to memorize the grammatical rules in order to be successful in the exams (Participant 16; rule-based questions; memorization).The most common assessment format was multiple choice; we did not produce language enough (Participant 86; multiple-choice format; no production).The topics were so abstract nothing is concrete in our life (Participant 54; not meaningful).
The examples in the class were very easy and out of context (Participant 8; easy).Regarding participants' beliefs about their current assessment in grammar, nine themes were obtained from the data.The themes presented in Table 2 display a rather varying frequency of emergence.From the table, it can clearly be seen that the participants regard this experience a rather positive alternative assessment model in terms of testing in general and for its positive washback effect on learning.Among the nine themes emerged, the theme difficult appears to be a negative one, since some minimal percentage of participants believed that authenticity caused some comprehension difficulty.Insert Table 2 Here Table 2 illustrates that students mostly agreed with the idea that this type of assessment could lead to positive perceptions.The theme assessment reflects lecture, by 17.1 %, emerged as most prominent among others.This may be due to the fact that, naturally, all learners would probably prefer to be assessed for what they may have covered in courses and not for what they have not.Face and content validity seem to have been well appreciated by participants in this type of assessment.In the interviews, they added that they could not differentiate assessment form the course itself, such that they were learning as well as being assessed.Throughout the assessment session, most participants expressed that the assessment was as an extension to class lectures, and that it included tasks similar to class activities, which enhanced face and content validity, both regarded as two essential criteria to meet the standards of effective assessment (Heaton, 1988).The theme authentic texts and samples within grammar assessment was mostly regarded as positive since the participants could focus on meaning rather than rules.In the transcribed data, we were able to see that the most frequent complaint was about the rule-based grammar instruction, which was the reality in most of participants' previous grammar experience.With a text they believed that they could feel safer and could work out the language more effectively referring to meaning without having to focus on abstract rules.The participants also mentioned that they could understand the targeted grammatical structures better in authentic texts if they were bold-underlined as they were in their current grammar course.This result is parallel to the results obtained by Doughty and Varela (1998), where it was emphasized that highlighting certain features of grammatical input helps learners to easily notice target structures.In their comparison to previous assessment, composed mostly of traditional multiple choice items, participants could not analyze the language; all they could do was just memorize rules to prepare for exams.In this traditional type of assessment, they found the process to be meaningless and very remote from representing real-life language.With the present experience, authenticity provided them with the opportunity to use language naturally in a real context where they could broaden their horizons in a target language setting more effectively compared to course book materials where the language was probably artificial and information outdated.Ellis (1994) does well emphasize the significance of authenticity by suggesting that teachers, apart from sticking to the syllabus, should incorporate authentic sources into their teaching of courses.The themes meaningful (12.7%) and informative (12.0%) were also found to be positive in elicited responses.The participants believed that what is taught meaningfully in class should be assessed in a meaningful fashion.Due to being meaningful, participants believed that they learned much about politics, history, economics and culture of different countries.No memorization (11.1 %) is another theme reflecting participants' positive perception about the current assessment experience.Some highlighted the shortcomings of memorization in their previous grammar tests and added that in their current assessment they did not have to memorize since they could digest targeted grammatical features with great ease.An important point voiced by some participants concerned content of university entrance exam in Turkey.Participants believed that this exam was rather traditional and adversely affected their production and creative thinking, a perception highlighted in the theme higher level thinking skills (10.9%).They held the belief that using authentic news for assessment could enable them to understand grammatical points without getting bored or frustrated.Related to this theme, some participants expressed positive ideas regarding the type of questions posed in the assessment.These ideas are well illustrated by the theme authentic-text-based questions (10.6%).In their previous assessment they stated that they were exposed to rule-based questions only.They believed that by being exposed to this opportunity of alternative assessment, they were able to comprehend and answer questions due to the presented meaningful language in assessment sheets.Some 6.6 per cent of the participants thought that such assessment was somehow detailed.They believed that it seemed rather long and required detailed reading.The only seemingly negative theme to emerge in the elicited data is regarding this type of assessment as difficult (5.4 %).In this category, the participants expressed views suggesting that the difficulty in this assessment arose from reading comprehension and meaning association with targeted grammatical structures.The following verbatim excerpts related to these nine themes are presented below: I can learn new things even in our exams and this makes me happy (Participant 43; informative).Texts are difficult to understand because there were so many new words (Participant 98; difficult)

Conclusion
If one goal of language instruction is to include teaching students to produce grammatically and socially appropriate constructions, then a compelling case can be made for teaching and assessing grammar.Instead of viewing grammar as a static system of arbitrary rules, it should be seen as a rational and a dynamic system comprised of structures characterized by three dimensions of form, meaning, and use.This study has implemented an alternative type of grammar assessment and portrayed perceptions of EFL learners.The results suggest that learners feel more successful and comfortable if assessment reflected classroom language with content and face validity.Learners who are exposed to such grammar training and assessment do believe that the process is challenging, meaningful, and informative not only in terms of targeted structures but also concerning current issues such as politics, history, economics and culture.Implementation of authentic texts in classes for teaching grammar or any other skill necessitates an authentic assessment approach where both form and meaning are equally assessed.Based on the premise that language learning and assessment are positively interrelated, we thus suggest that our model may prove to be one alternative assessment to traditional approaches in the field of language learning and teaching.
The lectures and the exam are very alike so having an exam or quiz in class do not bother us (Participant 27; assessment reflects lectures).This course makes me think deeply about the structures and their positions in authentic sources (Participant 13; authentic texts and samples).Meaningful contexts make the topic easier to comprehend.Now I can understand what is adjective what in noun much more effectively (Participant 7; meaningful).We improve our knowledge especially in politics because we are dealing with news mostly (Participant 117; informative).I understand every topic in the class there is not anything abstract (Participant 17; no memorization).Our class is a very detailed course we learn the basis of each topic in the syllabus (Participant 45; detailed).

Table 1 .
Participants' previous grammar assessment experience