A Needs-Based Approach to the Development of a Diagnostic College English Speaking Test

This paper investigated the current situation of oral English teaching, learning, and assessment at the tertiary level in China through needs analysis and explored the implications for the development of a diagnostic speaking test. Through random sampling, the researcher administered both a student questionnaire and a teacher questionnaire to over 200 students and 30 teachers respectively. Results revealed that students regarded the use of communicative strategies, range of vocabulary and fluency as the most difficult things in their oral English learning. Results also demonstrated that the majority of college English teachers paid much attention to oral English teaching. However, most college English teachers seldom administered classroom oral assessment, and their practices of classroom oral assessment were considered as lack of systematic and standardized procedures. Implications for further improvement in oral English teaching and assessment were also discussed.


Introduction
With regard to oral English teaching and learning in China, the development of a diagnostic English speaking test with high reliability and validity will enable teachers to diagnose the strengths and weaknesses of students' English speaking ability, which can help them to determine appropriate types and levels of teaching and learning activities and improve teaching efficiency.However, most of the oral English tests in practice are oriented towards certifying students' oral English performance rather than diagnosing their English speaking ability.The College English Test-Spoken English Test (CET-SET), a large-scale high-stakes standardized oral English proficiency test for college students in China, is an example of such kind of test.The CET-SET test takers are awarded a graded score ranging from D to A+ (National College English Testing Committee, 1999).Although a detailed grade description is provided on the test certificate, there is little information about individual test-takers' strengths and weaknesses in oral English.Among the few diagnostic tests that are available, DIALANG (Alderson & Huhta, 2005) is a standardized language assessment system which provides detailed diagnostic feedback to individual test-takers.DIALANG is an online test system containing tests of reading, listening, vocabulary and structure in 14 European languages.However, it can be seen that the focus of the assessment system is on the measurement of test takers' receptive abilities, instead of productive abilities like speaking and essay writing.In addition, Pearson Test of English Academic (PTE Academic), an academic English proficiency test, also provides a detailed score report to test takers in terms of their Overall Score, scores of Communicative Skills and Enabling Skills.The score report displays the scores in a graph which allows test takers to quickly see their strengths and weaknesses, and how each skill relates to their overall performance.
In this study, an extensive needs analysis is first to be carried out to investigate the current situation of oral English teaching, learning and assessment at the tertiary level in China.The implications of the needs analysis for the development of a diagnostic college English speaking test will also be discussed.Most importantly, types of information to be provided as feedback to test-takers and methods of providing the feedback will also be explored.
The specific questions to be addressed in this study are as follows: 1) What is the current situation of oral English teaching, learning and assessment at the tertiary level in China?
2) Is there an urgent necessity for the development and use of a diagnostic college English speaking test for oral English teaching and learning at the tertiary level?
3) What are the implications of the needs analysis for the development of a diagnostic college English speaking test?

Method
The design of both the student questionnaire (SQ hereafter) and teacher questionnaire (TQ hereafter) was first described in this section.

Design of the Questionnaires
The design of the questionnaires (SQ and TQ) was based on a synthesized view arrived at through a review of the literature on communicative language ability, learning difficulty, speaking test tasks and rating criteria (e.g., Bachman, 1990;Bygate, 1987;Fulcher, 1997Fulcher, , 2003;;Luoma, 2004;Munby, 1978;Weir, 1993) and two unstructured interviews with teachers and students from a top-level national key university.For selected-response questions used in the surveys, three-point or five-point Likert scales were adopted.Both the SQ and TQ were written in Chinese.They began with a statement of the purpose of the survey and an elicitation of subjects' background information.For teachers, the background data include mainly their College English teaching experiences.For students, the background data include mainly their English learning experiences at college.
The TQ survey aims at finding out the teachers' evaluation of undergraduates' oral English proficiency and their methods of teaching and assessing oral English in class.The TQ is composed of three parts, namely, teacher's evaluation of their students' oral English proficiency and learning difficulties, methods of oral English teaching, methods of oral English assessment in class (see Table 1).The purpose of the SQ survey is to investigate students' evaluation of their oral English proficiency and learning difficulties, and to elicit their comments on oral English teaching and assessment.The SQ is composed of four parts: students' self-assessment of their oral English proficiency and learning difficulties, learning motivations, learning methods, and comments on oral English teaching and assessment (see Table 2).Both questionnaires were piloted by asking four potential respondents to respond and react to a draft of the questionnaire, and encouraging them to make suggestions and comments.Based on the feedback collected in the pilot surveys, the researcher made minor revisions to the format, wording, and item order of the questionnaires.

Participants and Administration of the Questionnaire Survey
A total of 200 non-English major college students from three universities took part in the SQ survey.Among them, 102 students were from a top level national key university, 55 from another national key university, and 43 from a provincial university.As for the students' majors, 50 majored in material science, 100 in arts and humanities, and 50 in economics and management.Apart from one participant who did not complete the questionnaire, all the remaining 199 students produced valid questionnaire data in the survey.
As for the TQ survey, a total of 30 teachers from the same three universities as the student respondents, 20 from the two key universities and 10 from the non-key university, participated in the survey.Except for one teacher who didn't complete the questionnaire, the remaining 29 teachers provided valid data in the survey.
Altogether, the raw data collected in the present study included 199 student's questionnaire and 29 teacher's questionnaire.To facilitate the data analysis, the researcher sorted out the quantitative data using SPSS 18.0.All the statistical analyses were performed using SPSS 18.0.

Data Analysis and Discussion
The results of data analyses of both SQ and TQ were reported in detail in this section.The demographic data indicated that the students' English learning experience ranged from 6 to 15 years.On average, they have been learning English for 9 years.All the student participants passed the CET-4 and would take the CET-6 soon.
Responses to SQ Question 1 showed that students' self-evaluation of their overall oral English proficiency was lower than what was expected of them (see Table 3).Among the 199 students, only 10 (5.2%) rated their oral English proficiency as upper-intermediate.None considered their level of oral English proficiency as advanced.About 89 (46.4%) rated their oral English proficiency as intermediate and another 85 (44.3%) lower-intermediate.And the teachers' rating of their students' oral proficiency was slightly better than students' evaluation (also see Table 3).Although only 4 (13.8%)teachers thought that their students' oral English proficiency level was at the upper-intermediate level and non at the advanced level, the majority of them (62.1%)rated the level of their students' oral English proficiency as intermediate.Note: The scale used for the questions is: 1=very poor, 2=poor, 3=so so, 4=good, 5=very good.
Following an overall evaluation, students were asked to evaluate their linguistic skills and knowledge (see Table 4).Most students thought they were reasonably good at pronunciation, grammatical accuracy and intonation, whereas fluency, range of vocabulary and use of communicative strategies were regarded as the weakest aspects of their oral English ability.
As for their learning difficulties, students considered pronunciation, intonation and grammar as the least difficult, whereas use of communicative strategies, range of vocabulary and fluency as the most difficult (see Table 5).It can be seen that students' of various aspects of their oral English ability corresponds well to their evaluation of learning difficulties.
However, the teachers' evaluation of their students' learning difficulties varied slightly from students' judgment (also see Table 5).The teachers thought that the use of communicative strategies, intonation, and accuracy of vocabulary were the most difficult aspects.Note: The scale used for the questions is: 1=very difficult, 2=difficult, 3=so so, 4=easy, 5=very easy.
As for the causes of learning difficulties, students considered that lack of practice outside the classroom was one of the major reasons for their unsatisfactory oral English proficiency (see Table 6).More than half of the student respondents also reported that they did not know how to learn oral English effectively.Note: More than one option can be selected for the question.Note: More than one option can be selected for the question.
In the survey questions on students' motivation of improving their oral English proficiency, four broad categories of learning motivation were investigated: professional (e.g., to help advance their careers), communicative (e.g., to communicate with foreign friends), academic (e.g., to meet course requirements), and interest in learning English (Malone, 1999).The survey results indicated that most students learned English for professional purposes, that is, they wanted to be able to use English fluently in workplace (see Table 7).They hoped to be better prepared for the job interview and found a better job through improving their oral English proficiency.It was somewhat out of our expectation to learn from the survey that only a small percentage of students (27.1%) reported that they were interested in learning English.So it can be said that the students being surveyed in this study were learning English mainly for external types of motivation.
As for the ways of learning oral English, most students reported that they got chances to practice oral English mainly through classroom activities such as answering questions or discussing with classmates, etc., or watching English TV and movies after class (see Table 8).It is worth noting that chances of practice oral English through deliberately organized classroom oral activities seem less frequent than we had expected.Only 23.6% of the students reported that they participated in oral English communicative activities in class.Note: More than one option can be selected for the question.
When students were asked to evaluate their oral English teaching and assessment, only 5.6% thought their English teachers paid sufficient attention to oral English teaching.About 63.3% of the students reported they had speaking classes every week.As for classroom-based speaking assessment, 79.9% of the students reported that their teachers seldom administered speaking tests or assessments to them.And only 5.6% of the students reported that their teachers would often provide feedback to them after tests.Almost all students (95.5%), however, thought that the provision of feedback from their teachers about their oral English proficiency would be helpful.Note: More than one option can be selected for the question.
As for the teachers, most of them reported that they often used reading texts followed by answering questions and group discussion as classroom practices of oral English (see Table 9).
About 69% of the teachers seldom gave oral tests to students.Table 10 presented the frequencies of the types of speaking test tasks as reported by the teachers to be used for classroom assessment.Group discussion seemed to be the most popular form of assessment for the teachers.Note: More than one option can be selected for the question.
Table 11 indicated that the rating criteria adopted most frequently by the teachers were fluency, accuracy and communicative effectiveness.As for the provision of feedback, more than half of the teachers (58.6%) reported that they would provide feedback to students and thought that students responded positively to their feedback.Note: More than one option can be selected for the question.
To sum up, the analysis of the SQ and TQ survey data provided us with an overall picture of the current situation of teaching, learning and assessment of oral English in universities.The next section discussed the important implications of the survey data analysis for the development of the diagnostic speaking test.

Implications for the Development of Diagnostic Speaking Test
The survey results indicated that the overall oral English proficiency as perceived by the college students and their English teachers was not quite satisfactory.The majority of the students thought their oral English proficiency were at the lower-intermediate or intermediate level.The majority of the teachers thought that students' oral English proficiency was at the intermediate level.And it was found that students had difficulties mainly in fluency, range of vocabulary and use of communicative strategies.The survey results revealed that there was an urgent need for the development and administration of a diagnostic test for the purpose of promoting students' learning autonomy.
Since the purpose of diagnostic tests is to identify the strengths and weaknesses of students' English proficiency and provide diagnostic feedback for improvement, it is necessary that the students' learning difficulties be taken into serious consideration while designing the test in order for the diagnostic speaking test to be useful to students.The test's construct, for example, should encompass the salient aspects that students found most difficult to learn.The test tasks should be so designed that they enable teachers to detect the problems related to students' learning difficulties.The rating criteria and feedback reports should enable teachers to provide relevant diagnostic information on students' learning difficulties.What follows in the next section is a description of the design and development of the diagnostic speaking test's feedback descriptors.
Feedback on test-takers' performance is an integral part of a diagnostic test.Black and Wiliam (1998) suggest that a feedback system should be made up of four elements:  Data on the actual level of some measurable attribute;  Data on the reference level of that attribute;  A mechanism for comparing the two levels, and generating information about the gap between the two levels;  A mechanism by which the information can be used to alter the gap.
The first element refers to the description of test takers' actual performance.The second element describes the reference level of performance, which informs test takers of the expected performance at each level.The third element requires a comparison of the actual performance with the reference performance, which generates relevant diagnostic information.The last element suggests that advisory information be produced to help test takers take remedial actions to improve their performance.The information obtained from the last two elements can be called elaborated feedback, which explains why test takers' performances are satisfactory or unsatisfactory and provides suggestions for them to achieve good performance.Bangert-Drowns, C. Kulick, J. Kulick, and Morgan (1991) commented that elaborated feedback was crucial in developing deeper conceptual understanding, and helpful in applying rules in more complicated situations.Black and Wiliam's (1998) feedback system sheds some light to the development of the diagnostic speaking test feedback, which will describe the actual performance of test takers and provide detailed diagnostic information on their performance in accordance with the rating scale.
According to Luoma (2004), the use of rating checklists is helpful for developing more structured feedback mechanisms for speaking assessment.In the present study, therefore, the researcher decided to use the seven rating criteria as parameters of a rating checklist to develop the feedback descriptors.Each parameter was divided into five levels in accordance with the rating scale.Detailed descriptors were developed for each level so that test takers could learn about their strengths and weaknesses in all the aspects concerned in the diagnostic speaking test.
In sum, the feedback report of the diagnostic speaking test is composed of two parts: "your level" which reports students' scores in terms of the seven categories of rating criteria on the 5-level rating scale, and "feedback descriptors" which provides a detailed description of the student's performance on the diagnostic speaking test in the aspects concerned.
Keeping the diagnostic purpose in mind and considering the classroom assessment nature of the test, the researcher selects a number of speaking tasks to measure the linguistic knowledge and skills identified in the test construct.In terms of the task types, one structured task (reading aloud) and two open-ended tasks (individual presentation and role-play) would be employed.

Conclusion
The results of SQ and TQ surveys in this study presented a picture of the current situation of teaching, learning and assessment of oral English at the tertiary level in China.It was noticed that the majority of the college students who participated in the survey were not satisfied with their current oral English proficiency.Over 40% of the students thought that their oral English was at the lower-intermediate level.With regard to their oral English learning, the students reported that they experienced learning difficulties mainly in the following three aspects: oral English fluency, range of oral English vocabulary, and use of communicative strategies in oral interactions in English.And most of the students attributed their learning difficulties to lack of explicit instructions on how to learn oral English effectively and inadequate practice in class.As for the motivations of learning oral English, it was somewhat out of our expectation to learn that only a small percentage of students (27.1%) reported that they were interested in learning English; whereas the majority of them were learning English for external types of motivations such as being able to speak fluent English in working place or obtaining oral proficiency certificates in order to find a better job, etc.With regard to the assessment of speaking and the provision of feedback in class, about 79.9% of the students said that their teachers seldom used oral English assessments in class.Not surprisingly, 69% of the teachers in the survey admitted that they seldom administered oral English assessments to students.As for the teaching activities and speaking test tasks used in classroom teaching, most teachers reported that they often used reading aloud and answering follow-up questions, group discussions and debate as teaching tasks and they used group discussion, individual presentation and interview as speaking test tasks.And the rating criteria adopted most frequently by teachers included fluency, accuracy and communicative effectiveness.
The results of the needs analysis indicated that the instruction and the assessment of oral English were not given due attention.Therefore, it was of urgent necessity to raise the awareness among college English teachers of the importance of diagnostic assessment for teaching and learning, and to develop and validate a prototype diagnostic college English speaking test and to explore the impact of detailed diagnostic feedback on students' oral English learning.
Diagnostic testing has great potential for future classroom assessment.In this study, the researcher explores the current situation of college English teaching and learning in China.Due to the time limitation and lack of fund, the current study could not be administered to a larger sample.Therefore, future research should be conducted on a larger sample with students from a variety of majors and universities that could better represent Chinese undergraduates for the purpose of confirming the generalizability of the results of the present study.In addition to student participants, more college English teachers should also be invited to participate in future research to investigate the impact of test feedback on oral English teaching.
In sum, the present study represents a new endeavor to promote the assessment for learning and to facilitate student's learner autonomy in the field of oral English teaching and learning at the tertiary level in China.The positive results of the present study are but a starting point and continued research efforts into validating the effects of diagnostic testing on classroom teaching and learning are opportune and urgent.

Table 1 .
The structure and content of TQ

Table 2 .
The structure and content of SQ

Table 3 .
Descriptive statistics of oral English proficiency

Table 6 .
Causes of learning difficulties in oral English

Table 7 .
Students' motivation of improving oral English proficiency

Table 8 .
Ways of oral English learning as reported by the students

Table 9 .
Tasks used by teachers in oral English classroom

Table 10 .
Test tasks used by teachers in classroom assessment of oral English

Table 11 .
Rating criteria used by teachers