Construct Validation of the English Language Teaching Reflective Inventory with a Sample of Turkish University EFL Instructors

Although Reflective Practice is a popular approach to teacher education, there have been some recent criticisms regarding its effectiveness in students learning. Some researchers have put this down to the fact that the concept has not been operationalised, and it is thus impossible to set up empirical studies. The current study was conducted to examine the construct validity in a Turkish higher education context of a recent inventory which was designed to measure the reflective practice of teachers of English as a Foreign Language. The sample was a group of instructors (n=173) working on the English language preparatory programs of four state-run universities in Turkey. The original model suggested six factors: cognitive, meta-cognitive, critical, practical, affective and moral. The factor structure of the instrument was examined using exploratory factor analysis with SPSS 13.0, and confirmatory factor analysis with AMOS 16.0. The results of the current study showed similarity with those of the original study. Specifically, the cognitive, meta-cognitive and critical factors remained largely intact, and the affective and moral factors were not validated. However, the practical factor was reduced from 7 to 3 items. The implications of the findings are discussed.


Introduction
With the emergence of the postmethod era (Kumaravadivelu, 1994(Kumaravadivelu, , 2001)), the concept of method in the field of English Language Teaching (hereafter ELT) disappeared from the scene and researchers began to turn their attention to the contributions of the learners, teachers and larger socio-political context to the complex foreign language learning/teaching process (Williams & Burden, 1996;Dörnyei, 2005).Reflective practice (hereafter RP), which had been the subject of much research in mainstream education for some time already (Brookfield, 1995;Zeichner & Liston, 1996) was embraced by many in the field of ELT as a viable approach to teacher education which could empower teachers by placing them at the centre of their own development in their own contexts (see, e.g.Wallace, 1991;Farrell, 2007).
Despite the continuing popularity of RP, it has become the focus of recent criticism.Akbari (2007) and Cornford (2002) contended that there was a lack of empirical evidence to show the effectiveness of RP.This observation has been corroborated by more recent studies (see, e.g.Harvey, Coulson, Mackaway & Winchester-Seeto, 2010).Researchers (see, e.g.Cornford, 2002;Akbari, Behzadpoor & Dadvand, 2010) have attributed this lack of empirical evidence to the lack of consensus on what reflection entails.Largely because of its roots in qualitative inquiry (Gore, 1987), there have been very few attempts to operationalise the construct of RP.Hence, what the components of RP are and what they entail are still unclear.Motivated by this gap in the literature, Akbari et al (2010) developed the English Language Teaching Inventory (hereafter ELTRI) to allow for the quantification of the construct and thus its empirical investigation.
ELTRI is a very new instrument and a unique attempt at operationalising foreign language teacher reflection.As Akbari et al (2010) pointed out, it is necessary to conduct replication studies in different cultural contexts in order to test its relevance and validity.To the researcher's knowledge, there has been no attempt to do so as yet.The aim of the current study sets out to investigate the construct validity of ELTRI with a group of Turkish university EFL instructors by means of exploratory and confirmatory factor analysis.

The Origins of Reflective Practice
RP is most frequently traced back to the works of Dewey (1991Dewey ( /1933) ) and Schön (1991Schön ( /1983)).Differentiating it from the loose string of ideas that go through our minds, Dewey (1991Dewey ( /1933) defined reflective thought as the "active, persistent, and careful consideration of any belief or supposed form of knowledge in the light of the grounds that support it, and further conclusions to which it tends" (p. 6).For Dewey, reflective thought was a means to avoid impulsive, routine thought.He saw it as a civilising force for the improvement of society.
To counterbalance the occurrence of teacher burnout in the 1980s, Schön (1991Schön ( /1983) ) emphasised the importance of adopting a reflective approach to teaching over more traditional approaches.According to Schön, reflection is usually triggered by a problem in teaching, and the ability to frame and reframe such problems is an important part of developing RP.He distinguished between two time-frames of reference for reflection: reflection-in-action, or thinking about something while doing it; and reflection-on-action, or thinking about an event retrospectively.
RP was embraced in the field of ELT with Wallace's (1991) milestone work 'Training foreign languages teachers: A reflective approach', in which he adapted Schön's (1991Schön's ( /1983) ) model.It reached maturity in the mid to late-1990s with the works of Richards and Lockhart (1996) and Gebhard and Oprandy (1999) and currently remains popular (see, e.g.Farrell, 2007).

Defining Reflection
Despite the popularity of RP in both mainstream and ELT, it has been a notoriously difficult concept to define.Many researchers (e.g.Zeichner & Liston, 1996;Fendler, 2003;Akbari, 2007;Farrell, 2007) have drawn attention to the diverse range of meanings that RP carries.Zeichner and Liston (1996) identified five distinct orientations in RP which have lead to this diversity, each one emphasising a different aspect of teachers' thinking.For example, subject matter (academic tradition); the thoughtful application of teaching strategies suggested by research (social efficiency tradition); teaching with sensitivity to students' backgrounds and patterns of developmental growth (developmentalist tradition); reflection on the social and political context of schooling with the aim to enhance equity and justice (social reconstructionist tradition); finally, simply thinking about what we are doing without consideration for the quality and content of that thinking (generic tradition).Zeichner and Liston (1996) stressed that these traditions represent 'intellectual distillations of teachers' reflective practice and historical developments' (p.52) rather than niches into which teachers should fit themselves or be fitted.
Different attempts to define the concept of reflection in both mainstream and ELT teacher education have led to a distinction between descriptive reflection, which provides an account of events; analytic reflection, which searches for reasons, provides alternatives and evaluates the result of teaching; and critical reflection, in which the larger socio-political context is taken into account (see, e.g.Hall, 1997;Hatton & Smith, 1995;Jay & Johnson, 2002;Stanley, 1998;Ward & McCotter, 2004).There is a particular emphasis on the critical and moral aspects of reflection.As Akbari (2007) pointed out, if RP remains at a purely cognitive level, then it cannot contribute to the improvement of society which Dewey (1991Dewey ( /1933) ) had envisioned.However, some researchers (see, e.g.Sanger & Osguthorpe, 2011) have stressed that the moral aspect of teaching is largely overlooked in teacher education, and some teachers are unaware of the ethical effects of their actions (e.g., Husu & Tirri, 2007).Moreover, Day (1993) commented that teachers' workloads made it very difficult for them to go beyond practical reflection to consider the moral aspects of teaching.
A similar argument could be put forward for the affective aspect of teaching.As Hawkey (2006) remarked, emotion has always been viewed with suspicion in Western culture.However, it is now widely recognised that emotion and cognition are in fact interwoven (see, e.g.Mayer, Caruso & Salovey, 1999), and researchers have started to suggest that the emotional aspects of teaching should be given more emphasis in teacher education and development (see, e.g.Golombek & Johnson, 2004;Hawkey, 2006).
To summarise, essential elements of reflection can be given as cognitive and affective engagement with practical experiences in such a way as to give meaning to the often messy events beyond a common sense level with the view to learning and professional development to the ultimate aim of the improvement of society (Dewey, 1991(Dewey, /1933;;Boud, 2001;Brookfield, 1995;Osterman & Kottkamp, 2004).

Criticism of Reflective Practice
Despite the popularity of RP in teacher education, it is not without its critics.Cornford (2002) and Fendler (2003), for example, remarked on the central role in RP of Western belief systems and philosophies, hence its popularity in many Anglo countries in particular.The question springs to mind of how a paradigm with such connections may be accepted by cultures holding different values and beliefs to those that underpin that of RP.Çakıroğlu and Çakıroğlu (2003) described the difficulties faced by Turkey, the context of the current study, which has relied heavily on the Western knowledge base for both teacher education and research.It might be that non-Western European cultures approach RP in different ways because of cultural factors.Cornford (2002) also warned that despite the large number of qualitative studies reporting the enthusiasm of teachers for engaging in reflection, there has been very little empirical research to show that RP is more effective than any other approach to teacher education.He attributed this to the lack of consensus on a definition of RP, stressing that "without adequate definition, and agreement on ideals and meanings, it becomes impossible to operationalise a paradigm and translate it into practice" (pp. 220-221).Akbari (2007) echoed Cornford's (2002) criticism, with particular reference to the field of ELT.More recently, Harvey et al ( 2010) made a similar observation.This would suggest that it might be timely to adopt alternative research approaches to complement the currently largely qualitative ones on the construct of RP.

Can Quantitative Measures Be Used to Measure Reflective Practice?
Both Cornford (2002) and Akbari (2007) have emphasised the need for empirical research into RP.Owing its existence to a break away from the positivistic effective teacher movement (Williams & Burden, 1996), the RP paradigm is essentially diametrically opposed from the empirical research tradition.Indeed, early researchers (see, e.g.Gore, 1987), argued that quantitative measures could not be used to measure reflection.Cornford (2002), however, purported that such a stance was both extreme and "educationally undesirable and logically difficult to defend" (p.221).He added that the large number of qualitative studies conducted to categorise different views on RP had served to increase its diversity rather than reach a consensus.Given that the current research climate favours the use of both qualitative and quantitative methods "to draw from the strengths and minimise the weaknesses of both in single research studies and across studies" (Johnson & Onwuegbuzie, 2004, p. 15), it seems an appropriate time to make an attempt to operationalise the RP paradigm.

Akbari et al's (2010) Proposed Model of Reflection
While some researchers have developed instruments to measure the level of reflective thinking in university students (see, e.g.Kember, Leung, Jones, Loke, Yuen et al, 2000); there have been very few attempts to operationalise the construct of RP in teacher education since Korthagen and Wubbels (1995) set out to identify the critical attributes of reflective teachers.Motivated by this gap in the research, Akbari et al, (2010) developed the English Language Teaching Reflective Inventory (ELTRI), a 42-item model of six factors: practical, cognitive, affective, meta-cognitive, critical and moral.In the validation study with a group of Iranian English Language teachers (Akbari et al, 2010), the proposed model survived exploratory factor analysis intact.However, during confirmatory factor analysis the moral dimension had to be deleted from the model and the affective dimension was reduced from 7 to 3 items.ELTRI is currently the only instrument available to measure teacher reflection in the field of ELT.Due to the novelty of the instrument, Akbari et al (2010) strongly recommended that further studies be conducted in different contexts to test its relevance and validity.Hence, the aim of the current study is to contribute to the construct validity of ELTRI by administering it to a group of Turkish university EFL instructors.A single research question has been formulated to this aim: "What is the construct validity of ELTRI with a sample of Turkish university EFL instructors?"

Participants
Based on the survey method, a convenience sample (Dörnyei, 2007) of 173 ELT instructors (36 males, 137 females) working at the schools of foreign language of two large state-run universities in Ankara, and the departments of Basic English of two medium-sized state-run universities in the Western Black Sea region of Turkey voluntarily participated in the current study.32.9% of the instructors had 0-5 years of teaching experience, 28.3% had 6-10 years, 17.9% had 11-15 years, 12.7% had 16-20 years, and 8.1% had over 20 years of.53.8% of the participants held an undergraduate degree in ELT, 20.2% held a master's degree in ELT, 1.2% held a doctoral degree in ELT, and 24.9% held degrees in other foreign language related subjects, for example English Language and Literature, American Language and Literature, American Cultural Studies.

Data Collection Instrument
The instrument used in the current study is the initial tentative form of the English Language Teaching Reflection Inventory (ELTRI) developed by Akbari et al (2010).Permission to use the instrument was granted by the corresponding author via e-mail.ELTRI is a six-factor model with 7 manifest indicators for each factor.The factors with sample items are as follows: Practical (e.g., 'I write about my teaching experiences in a diary or a notebook'); Cognitive (e.g., 'I think of using/introducing new techniques in my classes'); Affective (e.g., 'I think about my students' emotional responses to my instructions'); Meta-cognitive (e.g., 'as a teacher, I think about my teaching philosophy and the way it is affecting my teaching'); Critical (e.g., 'in my teaching, I use less-discussed topics such as old age, AIDS, discrimination against women and minorities, and poverty'); and Moral (e.g., 'I provide equal opportunities for all my students in the class regardless of their capabilities').Participants rated their responses on a 5-point Likert-type scale ranging from 5 (always) to 1 (never).

Data Collection Procedures
310 instruments were distributed by post to the ELT instructors of the four universities involved in the current study with permission from the institutions in the first half of the autumn term of the 2010-2011 academic year.187 instruments were completed (56.67% response rate) and returned to the researcher by the middle of the spring term of the same academic year.After initial inspection of the instruments, 14 were discarded due to incomplete responses.This left 173 instruments to be subjected to the data analysis procedures.

Data Analysis Procedures
Data analysis procedures were carried out using SPSS 13.0 and AMOS 16.0.After the data had been entered into SPSS, Missing Variable Analysis revealed that less than 5% of the data were missing, which meant that it was acceptable to replace the missing values using the series mean method (Tabachnick & Fidell, 2007).The data were then checked for kurtosis, skewness and normality.Following Akbari et al's (2010) study, a two-step data analysis framework of Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA) was followed.
Given that the data were normally distributed and there was expected correlation between the factors, EFA with maximum likelihood factor extraction (MLFE) and promax rotation (cut off .40)(Costello & Osborne, 2005) was conducted in order to determine the underlying factor structure of the ELTRI with the current sample.The scree plot test was also applied.The reliabilities of the factors yielded from the EFA were computed using Cronbach's Alpha.The factors then underwent CFA, a data reduction procedure in which variables are fixed a priori to load on specific factors for model validation (Bandalos, 1996).When assessing model fit, a number of indices are generally used.In the current study, the most commonly preferred normed χ² statistic, Comparative Fit Index (CFI), Tucker Lewis Index (TLI) and Root Mean Square Error of Approximation (RMSEA) were calculated (see, e.g.Kline, 2005).The minimum cut off point for model validation is≤3 for the normed χ² statistic, and ≥ .90,.90 and <.06 to.08 for CFI, TLI and RMSEA respectively (Kline, 2005).

Exploratory Factor Analysis
Prior to the EFA, the absence of kurtosis and skewness, and the normality of distribution of the data were determined.The Kaiser-Meyer-Olkin measure of Sampling Adequacy was .84,and Bartlett's Test of Sphericity was significant (p=.000), indicating that the data were factorable.MLFE with promax rotation (cut off .40) on the 42 items initially yielded 11 factors with eigenvalues greater than 1 accounting for 67.65% of the total variance.However, as Costello & Osborne (2005) pointed out, there is a broad consensus that retaining all values with eigenvalues of over 1 is among the least accurate methods for retaining factors.Hence, the scree test was used to determine which factors should be retained for rotation.The scree test suggested that 5 factors would be more parsimonious.Promax rotation of these factors showed they accounted for 49.78% of the total variance (see Table 1) and were identifiable with Akbari et al's (2010) tentative model.These factors were Meta-cognitive (Factor 1 with 6 items accounting for 26.36% of the variance); Practical (Factor 2 with 5 items accounting for 8.13% of the variance); Cognitive (Factor 3 with 7 items accounting for 6.2%) of the variance; Critical (Factor 4 with 6 items accounting for 5.04% of the variance); and Moral (Factor 5 with 4 items accounting from 4.05% of the variance).10 items failed to reach the acceptable loading value in their given factor and were thus discarded from the subsequent analyses after discussion with a domain expert.None of the items of the Affective factor of the tentative model loaded under a single distinct factor.Items 16 'When a student is having an emotional problem or is neglected by his/her peers, I try to spend more time with him/her', 19 'I talk to my students to learn about their learning styles and preferences', 20 'I talk to my students to learn about their family backgrounds, hobbies, interests and abilities', and 21 'I ask my students whether they like a teaching task or not' did not load significantly.Items 15 'I think about my learners' emotional responses to my instructions' and 17 'Before and after teaching, I think about aspects of my lessons my students liked/disliked' cross-loaded under the Meta-cognitive factor.Item 18 'I ask students to write/talk about their perceptions of my classes and the things they liked/disliked about it' cross-loaded under the Practical factor.
The remaining items which did not load significantly were items 3 'I talk about my classroom experiences with my colleagues and seek their advice/feedback' and 5 'I discuss practical/theoretical issues with my colleagues' from the Practical factor; item 28 'I think of inconsistencies and contradictions that occur in my classroom practice' from the Meta-cognitive factor; item 29 'I think about instances of social injustice in my own surroundings and try to discuss them in my classes' from the Critical factor; and item 40 'I establish a clear set of rules for my students to follow in terms of their classroom attendance and the way they will be evaluated at the end of the course' from the Moral factor.

Confirmatory Factor Analysis
EFA revealed that a 5 factor model of reflective practice could be extracted from the current data set.In the second stage of the analysis, CFA was conducted using AMOS 16.0 to validate the hypothetical model.Prior to this analysis, the Cronbach Alphas for the Meta-cognitive, Practical, Cognitive, Critical and Moral factors and the total factors were calculated and found to be .88,.81,.88,.74,.73 and .89respectively.These values suggest that the 5 factor model had high internal reliability.
The results of the CFA suggested a four-factor model, i.e.Meta-cognitive, Practical, Cognitive and Critical, in which all the loadings between the indicators and the latent factors as well as the covariance among the factors were significant at α = .001(p≤.001).However, none of the items loading under Moral factor during the EFA stage showed any significant relationship with this factor.Moreover, the Moral factor showed no significant covariance with the other latent factors, so it was deleted from the model.In addition, only 3 of the items of the Practical factor, 1 'I write about my teaching experiences in a diary or a notebook', 6 'I observe other teachers' classrooms to learn about their efficient practices' and 7 'I ask my peers to observe my teaching and comment on my teaching performance' turned out to be salient in the CFA, hence items 2 'I have a file where I keep accounts of my teaching for reviewing purposes' and 4 'After each lesson, I write about the accomplishments/failures of that lesson or I talk about the lesson to a colleague' were discarded.Given that a minimum of 3 indicators is necessary for the stability of a latent factor (DiStefano & Hess, 2005), the Practical factor was retained.These analyses resulted in an instrument of 4 factors and 21 items.The zero-order correlations among the indicators outlasting the CFA are given in Table 2. Examination of the correlation matrix reveals that the indicators of a given factor were generally more strongly correlated with each other than with the indicators of the other factors.Based on modification indices (MI>10), errors of 3 indicator pairs were allowed to correlate (MC25-MC26, MC27-Crit33, Cog8-Cog11, Cog8-Prac7).
Following CFA, a number of model fit indices were used to cross-validate the findings.The results are as follows: normed χ² (1.71), TLI (.90), CFI (.91) and RMSEA (.06).These results show that the model had acceptable fit (Kline, 2005).

Discussion
The most striking similarity between the findings of the current study and those of the Akbari et al (2010) study is the deletion of the moral factor during the CFA stage despite the corroboration of this factor by EFA.As the literature has shown, a moral dimension is essential if RP is to bring about improvement in society (Akbari, 2007), and that this crucial construct has not been supported empirically in either the current or the original study (Akbari et al, 2010) is noteworthy.This finding could be attributed to the fact that moral aspects of teaching are not widely covered during teacher education (Sanger & Osguthorpe, 2011), hence the teachers might not be aware of the importance of these aspects (Husu & Tirri, 2007).Their busy workload could also act as a hindrance (Day, 1993;Çakıroğlu & Çakıroğlu, 2003).
Although the affective factor emerged as an independent factor with only 3 of its initial 7 indicators being confirmed in the original study (Akbari et al, 2010), this factor was eliminated from the model during the EFA stage in the current study.This suggests that the affective dimension was problematic to a certain extent in the original study, and more so with the current sample.Given the predominant place given to cognition over emotion in teacher education (Golombek & Johnson, 2004;Hawkey, 2006), this finding is not surprising.
The 7 item practical factor of the tentative model was reduced to 5 items during EFA, and subsequently further reduced to 3 items during CFA, with only items 1, 6 and 7 outlasting the validation processes.In the original study (Akbari et al, 2010), the factor retained 6 of the 7 items.A number of participants remarked on the instruments that they found items 3 and 4 ambiguous, which suggests that the wording of these items needs reviewing.

Conclusion
The current study reported the validation of the ELTRI (Akbari et al, 2010), an instrument designed to measure teachers' reflection in the field of ELT, with a group of Turkish university EFL instructors.EFA and CFA were used to examine the construct validity of the tentative 6 factor model: Meta-cognitive, Cognitive, Practical, Critical, Moral and Affective.EFA suggested a 5 factor model, with the items of the proposed Affective factor either failing to load significantly or cross-loading under other factors.CFA reduced the model further to a 4 factor model with 21 items, with the Moral component failing to survive this stage.The calculated model-fit indices suggested that this 4 factor model had acceptable fit, given the small size of the current sample.These findings show similarities to those of the original study (Akbari et al, 2010), which suggests that the instrument can be used as a valid instrument in the context of Turkish higher education.
There are a number of limitations to the current study, each of which provides a direction for future studies.First, it was conducted with only university EFL instructors.Further studies could be conducted with a sample of EFL teachers at primary and secondary level.Second, all the participants were Turkish nationals.It would be interesting to replicate the study with samples of teachers from a more diverse range of cultures.If, as the literature has suggested, the RP paradigm is closely tied to Western belief systems and philosophies, it would be particularly interesting to administer ELTRI to teachers from these cultures.The current study revealed problems related to the validity of the affective and moral factors.However, it is beyond the scope of this study to determine the reasons behind this lack of validity.Further replication studies would serve to investigate the validity of the affective and moral factors of ELTRI with different samples.The results of these studies would help to make any necessary modifications to the model.

Table 1 .
Results of the exploratory factor analysis

Table 2 .
Zero-order correlations among the indicators of ELTRI outlasting confirmatory factor analysis