Evaluating the Effectiveness of the Chemistry Education by Using the Analytic Hierarchy Process

In this study, an attempt was made to develop a method of measurement and evaluation aimed at overcoming the difficulties encountered in the determination of the effectiveness of chemistry education based on the goals of chemistry education. An Analytic Hierarchy Process (AHP), which is a multi-criteria decision technique, is used in the present study. The proposed method consists of four main steps: (1) Determination of the main goals and sub-goals of chemistry education; (2) Formation of the AHP model; (3) Making pairwise comparisons and the calculation of local weights; (4) Calculation of global weights and the effectiveness of chemistry education. Research findings indicated that it is possible to determine the effectiveness of chemistry education based on the goals set through the proposed method.


Introduction
In its most general sense, chemistry education aims to enable students to acquire the means to understand, explain, predict and control nature.Specifically, chemistry education goals are set to give direction to chemistry instruction, the curriculum, instructional materials and evaluations, and to the goals which are to be achieved in the future (Gayon, 2010).As the perceptions of chemistry, and developments within it have changed over time, so too the goals of chemistry education have changed in parallel with them (Mitchell, 1993).The changes encountered have also modified the content and methods of chemistry education.For example, the context-based science education approach recently proposed for chemistry education requires changes in the curriculum as well as the re-examination of the goals of chemistry education, the renewal of subject content, and new approaches to assessment (Coenders, Terlouw, & Dijkstra, 2010).For these reasons, it is impossible to consider the goals of chemistry education as fixed.On the contrary, it is seen that goals of chemistry education have a dynamic structure.
Different problems concerning chemistry education are the subject of some research in the literature on the goals of chemistry education (Boyer & Tiberghien, 1989;Tamir, 1990;Horak, 1993;Mitchell, 1993;Coenders et al., 2010;Gayon, 2010).However, besides the content of goals which give direction to chemistry education, it is also important to determine to what extent these goals have been reached.In a traditional context, in the field of chemistry, the level of reaching educational goals is measured by the academic achievement level of students, as is the case in other fields of science (Muijs, 2006).For this reason, many methods for, and approaches to, the assessment and evaluation of academic achievement of students have been developed (Kilmen, Kösterelioğlu, & Kösterelioğlu, 2007;Gelbal & Kelecioğlu, 2007;Nazlıçiçek & Akarsu, 2008;Çelikkaya, Karakuş, & Demirbaş, 2010;Okur & Azar, 2011).Assessing and evaluating student academic achievement in the education process is an important indicator that reveals the extent to which a curriculum's purposes have been achieved (Çelikkaya et al., 2010).However it is impossible to say that such a method is adequate.Traditional assessment and evaluation methods are insufficient for determining the outcomes aimed at in the curriculum (Gelbal & Kelecioğlu, 2007;Nazlıçiçek & Akarsu, 2008;Çelikkaya et al., 2010).This is because there are many factors concerning students that must be evaluated within the framework of the goals of education.Evaluating and assessing them with only one technique is generally impossible (Kilmen et al., 2007).However, alternative approaches are developed for the assessment and evaluation of learning based on the changes in epistemological theories (Baki & Birgün, 2002).However the alternative approaches proposed (Baki &Birgün, 2002;Nazlıçiçek & Akarsu, 2008) are also generally considered to be for the assessment and evaluation of student academic knowledge.
As well as measuring student academic knowledge, it is important to measure to what extent the goals of the curriculums are achieved in order to determine the effectiveness of education.This is because the effectiveness of education is also among the purposes of educational evaluation (Baykul, 1992).However, a series of difficulties and limitations are encountered in the measurement and evaluation of the effectiveness of chemistry education on the basis of the goals of chemistry education.The first problem results from the nature of the effectiveness concept.Effectiveness is defined in different contexts in the literature.However, in its most general sense, it is defined as the extent of the achievement of the purposes.In other words, it shows to what extent the expected accomplishments are achieved or how well the outcomes meet the stated goals (Cowan, 1985;Baykul, 1992;Muijs, 2006;Yılmaz & Taşdan, 2006;Rupsiene & Bartuseviciene, 2008).However effectiveness studies are sophisticated as they generally cover complex activities (Rupsiene & Bartuseviciene, 2008).For this reason, difficulties can be encountered in the measurement of effectiveness.There are also difficulties in converting effectiveness into a numerical index as it covers different goals with different features (Baykul, 1992).
Another problem encountered in determining the effectiveness of chemistry education on the basis of its goals results from the excessive number of goals in chemistry education.Numerous and different chemistry education goals necessitate the measurement and evaluation of the extent to which these goals are achieved from a holistic perspective.This is because evaluating the effectiveness of chemistry education on the basis of each goal separately will not lead to a holistic evaluation.In the event where the effectiveness of chemistry education is evaluated separately for each goal, only the degree to which the goal taken into consideration is realized can be determined.However, the relative importance of each goal may vary.For example, it is not possible to say that the goal of "presenting fundamental knowledge" and the goal of "promoting awareness of chemistry-related careers" have the same relative importance in chemistry education.A similar situation may be true for other goals in chemistry education too.A second problem derives from the fact that the structures of the goals of chemistry education are generally qualitative and conceptual.Certain difficulties are encountered in the quantitative measurement of the goals of chemistry education in this kind of a structure.For example, it is not possible to directly quantitatively measure the goal of "emphasizing environmental concepts and world problems" in chemistry education.Accordingly, the structure of the measurement technique must be congruent with the structure of the goals.
In the light of the abovementioned information, the present study aims at developing a method of measurement and evaluation which overcomes the difficulties and solves the problems encountered in the process of determining the effectiveness of chemistry education based on the goals of chemistry education.The Analytic Hierarchy Process (AHP), which is a multi-criteria decision technique, developed by Saaty (1980) was used for realizing this, the main purpose of the present study.The reason for the use of this technique is that the AHP concept is appropriate for the purposes and nature of this research.It was seen in the literature review that the AHP technique was used for the solution of some of the problems encountered in the field of education, though in a limited number (Feng, Lu., & Bi, 2004;Yetim, 2004;Begicevic and Divjak, 2007;Klein, 2007;Suryadi, 2007;Warwick, 2007;Uzoka, 2008;Li, Li, & Zhang.,2009;Begicevic, Divjak, & Hunjak, 2010;Tsinidou, Gerogiannis, & Fitsilis, 2010;Ho, & Hsu, 2011).
The remainder of this paper is organized as follows.The second section presents the method of measurement and evaluation proposed in accordance with the purpose of the study.This section describes the AHP technique used in the study and the basic steps of the proposed method.The third section of the study presents the application of the proposed method.Finally, in the fourth section, conclusions and suggestions are presented.

Method
This section presents the method proposed for the measurement of the effectiveness of chemistry education.As expressed in Section 1, the main technique used in the proposed method is the Analytic Hierarchy Process (AHP).AHP, which is a multi-criteria decision-making technique, was developed by Saaty (1980).
In AHP technique, firstly, the problem is defined, and the scope of the problem is determined.In the next stage, the hierarchical structure of the problem is formed.There is a purpose in the first level of the hierarchical structure, and there are elements and alternatives in the lower levels.The hierarchical structure in AHP may vary according to the complexity of the problem and the number of elements and alternatives it includes.Another stage in the AHP technique is the formation of pairwise comparison matrixes according to the hierarchical model constituted.Pairwise comparison matrixes are square matrixes in n x n size.The comparison of elements of matrixes with one another is made according to 1-9 scale of Saaty (1980) (Table 1).

Comparative judgment
Intensity of importance a i and a j are equally important 1 a i is weakly more important than a j 3 a i is strongly more important than a j 5 a i is demonstrably or very strongly more important than a j 7 a i is absolutely more important than a j 9 Intermediate values between adjacent scale values 2, 4, 6, 8 Accordingly, a ij indicates pairwise comparison value of i. element and j. element.aji is comparison value of j. element and i. element.In AHP, a ji value is obtained via a ji = 1/a ji equality (Saaty, 1980;Saaty, 1982).The appearance of the pairwise comparison values in the matrix is as given in equation (1).
In AHP, the pairwise comparison number in each matrix varies by the number (n) of elements of the matrix.Accordingly, in a matrix containing (n) elements, n (n-1) /2 pairwise comparisons are made.The diagonal values of the matrixes are equal to 1.Following the formation of pairwise comparison matrixes in AHP, priority vectors of the matrixes are determined.In AHP, priority vector is obtained via the solution of the (A ˗  max I) W = 0 equality (Saaty, 1980).In this equation, A indicates pairwise comparison matrix, W indicates eigenvector and  max indicates the biggest eigenvalue of A matrix.
According to the assumptions of AHP technique, pairwise comparisons must be consistent.Acceptable inconsistency levels suggested by Saaty (1994) are as follows: for 3x3 sizes the matrix is 0.05, for 4x4 size the matrix is 0.08, and for all other matrix it is 0.1.If the inconsistency ratio of pairwise comparison matrixes is lower than the suggested value, the matrix can be considered consistent.Otherwise, pairwise comparisons are to be reviewed and renewed by the decision maker.Consistency ratio (CR) is calculated depending on the consistency index (CI) and the random index (RI) values.Consistency index is determined by using the eigenvalue ( max ) as following: CI= ( max -n) / (n-1), where n is the matrix size.Then consistency ratio is calculated by dividing CI to Random index (RI) value (CR = CI / RI).RI is determined depending on the size of matrix (n) as seen in Table 2.The operation of the AHP technique takes place in accordance with the abovementioned rules.
Table 2. Values of random index (RI) (Saaty, 1980) RI 0 0 0.58 0.9 1.12 1.24 1.32 1.41 1.45 The method proposed in this study consists of four main steps: Step 1: To determine the main goals and sub-goals of chemistry education.In AHP technique, firstly, the elements making up the structure of the problem are determined through an analytical examination.In AHP technique, after the elements are determined, they are categorized according to their common features.In the present study, the goals of chemistry education were determined and categorized in the first step.The determination and categorization of the main goals and sub-goals were in the base of the effectiveness of the chemistry education and were made in accordance with the methods of the study by Gayon (2010).
Step 2: To form an AHP model in accordance with the purpose of the study.The elements that make up the problem according to the assumptions of the AHP technique are converted into a hierarchical model in accordance with the purpose of the study.In the present study, an AHP model was constituted according to the categorization set in the first step.
Step 3: To make pairwise comparisons of the matrix of each of the main goals and sub-main goals with a 1-9 scale, and to calculate the local weights and consistency ratio (CR).In AHP technique, evaluations start with pairwise comparisons.In this study, too, pairwise comparisons were made according to the 1-9 scale proposed by Saaty (1980) by taking into account the constituted AHP model.Another process at this stage is the calculation of the local weights of each goal.Local weight indicates the relative importance of each goal within the group it exists in.In this stage, lastly, the consistency ratio of pairwise comparisons was calculated.
Step 4: To calculate the global weights for the sub-goals of chemistry education and the effectiveness of chemistry education.In this stage, firstly, the global weights related to sub-goals were calculated.Global weight indicates the relative importance of each goal within the whole.In the AHP technique, the value of the whole is equal to 1.The final process in this step is the determination of the effectiveness of the chemistry education.As stated in the literature review, effectiveness can be explained in different ways.In this study, what is meant by the concept of effectiveness is as follows: The degree to which goals anticipated in the curricula are actually performed.In the present study, firstly, the expected realization level of the goals was calculated by means of the AHP technique.In the second stage, the actual realization level was calculated by taking into account the relative weights of goals within all of the goals.In the present study, the expected theoretical value range of the effectiveness is 0 -1.Nearness of effectiveness value to 1 means the perfect realization level of goals, and nearness of effectiveness value to 0 means an insufficient realization level of goals.

Results
In this section, application results related to the proposed method are presented.Results related to the application were given according to four steps covered by the method proposed in the second section.The application of the proposed method was performed within the scope of the 9th grade chemistry course of public high schools in Turkey.
Step 1: Determination of the main and sub-main goals of the chemistry education.
In this step of the study, goals and sub-goals taken as a basis for measuring the effectiveness of chemistry education were determined.As expressed in the literature review, there are many goals intended to be realized through chemistry education.These goals may vary according to progress in the science of chemistry, societal perception, and the knowledge level of society.The curriculum and goals of chemistry education in Turkey are set by the Ministry of National Education (2007).It may be seen that the curriculum in Turkey aims at the at the acquisition of knowledge appropriate to present levels of knowledge.According to the actual curriculum in Turkey, the main goal of chemistry education is not simply for the students to gain a specific knowledge of chemistry.There are also other goals covered by the curriculum, these concern an understanding of different domains in life such as the role of the individual, society, the environment, the economy and technology.
In the literature review, two main empirical studies (Mitchell, 1993;Gayon, 2010) examining and classifying the goals of chemistry education were encountered.The classification made by Gayon (2010) was used in this study as it was in accordance with the goals of the actual curriculum for the secondary education chemistry course in Turkey.Gayon (2010) grouped the goals of chemistry education according to the five major categories of science education goals.These are scientific knowledge, scientific methods, societal issues, personal needs, and career awareness.These five main goals consist of 30 sub-goals.The coding devised for the constitution of the AHP model in this study was based on Gayon's (2010)  Step 2: The formation of the AHP model for evaluating the effectiveness of the chemistry education based on the goals.
In AHP technique, after the factors and sub-factors constituting the problem are determined, it is necessary to form a model to show the relationships of the factors to one another (Saaty, 1980).Therefore, in this study too, the goals and sub-goals, which were hierarchically categorized in the first step, were converted into an AHP model.As seen in Figure 1, the AHP model for the effectiveness of chemistry education based on the goals consists of three levels.The first level includes the objective function.In the present study, the objective of the model is "to evaluate the effectiveness of chemistry education based on the goals of chemistry education".The second level contains the main goals.There are five main goals in the model: scientific knowledge, scientific methods, societal issues, personal needs, and career awareness.The third level of the model consists of sub-goals clustered under the main goals.As seen in the third level of the model, there are 30 sub-goals in total.

Figure 1. AHP model for the goals of chemistry education
Step 3: To make pairwise comparisons of the matrix of each of main goals and sub-goals with a 1-9 scale, and to calculate the local weights and the consistency ratio (CR).
This step consists of the formation of pairwise comparison matrixes of main goals and sub-goals included in the proposed model, evaluation of each matrix via 1-9 scale, calculation of local weights, and determination of consistency ratio.
There are six matrixes that must be formed according to the AHP model of this study (Figure 1).The first one is the matrix in 5x5 size that covers five main goals existing in the second level of the model.As seen in Figure 1, the components of the matrix are scientific knowledge, scientific methods, societal issues, personal needs, and career awareness.The other five matrixes consist of sub-goals existing in the third level of the model.The first one is the matrix in the 7x7 size that covers seven sub-goals included in the scientific knowledge main goals.The second matrix is the one consisting of sub-goals covered by scientific methods.The scientific methods matrix is also a 7x7 size matrix.The third matrix consists of sub-goals included by the societal issues main goal, and this matrix has a 7x7 size.The fourth matrix in the third level of the model is associated with sub-goals covered by the personal needs main goal.As is seen in the model, the personal needs matrix has a 4x4 size.The last matrix in the third level of the model is a 5x5 matrix related to the career awareness main goal.
After the matrixes included by the model have been determined according to AHP technique, it is necessary to make pairwise comparisons.In other words, the elements making up the matrixes are to be compared with one another.Pairwise comparison is made according to a 1-9 scale (Table 1).In AHP, pairwise comparisons are made by relevant specialists who know the nature of the research problem.In the present study, pairwise comparisons were made by an expert team consisting of three teachers giving chemistry courses for the 9th grade in secondary education schools.The goals of chemistry education were evaluated based on the opinions of the expert team according to 1-9 scale ( assigned to relevant matrixes. In AHP, pairwise comparisons are respectively made between two elements of the matrix.In this study, goals and sub-goals were indicated as elements.Accordingly, comparison of i. element located in the line of the matrix and j. element located in the column of the matrix is made according to Table 1 and the result (a ij ) is expressed as numerical value.On the other hand, the comparison between j. element located in the column of the matrix and i. element located in the line is equal to the inverse (a ji = 1/a i ) of a ij .In the light of the above-mentioned information, firstly, a pairwise comparison matrix associated with the main goals was formed (Table 3).Pairwise comparisons of main goals matrix (Table 3) were made by an expert team consisting of experienced teachers.These pairwise comparisons were as follows: How important the goals were in comparison to one another in the effectiveness of the chemistry education was determined according to Table 1.For example, how important is the scientific knowledge (SK) main goal in comparison to the scientific methods (SM) main goal when the scientific knowledge (SK) main goal and the scientific methods(SM) main goal in the matrix (Table 3) are compared?As in this study, if the answer is "weakly" more important, the numerical value to be assigned to matrix is 3 according to Table 1.As the comparison of j. element with i. element is equal to a ji = 1/a i if the comparison of i. element and j. element is equal to aij according to the AHP technique, pairwise comparison value related to the comparison of the scientific methods (SM) main goal in the column with the scientific knowledge (SK) main goal is 1/3.Other elements of matrix were compared in a similar way.Accordingly, pairwise comparisons, local weights and consistency ratio (CR) related to the main goals are given in Table 3.In the present study, local weights and consistency ratios were calculated via Expert Choice (2000).When Table 3 is examined, pairwise comparisons of the main goals can be seen.In addition, the local weights of each main goal can be seen in the last column.Local weight indicates the relative importance of each main goal.In other words, it indicates the level of each goal within the whole goal.The total of local weights of goals is equal to 1.
For the present study, the relative importance of scientific knowledge, scientific methods, societal issues, personal needs, and career awareness are respectively 0.312, 0.308, 0.171, 0.116, and 0.093.The consistency ratio of the matrix was found as 0.07.This value indicates that the pairwise comparisons are consistent according to the threshold values anticipated by AHP assumptions.
In the present study, after the pairwise comparison matrix related to main goals was formed and local weights and consistency ratios were set, pairwise comparisons of the goals in the third level of the model were made.
Table 4 involves pairwise comparison matrixes and local weights respectively; these are related to scientific knowledge, scientific methods, societal issues, personal needs, and career awareness sub-goals.A pairwise comparison of scientific knowledge sub-goals was made by the team of experts as follows: How important are sub-goals in comparison to one another in terms of scientific knowledge?For example, when reflecting the nature and structure of chemistry (SK1) the sub-goal was compared with the "presenting chemistry as a body of scientific knowledge" (SK2) sub-goal, the SK1 sub-goal was not found to be more important than SK2.However, when the SK2 sub-goal was compared with SK1, SK2 was found "weakly" more important than SK1.Therefore, while the value related to the comparison of SK1 with SK2 is 1/3, the value obtained at the end of comparison of SK2 with SK1 is 3.In the last column of Table 4, there are local weights related to sub-goals.For example, the local weight of "reflecting the nature and structure of chemistry" (SK1) sub-goal is 0.089.This value indicates that, for this study, the relative importance of reflecting the nature and structure of chemistry (SK1) sub-goal within scientific knowledge goal is 8.9%.Other local weights can be interpreted in a similar way.It was seen that pairwise comparisons related to sub-goals forming the scientific knowledge are also consistent (CR: 0.05).
Pairwise comparisons and local weights of sub-goals related to scientific methods, societal issues, personal needs, career awareness have been determined as in the above-described main goals and scientific knowledge sub-goals.The consistency ratio of the scientific methods (CR: 0.08), societal issues (CR: 0.07), personal needs (CR: 0.04), career awareness (CR: 0.04) sub-goals was found within acceptable limits.
Table 4.The pairwise comparison matrix and local weights for sub-goals Step 4: Calculating the global weights for the sub-goals of chemistry education, and determining the effectiveness of chemistry education.
Operations performed in this step are given in Table 6.As may be seen in Table 6, there are five columns: goals of chemistry education, global weights, levels of realization, value of level, and level of effectiveness by sub-goals.In this step, firstly, the global weights of sub-goals in the third level of the AHP model (figure 1) were calculated by means of Expert Choice (2000).Global weights are calculated according to the local weights of the main goals and sub-goals.And the calculated global weights of sub-goals are given in the second column of Table 6.Global weight indicates the relative importance of each sub-goal within all of the goals.In the present study, global weights demonstrate anticipated or intended realization levels related to the goals of the chemistry education.For example, in this study, the global weight of "reflecting the nature and structure of chemistry" (SK1) sub-goal was calculated as 0.028.It means that the reflecting the nature and structure of chemistry (SK1) sub-goal has a 2.8% relative importance within all of the goals, and the expected realization level is 2.8% when all the goals are taken into consideration.

Conclusion
In this study, an attempt was made to develop a method of measurement and evaluation that overcomes the difficulties encountered within the process of determining the effectiveness of the chemistry education based on the goals of chemistry education.In the proposed method, Analytic Hierarchy Process (AHP), which is a multi-criteria decision technique, developed by Saaty (1980) was used.
In the present study, a large number of the goals of chemistry education were hierarchically grouped according to their common features from a holistic perspective by means of the proposed method.Thirty chemistry goals taken as a basis in the study were converted into an AHP model in five main groups.The constituted AHP model is considered important in terms of determining the position of each goal within the whole.That is because the separate evaluation of the effectiveness of chemistry education for each goal will not allow for an evaluation related to the whole.Just the level at which the goal taken into consideration is realized can be determined.
Another advantage of the proposed method is the possibility of determining the local weight of each goal in the AHP model constituted from a holistic perspective.Local weights indicate the relative importance levels of goals within the group they exist in.As is stated in the introduction section, it is not possible to say that the goals of chemistry education have the same importance levels.Therefore, it is thought crucial to determine the level at which each goal is important for chemistry education.
The relative importance level of each qualitative goal can be calculated by means of the AHP technique used in the study.As a matter of fact, one of the advantages of the AHP technique is that it can model factors with both qualitative and quantitative structures.In addition, the consistency or inconsistency of pairwise comparisons allowing for the calculation of relative importance can also be calculated by means of AHP technique.This characteristic of AHP allows for an awareness of the presence or absence of any inconsistency in the effectiveness research.As a matter of fact, inconsistencies are to be expected in evaluations made in accordance with opinions of human factor.By means of the AHP technique used in this study, the consistency levels of pairwise comparisons were determined, and they were found to be within acceptable threshold values.
In the study, the global weights of each goal were determined via AHP technique.The determination of global weights answers the question as to what is of relative importance in each goal within all of the goals.For example, it is not possible to say that the "reflecting the nature and structure of chemistry" and the "enabling students to use the scientific method to solve daily problems" goals can have the same relative importance in chemistry education.Accordingly, it is necessary to consider the differences between the relative importance of goals during the evaluation.As a matter of fact, the findings of this study indicate that the relative importance of "reflecting the nature and structure of chemistry" goal within all of the goals is 0.028, and the relative importance of "enabling students to use the scientific method to solve daily problems" goal within all of the goals is 0.097.On the contrary, the separate evaluation of the effectiveness of chemistry education for each goal does not provide an opportunity to make an evaluation of the whole.However, the relative importance of each goal may vary.
In this study, the effectiveness of chemistry education based on goals was determined by using the advantages provided by the AHP technique.In the present study, the expected theoretical value range of the effectiveness is 0 -1.The nearness of effectiveness value to 1 means the perfect realization level of goals, and the nearness of effectiveness value to 0 means an insufficient realization level of goals.The effectiveness level was calculated as 0.6896 in this study.In other words, 31.04% of the goals anticipated in the chemistry education could not be realized.This value shows that there is not a perfect effectiveness level in terms of the grade where the study was conducted.The acceptability of the calculated effectiveness level may vary according to the judgments of the decision-makers and the levels set.Just as the effectiveness of chemistry education can be determined, based on all of the goals by means of the method proposed in the study; the level of effectiveness in terms of each goal can also be calculated by this method.For example, it was found in the present study that 31.04% of goals could not be realized.The goals making up the level (31.04%) that could not be realized can be determined in detail by the global weights of the sub-goals and the level of effectiveness by sub-goals values.The global weights of the sub-goals calculated by the proposed method indicate the expected realization level of each goal and the level of effectiveness by sub-goals values show the realization level of goals (Table 6).Accordingly, the difference between the level of effectiveness by sub-goals and the global weights of the sub-goals demonstrate the unrealized part of each goal.
It is possible to say that the criticisms of other measurement and evaluation techniques or methods in the literature (Gelbal & Kelecioğlu, 2007;Çelikkaya et al., 2010;Okur & Azar, 2011) can also be made of this study.Accordingly, it is not possible to say that there will be no criticism of the assumptions, content and implementation of the proposed method.The first criticism may be of the validation of the goals of chemistry education taken as a basis for the present study.As is expressed in the methods section of the study, in this study, the effectiveness level of chemistry education was determined based on the classification made by Gayon (2010).There are two main reasons for the use of this classification: The first one stems from the purpose of the study.That is to say, it was intended to develop a method for determining the effectiveness of chemistry education.The issue of what the goals of chemistry education should be or the content validation of the goals used in the study does not constitute the problem of the study.In this study, the chemistry education goals and classification as set by Gayon (2010) were assumed to be acceptable.The main reason for this assumption was that the classification made by Gayon (2010) concerning the goals of chemistry education showed a similarity with the goals included in the curriculum of the school where this study was conducted.Despite these reasons, there might be objections to the main goals and classification taken as basis in this study.However, that does not affect the operation of the proposed method.
Although the method proposed in this study intends to determine the effectiveness of chemistry education, it has a structure which can be implemented in other disciplines as well.The aspect which needs to be modified is the constitution of an AHP model according to the number and classification of goals to be taken as the basis in the study.There is no difference in other operations of the method.
In future studies, the issue of effectiveness could be examined through a fuzzy logic approach.The main reason for the use of such an approach is that some of the goals of education may be vaguely stated (Baykul, 1992), education services are often intangible, and accordingly direct measurement is difficult (Tsinidou et al., 2010).Therefore, in future research, the local and global weights of chemistry education goals can be calculated by means of fuzzy numbers.Moreover, overall evaluations can be made on the subject of effectiveness by using data that represent the views of students as well as those of course or school directors.
classification of main goals and sub-goals and is as follows: Scientific knowledge (SK)  Reflecting the nature and structure of chemistry (SK1)  Presenting chemistry as a body of scientific knowledge (SK2)  Being interdisciplinary in nature (SK3)  Presenting fundamental knowledge (SK4)  Developing fundamental understanding of chemical systems (SK5)  Providing opportunities for students to see connections among science disciplines and between science and other disciplines (SK6)  Developing fundamental understanding of chemistry concepts (SK7) Scientific Methods (SM)  Making students understand scientific inquiry as a process (SM1)  Improving students' ability to think critically about chemistry-related problems (SM2)  Focusing upon student experiences with process skills such as inferring, identifying variables, etc. (SM3)  Providing students with the ability to form a hypothesis and plan an experiment to test the hypothesis (SM4)  Enabling students to use scientific methods to solve daily problems (SM5)  Emphasizing the methods and processes of scientific investigation (SM6)  Providing opportunities for students to discover chemistry concepts through laboratory activities) (SM7) Societal Issues (SI)  Promoting the protection, conservation, and improvement of the environment (SI1)  Developing a sense of community from the local to international levels (SI2)  Emphasizing environmental concepts and world problems (SI3)  Including contemporary social problems and solutions for those problems (SI4)  Providing students with an understanding of the crucial role of chemistry and technology in our society (SI5)  Recognizing and emphasizing the moral obligation of chemistry and technology to the individual and society (SI6)  Presenting chemistry as a value-laden subject that has moral and ethical dimensions (SI7) Personal Needs (PN) -Developing the intellect of students (PN1) -To contribute to the creation of a favorable environment that will promote the development of students as human beings (PN2) -Providing information and training that will be useful for the students' future employment (PN3) -Providing opportunities for the total personal development of the students (PN4) Career Awareness (CA) -Preparing students for chemistry-related careers (CA1) -Emphasizing the importance of chemistry-related careers to the advancement of science and technology (CA2) -Informing students of the demand for chemists and chemistry teachers (CA3) -Being geared to the training of future scientists (CA4) -Promoting awareness of chemistry-related careers (CA5)

Table 1 )
, and numerical values corresponding to the evaluation findings were

Table 3 .
The pairwise comparison matrix and local weights of the main goals