An Analysis of the Studies Conducted in the Field of Curriculum Evaluation from a “Methodology” Perspective

,

When the research in the CE field is analyzed, a direct study on comprehending the paradigms and methodologies in the field of CE cannot be attained.The studies (Akıncı & Köse, 2021;Tok et al., 2020;Yetkiner et al., 2019;Özüdoğru, 2018;Dündar & Meriç, 2017;Koç, 2016, Kurt & Erdoğan, 2015Gökmenoğlu, 2014) show that the descriptive analysis of CE investigations is typically carried out. A few program evaluation components were reviewed, studies were conducted with a small sample, and some were restricted to doctorate theses. CE studies were also confined to programs connected to just one course in these studies.

Purpose of the Research
This study aimed to assess the studies in a methodological context by posing the question, "How?" these studies were conducted in curriculum evaluation. In line with the primary purpose of the research, answers to the following questions were sought: 1) What is the current situation in the context of CE approaches/methods and models in studies in CE? i) Which CE approaches/models are used?
ii) Which approaches, methods and models are used in studies conducted in the field of CE according to years and CE purpose?
iii) What are the reasons for using approaches/methods and models by those working in the field of CE?
2) What is the situation in the context of the scientific research approach used in CE studies? i) Which research approaches are used (qualitative, quantitative, mixed)? What is the scientific research approach according to years, CE purpose, and used CE model?
ii) What is the current situation regarding the sampling methods used in CE studies? Sample size and data source, what is the current situation in the number of data sources?
iii) What is the situation regarding the limitations stated in the studies conducted in CE? iv) What are the data collection tools used in studies in the field of CE? What is the situation in data collection tools according to CE models? v) What are the data analysis methods used in CE studies? vi) What is the current situation regarding the reliability and validity method techniques used in CE studies?

Method
The study was carried out in two stages. the "Ethics" curriculum received the lowest evaluations. The subject of the thesis is 228 codes for the program part, and "All" (82.9%) of these codes receive the most attention. The program's "Learning-Teaching Process" (5.3%), "Aim/Outcome" (4.8%), "Content" (3.9%), and at least "Assessment-Evaluation" (2.6%) were then the main points of attention. It can be noticed that 0.4 does not specify it.
iii) Identifying key analysis areas The theses evaluated as part of the research were decided to be analyzed under the headings of approach/model, scientific research approach, sampling, data source, limitation, data collecting, data analysis, and validity-reliability in a "methodical" context. They were transformed into a "thesis review form." Coding the document Theses were examined using a "thesis review form." First, surface reading was used to check the titles of the analyzed theses during coding. The evaluation model, scientific study technique, sampling method, sample size, data source type, restriction, data collection tool type, data analysis type, and validity/reliability type were all coded in a Microsoft Excel file.

v) Verification and analysis
The research was conducted by the analytical units chosen at the start of the investigation. In this case, descriptive analysis was employed. The codes were inserted into the appropriate themes per the specified themes. The frequency and percentage values of these codes were then calculated.

B. Interview
In order to corroborate and elaborate the findings we acquired in the first part of the research in the context of data diversity and to highlight the challenges in the field, semi-structured interviews were performed with the pertinent stakeholders. So why and how? In order to get the responses, interviews were performed.

i) Preparation of the Interview Form
A semi-structured interview form was created for the research questions based on the outcomes of the pertinent literature review and the findings from the first stage. An expert's opinion was sought to guarantee the legitimacy of this interview form, and any necessary adjustments were made in response to the comments. The interview form was tested in a trial application to guarantee its authenticity. Once more, changes were made per the results, and the final interview form was received. The interview form's final version has 22 questions with five subtopics in each (Annex-1).
ii) Determining the Individuals to be Interviewed The maximum variation sampling method-one of the purposive sampling techniques-was employed because selecting the sample for the study's interviews required a comprehensive methodology. In order to achieve a uniform distribution from both the students who authored the thesis (16) and the advisers to the thesis, the period between 1991 and 2020 was divided into five-year procedures according to the titles of the faculty members, the education level of the program in which the thesis was assessed, and whether it was a master's or doctorate thesis.
A sample of 32 people from faculty members (16) was chosen. Twenty-nine of the individuals took PD-related courses, while three did not, according to an analysis of their academic education processes. Additionally, it was found that 12 participants were exclusively applied, ten individuals were only theoretical, and 18 participants were both theoretical and practical.

iii) Application of the Interview Form
In the study's second phase, appointments were set voluntarily by phone or email, and interviews were conducted and videotaped over the phone and using the zoom application. On average, interviews took 40 to 45 minutes to complete. The interview transcripts were turned into interview texts and distributed to the participants through email or phone. The participants were requested to examine the texts, and after their review, the required portions received the necessary adjustments. During the interviews, the participants were given information regarding the subject, such as explaining what the term "purposeful context" means.

iv) Analysis of the Data in the Interview Form
Descriptive and content analysis approaches were employed for the study's interview component. Prior to analysis, the data were prepared. The Word document was dictated for each of the recorded interviews. Thesis writers (students) are identified as A1,..., and A16, respectively, while adviser faculty members (field specialists) are identified as T1...T16, respectively, using pseudonyms. The analysis was aided by the use of the MAXQDA 2020 qualitative data analysis tool. The codes were created by looking at the responses, and then the categories and topics were determined. The advisor for the thesis and the faculty member often communicated throughout the coding process.

C. Validity-Reliability
To assure the research's objectivity, extensive interaction, depth-focused data collecting, diversification, expert evaluation, and participant confirmation were conducted. Detailed descriptions and purposeful sample techniques are advised to ensure transferability. For the research, a thorough description and purposeful sampling were employed to guarantee transferability. To ensure consistency throughout the investigation, a consistency analysis was done. One month after the initial data coding was finished, the researcher coded the quantitative and qualitative data again for this purpose. To verify confirmability, a confirmation review was completed.

Findings Related to the First Research Question
In order to answer the first research question, we looked at which CE methodologies and models are employed in CE studies, how they are distributed among years and CE objectives, and the rationales behind why researchers choose to employ these approaches.

i) Findings on approaches and models used in CE studies
The analysis is presented in the context of the CE assessment model used in studies in CE. that the CIPP model (15%) and "management-based techniques" (14%) were utilized most frequently in theses. It has been observed that theses frequently use methodologies, strategies, and expressions for CE that are unrelated to CE approaches and models, including delphi, bloom, total, product, and process.
ii) Findings regarding the approaches/methods and models used in studies conducted in the field of CE by years and type of CE purpose The analysis in the context of CE evaluation approaches/methods and models used periodically by years in studies in the field of CE is presented in Table 2. Table 2. CE approach/methods and models by years 1991-1995 1996-2000 2001-2005 2006 The most significant intersection in Table 2 was between the 2011-2015 time period and the statement "no model." This intersection can be attributed to the fact that models were not used in 134 theses. Additionally, it was discovered that from 1991 to 1995, no models were utilized and that between 2016 and 2020, when 34 researches were conducted, most models were employed. Additionally, it is significant that studies typically use the CIPP model, but alternative models are occasionally used. Table 3 analyzes the CE evaluation approaches, methods, and models utilized in CE research according to the goal type. The essentialist inquiry type and the no model statement have shown the most significant amount of intersection in Table 3. The UCLA approach employed comparison, idealization, and essentialist questions kinds. There was no use of a decision-making question type. The CIPP model used the most essentialist question type but hardly ever used the question type with decision-making utility. Instrumental and essentialist question types were most frequently utilized in the Tyler model.
iii) Findings regarding the reasons for using approaches and methods/models of CE students The CE approach used by those working in the field of CE and the reasons for using it is presented in Table 4. According to the interviews with researchers working in the field of CE, when Table 4 was evaluated, 32 researchers provided 31 codes. The majority of techniques are "purpose-based," followed by "management-based," "participant-oriented," "expert-oriented," and "consumer-oriented." However, it was discovered from examining the table that many researchers cannot defend their methods. There were a total of 8 codes mentioned by those who gave their justifications. It can be seen from the table that the researchers take these techniques into account for their studies since, for instance, utilizing a goal-based approach is crucial to achieving the aims. ies.ccsenet.org International Vol. 16, No. 1;2023 In this regard, "A10," one of the researchers, claims that he employed the "purpose-based" methodology in the following manner: "I particularly tried to do goal-oriented CE because it is important to achieve goals, especially in the education program I studied, which is aimed at gaining communicative competence and at students becoming proficient in all of their reading, writing, speaking, and listening skills" (A10 Interview Transcript, Position 42).
Upon examining the A10 opinion, it is apparent that the author uses a purpose-driven approach in his/her thesis. However, he fails to state why he holds this opinion.
When the justifications listed in the table are taken into consideration, it becomes clear that "it is crucial to achieving the goals" for the goal-based approach, "it gains importance with constructivism" for the participant-oriented approach, "it is a professional process" for the expert-oriented approach, and "to expose how the program is processed" for many different methods in collaboration with the team. Gave their justifications.
The findings related to the CE model used by the researchers who conducted CE studies and the reasons for using this model are presented in Table 5. According to the data gathered from the interviews with academics working in the field of CE, 32 researchers provided 54 codes when Table 5 was analyzed. The "CIPP" model was most frequently used, and it was followed by "No model," "Kirkpatrick," "Provus' Differences Model," "Tyler's Model," "Stake's Suitability-Probability Model," "Alkin's Model," "Hammond's Cube Model," "Demirel's Analytical Model," "Eisner Educational Criticism Model," "Illuminating Model," "Target-free Model," However, a closer examination of the table reveals that many researchers cannot justify why they use these models. Those who presented their justifications also stated a total of 22 codes.
The table shows that the researchers' motivation for considering these models in their studies, such as the CIPP model, is "to be very comprehensive".
In this regard, one of the researchers states that he used the "CIPP" model in his study "A16" as follows: "I used the CIPP model. I used it because it offers the opportunity to evaluate each step of the program separately and is quite comprehensive." (A16 interview form, Position 42).
When the A16 opinion was analyzed, it was discovered that the author employed the CIPP model in his thesis and articulated it in this manner, but he did not provide a reasonable explanation for why.
The most frequent justifications for the CIPP model when the justifications listed in the table are examined are "to be very comprehensive," "the models are not valid today" for no model, "a model suitable for study" for the Kirkpatrick model, "guidance" by Hammond for the cube model, Demirel's argument, and "a model suitable for study" for the Kirkpatrick model. For Eisner's analytical model and "being systematic" for his educational criticism model, they gave the justifications for "being in Turkey" and "being systematic," respectively.

Findings Related to the Second Research Question
In the second research question, the analysis of the studies in the field of CE was presented in the context of the scientific research approaches used.

i) Findings on scientific research approaches used in CE studies
Findings related to scientific research approaches used in CE studies are presented in Table 6. The quantitative approach was used in 58% (126) of the 215 theses examined in Table 6; 22% (47) of them used the mixed approach; It is seen that 20% (42) used the qualitative approach.

Findings of scientific research approach according to year periods in studies in the field of CE
The findings of scientific research approaches according to year periods in studies conducted in CE are presented in Table 7. The quantitative method for 2011 to 2015 shows the most significant overlap, according to Table 7. Most thesis research using quantitative, qualitative, and mixed methods was completed between 2011 and 2015.
Findings on scientific approaches used according to the aims of studies in the field of CE The findings of CE models according to scientific research approaches in studies conducted in CE are presented in Table 8.  The qualitative approach and the phrase "no model" have been found to intersect most frequently when Table 8 is evaluated. The UCLA model-using study is an empirical investigation. The CIPP model was mainly used in mixed research. Quantitative research has typically employed the Eisner model. Most importantly, qualitative research has been used to apply the Tyler model.

ii) Findings on sampling methods used in studies in the field of CE
The findings regarding the sampling methods used in the studies conducted in the field of CE are presented in Table 9.  Table 9 shows that there are 291 codes linked to the sampling method in the theses, with 18% of these codes being "unspecified." It can be shown that "basic random sampling" (17%) is the most chosen sample technique. It may be shown that at least other sampling techniques are employed.
Findings regarding the sample size used in studies in the field of CE Findings regarding the sample size used in studies conducted in the field of CE are presented in Table 10.  The sample size between "11-50" (29%) is the most frequently used among the 584 codes relating to the sample size in the theses, according to Table 10. As observed, at least a sample size of "1001 and above" (2%) is being used.
Findings related to the type of data source used in studies in the field of CE Findings related to the type of data source used in studies conducted in the field of CE are presented in Table 11.  Table 11 shows that there are 429 codes connected to the data source in the theses, with "teacher" accounting for 32% of these codes. The group with the fewest students is the "lecturer" (1%).
Findings on the number of data source types used in studies in the field of CE The findings regarding the number of data source types used in studies conducted in the field of CE are presented in Table 12. In 48% (103) of the 215 theses examined in Table 12, one; 26% (55) had two; 13% (28) had three; 8% (18) had four; 3% (7) five; six (3) in 1%; 1% of (1) seven; data source type is used. ies.ccsenet.org International Education Studies Vol. 16, No. 1;2023 iii) Findings on the limitations of studies in the field of CE Findings related to the limitations of studies conducted in the field of CE are presented in Table 13.  Table 13 reveals that there are 332 codes connected to the theses' constraints, with "no model" (40%) being the most restrictive of these categories. It can be seen that there are, at the very least, "single unit" and "other" constraints (only the administrator's opinion and only the parent's opinion, respectively) (1%).
iv) Findings on the type of data collection tool and data collection tool used in studies in the field of CE The type of data collection tool used in CE studies and the findings regarding the data collection tool is presented in Table 14. When Table 14 is examined, it is seen that there are 642 codes related to data collection tools in theses, and among these codes, "interview" (31%) is used as the data collection tool type. It is seen that forms are used at least.
Findings of CE models according to the data collection tool in the studies in the field of CE The findings of CE models according to the data collection tool included in the studies conducted in the field of CE are presented in Table 15.    Table 15 analysis revealed that the questionnaire and no model had the most outstanding amount of intersection. The study, which applied the UCLA model, used interview forms and materials. The interview form was employed more frequently than the achievement test in the research utilizing the CIPP paradigm. The Eisner model employed the interview form more frequently than the achievement test and observation form. The achievement test and observation form were used sparingly in research based on the Tyler model. v) Findings related to data analysis methods in studies conducted in the field of CE Findings related to data analysis methods in studies conducted in the field of CE are presented in Table 16. When Table 16 is studied, it can be observed that there are 1074 codes relating to data analysis methods in the theses, with the majority of these codes (59%) using the descriptive statistics data analysis method. Other research has demonstrated adopting the "inferential statistics" data analysis method (41%). vi) Findings on the number of validity types in studies conducted in the field of CE The findings regarding the number of validity types included in studies conducted in the field of CE are presented in Table 17. The validity type was not utilized in 6% (14) of the 215 theses that were looked at in Table 17. Instead, 18% (38) had one, 25% (53) had two, 23% (50) had three, 9% (19) had four, 8% (17) had five, below 5% (11), and 4% (9) ate. It is also noticeable that 2% (4) use eight varieties of validity.

Findings Related to Types of Validity in Studies in the Field of CE
The findings regarding the types of validity in the studies conducted in the field of CE are presented in Table 18. There are 616 codes connected to the type of validity in the theses, and "expert opinion" (32%) is the most frequently used. This information is shown in Table 18. We can see that "other" validity kinds (1%) are utilized the least frequently.
The number of reliability types included in studies conducted in the field of CE, related findings The findings related to the number of reliability types included in studies conducted in the field of CE are presented in Table 19. ies.ccsenet.org International Education Studies Vol. 16, No. 1;2023  Reliability type was not used in 29% (63) of the 215 theses examined in Table 19; one in 48% (104); 16% (34) had two; 6% (13) had three; It is seen that 0.5% (1) use four reliability types.
Findings related to reliability types in studies conducted in the field of CE Findings related to reliability in studies conducted in the field of CE are presented in Table 20. Total 215 100 Table 20 shows that Cronbach's Alpha (53%) reliability type is used the most, and correlation (4%) reliability type is used the least.

Discussion
CE studies were analyzed "methodologically," which corresponds to the answer to the question "How?" We were seeking answers to how CE research approaches and models were applied.
Based on the document analysis, more than half of the theses examined do not use CE models, while CIPP models are primarily used. Özüdoğru (2018), Kurt and Erdoğan (2015), and Taş and Duman (2020) concluded from their studies that most theses did not use models, and the CIPP model was prevalent in those that did. Akıncı and Köse (2021) examined the studies using the model, and it was determined that the CIPP model was most preferred. Considering that researchers mostly use the CIPP model, it is reasonable to assume that other researchers have adopted it. It was given in the findings that the expressions that are not related to the CE model in the examined theses are also used as the CE model type. This shows that some researchers lack cognitive awareness about CE models.
From a year perspective, 1991-1995 had no models, whereas 2016-2020 had the most models. CE researchers gradually become more cognizant of CE models with each passing day. According to the question type, While the most essentialist question type was used in the CIPP model, the question type with decision-making value was used very little. In the Tyler model, instrumental and essentialist question types were used the most. However, CIPP is a management-based model, and questions with decision value should be used more. The Tyler model is goal-based, and according to this approach, the outputs are more important; it can be said that question types with instrumental value can be used in this model.
The theses under review also use phrases unrelated to the CE model.
The interviews' findings show that participants primarily employ the "purpose-based" CE method in their research. The goal-based CE strategy is said to be employed for many reasons, including that "it is important to attain the goals, the planning is explicit, and it is applicable." They employ "many ways based on management, participant-oriented, expert-oriented, together with the team," despite poor rates. It can be noted that participants primarily employ the CIPP model in their research. Most respondents who chose the model said they did so because it was "extremely complete." They said the participants' response, "There is no model," was given because "models are specific examples of the input-process-output structure." Akıncı and Köse (2021) examined the studies using the model related to curriculum evaluation and found that the CIPP model was preferred the most, and the reason for the "multidimensionality" was presented as the reason for choosing CIPP. Christie and Lemire (2019) also emphasized that models are theories that draw attention to the context-mechanism-outcome configuration.
It is seen that the reasons why the participants use the CE approaches and models are not clear enough, and some of the reasons do not fully meet the relevant CE approach and model.
The study observed that the code "depends on the context" was obtained about which model the participants used. Nouraey et al. (2020) concluded that selecting an appropriate evaluation model will depend on several criteria, such as the evaluation's context, purpose, and expected outcome.
It can be seen that the participants agreed that every aspect of CE is crucial. This indicates that the participants believe no component is more significant than the others. Document analysis reveals that more than half of the theses studied as documents in the "methodological" context take a quantitative approach; this correlates to the response to the "how" issue in CE studies. It was found that the most quantitative research methodology was applied in the studies of Koç (2016) and Kurt and Erdoğan (2015). Kurt and Erdoğan (2015) interpreted this to mean that the quantitative approach's prominence is because it yields findings more quickly than other approaches.
Again, Azzam (2011) discovered that the evaluators chose a more quantitative methodology in his study, which studied the impact of specific demographic characteristics of the evaluators on their choice of methodology. This result supports the quantitative approach to taking the lead.
Most of these studies lack the characteristics of formal CE studies; in real CE studies, philosophy, ideology, question type, and more qualitative studies should predominate. When CE is generally mentioned in studies and success tests, there is logic to obtaining numerical data by using CE, but it is seen that the quantitative approach is more common.
In the study of Yetkiner et al. (2019), the study of Tok et al. (2020), the study of Akıncı and Köse (2021), and the study of Özüdoğru (2018), the number of mixed studies were found to be higher. Some researchers also state that paradigms tend to coexist and are used when appropriate; for example, they argue that while some research questions are suitable to be analyzed by quantitative methods, some questions are suitable to be analyzed by qualitative methods (Erickson, 1986;Lakatos, 1978). This may explain the greater use of mixed studies in these studies.
It can be observed from the year periods that the years 2011 to 2015 are when the quantitative technique is most frequently applied. The research that used the most qualitative method did not use the model according to the PD models. A quantitative approach can be applied in graduate studies between 2011 and 2015.
It is common to find that the sampling technique employed in CE studies is "basic random sampling," which is not stated in CE studies to a great extent. Similar findings were observed in the investigations of Kurt and Erdoğan (2015) and Özüdoğru (2018), where the majority of the studies did not specify the sampling method and the random (simple random sampling) method predominated. The bulk of the studies that Akıncı and Köse (2021) looked at failed to mention the sampling technique. It was observed that the population received the most significant direct application in the studies listed.
that this is why more teachers are chosen as data sources. In the studies of Yetkiner et al. (2019), it was seen that academics were more prominent. In the studies of Taş and Duman (2020), it was determined that most documents were used as data sources.
There is no model in more than half of the investigations, which is a restriction. The most popular methods for gathering data are questionnaires and interviews. In studies that do not employ models, it is evident from the models that surveys are mainly used.
The two techniques most frequently utilized in data analysis are descriptive statistics and content analysis. Descriptive statistics was the study's most popular data analysis technique, followed by content analysis. Kurt and Erdoğan (2015) found that descriptive statistics are at the forefront of the quantitative method and content analysis is in the foreground of the qualitative method. In the study of Özüdoğru (2018), it was seen that descriptive statistics were also in the foreground.
It can be shown that more than half of the research used multiple types of validity, with the expert opinion being the most prevalent. In more than half of the studies, reliability was not specified, or only (1) reliability type was stated. It is seen that Cronbach Alpha reliability is the most used reliability type. In the study, it was determined that expert opinion came to the fore in validity methods. Özüdoğru (2018) found that expert opinion was used the most as a validity method in the theses he examined. As for reliability, it was seen that Cronbach's Alpha method was mainly used.
In light of this, it can be argued that the evaluation approach and model used in research undertaken in CE have flaws and shortcomings. In conclusion, it can be claimed that the methodological framework of the studies done in CE has issues and deficiencies.