The Effects of the Integration of Indirect Feedback and Concordances on Improving Grammatical Accuracy in Thai EFL Students’ Writing

This study aimed to investigate the effects of the integration of indirect feedback and concordances on improving grammatical accuracy in Thai EFL undergraduate students’ writing and to explore the students’ perceptions toward the intervention. Thirty third-year undergraduate students who studied at a government university in Bangkok were the participants of this study. During the experiment, participants were required to perform three writing tasks. In each writing task, they wrote the first draft and then received both direct and indirect feedback. For indirect feedback, participants were asked to look at concordances to help revise their first draft. The pre-test and the post-test were used to assess grammatical accuracy in the students’ writing before and after the experiment. Additionally, the questionnaire and the semi-structured interview were used to find out the students’ perceptions toward the intervention. The findings revealed that the number of grammatical errors showed a significant decrease in the post-test compared with the pre-test. As a result, the integration of indirect feedback and concordances was seen to be able to improve grammatical accuracy in the participants’ writing. Besides, the results from the questionnaire and the semi-structured interview showed that the participants had positive perceptions toward the intervention on improving grammatical accuracy in their writing because they had a chance to learn from their mistakes after receiving indirect feedback and concordances helped them to induce grammatical patterns.


Introduction
In the 21 st century, the global trend is changing rapidly, so this situation leads to the competition of countries around the world. Although Thai society has become more international, Thai workers still have lower-than-expected performance in terms of English language skills (The Twelfth National Economic andSocial Development Plan (2017-2021) of the Office of the Prime Minister, p. 51). Consequently, in Thailand, the Minister of Education has created Thai Qualifications Framework for Higher Education (TQF: HEd), which is related to students who study in the English Language Program in Thai universities in order to promote English language education so that students will have abilities to catch up with the global trend. In order to standardize the quality of education, this framework indicates that the students should be able to achieve CEFR C1 level (Proficient User) before finishing their degree. In other words, students should be able to listen and speak in English in various contexts (i.e., academic, social, and careers), summarize the main ideas after listening to those contexts, analyze the information from different kinds of media, and give both oral and written presentations accurately (translated from Thai Qualifications Framework for Higher Education of the Ministry of Education, p. 3).
In order to achieve CEFR C1 level, Thai universities need to provide students in the English Language Program with knowledge of four skills: listening, speaking, reading, and writing. Among all the four language skills, writing has an important role for students' academic achievement. They need to master writing skills because this level requires an ability to give written presentations accurately. However, it is still challenging for them to develop their writing skills because English language is different from Thai language in the rules of constructing sentences. Thai EFL students have serious problems with writing skills because they think that they can translate sentences word-by-word to construct each sentence. They have grammatical errors in their writing due to the influence of L1 grammatical patterns (Iamsiu, 2014). As a consequence, it is important to help Thai EFL students improve grammatical accuracy. The use of different kinds of technology in English language teaching has been proven by many researchers as one of the most effective methods to help Thai EFL students improve English language proficiency (Deerajviset, 2014). Before conducting the study, the researchers reviewed a number of previous research studies on the use of technology in English language teaching and found that a few research on a corpus, which is one of effective technology tools, had been conducted in Thailand. Hence, a corpus should not be neglected by Thai EFL students and teachers, so this study attempted to improve grammatical accuracy in Thai EFL students' writing by integrating concordances with indirect feedback.

Significance of the Study
It is expected that the findings of this research will be useful to two groups of people including EFL undergraduate students and teachers who teach English writing. Firstly, it is beneficial to EFL undergraduate students who have problems in grammatical accuracy. By understanding the effectiveness of corpora, students can use concordances to observe grammar usages. In addition to students in the classroom context, students who are out of the schooling system can use a corpus to improve their writing skills. Secondly, this study may be helpful to EFL teachers to promote the use of corpora in writing classrooms. By understanding the effectiveness of corpora, teachers can develop the writing course that fits students' needs. For example, teachers can integrate concordances into indirect feedback. Students can discover usage patterns and rules from concordance data.

Purposes of the Study
In order to improve grammatical accuracy in Thai EFL undergraduate students' writing, the integration of indirect feedback and concordances can be employed in writing classrooms. Therefore, the researchers decided to first investigate the effects of the integration of indirect feedback and concordances on improving grammatical accuracy in Thai EFL undergraduate students' writing. Secondly, Thai EFL undergraduate students' perceptions toward the integration of indirect feedback and concordances on improving grammatical accuracy were also investigated.

Research Questions
1) How does the integration of indirect feedback and concordances affect grammatical accuracy in Thai EFL undergraduate students' writing?
2) What are Thai EFL undergraduate students' perceptions toward the integration of indirect feedback and concordances on improving grammatical accuracy?

Direct and Indirect Feedback
Types of feedback can be created under how teachers correct students' writing. It can be categorized as direct and indirect feedback (Ellis, 2009). The details of these two types of feedback are pointed out as follows: a. Direct Feedback: According to Ellis (2009), direct feedback involves the correct form which is given to students by the teacher. The teacher can cross out, rewrite, and add words or phrases to correct errors. The way that teachers make changes to students' writing tasks is defined as editorial feedback (Hughes, 2013). When teachers use editorial feedback, the explanation of the changes is not presented in students' writing tasks. Additionally, using corrective symbols is considered as direct feedback as teachers can use them to identify students' errors (Boramy, 2013). b. Indirect Feedback: As opposed to direct feedback, indirect feedback means that the teacher does not provide the correct form (Chatranonth, 2007 as cited in Sommanochai & Meesri, 2018). Without giving the correct form, teachers can underline, circle, or highlight words or phrases (Ellis, 2009). Indirect feedback has the advantage that it motivates students to reflect linguistic errors. An error correction code can help students reflect errors because it can be used as a clue to identify the type of mistakes students made (Lackman, 2010). According to Shirotha (2016), indirect feedback should be promoted in EFL writing classrooms because it elicits autonomous learning by allowing students to be more reflective and analytical about their errors, and thus leads to lifelong learning.

Application of Corpora in EFL Writing
A corpus is a body line of texts which represent an aspect of language in both spoken and written forms (Spiri, 2012). A large and carefully gathered corpus is known as a beneficial resource (Pravec, 2002 as cited in Divsar & Heydari, 2017). The corpus has changed English language instruction due to the rapid development of technology. The corpus is built and utilized by teachers, learners, researchers, and anyone who has an internet connection. It is available on the worldwide web, so it is convenient for users to make use of it.
Corpora have been widely used in English language teaching (Xiao & Chen, 2018). Users can access corpora on the worldwide web and use a variety of corpora via concordance programs. According to Spiri (2012), users can see how a word is used in context by using concordance programs. There are many online concordance programs that students can choose in order to develop their English language skills. Corpus of Contemporary American English (COCA) is seen as representative of American English with over 560 million words (Davies, 2018), and it is also popular for EFL researchers. COCA can be used to develop students' writing skills. According to Xiao and Chen (2018), COCA can be applied in EFL writing instruction at three stages of writing process including pre-writing, while-writing, and post-writing. Firstly, application of COCA at pre-writing can help students inspire their ideas around the topic of writing. Students can use the searching of key words and collocations in COCA to search lexical items closely related with the key words, and this way can inspire their ideas. Secondly, at while-writing stage, students' errors in terms of violation of semantic prosody, mismatching collocation, lexical usage in writing, and genre appropriateness can be resolved by using COCA. Lastly, at post-writing stage, COCA can be used for providing effective feedback for students' writing. Feedback can be given to students in terms of content and language.

The Role of Corpora in Grammatical Accuracy
As mentioned in the previous section, a corpus can be used to develop students' writing skills in the writing process as it can be used to help correct lexical errors, such as mismatching collocation (Xiao & Chen, 2018). In addition to lexical errors, a corpus can be used to help correct grammatical errors. According to Jones, Myhills, and Bailey (2013), grammar plays a significant role in the writing process because it is a meaning-making resource. However, due to its difficulty, EFL students always have grammatical errors. Teachers can help students reduce grammatical errors by using a corpus in writing classrooms. The role of teachers is a facilitator who gives guidance on how to use a corpus. Students are asked to use a concordance program to explore authentic language and observe patterns. This method is called discovery-based learning as students have a chance to induce grammatical rules in order to correct grammatical errors by themselves (Chujo, Anthony, Oghigian, & Uchibori, 2012).
There were research studies focusing on the effectiveness of corpora on grammartical accuracy in students' writing. Firstly, Liu and Jiang (2009) investigated the effects of integrating corpora and contextualized lexicogrammar in foreign and second language teaching. Participants of this research were 236 EFL/ESL students and 8 instructors. There were three teaching activities in the classes. The first activity related to concordance data used to help students understand the difference between the two words' meanings and usages. The second activity was used to help students address the lexicogrammatical errors they made in their writing with the help of corpus data. The last activity was used to find the meanings, usage patterns, and register information by using the BNC. The study showed that the use of corpora and lexiocogrammar can enhance students' language awareness, improve their command of lexiocogrammatical rules and usage patterns, increase their appreciation of context in language use and their critical understanding of grammar, and promote discovery learning.
The next study was conducted by Tasanameelarp and Laohawiriyanon (2011) using 37 Thai EFL students at a private high school in the southern Thailand to examine whether students were able to self-correct grammatical errors and retain required grammatical rules after using concordances. Students were asked to compose a narrative so that the researcher could find the five most common types of grammatical errors and design the concordance exercises. After that, the researcher trained students how to use the concordance and practice how to induce grammatical patterns from concordance outputs. Finally, students were asked to self-correct their own work, and only the five types of grammatical errors underlined. The findings showed that among the five types of errors including nouns, verbs, prepositions, articles, and subject-verb agreement, subject-verb agreement was corrected the most accurately.

Related Studies
The aforementioned studies were conducted on the impact of concordances on grammatical accuracy in students' writing. Hence, to make indirect feedback more effective, it is important to find studies focusing on the effectiveness of the integration of indirect feedback and concordances on grammatical accuracy in students' writing. Firstly, Tono, Satake, and Miura (2014) explored the effectiveness of corpus use on the correction of grammatical errors. A total of 93 undergraduate students at a private university in Tokyo were asked to write an essay, and then they were asked to revise their essays based on indirect feedback. Each student received only two problematic segments. The first one was intended to be corrected by using a corpus, whereas students were asked to correct the second one without the use of a corpus. Each of the two errors was highlighted and special codes were also used to help identify types of errors. Students were instructed to search for the highlighted words and examine the concordances lines to identify correct grammatical patterns. The results showed that students successfully corrected most of grammatical errors in prepositions by using corpus data.
Another interesting study provided the appropriate way in utilizing a corpus with indirect feedback. Satake (2020) investigated the use of a corpus as a reference tool by 55 undergraduate students from Tokyo. Students were asked to write an essay. After that, they were asked to revise their essays using COCA. The researcher gave indirect feedback by highlighting two errors in each essay. Although each student received indirect feedback on only two errors, the researcher intended to identify as many different kinds of errors as possible. Then they were asked to use COCA to search for underlined words and interpret the concordance lines. The results revealed that students seemed to use COCA effectively to correct omission errors.
Even though there have been research studies on the effects of the integration of indirect feedback and concordances on improving grammatical accuracy in students' writing, the perceptions of the intervention have not been deeply investigated. Besides, it is acceptable to state that there is the lack of research on the effects of the integration of the invention on improving grammatical accuracy in Thai EFL undergraduate students' writing.
Consequently, a study of the integration of indirect feedback and concordances provided by COCA on improving grammatical accuracy in Thai EFL students' writing will be useful for both teachers and students in the 21 st century with the meaningful information from the findings.

Research Design
This study employed a mixed-method quantitative and qualitative research approach, with the main aim of examining pre-to post-changes in grammatical accuracy and students' perceptions toward the integration of indirect feedback and concordances on improving grammatical accuracy. The data from quantitative research approach were collected from a pre-test, a post-test, and a questionnaire while the data from qualitative research approach were collected from a semi-structured interview. This study aimed to look for a causal relationship between the independent variable (=the integration of indirect feedback and concordances) and the dependent variables (=grammatical accuracy in students' writing).

Participants
Approximately ninety students who enrolled in Paragraph Writing in the academic year of 2020 were the population of this study. The researchers chose only thirty third-year undergraduate students. The participants were an intact group of students who studied in the English Language Program at a government university in Bangkok. Before all participants studied Paragraph Writing, they were required to enroll in Controlled and Formulaic Writing. Consequently, all of them had the same background knowledge of writing because they passed Controlled and Formulaic Writing before they studied Paragraph Writing. However, as mentioned earlier, students in the English Language Program should achieve CEFR C1 level, but the participants' scores were equivalent to B1 to B2 in the CEFR. Therefore, their scores did not meet the standard.

Writing Test
The aim of the writing test in this research was to be used as the pre-test and post-test. It was intended to measure grammatical accuracy in participants' writing. To further explain this, the pre-test and post-test had the same instructions. In other words, the participants were asked to write a paragraph with the topic, My Favorite Place to Relax. In both the pre-test and the post-test, participants were required to write 100-120 words within 45 minutes. If students wrote less than 100 words or more than 120 words, the test would not be scored. During the pre-test and post-test, participants were not allowed to use dictionaries. In the context of the present study, grammatical accuracy depends on a number of their writing errors. In order to analyze grammatical accuracy in participants' writing in both the pre-test and the post-test, the researchers used an error classification system created by the researchers in order to find a number of their writing errors. The researchers needed to classify each error based on one of three categories. The categories included a) grammar which refers to articles, nouns, subject-verb agreement, prepositions, infinitives/gerunds, and verb forms b) mechanics which refers to punctuation and capitalization and c) sentence structure which refers to sentence fragment and word order.

Concordances
A concordance program investigated in this study is Corpus of Contemporary American English (COCA).
Although there are various concordance programs, the researchers chose COCA because it represents American elt.ccsenet.org Vol. 15, No. 4; English, and it is popular for EFL researchers to develop students' English writing skills. In addition, all the language resources in COCA are accessible to users for free via the registration on the website. Users can make use of seven functions in COCA to develop their English language skills. In this study, "keyword in context (KWIC)" was chosen for the experiment because this function provides concordances which can help Thai EFL students improve grammatical accuracy. Concordances can be used to identify structural usage patterns. Students can examine the concordance data regarding the word that they have difficulties.

Questionnaire
Twelve questionnaire items were used to survey the participants' perceptions toward the integration of indirect feedback and concordances on improving grammatical accuracy. The questionnaire was adapted from Wang, Shang, and Briody (2013). It consisted of a four-point Likert scale, ranging from 4 (strongly agree) to 1 (strongly disagree). Obviously, a four-point Likert scale was used in this study because the researchers needed the respondents to choose whether they 'agree' (i.e., strongly agree, agree) or 'disagree' (i.e., strongly disagree, disagree). If the researchers use a five-point Likert scale, the respondents might choose 'moderate' and it might be difficult for the researchers to analyze the data. As a result, the researchers used a four-point Likert scale because the results would be more accurate and easier to analyze. Besides, all questionnaire items were translated into a Thai version in order to avoid misinterpretation or confusion among participants. Therefore, questions were written in both Thai and English. Questions 1-10 asked what the participants think about using COCA to gain knowledge of articles, nouns, subject-verb agreement, prepositions, infinitives/gerunds, verb forms, punctuation, capitalization, word order, and sentence fragment. Questions 11-12 asked whether the participants think it is useful to use COCA or not.

Semi-Structured Interview
A semi-structured interview was employed as the means of qualitative data collection to find out whether the integration of indirect feedback and concordances could help Thai EFL students improve grammatical accuracy in their writing. There were the five most improved participants and the five least improved participants. The interview session was done privately on a one-to-one basis. Each participant was interviewed for about fifteen minutes. During the interview process, each participant could answer the questions in Thai.

Research Procedure
At the outset of the research, participants were asked to do the writing pre-test and then submitted it online. One week later, they learned how to use a corpus for the revision. Before participants learned how to use COCA, they learned how to identify types of grammatical errors from indirect feedback. This study focused on ten most common types of grammatical errors (i.e., capitalization, prepositions, nouns, sentence fragment, verb forms, infinitives/gerunds, punctuation, subject-verb agreement, articles, and word order). As a result, the researchers used indirect feedback to correct only those ten types of errors. For other types of errors, the researchers used direct feedback. An example of a paragraph with both direct and indirect feedback was given to participants. For direct feedback, the researchers rewrote words, so the correct ones were given to participants. For indirect feedback, instead of giving the correct form, the researchers circled words and wrote an error correction code. According to Evans, Hartshorn, McCollum, and Wolfersberger (2010), a circle and an error symbol can be used together to make indirect feedback more meaningful. As a result, the researchers used both a circle and an error correction code.
As mentioned earlier, participants needed to learn how to identify types of grammatical errors from indirect feedback before they learned how to use COCA. Consequently, they needed to understand indirect feedback before they corrected errors, so they were asked to study an error correction code (see Appendix A) before they learned how to use COCA. After that, participants learned only one function of COCA which was "keyword in context (KWIC)". The researchers taught participants how to type words in "KWIC". Participants needed to type words in a circle in order to look at the concordance lines. In addition to words in a circle, for some types of errors, participants needed to type some specific codes to identify parts of speech. Next, the researchers trained participants to use COCA to help correct each type of errors. An example of indirect feedback is shown in Figure  1. Participants were asked to type the words "a lot of *_nn*". Then they had to look at the concordance lines. A computer screenshot of concordance lines is illustrated in Figure 2. Participants had to correct the error by using a plural noun after "a lot of". One week later, participants practiced using COCA for the revision. Another example of a paragraph with both direct and indirect feedback was given to participants. They were asked to use COCA to help revise the paragraph. The training session was done for two weeks.
One week later, participants were asked to perform the first writing task. During the experiment, the researchers chose the writing process (i.e., introductory session, writing session, revising session) created by Lai (2009 as cited in Wang et al., 2013). In this process, participants wrote only two drafts for each type of writing. They were asked to write the first draft based on the topic given by the researchers. They had to write the first draft and then submit their first draft online within one hour. One week later, participants received both direct and indirect feedback from the researchers. The researchers gave indirect feedback to participants by circling words and using an error correction code. After that, participants were asked to revise their first draft by using COCA. After they revised their first draft, they were asked to submit their final draft online. These steps had to be completed within four hours. Participants had two class sessions to finish the first writing task. However, they were required to perform three writing tasks in total, and thus, the writing tasks were done for six consecutive weeks. Similar procedures were followed for the other two writing lessons and tasks.
One week after three writing tasks, a post-test with the same writing instruction was conducted online. One week after the post-test, participants were asked to fill in the questionnaire. One week later, only ten participants were selected for a semi-structured interview. Since the researchers set the clear procedure in advance, participants received the pre-test, the treatment, and the post-test within the timeframe of their regular class sessions.

Writing Test
The number of participants' grammatical errors in the pre-test and the post-test was counted and classified using an error classification system. After that, the result of the pre-test and the post-test was compared. A chi-square test was used to examine the difference between the number of errors of the pre-test and the post-test.

Questionnaire
Frequency was computed from the questionnaire to collect self-report data about the effects of the integration of indirect feedback and concordances on improving grammatical accuracy.

Semi-Structured Interview
The researchers collected and analyzed the semi-structured interview. The method of investigating the experiences of participants with reference to the use of the integration of indirect feedback and concordances on improving grammatical accuracy was qualitative content analysis. According to Dörnyei (2007), qualitative content analysis refers to an analysis of the deeper meaning of the data. All of the participants' responses from the semi-structured interview were recorded and transcribed in order to explore their perceptions toward the integration of indirect feedback and concordances on improving grammatical accuracy.

Data from the Pre-Test and the Post-Test
This section presents quantitative research results obtained from the pre-test and the post-test. Before and after the learning through the integration of indirect feedback and concordances, the participants had a pre-test and a post-test that aimed to assess grammatical accuracy in their writing. This study focuses on ten types of grammatical errors including 1) articles 2) nouns 3) subject-verb agreement 4) prepositions 5) infinitives/gerunds elt.ccsenet.org Vol. 15, No. 4; 6) verb forms 7) punctuation 8) capitalization 9) sentence fragment, and 10) word order. Therefore, the researchers counted errors in the pre-test and the post-test based on only ten types of grammatical errors. In this study, grammatical accuracy depends on the number of grammatical errors. The lower score shows the higher grammatical accuracy. Table 1 presents error counts of the participants' pre-test and the post-test according to ten types of grammatical errors. It clearly shows that the participants made the lower number of grammatical errors in the post-test compared with the pre-test. Before the learning through the integration of indirect feedback and concordances, the total number of errors was 285. After the learning through the integration of indirect feedback and concordances, the participants made the lower number of errors as the total number of errors was only 190. Additionally, before the learning through the integration of indirect feedback and concordances, participants made the highest number of errors in articles (67 errors), whereas they made the lowest number of errors in verb forms (6 errors). After the learning through the integration of indirect feedback and concordances, participants still made the highest number of errors in articles (41 errors), but they made the lowest number of errors in verb forms and word order (2 errors). Examples of some grammatical errors in the pre-test and the post-test are illustrated as follows.
(1) I see sunset in evening.
In (1), there is an article omission error. The participant did not put an article before "evening". In fact, the participant needed to put "the" before "evening". In (2), the participant used the wrong verb form. In fact, the participant needed to use the verb in the infinitive without to after "me". A chi-square test for independence was employed to determine the statistical significance of the difference between the number of grammatical errors in the pre-test and the post-test. As shown in Table 2, there was a significant difference between the number of grammatical errors in the pre-test and the post-test since the p-value .024 (marked as Sig.) was less than the level of significance .05. According to the statistical analysis, the number of grammatical errors shows a significant decrease in the post-test compared with the pre-test. As a result, the integration of indirect feedback and concordances was seen to be able to improve grammatical accuracy in the participants' writing.

Data from the Questionnaire
To provide the findings regarding Thai EFL undergraduate students' perceptions toward the integration of indirect feedback and concordances on improving grammatical accuracy, the data were collected through the questionnaire and the semi-structured interview. For the questionnaire, the mean scores were computed and categorized into four levels as follows.   Table 4 shows the results from the questionnaire regarding the use of a corpus on improving grammatical accuracy. The participants strongly agreed with the overall statements regarding the use of a corpus on elt.ccsenet.org Vol. 15, No. 4; improving grammatical accuracy (Mean = 3.53 and S.D. = 0.58). According to the results of the participants' responses to the twelve Likert-type items, all strongly agreed on 11 of the items.
In comparing the first ten questionnaire items related to the use of COCA to gain knowledge of articles, nouns, subject-verb agreement, prepositions, infinitives/gerunds, verb forms, punctuation, capitalization, word order, and sentence fragment, item 8 (Mean = 3.70 and S.D. = 0.46), item 2 (Mean = 3.60 and S.D. = 0.55), and item 6 (Mean = 3.60 and S.D. = 0.55) are the top three highest-scoring items. On the other hand, there was only one item which was item 5 (Mean = 3.20 and S.D. = 0.80) on which the participants agreed less.
Besides, the results from the questionnaire indicate that the participants have positive perceptions toward the use of a corpus on improving grammatical accuracy in their writing. Seventy percent of participants found that COCA provides data which helps them to induce grammatical patterns (Mean = 3.70 and S.D. = 0.46). However, only fifty percent of participants strongly agreed that they usually found what they needed when they searched for information in COCA (Mean = 3.47 and S.D. = 0.56).

Data from the Semi-Structured Interview
In this study, ten participants including the five most improved participants and the five least improved participants were selected for a semi-structured interview. All of the participants' responses from the semi-structured interview were transcribed in order to explore their perceptions toward the integration of indirect feedback and concordances on improving grammatical accuracy. There were four interview questions, and the researchers applied qualitative content analysis created by Dörnyei (2007) for an analysis of the semi-structured interview.

Question 1: How do you feel about giving feedback by circling words? Why?
On the question that asked about the participants' feelings toward circles given by the teacher, both groups of participants were satisfied with this type of feedback.
The participants' feelings from the most improved group: When the participants were asked if a corpus could lead to their improvement in grammatical accuracy, most of them had positive responses. They felt that a corpus made their writing better. They could reduce future mistakes and avoid repeating the same mistakes.
The participants' feelings from the most improved group: According to the results gained from the interview, there were two aspects that both groups of participants noted in the same way. When the participants were asked whether they had the same writing mistakes in the last writing task or not, most of them reported that they still had the same mistakes, but they made the lower number of errors.
The participants' responses from the most improved group: S10: I still have the same old mistakes but fewer than before.
S24: I still make the same old mistakes but a lot fewer than my first two writing tasks.
The participants' responses from the least improved group: S15: I still make the same old mistakes, but it's actually a lot less than before.
S28: I still have the same old mistakes but fewer than before.
Question 4: Do you think a corpus will lead you to better writing for your further study and work (in the future)?
On the question that asked whether a corpus could lead the participants to a better writing in the future or not, all of them were very positive. They stated that they can consult a corpus to check for grammatical errors. They can look at concordances when they are not sure whether they write correctly or not. Below are responses from each group.
The participants' opinions from the most improved group:

Discussion
In order to answer the two research questions, this section discusses the findings of the present study regarding the effects of the integration of indirect feedback and concordances on improving grammatical accuracy in Thai EFL undergraduate students' writing and their perceptions toward of the instruction.

Research Question 1
The first question that this study needed to answer was the following: How does the integration of indirect feedback and concordances affect grammatical accuracy in Thai EFL undergraduate students' writing? The present study indicates a significant decrease in the number of grammatical errors in the post-test compared with the pre-test. Consequently, the integration of indirect feedback and concordances was seen to be able to improve the overall grammatical accuracy in the students' writing.
As mentioned previously, the participants were asked to do the post-test which was exactly the same task as the pre-test; that is, the participants were asked to write a paragraph with the same topic. According to James (2008), task similarity does not affect the transfer of writing skills. Hence, the participants' improvement might not result from task similarity. Concentrating on implementing concordances in order to improve grammatical accuracy, the result of this present study is similar to Bridle's (2019) result. Bridle (2019) indicated that due to a high percentage of correct corrections, a corpus could be used as a reference tool in the correction of grammatical errors in students' essays.
As mentioned earlier, a corpus was considered an effective tool to improve grammatical accuracy in the students' writing. Although there were various concordance programs, the present study used COCA. This concordance program has been a popular concordance program in an EFL setting because it provides users with useful functions. Students can search for words or phrases and learn how they are used in different context. Concordances are presented to users based on various sources of information, for example, academic journals, magazines, news, and so on (Oghigian & Chujo, 2010). Therefore, it is beneficial to students as they can learn how English sentences are constructed in an authentic way. Besides, each concordance line in COCA provides users with color codes to make it more comprehensible. Different colors show different parts of speech, such as nouns, verbs, prepositions, adjectives, adverbs, and so on. Consequently, it is easy for students to learn sentence structure from concordance lines. When students spend a lot of time paying attention to concordances, they may deepen their understanding about words or phrases related to their errors and try to avoid making those errors again in other writing tasks (Luo & Liao, 2015).

Research Question 2
The second research question asked the followings: What are Thai EFL undergraduate students' perceptions toward the integration of indirect feedback and concordances on improving grammatical accuracy? In order to answer this question, this study implemented a questionnaire with all the 30 students and a semi-structured interview with only ten students.
Since there was no previous research aimed at integrating indirect feedback with concordances, the researchers found that this intervention was beneficial in terms of improving grammatical accuracy in the students' writing. From the results of the analysis of the questionnaire and the coding of the semi-structured interview, most of the students indicated they had a positive perception toward both indirect feedback and concordances.
In this study, the students had to perform three writing tasks, and in each writing task, they received teacher feedback after finishing the first draft. After the experiment, students reflected in the semi-structured interview that they had a chance to learn from their mistakes after receiving indirect feedback. The students' responses in the present study are supported by the study by Shirotha (2016). The researcher explored the effectiveness of indirect written corrective feedback on students' writing accuracy. The results showed that indirect written corrective feedback did not only improve students' writing accuracy but also led to an autonomous learning. Students tended not to make the same mistake for the second time because indirect written corrective feedback reminded them of their mistake.
According to the students' responses in the questionnaire and the semi-structured interview, most students mentioned that a corpus was an effective tool to help them improve grammatical accuracy in their writing. As evidenced by the quantitative findings from the questionnaire, the students strongly believed that concordances could help them improve their writing performance. Errors in capitalization, nouns, and verb forms are the top three types of errors that they thought they could figure out. The findings of this present study are similar to those of Yoon and Hirvela (2004), Nasution (2018), and Liou (2019). The findings of Yoon and Hirvela (2004) indicated that the students who studied a writing course at a large American university had a positive feeling about the use of corpus activities in L2 writing. They agreed that corpus use was useful for acquiring usage patterns of words and enhancing their writing skill. In the study by Nasution (2018), the fourth-year students at a university in Indonesia reported that they improved their grammar with the help of a corpus, and using a corpus helped in a way that fixed their structures. Liou (2019) explored the students' perceptions on corpus tools through the questionnaire. They claimed that their grammar improved due to corpus use because a corpus allowed them to memorize grammar and word usage.

Conclusion
Due to the difference between English language and Thai language, Thai EFL students have errors in their writing, particularly grammatical errors. In order to improve grammatical accuracy in students' writing, it cannot be denied that it is a teacher's responsibility to help the students correct grammatical errors by giving feedback. The teachers can choose an appropriate type of feedback, which is indirect feedback, to help the students improve grammatical accuracy. In order to correct grammatical errors based on indirect feedback, the students can use a corpus to find correct grammatical patterns by themselves. They can learn from concordances to improve their grammatical accuracy. The data of this study have answered questions on the effects of the integration of indirect feedback and concordances on improving grammatical accuracy the students' writing as well as their perceptions toward indirect feedback and concordances. Descriptively, the intervention could improve grammatical accuracy, and the students had a positive perception toward both indirect feedback and concordances. All in all, the essence of indirect feedback incorporated with the function of COCA while studying the paragraph writing course among Thai EFL undergraduate students has accomplished the aim of improving grammatical accuracy.

Copyrights
Copyright for this article is retained by the author(s), with first publication rights granted to the journal.
This is an open-access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/4.0/).