Can I MOOC to Catch up? The Effects of Using an LMOOC as a Remedial Tool for EFL Students in Thailand

This study investigated the effects of supplementing a traditional EFL class with a grammar-focused LMOOC. It also investigated students’ attitudes to the LMOOC. Students taking a compulsory English course at a nursing college in Thailand were divided into two groups, a LMOOC group (n=33) and a non-LMOOC group (n=26). The LMOOC group engaged in a 4-week LMOOC as a supplement to their usual English classes. The non-LMOOC group continued with their usual English classes with no additional interventions. Final examination scores and gains since the midterm for the two groups were compared. Attitudes to the LMOOC were assessed using a questionnaire and interviews. Students in the LMOOC group experienced statistically significantly larger gains in grammar scores than the non-LMOOC group (M = 5.45, SD = 4.31, p < .001). Students reported very positive attitudes towards the LMOOC, in terms of enjoyment and perceived effectiveness. The estimated gains found in this small study were relatively modest, but our findings suggest that LMOOCs as a way to supplement in-class teaching may improve attainment and foster positive attitudes. Further controlled experiments to assess the wider applicability of our findings are needed.

remediating tool for falling-behind learners. We hypothesize that such learners might benefit from the opportunity to supplement their in-class learning with a structured adjunct to the work being undertaken in class, done at a pace of their choosing, away from the bustle of the classroom. LMOOCs may, therefore, be a way to operationalize this. In Thailand, an EFL context, there are few opportunities for students to be exposed to English in their environment, and, with typically very large class sizes with a wide range of proficiencies among students, language teachers can struggle to offer differentiated support that stretches the most able and addresses the specific challenges faced by the least able; those who might be falling behind. A common assumption among teachers in these contexts is that encouraging students who are at risk of falling behind to take online courses to supplement their in-class learning is worthwhile. Therefore, it is both empirically interesting and practically beneficial to investigate whether integrating an LMOOC into a formal language course is an effective use of time and resources.
The objective of this study was to investigate the effects of using an LMOOC as a supplement to in-class English teaching for falling-behind Thai EFL students, and to examine their attitudes toward such integration of an LMOOC. The study was conducted among Thai nursing students taking a compulsory English for Everyday Use course in a residential nursing college in Bangkok, Thailand. It is an expectation that all qualified nurses in Thailand will have at least basic competence in English. Many student nurses have very limited English proficiency when they start their formal nursing training. The English for Everyday Use course is thus designed to bring all students up to a miminum standard before they qualify, roughly equivalent to level A1/A2 on the Common European Framework of Reference. The course is, therefore, considered an introduction to English, with the emphasis on everyday use, rather than an English for Specific Purposes (ESP) course, or other vocantionally oriented language course, focusing on English use for nursing specifically. The course covers the basics of listening, speaking, reading, writing, vocabulary and grammar.
The study aimed to address two research questions: 1) To what extent does the use of an LMOOC as a supplemental teaching intervention help falling-behind students catch up with their learning as measured by their scores in standard end-of-year assessments?
2) What are students' attitudes toward the use of an LMOOC as a remediating tool for their English language learning?

LMOOCs as an Additional Learning Resource
According to Bárcena and Martin-Monje (2014), LMOOCs are defined as "dedicated web-based online courses for second languages with unrestricted access and potentially unlimited participation" (p. 1). Despite the growing interest in LMOOCs, the MOOC educational model has sometimes been criticised as being 'problematic' for language learning (Bárcena & Martin-Monje, 2014;Bárcena, Read, Martın-Monje, & Castrillo, 2014;Lin & Chang, 2014;Sokolik, 2014). This is because most available LMOOCs follow a cognitive-behaviourist approach (xMOOC pedagogy), which focuses on knowledge acquisition. Language learning, however, is skill-based and requires active participation and interaction with other learners (Martin-Monje, . In other words, the social and collaborative component seems to be lacking in available LMOOCs. These criticisms are theoretically valid, especially if such LMOOCs are to be adopted as stand-alone courses. However, there are many other possibilities for LMOOC implementation including the use of LMOOCs as an additional learning resource in a formal education context. Regarding the use of LMOOCs as an additional learning resource, there are several ways in which an LMOOC can be adopted. At the less integrated end of the spectrum, language teachers can make their students aware of the language learning opportunities available in LMOOCs and encourage them to explore these LMOOCs at their leisure. A slightly more direct use of this approach is for language teachers to recommend specific LMOOCs that are relevant to the course they are teaching and that students can take in partial completion of the class (Jitpaisarnwattana and Reinders, 2021). In addition, LMOOCs can be made a formal supplement to language courses, either to provide additional opportunities for practice, or to support students who need extra help, as a form of remedial support. At the more integrated end of the spectrum LMOOCs can be made a fully integrated part of language courses. If certain topics or skills are dealt with in existing LMOOCs, teachers may instruct their students to study the topics independently and use class time to initiate discussion and offer more personalised support.

LMOOC(s) Implementation: Research and Practice
Pedagogically, LMOOCs are a relatively novel and evolving innovation, in that they can serve both as a new education model that can replace traditional face-to-face L2 classes and as a piece of educational technology that can be integrated into L2 classes to serve different pedagogical objectives. Such flexibility in the way LMOOCs can be used has resulted in two main lines of LMOOC research: LMOOCs used as an alternative to (or a replacement for) a traditional L2 course and LMOOCs as an integrative tool for L2 classes (Jitpaisarnwatta et al., 2019). Given the fact that MOOCs were originally developed as courses in their own right, it is understandable that a greater proportion of LMOOC studies fall into the former category. However, an integrative approach to using LMOOCs in L2 classes is expanding and research on such an approach is also growing. To this end, several language learning constructs have been researched including motivation, pronunciation, interaction, flipped learning and blended learning.
In the first line of research (LMOOCs as an alternative to traditional L2 courses), students' interaction has been investigated quite extensively in several studies. The findings from these studies reveal that the interaction among learners was very low in both learning activities and discussion forums (Martin-Monje et al., 2013) and that learner-learner interaction was very low despite high levels of learner-content and learner-instructor interaction (Rubio, 2015). A more recent study also reveals a low level of interaction among learners (Martin-Monje, Castrillo, & Manana-Rodigrez, 2018), but indicates that learners who were active in their participation and interaction were more likely to be successful in the course than those who were not. In addition to interaction, motivation to register for, participate in, and complete a MOOC has also been examined (Uchidiuno et al., 2017). Both intrinsic and instrumental reasons play a role in attracting language learners to register for MOOCs and LMOOCs. Taking part in an LMOOC was seen as an opportunity to enhance learners' professional qualifications and equip them with important skills for social, economic and geographical mobility. In addition, the LMOOC environment itself was seen as an interesting platform for learners to develop their language and technological skills and a place where intercultural learning through interaction between learners from different backgrounds could take place. Teaching of a specific language skill such as pronunciation has also been investigated in LMOOC environments. Rubio (2014) compared gains in learners taking a traditional face-to-face Spanish Pronunciation course with those taking an LMOOC and found that both learning formats led to statistically significant improvements in their pronunciation comprehensibility, with greater gains seen in the LMOOC format. A greater amount of individualised feedback from the instructor and peers in the LMOOC format was believed to have contributed to these gains.
The second line of LMOOC research (integrating LMOOCs into traditional L2 courses) is much more limited, and the use of (L)MOOCs is more often than not associated with flipped/ blended learning approaches. Students' attitudes towards (L)MOOC integration have frequently been investigated. Zhang (2017), for example, examined students' attitudes towards a LMOOC-embedded flipped classroom at a university in China. The self-developed LMOOC in this study focused on vocabulary, reading comprehension, text analysis and writing skills. Zhang (2017) reported positive attitudes towards the integration of the LMOOC in this way, and a perception among students that having a LMOOC component embedded in the course helped them improve their reading and writing skills. LMOOCs were also used to complement a blended Content and Language Integrated Learning (CLIL) programme (Titova, 2017). Though the LMOOC was not technically a blended component of the course in this study, it served as a platform to afford students opportunities to communicate with other learners outside the course. The findings revealed that students perceived LMOOC integration positively. The opportunity to communicate with other learners via the LMOOC forum was seen as a main contributing factor to these positive attitudes. Wang and Wright (2018) investigated the use of a LMOOC as a flipped component in a Chinese language course for adult learners. It was found that flipped instruction enhanced learners' oral proficiency and resulted in a more positive attitude towards the course overall. Changes in learning behaviours attributed to MOOC intervention have also been observed. Though not essentially an LMOOC intervention, Luo (2020) found a positive change in students' language learning behaviours in terms of learning techniques following participation in a four-week MOOC called Learning How to Learn. The change in learning behaviours attributed to the MOOC was considered to have led to better academic performance in the following semester.
The evidence discussed in this section demonstrates the potential of LMOOCs in both types of implementation and point to several potentially fruitful avenues for further research. At a methodological level, there is a clear need for research into LMOOC implementation and to start looking at the actual effects of LMOOC on language learning (rather than proxies for this, such as engagement and enjoyment). This is because, despite showing positive results, most existing studies (Titova, 2017;Zhang, 2017) have only examined the effects of LMOOC implementation in terms of attitudes rather than actual success in learning the target language. While Wang and Wright (2018) did report a positive effect of an LMOOC on oral proficiency and Luo (2020) on learning behaviours, we argue that there remains considerable uncertainty about the substantive educative effects of implementing LMOOCs in the ways that we have described. This study aims to help address this gap in our understanding of the potential role of LMOOCs.

Methods
The purpose of the study was to assess the extent to which supplementing usual in-class teaching with an appropriately aligned LMOOC supports attainment among students who were deemed at risk of failing the course. We compared two groups of students who had failed the English for Everyday Purposes mid-term exam. One group was enrolled in a four-week LMOOC, which they engaged in in their spare time in addition to the main class. The other group continued with business as usual. The primary outcome was attainment in the end of term exam. The first part of the study thus adopted a quasi-experimental, pre-test post-test, non-equivalent groups design. In addition, we sought to understand the experiences of and attitudes towards using the LMOOC of those who had been allocated to it. To do this we designed a questionnaire, using 5-point Likert scales and free text responses. These were followed up by interviews with participants who had been selected for maximum variation. That is, participants who had responded favourably to the LMOOC in the questionnaire, participants who had responded largely neutrally to the LMOOC, and the single respondent who had expressed a negative view.

Participants and Allocation
Participants were drawn from two classes (Class A and Class B) of adult nursing students enrolled on English for Everyday Use, a 15-week compulsory English course, at a residential nursing college in Thailand. Students were eligible for inclusion if they had failed their mid-term exam (scored less than 50% on this 70-item multiple choice test). Of the 212 students enrolled on the course, 62 met this criterion (35 in Class A and 27 in Class B), all of whom were recruited to the trial. Four participants dropped out of the whole course during the study, leaving 59 who participated in full and were available for post-intervention assessment (33 in Class A and 26 in Class B).
While it would have been methodologically preferable to randomly allocate individual participants to either the treatment group (LMOOC) or the control group (non-LMOOC), this would have introduced a threat of contamination, undermining the independence of each group. At this residential nursing college, students are assigned dorms on the basis of their classes. That is, all members of Class A lived in one dorm and all members of Class B lived in a different dorm. The LMOOC was intended to be carried out in participants' spare time, which probably meant it would be done in their dorms. To reduce as far as practical, the possibility that members of the non-LMOOC group would be exposed to the intervention via members of the LMOOC group with whom they shared a living space, allocation was instead made at the level of the class. That is, all members of Class A were allocated to the LMOOC treatment group and all members of Class B were allocated to the non-LMOOC control group. Allocation was made on the basis of chance, using the random number generator at www.randomizer.org (Urbaniak & Plous, 1997. While it is fair to say that contamination could arise via other means (students conversing on social media, for example), we considered this approach to minimise the threat of contamination based on what was practical in the context. See CONSORT Flow Diagram in Figure 1 for the flow of participants through the study.

CONSORT Flow Diagram
A between samples t-test was conducted to assess the level of similarity between the two groups at the baseline, based on their midterm scores. On average, the non-LMOOC group (n=26, M = 29.80, SD = 3.18) had very slightly higher midterm scores than the LMOOC group (n=33, M = 29.12, SD = 3.27). This difference was not statistically significant (t (57) = -.810, p = 0.42). On this basis, we are satisfied that both groups were approximately similar in their English proficiency at baseline.
At the end of the intervention period, members of the LMOOC group were invited to participate in a follow-up study to explore their attitudes towards the platform and their perceptions about its effectiveness through a questionnaire and interviews. All members of the LMOOC group agreed to participate in this phase of the study. All recruitment procedures and practices used in the study were conducted in line with, and with the approval of, the Central University Research Ethics Committee at the University of Oxford.

Intervention Study
Verb Tenses and Passives, an LMOOC offered by the University of California Irvine through the MOOC platform Coursera (Coursera Inc., 2020), was chosen for the LMOOC component of the intervention. The course, following an xMOOC pedagogy, is self-paced and estimated by the provider to take between 4-5 hours per week over four weeks to complete. The four units in the course cover simple, progressive and perfect tenses, perfect progressives, passives and perfect modals, and blending tenses. Activities consisted of short video lectures, reading tasks, a series of practice quizzes and a graded quiz at the end of each unit. The full syllabus can be viewed at https://www.coursera.org/learn/verb-passives. This LMOOC was chosen because it was written for beginner to intermediate EFL learners and its content was well-matched to the grammar component of the English for Everyday Use course on which participants were enrolled, and, accordingly, the content of the mid-term and final assessments. Participants, by virtue of having received failing grades in the mid-term assessment, demonstrated that the content of the LMOOC was appropriate to address their language learning needs relative to the overall goals of the course. Although the content was delivered in English, it was carefully produced for EFL learners, and English subtitles and transcripts of the video lectures were available. In addition to continuing to attend regular classes, the LMOOC group was asked to take the LMOOC in their spare time in the final four weeks of the college term leading up to the final exam. The non-LMOOC comparison group continued their study in class as usual, with no additional instruction.
In the week prior to the start of the intervention the researcher delivered an introductory session on using Cousera to the LMOOC group to ensure they knew how to sign up for the course and how to navigate it once they had started. At the end of each week of the intervention, the LMOOC group was asked to screen-capture confirmation that they had completed each activity for that week, then send it to the researcher.

Follow-Up Attitudes and Perceptions Study
On completion of the intervention period, members of the LMOOC group were invited to complete a questionnaire to report on their experiences using the LMOOC. The questionnaire (see Appendix 1) consisted of 12 five-point Likert scale items in three categories: learning grammar, usefulness for their learning in the course, the learning experience. Three open-ended questions allowed for longer-form insights from the participants. Five-point Likert scales were chosen over even numbered, forced-choice scales. This was done mainly for cultural reasons. The socially accepted nature of the relationship between students and teachers in Thailand is such that it is generally considered bad form for the former to criticise the latter, either explicitly or implicitly. Given that the first author was also the participants' teacher, had a forced choice been offered, the influence of this Thai social script might have inflated the frequency of socially acceptable responses at the expense of honest ones, or participants may simply have avoided answering some questions at all. The odd numbered scale with its neutral option allowed participants a 'get out clause' should they have found naked disagreement unpalatable. All invited participants (n=33) completed the questionnaire. The paper and pencil questionnaire was administered at the end of the intervention, face-to-face with the whole group together. The researcher was in the room with the participants to provide clarification on any points that the participants did not fully understand. The participants spent 10-15 minutes completing the questionnaire.
On the basis of the questionnaire responses, four participants with generally heterogeneous responses were identified and invited to attend interviews (see Appendix 2 for interview questions) with the researcher. All agreed. Purposive sampling was chosen to ensure a range of views about that intervention, from those with generally positive responses, those with generally neutral responses, and the one participant who expressed a negative response. Interviews were conducted either face-to-face (n=2) or online using Skype (n=2). The interviews were conducted in Thai and transcribed and translated by the researcher. To check for accuracy, the transcripts were back translated by another lecturer at the university. The interview scripts were coded into themes based on the categories in the questionnaire.

Data Collection and Analysis
Data from the midterm and final examinations were collected through score reports from the college. Both examinations followed the same test structure, consisting of 70 multiple-choice items and covering three areas of the course content: grammar, vocabulary and reading. Both examinations consisted of 40 grammar items, 17 vocabulary items and 13 reading items. These test items were written by the course instructor (who was also the researcher) and approved by the college committee.
Statistical analysis was conducted using the Statistical Package for Social Sciences 20.0 (SPSS). Between samples t-tests were used to compare group scores on the final exam, and to compare the extent of any change between the midterm and final exam scores. In addition, post hoc analysis of scores in different parts of the final exam (grammar, reading and vocabulary) was carried out using repeated measures t-tests to assess whether the effects of the LMOOC were limited to the grammar component of the final exam, or whether they extended to other parts of the test as well.
Fundamental descriptive statistics (frequency, mean scores and standard deviations) from the questionnaire data were calculated. The data from the open-ended questions were coded deductively based on three categories: learning grammar, usefulness for their learning on the course, the learning experience, to allow for thematic descriptions of the participants' responses. Answers that did not match one of these categories were categorised inductively into emerging themes. To ensure the reliability of the coding process, two people independently coded the responses, the researcher and another English lecturer at a university in Thailand.
After transcription, the interviews were coded deductively based on three categories: learning grammar, usefulness for their learning on the course, the learning experience. Answers that did not match one of these categories were categorised inductively into emerging themes. Again, these were dual-coded with assistance from an English lecturer at a university in Thailand. Data from the interviews are used here to illustrate and expand on the findings from the questionnaire data.

Effects of the LMOOC Intervention on Students' Overall Scores
Having established statistical equivalence between the groups on their midterm scores at the baseline (see section 3.1), statistical analyses were conducted to assess whether participation in the LMOOC was associated with improved outcomes in the final exam, relative to the non-LMOOC group. Data were assessed for normal distribution using a Shapiro-Wilk test for normality, and were found to be within acceptable levels to meet the assumptions required for paired and independent t-tests.
The LMOOC group obtained a mean score of 34.15 points (SD=4.62) in the final exam. The non-LMOOC groups obtained a mean score of 30.50 (SD=4.91). This difference was statistically significant (t (57) =2.93, p=.005). Participation in the LMOOC was thus associated with statistically significantly higher final scores compared with teaching as usual. See Table 1. In addition to comparing final scores, the extent of change between the midterm and final exams in each group was compared. The LMOOC group made a mean gain of 5.03 points (SD=3.82). The non-LMOOC group made a mean gain of 0.65 points (SD=4.08). This difference in gains was also statistically significant (t (57) = 4.24, p < 0.001). Participation in the LMOOC was thus associated with statistically significantly greater gains compared to teaching as usual. See Table 2.  The results of this analysis demonstrate a positive, statistically significant overall effect of the LMOOC on the student's final exam scores.

Effects of the LMOOC Intervention on Different Elements within the Final Exam
As the content of the LMOOC dealt primarily with grammar, we examined gain scores to assess whether participation in the LMOOC conferred an advantage only in terms of scores in the grammar component of the test, or whether it was associated with gains in reading and vocabulary scores as well. As only the LMOOC group made statistically significant gains between the midterm and final exams, the scores of this group were disaggregated by question focus, and the grammar component of the test was considered separately from the reading and vocabulary components. As multiple comparisons were conducted, and because it is theoretically plausible that engagement in the LMOOC's reading materials might have had an effect on the participants' reading development and vocabulary knowledge, a Bonferoni adjustment to the alpha levels of .025 per test (.05/2) was applied.
The average gain in the grammar component of the test was 5.45 points (SD 4.31). This gain was statistically significant (t (32) = 7.27, p < 0.001). The average gain in the reading and vocabulary component was -.42 (SD=5.96). This very small tendency to do worse in the final exam on reading and vocabulary was not statistically significant (t (32) = -.41, p = .685). See Table 3. The results from this analysis suggest that participation in this grammar-focused LMOOC was associated with gains in the grammar knowledge of the participants, but that engagement in the LMOOC did not extend that advantage to their reading and vocabulary knowledge.

Students' Attitudes towards LMOOC Intervention
This section analyses the students' attitudes toward the use of the LMOOC according to three categories: learning grammar, usefulness for their learning in the course, and learning experience.

Learning Grammar
Given the importance of grammar knowledge for passing the final exam, the first four questions aimed to explore how well participants felt the LMOOC had addressed their needs in that domain, specifically. The summary in Table 4 demonstrates that the students tended to agree with all the statements in this category. The vast majority (90.90%) agreed or strongly agreed that the LMOOC was helpful in improving their grammar knowledge. In addition, a large proportion of participants (81.82%) thought that the LMOOC made them feel more confident in learning English grammar and that their knowledge was better after taking the LMOOC. A similar number of students (84.85%) believed that the LMOOC was a useful tool for learning grammar. Notes. F = frequency, SD = strongly disagree, D = disagree, N = neither agree nor disagree, A = agree, SA = strongly agree, StD = standard deviation.

Q1:
The MOOC helped me improve my knowledge of English grammar.

Q2:
The MOOC made me feel more confident in learning English grammar.
Q3: I have better knowledge of English grammar after taking the MOOC.

Q4:
The MOOC was a good tool to learn about English grammar.
In answers to the open-ended questions on this theme, most of the participants reported that they felt the LMOOC was beneficial for their grammar learning, and was a good tool to learn with. In particular, participants felt that the opportunities for extra practice provided by the LMOOC were valuable in a context where few opportunities exist outside of class. As one participant mentioned in an interview:

Usefulness of the Course
One of the prime motivators for considering the LMOOC with this group of participants was its potential to help them catch up with course content they had not fully grasped. Questions 5-8 explored their opinions on this theme. The majority of the participants (84.85%) agreed that the LMOOC intervention was helpful in preparing them for the final examination. Importantly, given our motivation for this study, an even higher number of participants (90.90%) thought that the LMOOC intervention allowed them to catch up with content they had not understood in class. Strong agreement can also be found with the notion that the LMOOC helped them to understand the course content better (Q7 and Q8), with 87.88% agreeing or strongly agreeing with both statements. Responses are summarised in Table 5.  Notes. F = frequency, SD = strongly disagree, D = disagree, N = neither agree nor disagree, A = agree, SA = strongly agree, StD = standard deviation.

Q5:
The MOOC helped me prepare for the final examination.

Q6:
The MOOC helped me catch up with content I did not understand in class.

Q7:
The MOOC helped me understand the content in "English for Everyday Use" better.

Q8:
The MOOC made it easier for me to understand the course content.
The long-form response data lend support to the questionnaire findings, with students reporting that they found the LMOOC useful for their studies and for preparation for their final exam. In particular, they reported appreciating the opportunities to review what they learned in class. One participant reported in an interview:

The Learning Experience
No matter how effective an intervention is, if it is expected that students will engage with it in their own time in addition to their other study, it ought to be reasonably enjoyable, motiviating, or engaging, to maximise the chance that they will persist with it through to completion. The final four questions addressed this ascpet of the intervention. The results presented in Table 6 demonstrate that, overall, the students had a positive learning experience with the LMOOC. Most of the participants (90.90%) saw the LMOOC as relevant to their learning. A substantial proportion of the students felt that the LMOOC was interesting and that they were motivated to learn by participating in it (84.85% and 81.82%, respectively). Interestingly, this section of the questionnaire attracted the only negative response, with one participant (3.03%) disagreeing with the statement 'The MOOC motivates me to learn English'. As for the learning experience, the majority of the students (81.82%) agreed that the LMOOC learning experience was 'good'. Notes. F = frequency, SD = strongly disagree, D = disagree, N = neither agree nor disagree, A = agree, SA = strongly agree, StD = standard deviation.

Q10:
The content of the MOOC was relevant to my learning.

Q11:
The MOOC motivates me to learn English.
Q12: I had a good experience taking the MOOC. The long-form response data confirmed the questionnaire findings, with many informants describing the LMOOC as enjoyable and motivating. By way of example, one participant commented: It was easy to learn. Also, it motivated me to learn and was a good tool to learn English. [Questionnaire,Student 22]

Potential of LMOOC as a Remediating Tool
Given the generally positive effects associated with using the LMOOC to help falling-behind students catch up, some themes emerge from the qualitative data that add important information about implementation, for teachers considering adopting this approach. These themes are categorized and presented in this section. One issue that participants raised in the interviews concerned implementation. They felt that the LMOOC was a good remedial support for falling-behind students, but was perhaps too demanding if it were to be used as a full supplement to the course. However, language proficiency was seen as a challenge, particularly due to a lack of vocabulary knowledge, which participants said affected their comprehension of the LMOOC content. In addition, some participants reported having problems because there was no teacher to help them when they did not understand. More positively, several forms of independent and reflective learning were reported by the participants as a result of taking the LMOOC. Participating in the LMOOC allowed them not only to learn independently, but also to reflect on that learning. Many students reported valuing the catalyst provided by the LMOOC for reflecting on their current knowledge. As two participants commented:

Discussion
Following a four-week LMOOC intervention in addition to business as usual for 'falling-behind' nursing students enrolled on an introductory English course in Thailand, participants' performance in their final exams improved to a statistically significant extent, compared with similar students who did not take the LMOOC, and who continued with learning as usual only. This improvement was in terms of the overall score in the final exam. Post hoc analysis of those scores, disaggregated by exam focus, revealed that the difference in scores between baseline and endpoint was attributable to improvement in the grammar component but not in other areas. Given that both groups were statistically similar in terms of scores on their midterm exam prior to the intervention, and received identical teaching with the exception of the LMOOC, the improvement in grammar scores can be reasonably attributed to participation in the LMOOC. It is important to note that although the LMOOC group statistically significantly improved, most of the participants in that group (N =27, 81.82%) still failed to attain the 35-point (50%) pass score in their final exam.
Several observations can be made from these results. First, it came as no real surprise that gains were evident in the grammar component of the examination and not the other components, as the content of the LMOOC was grammar-focused. At the outset, we considered it plausible that engagement in the LMOOC (and therefore exposure to additional reading materials and spoken English in the videos) might have had an effect on areas beyond just grammar. This was not the case. Second, while the change in mean scores on the reading and vocab component was not statistically significant, it was nonetheless interesting to note that participants experienced a slight decrease in their vocabulary and reading scores (M = -.43). This might be because their primary focus was drawn to the grammar aspect of the class content and thus their preparation for the final exam may have prioritised this over other aspects of the exam. Although some students (18.18%) passed the final exam, the majority in the LMOOC group did not (81.82%) (though the number of points by which they fell short of a pass grade tended to be smaller than the equivalent shortfall in the non-LMOOC group). We attribute this to the degree of difference between their performance at the baseline and the knowledge needed to pass the exam. It seems that four weeks with a focus only on grammar, while positive, was simply not sufficient in the context of this course.
Despite our disappointment that the LMOOC did not achieve the outcome hoped for these students, and despite criticism in the literature claiming that LMOOCs are problematic for language learning (Bárcena & Martin-Monje, 2014;Sokolik, 2014), we have provided evidence that LMOOCs that follow a broadly cognitive behaviourist model can be beneficial for language learning in terms of the outcomes we investigated here. This provides important evidence about the effectiveness of such an approach for teachers with students they believe are at risk of falling behind in similar courses. Given the differential effects of the LMOOC on different domains elt.ccsenet.org English Language Teaching Vol. 15, No. 1; within language learning (grammar, reading and vocabulary), we suggest that the qualified success of this intervention was a result of adopting an LMOOC for a clear pedagogical focus (helping falling behind students) and addressing a clearly defined area (grammar). In practical terms, we think this means that in formal language learning contexts, LMOOCs might be more beneficial if they are used as integrative tools to complement regular language classes rather than as stand-alone courses. This, of course, remains to be investigated more fully in other experiments; the type of research that we have noted as scarce in the literature. We would urge the research community to invest in more and larger controlled comparisons so that we can better understand the educative effects of LMOOCs in all their various forms on supporting language learners, but in particular as a remedy for falling-behind learners.
With regard to students' attitudes, the learners who took the LMOOC reported positive attitudes toward the intervention in all three categories: learning grammar, usefulness for their learning in the course, and learning experience. The participants felt that their grammar knowledge improved as a result of the intervention. The strong link between LMOOC content and course content allowed the students to make a direct connection between the LMOOC and their in-class learning, making the focus on grammatical improvement more salient for them. As for the usefulness of the LMOOC, participants saw benefits in terms of helping them catch up with course content and preparing for their final exam. This could be attributed to opportunities to review the lessons they learned in class provided by the intervention. We believe that this reflects the importance of choosing LMOOCs that directly contribute to the themes being taught in the main classroom. As for their learning experience, the participants reported having a good experience in taking the LMOOC and found it interesting and motivating. Such positive attitudes were ascribed to the user-friendliness of the LMOOC, the portability of mobile technology, and the relevance of the LMOOC content. Empirically, the study elaborates, though from a rather different point of view, the results from previous studies that people are motivated to join MOOCs for both intrinsic and instrumental reasons (Beaven et al., 2014;Uchidiuno et al., 2017). Although the reason for using the LMOOC in this study was operationalised by us as chiefly instrumental (passing the final exam), a clear intrinsic aspect was reflected in participants' attitudes as reported in the questionnaire.
However, if LMOOCs are to be considered as an adjunct to the language classroom, it is important to note some of the challenges reported by the participants. The most prominent of these was the participants' relative lack of English proficiency. This made it difficult for them to fully comprehend the LMOOC content. Frequently, they cited inadequate vocabulary knowledge as responsible for this. Such a finding is understandable given that these students' level of English proficiency was relatively low -that is why they were encouraged to do the LMOOC in the first place. This suggests that providing more first language (L1) support as a part of LMOOCs may ease the cognitive load on learners as they navigate the learning materials and thus allow them to concentrate more fully on the specific learning objectives of the lesson (Roussel et al., 2017). We note that the LMOOC we chose for these students provides subtitles in several languages for the videos. Based on the experiences of our participants, and on the evidence that using L2 learners' first languages to support learning can be productive (Chalmers, 2019), a logical next step for this type of research would be to assess the effects of providing L1 translations of key materials on students' outcomes. Such are the affordances of this kind of technology, that L1 translations into a functionally limitless number of languages could be provided. Another challenge that emerged from the data was the lack of instructor presence. The students reported that they could not ask anyone when they did not understand the content. This resonates with findings reported in previous studies that MOOCs often lack support and feedback from course instructors (Castrillo, 2014;Martin-Monje et al., 2013). This seems to be an important issue that future implementations of MOOCs should aim to overcome.

Limitations
Although a comparison of the LMOOC and non-LMOOC groups' scores on the students' midterm exams prior to the intervention suggested that, on this measure alone, they were similar, it is impossible to tell if there were other systematic differences between the classes that might be responsible for the differences in outcomes. It is therefore not possible to say that the changes in the scores were caused only by the LMOOC intervention. Hence, future studies should adopt a randomised trial design (with either random allocation at the level of the individual or cluster randomisation with much larger samples) to reduce the possibility of systematic differences between comparison groups, and thus allow more confident causal inferences to be drawn. In addition, while the results of this trial are promising, they were based on a relatively small sample of learners. Future studies should investigate the issues reported here with much larger sample sizes.

Conclusion
This study examined the effect of using an LMOOC as an adjunct to usual teaching to help falling-behind Thai EFL students. The results demonstrate that the LMOOC intervention had a positive impact on the students' performance in their final exam, raising their average attainment to a statistically significant extent, compared to similar students who did not take the LMOOC. In addition, we found that these students held positive attitudes towards the LMOOC as a way to support their learning in class. These findings add important information to our understanding of the potential that LMOOCs hold for supporting more traditional pedagogical approaches. A practical implication of our findings for teachers is that, when provision of teacher led catch-up opportunities for learners who need them is limited, encouraging them to take an appropriate LMOOC is beneficial. We suggest that part of the success of the LMOOC in this study was its close alignment with the main class content and the flexibility it offered students in terms of how, where, and when they studied. Based on the responses to the questionnaire, we also believe that the LMOOC may have provided a motivating adjunct to their usual teaching. Tentatively, we suggest that adding L1 support to LMOOCs may further enhance the effects of this type of intervention, and that opportunities to review learning in the LMOOC with a 'live' teacher might be beneficial.
This study offers the LMOOC and CALL research community much needed evidence on the effects of LMOOCs on substantive educational outcomes, and triangulates this against more commonly studied outcomes, such as student enjoyment and engagement. While we do not claim that high levels of engagement and enjoyment necessarily lead to improved language outcomes, we note that the findings of both our experiment and analysis of students' attitudes lend support to the claim made by Al-Shammary (2007) that the success of any technological implementation is determined by students' attitudes towards it. More research that explores the relationships between these nominally quantitative and qualitative findings to generate a fuller picture of the affordances of LMOOCs as a part of blended learning is warranted.

Part two: Closed questions
Indicate whether you agree with the following statements. Circle the response that is correct for you (SD = strongly disagree, D = disagree, N = neither agree nor disagree, A = agree, SA = strongly agree)

Guided Interview Questions
• Did you enjoy taking the MOOC?
• What did you like most about the MOOC?
• What did you find the most challenging in taking the MOOC?
• Have you learned anything new from taking this MOOC?
• Did you encounter any problems using the MOOC?
• Do you have anything to suggest about using MOOC to help students improve their scores?
• Questions emerged from the questionnaires

Copyrights
Copyright for this article is retained by the author(s), with first publication rights granted to the journal.
This is an open-access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/4.0/).