Students’ Evaluations of Multilingual Educational Slides and Their Visual Attention Distribution on Slides with Different Layouts

,


Introduction
Since the beginning of this century, internationalization has become a key policy at academic institutions, and international student mobility has been strongly increasing.As a result, in non-English speaking countries where teaching has been mainly done only in the native language, there is a growing demand to provide educational slide presentations (e.g., PowerPoint) in multiple languages.English is the 'lingua franca' in academic institutions and has traditionally been used in international educational programs (de Wit & Altbach, 2021; Morley et al., 2019;Rose & McKinley, 2018).Offering English in such programs has led to vast increases in international student numbers in non-English speaking countries.For example, in Japan, where academic teaching is predominantly performed in Japanese, in 2023 there were about 230,000 international students (about 8% of the total student population) enrolled at formal educational institutions (Japan Student Services Organization [JASSO], 2023).The Japanese government aims to increase this number to 400,000 in 2033 (Nikkei Asia, 2023).Because of this, educational institutions are setting up and marketing their programs with bilingual classes.In the case of Japan, these often need to be taught by Japanese-native speakers or non-native speakers of English with a different nationality (Kojima, 2023;Liddicoat, 2007).

Literature Background
Multilingualism in education is considered to be beneficial for students.For example, research has shown that bilingual speakers tend to have better developed verbal and social communication (Okal, 2014), have increased cultural engagement associated with a language (Baker, 2008;Buttaro, 2014), and can obtain cognitive benefits, such as increased attentional control (Dong, 2023;Little et al., 2014).Furthermore, multilingual education promotes academic achievement and facilitates literacy development in English (Bhatia & Ritchie, 2013;Birulés-Muntané & Soto-Faraco, 2016;Kirss et al., 2021;Tang & Qu, 2020;Wang & Pellicer-Sánchez, 2023).Some potential disadvantage of multilingual education is that students may lose some of their cultural identity and/or some native-language ability (Baker & Wright, 2021;Bhatia & Ritchie, 2013).
The common electronic teaching media in educational institutions is the use of slides (e.g., PowerPoint), which require special attention in the case of multilingual education.There are practical advantages to using slides; however, there are certain boundaries as well (Baker et al., 2018;Craig & Amernic, 2006).When not designed effectively, slides that contain an extensive amount of information may be complex due to text crowding (Adamov et al., 2012;Mayer, 2020;Mutlu-Bayraktar et al., 2019).When two or more languages are used on slides for multilingual education, (text) crowding is a natural consequence.Even if the amount of information on a slide is summarized into a few main points, still the text font size and the layout need to be considered for visibility and effective information processing (Durso et al., 2011;Kahraman et al., 2011;Marchack, 2002;White, 2018).Many (commercial) resources have been developed to assist multilingual education, e.g., related to translation and presentation (Boichura & Lopatina, 2023;Degani & Goldberg, 2019;Steigerwald et al., 2022).When preparing multilingual educational slides, educators typically adopt the layout style from bilingual or multilingual scripts such as those used for road signs, bilingual newspapers, billboards, product manual books, and global advertisements (Bhatia & Ritchie, 2013;Qiu et al., 2018;Sebba, 2013), consisting of the separated-block and the mixed layout (Figure 1, Method section 2.3).The separated-block layout is one in which the same information is grouped according to language, with information in each language in a separate text block.The mixed layout refers to having the same information in a single block consisting of alternating lines according to language, i.e., with the same information in a different language for each line.
In order to better understand how multilingual information is processed, previous research has focused on how subtitles in video material influence information processing.In a study with educational videos for engineers, García (2017) found that students preferred audio in their non-native language (L2), but with subtitles in their native (L1) and their L2 languages.These bilingual subtitles not only improved L2 vocabulary learning, but were also found to be beneficial for understanding the video content.Similar positive effects of subtitles on content comprehension have been reported for subtitled film (Kruger et al., 2014;Perego et al., 2010).
Using eye tracking, Kruger et al. (2014) investigated the impact of subtitles on students' visual attention distribution and comprehension of materials presented in multilingual academic education.Native speakers of Sesotho with English as L2 watched a lecture video in English in three groups: without subtitles, with L1 subtitles, or with L2 subtitles.Two tests were performed, one shortly after the participants had watched the video and the other one two weeks after the video.As for the effect of the subtitles, no significant difference in the groups' understanding of the lecture was found.On a positive note, the authors argued that any redundancy caused by the subtitles in the lecture thus did not negatively impact the students' comprehension, e.g., through cognitive overload.
Another eye-tracking study on information comprehension through bilingual materials was done with video subtitling (Liao et al., 2020).In this study, native speakers of Chinese with English as L2 were shown video with English narration, either without subtitles, with subtitles in one language (i.e., Chinese or English), or with subtitles in the two languages (i.e., Chinese and English).The aim of the study was to investigate the impact of (uni-or bilingual) subtitles on participants' attention distribution, cognitive load, and comprehension by means of a recall test.Compared to having subtitles in one language, bilingual subtitles did not seem to heighten participants' cognitive load.This was in line with the findings of Kruger et al. (2014).In the study by Liao et al. (2020), however, compared to the video without subtitles, videos with subtitles benefitted the participants' comprehension, as based on the memory recall test.Consistent with the beneficial effects of bilingual education for students ' cognitive skills (e.g., attentional control, Dong, 2023;Little et al., 2014;academic achievement, Bhatia & Ritchie, 2013;Kirss et al., 2021;Tang & Qu, 2020;and second language acquisition, Birulés-Muntané & Soto-Faraco, 2016), similar to the findings of García et al. (2017) and Perego et al. (2010), bilingual subtitles thus may have a positive effect on the viewers' understanding.

Research Questions of the Present Study
The studies above that have shown a positive effect of bilingual subtitles on content understanding have used the same format consisting of two-line subtitles presented at the bottom of motion pictures.Educational slides in an academic context, however, typically consist of more lines with information, mostly without video material.In multilingual slides, the information is often arranged either in a separated-block layout or in a mixed layout.
Given the potential problem of text crowding, questions arise as to how students' visual attention is actually allocated over separated-block and mixed layouts, how students evaluate these layouts, and whether slide layout can influence the information they pick up from the slides.The expected outcome is not easy to formulate: one can expect that slides with a mixed layout are more difficult to navigate for students.However, since the same information is written in multiple languages in consecutive sentences, they may get a better understanding of the content, as in some of the research with video.
In order to investigate these questions, the aim of the present study is to obtain students' evaluations of multilingual educational slides, to analyze their gaze behavior using eye tracking, and to ascertain whether and how slide layout affects rote learning.The experiment was performed with 20 native speakers of Bahasa Indonesia and 20 native speakers of Japanese.Instead of bilingual slides, they were asked to evaluate multilingual slides consisting of the same text in English, Bahasa Indonesia, and Japanese, in both a separated-block and a mixed layout.Including three languages on the slides enabled us to have two different groups of participants with a different language background, yet both with English as their L2.To make sure that the students focused on the slides, the experiment was performed without narration by a teacher.

Participants
Forty participants, all enrolled at Kyushu University, Japan, joined the experiment.They were native speakers of Bahasa Indonesia (N = 20) and native speakers of Japanese (N = 20).They were divided into two groups: one group of participants (N = 20) viewed slides in a separated-block layout (10 Indonesian students and 10 Japanese students), while the other group (N = 20) viewed slides in a mixed layout (10 Indonesian students and 10 Japanese students).The members of each group were matched, as much as possible, in gender, age, and student grade.
The participants' statuses varied from research student (2 Indonesian students), bachelor student (5 Japanese students and 2 Indonesian students), master student (4 Indonesian students, 15 Japanese students), to doctoral student (12 Indonesian students).The ages of the participants ranged between 22 and 38 years (M = 26.08,SD = ±4.34).The participants reported their eyesight conditions as follows: not wearing glasses (40%), wearing glasses (32.5%), and using contact lenses (27.5%).Twelve (12) male and 28 female students reported English as their L2.Several Indonesian students (N = 8) reported having Japanese as L3.Four of them had completed the Japanese-Language Proficiency Test (JLPT) at level N4, and two had completed level N3.Meanwhile, 4 Japanese students had either French, Korean, Chinese (HSK Chinese test level 4), or Spanish (level 4) as L3.
The 40 participants had taken different English language proficiency tests, which included the International English Language Testing System (IELTS), the Test of English as a Foreign Language Institutional Testing Program (TOEFL ITP), (TOEFL iBT) and the Test of English for International Communication (TOEIC).Test results were referenced to the Common European Framework of Reference for Languages (CEFR) levels with acceptable levels ranging from B1-B2 ("independent user") to C1 ("proficient user").Each of the 40 participants received a gift card of 2000 Japanese yen as honorarium for their participation.After the instructions of the experiment were given, all participants provided written informed consent.The procedures of this study were pre-approved by the Ethics Committee of Kyushu University (Approval number: 131-9).

Materials
Multilingual slides were prepared with variations in two factors.The first factor was language, which consisted of three levels: English, Bahasa Indonesia, and Japanese.The second factor was the layout of the slides, with a separated-block layout (Figure 1A) and a mixed layout (Figure 1B).In slides with a separated-block layout, the First, each session was started with a calibration (the top left block) of the eye tracker for each participant individually.Here, the participant was asked to gaze at 16 points on the monitor one by one, until they disappeared from the screen.This was repeated a maximum of two or three times if problems were encountered with the calibration.When calibration was finished, opening information (slide 1) about the experiment was given for 10 seconds.Participants were then asked to gaze at a red fixation bar (slide 2) on a slide for 3 seconds before a multilingual slide was presented.The fixation bar was used to make sure that all participants' gaze position was the same at the start of the experiment.Following this, a multilingual slide was displayed for 10 seconds, and each participant could freely look at the slide anywhere they wished.This sequence consisting of a 3-s slide with a fixation bar and a 10-s multilingual slide was repeated another 8 times, i.e., presented 9 times in total (from slide 3 to slide 19).Lastly, a slide with closing information (slide 20) was presented for 10 seconds.
The whole slide presentation took precisely 90 seconds.
At the end of the slide presentations, the participant was asked to evaluate the slides by means of the same seven statements.Four statements (Statements 1 to 4 below) were answered by using an 11-point Likert scale with "0" representing "Strongly disagree" to "10" representing "Strongly agree", while the bottom three statements (Statements 5 to 7 below) were scored by using an 11-point Likert scale with "0" representing "not at all" and "10" representing "always".The statements were the following: 1) Amount of information: a) "The information on the slides was very complex."b) "There is so much information on the slides that I have trouble choosing what is important and what is not." 2) Information in more than one language: a) "Having the information in more than one language made the slides very complex."b) "I get distracted by the information in more than one language on the slides." 3) Layout crowdedness: When the experiment was finished and the evaluations were fully answered, participants were asked to answer a comprehension quiz.All participants were informed that the quiz was in English before it was given to them individually.The quiz consisted of 3 parts, handed out one at a time.Part 1 of the quiz consisted of ten questions in which words needed to be matched in the sentences.Part 2 consisted of ten questions that needed to be answered with "true", "false", or "not given", while Part 3 consisted of an open-ended question regarding the experiment.In total, the experiment took 35 minutes to 1 hour, depending on the participant's pace.

Data Collection and Statistical Approach
In order to see whether differences in the multilingual slide layout would affect the students' evaluations of the slides, their dwell time on the slides, and their quiz scores were obtained.Data were organized into two groups (the group that watched the multilingual slides with a separated-block layout and the group that watched the slides with a mixed layout).The variations in the multilingual slides (2 types of layouts × 3 languages) were determined as independent variables, while the students' evaluations (4 statements), the dwell time on the AOIs, and the comprehension quiz were the dependent variables.Data were analyzed also in two other groupings: one for the data of the Indonesian students (N = 20) and the other one for the data of the Japanese students (N = 20).The data were analyzed as follows: i. Evaluations of the statements.One-sample Kolmogorov-Smirnov tests showed that the scores indicating the students' prior knowledge were normally distributed for both participant groups (p > .05).The two evaluation scores for Statements 1-4 (i.e., concerning the "Amount of information", "Information in more than one language", "Layout crowdedness", and "Mental effort") were averaged for each participant.The resulting scores were normally distributed (p > .05)as well, with the only exception being the scores for the group who watched slides with a mixed layout (p < .05).Results were therefore analyzed with independent t-tests.
Regarding Statements 5-7 above ("On each slide, I read the information in English/Bahasa Indonesia/Japanese)", one-sample Kolmogorov-Smirnov tests showed that the data were not all normally distributed (p < .05)both for Indonesian students (separated-block slides: Bahasa Indonesia, p = .200;English, p = .065;Japanese, p = .007;and mixed slides: Bahasa Indonesia, p = .200;English, p = .036;Japanese, p = .000)and for Japanese students (separated-block slides: Bahasa Indonesia, p = .023;English, p = .000;Japanese, p = .000;and mixed slides: Bahasa Indonesia, p = .000;English, p = .000;Japanese, p = .000).The data were subjected to a non-parametric Friedman test for each layout per language, divided according to student group.A Kendall's W test was used to determine the effect size of the Friedman test, and categorized into small effect (0.1 to < 0.3), medium effect (0.3 to < 0.5), or large effect (≥ 0.5) (Cohen, 1988;Field, 2009).To compare the differences between each language, a Wilcoxon signed-rank test was performed for each language with the Bonferroni correction on the alpha level (α = .05/3= .0167).
ii. Cumulative dwell-time percentage over the Areas of Interest (AOIs).Average dwell time (milliseconds) was obtained separately for slides with a separated-block layout and with a mixed layout.Data were calculated by adding the sum of the amount of time each participant looked at each area of interest (2 types of layouts × 3 languages × 40 participants).The dwell time percentage was calculated by dividing dwell time in ms by the total duration of the nine multilingual slides (by 90,000 ms).One-sample Kolmogorov-Smirnov tests for each language in two different layouts showed that average dwell time for Japanese students was normally distributed (separated-block slides: Bahasa Indonesia, p = .200;English, p = .103;Japanese, p = .200,and mixed slides: Bahasa Indonesia, p = .200;English, p = .200;Japanese, p = .190).For Indonesian students, however, dwell time was not always normally distributed (separated-block slides: Bahasa Indonesia, p = .200;English, p = .000;Japanese, p = .200,and mixed slides: Bahasa Indonesia, p = .200;English, p = .047;Japanese, p = .007).A Mann-Whitney test was performed to compare the average dwell time on the AOIs per language between the two layouts.To connect the results of dwell time with the evaluation scores about language preferences, a non-parametric Friedman test was used.Post-hoc Wilcoxon signed-rank tests were used for pairwise comparisons with the Bonferroni correction on the alpha level (α = .05/3= .0167).The effect size was tested by Cohen's d (Cohen, 1988;Field, 2009).
iii.Comprehension quiz scores and prior knowledge.Overall, there were 20 questions with each correct answer corresponding to 5 points, totaling 100 points.The quiz scores were averaged for each of the two groups.
One-sample Kolmogorov-Smirnov tests showed that the quiz results were not normally distributed (p < .05)for each layout (separated-block slides: p = .037,and mixed slides: p = .024).An independent t-test was performed to analyze the participants' prior knowledge between the group who saw slides in a separated-block layout and a mixed layout.A non-parametric Mann-Whitney test was then used to compare the quiz results between participants who watched the multilingual slides in a separated-block layout and those who watched the same content in a mixed layout.To understand the correlation between the participants' prior knowledge and the quiz scores, a Pearson correlation test was performed for each layout separately.

Students' Evaluations of the Multilingual Slides via Statements 1-7
The Cronbach's alpha for the eight evaluation scores (two for each statement) stood at a high internal consistency of α > .8(Cohen, 1988;Field, 2009), with α = .810for slides with a separated-block layout, and α = .778for slides with a mixed layout.Results can be seen in Figure 3A-D.Regarding Statement 1 "The information on the slides was very complex" and "There is so much information on the slides that I have trouble choosing what is important and what is not" (Figure 3A), no significant difference was found between the scores for slides with a mixed layout (M = 6.3,SD = 2.261) and for slides with a separated-block layout (M = 5.900, SD = 2.01) [t (38) = -0.591,p = .558].Likewise, Statement 2 "Having information in more than one language made the slides very complex" and "I get distracted by the information in more than one language on the slides" (Figure 3B), there was no significant difference between students' evaluations of slides with a mixed layout (M = 6.25,SD = 2.473) and a separated-block layout (M = 5.175, SD = 2.272) [t (38) = -1.431,p = .161].As for layout crowdedness, regarding Statement 3 "The layout of the slides was very crowded" and "I get distracted by the crowded layout of the text on the slides" (Figure 3C), the difference between the evaluations of slides with a mixed layout (M = 5.775, SD = 2.638) and slides with a separated-block layout was on the border of significance, with mixed slides considered more crowded (M = 4.

Discussion and Conclusion
The purpose of this study was to obtain students' evaluations of multilingual educational slides and to investigate how these relate to their visual attention distribution on the slides.An experiment was performed in which native speakers of Bahasa Indonesia (N = 20) and native speakers of Japanese (N = 20) watched slides with information in three languages (English, Bahasa Indonesia, and Japanese), in two layouts typically used as slide formats.In the separated-block layout, the students watched the same information in separate blocks, one for each language.
In the mixed layout, the information was given line-by-line, alternating in languages.The students were asked to evaluate the slides on various variables, while their gaze behavior (i.e., attention distribution) was recorded using eye tracking.After the slide presentation, they performed a quiz to test their memorization of the slides' content.
Regarding the students' evaluation scores for the statements (see Figure 3), it was found that the multilingual slides with a mixed layout, compared to separated-block slides, were considered more crowded and requiring more effortful processing.The differences between mixed-layout and separated-block slides bordered on significance.Although Kruger et al. (2014), Liao et al. (2020), andPerego et al. (2010) did not report any clear increase in cognitive load between unilingual and bilingual subtitles, previous findings on multilingual texts have suggested that using multiple languages, including English, potentially could lead to visual complexity (Qiu et al., 2018;Sebba, 2013;White, 2018).Contrary to the research on subtitles, the present materials consisted of slides with multiple lines of text in three languages.When arranged in a single mixed-text block, the students seemed to have had difficulty navigating the slides.
The eye-tracking data of the students' gaze behavior, as an indicator of their visual attention distribution, confirm this.Regarding the students' attention distribution, first, regardless of whether they had read mixed-layout or separated-block slides, the students' self-reports (Figure 4) claimed that they mainly had read the text in L1 -significantly more than the other languages, including their L2 (English).When looking at the students' gaze behavior, we indeed see that for slides with a separated-block layout (Figures 5 and 6A), native speakers of Bahasa Indonesia significantly dwelled their gaze the most on the AOIs in Bahasa Indonesia.Likewise, the native speakers of Japanese significantly dwelled the most at the AOIs in Japanese.Overall, the Japanese AOIs are easiest to identify on the slides, due to the different typeface (Kanji, Katakana, and Hiragana characters for Japanese compared to the Latin/Roman alphabet for English and Bahasa Indonesia; Qiu et al., 2018).This may also have affected reading speed (Wakita, 2022) and therefore the dwell time of the Japanese participants.In line with Liao et al. (2020), who obtained gaze behavior for bilingual subtitles, on separated-block slides both the Japanese and the Indonesian participants thus had distributed their visual attention more to their L1 than to their L2 (English).The result was also consistent with Perego et al. (2010), where the viewers spent 67% examining the L1 subtitle areas.Although the participants in the present study were told that the quiz was in English, the dwell times on English (i.e., the participants' L2) were not significantly higher than on the other non-native language.
As for slides with a mixed layout, surprisingly, there was a vast discrepancy between the students' reports and their attention distribution (Figures 5 and 6B).Here, in spite of the self-reported focus on their L1, the gaze patterns showed longer fixations on the non-native language(s).This is in line with Kruger et al. (2014), who showed longer dwell time on L2 than L1 in a video with subtitles.Although the present study shows no significant differences between dwell times on the languages for slides with a mixed layout, the results are contrary to those for the separated-block slides and the other previous research with bilingual subtitles (e.g., Liao et al., 2020;Perego et al. 2010).Following the students' evaluations of the slides with a mixed layout as being more crowded and requiring more processing effort, the dwell time results confirm that the large block of AOIs, with alternating lines according to language, was indeed relatively difficult to navigate.This is likely due to the lack of blank lines that promote visual grouping of language blocks as in separated-block slides, with the added difficulty due to the (pseudo-)randomization of the order of the languages, which may have affected reading behavior (see Kwok et al., 2023).
In spite of the relatively negative evaluations and unclear attention distribution patterns for mixed-layout slides compared to separated-block slides, the students' quiz results were significantly higher after they had watched slides in the mixed layout.The result had a medium effect size, and the scores did not correlate to students' prior knowledge of the topics.Overall, this would imply that multilingual educational slides in a mixed layout can benefit students' learning more than separated-block slides.In the case of bilingual subtitles, the beneficial effect stems from the additional linguistic information added to either narration or the video content.In mixed-layout slides, similar to content with bilingual subtitles (García et al., 2017;Kruger et al., 2014;Liao et al., 2020;Perego et al., 2010), students watch the same information in different languages in successive or adjacent lines.
Although considered more crowded by the students, this apparently does not lead to cognitive redundancy, but instead might help them to semantically process or to memorize the information better.

Limitations of the present study
More research is needed to remedy the limitations of the present study.The topics and the quiz questions used here would benefit rote learning, and this does not dovetail with typical learning in higher education.Another major issue is that the slide presentation was short (90 s), without pictures, video, or formulas, colorful texts, and -most importantly -without narration or explanation by a teacher.In order to ascertain that the students watched the slides, no other visual or auditory information was given that would usually be present in the classroom, including other students and the whole lecture experience.Further investigations are needed to test how the text layout of multilingual educational slides affects learning in the presence of such other, realistic audiovisual sources.Until then, the main practical implication of the present study is that the use of multilingual slides in a mixed, line-by-line layout according to language, although visually crowded, seems to support content memorization.
a) "The layout of the slides was very crowded."b) "I get distracted by the crowded layout of the text on the slides."4) Mental effort: a) "I needed a lot of mental effort when watching the slides."b) "It was difficult to process the information on the slides."5) "On each slide, I read the information in English." 6) "On each slide, I read the information in Bahasa Indonesia."7) "On each slide, I read the information in Japanese." Figure 3 Figure 4.