Cognitive Synergy in Multimedia Learning

The main focus of our study was to investigate multimedia effects that had different results from the findings of existing multimedia learning studies. First, we describe and summarize three experimental studies we conducted from 2006 to 2010. Then we analyze our findings to explore learner characteristics that may impact the cognitive processes in multimedia learning. Third, based on this analysis, we identify individual student characteristics that offer a more coherent explanation for why and how multimedia students learn differently: affection, prior knowledge, metacognition, and cognitive synergy. Including these student characteristics in multimedia learning considerations can shed additional light on an issue of great importance, namely how cognitive synergy affects multimedia learning. It can also lead to the identification and development of methods for helping students overcome learning difficulties and thus has important implications for multimedia researchers and teachers who are trying to optimally align learner characteristics with pedagogical and practical approaches to multimedia learning.


Introduction
The traditional model of computer-assisted instruction (CAI) is changing, as demonstrated by the common use of advanced computer technologies with multimedia, and most significantly with Internet.With these technologies, educators have been able to expand their ability to send instructional messages in flexible ways to various devices, including netbooks, iPads, iPhones, and smart phones, that seem full of potential as tools to facilitate mobile learning (m-learning).Even young children get involved in m-learning experiences with these portable devices to engage with their favorite story books, movies, games, and the thousands of educational apps.In spite of the limitations of m-learning environments (e.g., the size of display, the limitation of input capabilities, etc.), learners are demanding more options to make learning easier to do what they want when they want to do it.As these student expectations continue to expand, Internet-based courses and supplemental online resources for traditional classrooms are being offered more often at both primary and secondary levels (Oliver, Osborne, & Brady, 2009).
In developing these digital learning environments, multimedia has been found to offer rich learning opportunities in order to meet student needs.However, because the human mind is limited in the amount of information that it can process at one time (Miller, 1956), multimedia certainly brings own unique challenges, such as how the multimedia learner effectively and efficiently integrates the presented material into a coherent mental representation in mental construction architecture.Many multimedia researchers have focused on Mayer's (2001) cognitive theory of multimedia learning (CTML) and Sweller's (1999) cognitive load theory to overcome the limitations of human memory and promote higher cognitive processes as meaningful learning (Tabbers, Martens, & van Merriënboer, 2004).
Mayer's CTML is based on three cognitive process assumptions of learning (Mayer, 2001): (a) dual coding assumption for visual/pictorial and auditory/verbal channel processes (Paivio, 1986;Sadoski & Paivio, 2001), (b) limited capacity assumption for each channel processing (Baddely, 1986;Chandler & Sweller, 1991), and (c) active processing assumption for carrying out a coordinated set of cognitive processes (Mayer, 2001).Mayer describes how the multimedia learner builds mental representations in the cognitive architecture (Tabbers, Martens, & van Merriënboer, 2004): meaningful learning occurs when a learner selects relevant information, organizes it into coherent representations, and integrates it with prior knowledge.
The basic assumption of cognitive load theory is that the cognitive architecture consists of multiple memory stores, including a very limited working memory and an extensive long-term memory.According to Schnotz and Kürschner (2007), the limitations of working memory disappear when dealing with information from long-term memory, where information is organized into higher order units called cognitive schemata.However, Sweller warned of the limitation of human working memory in that redundant memory load can be caused by presenting too many elements (Mayer & Moreno, 2002).In many types of multimedia instruction, the necessary mental integration of information from different channels leads to a high cognitive load; the extraneous load should be minimized as possible (Tabbers, 2002).
Simply adding a verity media formats of information does not assure an enhancement in learning.Mayer and his colleague demonstrated seven empirically tested basic multimedia design principles (referred to as multimedia design effects) that can be useful guidelines for designing multimedia presentations (Mayer, 2001, p. 184): (1) "Students learn better from words and pictures (e.g., text-with-pictorial annotation) rather than words alone" (Multimedia Principle); (2) "Students learn better when corresponding words and pictures are presented near rather than far from each other on the page of screen" (Spatial Contiguity Principle); (3) "Students learn better when corresponding words and pictures are presented simultaneously rather than successively" (Temporal Contiguity Principle); (4) "Students learn better when extraneous words, pictures, and sounds are excluded rather than included" (Coherence Principle); (5) "Students learn better from animation and narration than from animation and on-screen text" (Modality Principle); ( 6) "Students learn better from animation and narration than from animation, narration, and on-screen text" (Redundancy Principle); and ( 7) "Design effects are stronger for low-knowledge learners than for high-knowledge learners and for high-spatial learners rather than for low-spatial learners" (Individual Differences Principle).These multimedia design principles are consistent with the Cognitive Theory of Multimedia Learning (CTML).This theory explicates how students learn from both auditory/verbal stimuli (e.g., words) and visual/pictorial stimuli (e.g., graphics) under certain conditions and makes use of cognitive process assumptions of learning.Researchers refer to their results of multimedia studies as multimedia design effects: multimedia effect, spatial contiguity effect, temporal contiguity effect, coherence effect, modality effect, redundancy effect, and individual differences effect to explain their better learning outcomes under the design principles conditions.

Rationale for the Current Study
Mayer's CTML helps to explain these early multimedia design principles; for instance, the multimedia principle works because students experience meaningful learning when they select, organize, and integrate visual/pictorial and auditory/verbal information with their prior knowledge.Similarly, providing multimedia material is an innovative way for learners to make use of multimedia design effects to understand some concepts or content that would be very difficult to understanding from a single format material (e.g., visual text only).Because this concept seems so intuitively correct, multimedia instruction developers have added multimedia presentations in accordance with these multimedia design principles in an effort to help students learn better even though all multimedia messages are not equally effective (Mayer, 2001).Some researchers have expressed reservations about the generalizability of how multimedia design effects influence in student learning (e.g., Acha, 2009;Schnotz & Bannert, 2003;Schnotz & Kürschner, 2007;Tabbers, Martens, & van Merriënboer, 2004).For instance, Schnotz and Bannertdemonstrate that adding pictures (or graphics) to words does not always lead to higher learning outcomes because additional information like graphics can interfere with mental model construction created during non-verbal comprehension.Tabbers et al. also claim that either replacing words with spoken words or adding pictures to words does not easily adapted multimedia design effects such as modality and multimedia effects.
Why do some results differ from the findings of existing multimedia design studies?What causes the different learning outcomes?Why does multimedia not work as design principles promise?Although multimedia design principles or design effects have been considered as key guides in designing effective multimedia presentations, it is not sufficiently clear why multimedia messages are not equally effective.Little attention has been given to the human subject (individual learner) in multimedia learning because multimedia design effects studies have been focused on when multimedia does work.It is very important to reconsider how learning processes occur when a learner is able to learn with presented materials in media or multimedia (e.g.visual/pictorial and auditory/verbal information).The main focus of our study was to investigate multimedia effects that have had mixed results to explore common factors that contribute to instructional outcome differences.First, we describe and summarize three experimental studies we conducted from 2006 to 2010.Then we analyze our findings to explore learner characteristics that may impact the cognitive processes in multimedia learning.Third, based on this analysis, we suggest how cognitive processes are related to multimedia information as well as learner characteristics and introduce the concept of cognitive synergy in multimedia learning.Finally, we suggest further perspective about multimedia learning studies.

Experiment 1
The purpose of the first experiment (Kim & Gilman, 2008) was to investigate multimedia effects with six modes of Web-based vocabulary self-instruction for English as Foreign Language (EFL) students: visual text (Mode A), visual text and added spoken text (Mode B), visual text and added graphics (Mode C), visual text, added spoken text, and added graphics (Mode D), reduced visual text and added spoken text (Mode E), and reduced visual text, added spoken text, and added graphics (Mode F).The items of English vocabulary to be learned on the instruction program were: tether, wither, tumble, separate, gorge, fetter, beacon, crest, awl, ditch, entice, taut, quench, wizen, and waylay.The length of each lesson was a maximum of 30 minutes and the time was controlled by the computer user.The participants were 172 tenth-grade students from five classes in a public middle school in South Korea.Each student was required to complete several testing instruments (e.g., pretest, posttest, retention test, and student attitude inventory).

Experiment 2
In the second experiment (Kim & Kim, 2010), we concentrated on the multimedia effects within three different screen sizes (small: 320*240 pixels, medium: 480*320 pixels, and large: 600*800 pixels) in the Web-based vocabulary self-instruction that was created for the first study.This expanded study adopted two representation modes from the previous study: visual text (Mode A) and visual text and added graphics (Mode C).The participants were 135 tenth-grade students from five classes in a public middle school in South Korea.Each student was required to complete a pretest, posttest, and retention test for the study.

Experiment 3
The purpose of the third experiment was to investigate the multimedia effects with four modes of Web-based problem solving self-instruction in learning mathematics.This study used two representation modes (algebraic and geometric representations) often employed in learning mathematics within two different screen sizes (small: 320*240 and large: 600*800).The participants were 101 tenth-grade students from four classes in a public middle school in South Korea.Each student was required to take a pretest and posttest for the study.
In sum, we examined multimedia design principles in two experimental studies on vocabulary learning and one study on mathematics learning that focused on learning effects of the use of multimedia.Based on Mayer's CTML, it is commonly assumed that students are more likely to build representational connections between verbal and nonverbal information than through single-format information in order to increase higher cognitive processes.However, our evidentiary findings indicate that multimedia messages which consist of verbal and nonverbal information are not equally effective for all learners as shown in Table 1.Scores on the posttest and the retention test showed differences between the three screen multimedia instructions at the .05level of significance.In short, the large screen multimedia instruction helped the students to learn English vocabulary more effectively than the small screen instruction.However, there was little difference in vocabulary learning between the multimedia modes of "visual text" and "visual text and graphics." There was no significant difference in scores on the posttest.In short, there was little difference in mathematics learning between the four modes.

Multimedia Student Characteristics
Although we focused on multimedia learning particularly to learn and retain new English vocabulary and solve mathematics problems through presented multimedia messages, our contradictory findings in the three experiments lead us to conclude that current multimedia design effects do not occur automatically.For instance, from the cognitive theory of multimedia theory (e.g., dual-channel assumption) and multimedia design principles perspectives, presented material (i.e., multimedia messages) in the visual and auditory channel should trigger higher cognitive processes in the mental construction of learner, but our results in Experiment 1 indicated that presenting the information in "visual text and added spoken text" or "reduced visual text and added spoken text" significantly decreased learners' ability to learn and retain English vocabulary.Korean EFL students often have difficulty knowing exactly how words are pronounced because they are accustomed to memorizing new English vocabulary without knowing exactly how the word is pronounced, so we suggested that spoken text (e.g., narration) creates an unnecessary distraction for them (Kim & Gilman, 2008).Likewise, although graphics are often used to represent important information and support the information in visual text format, we found unexpected learning outcomes in Experiment 1 and 2 that there was a little difference in recognizing English vocabulary between the presentation modes of "visual text" and the "visual text and added graphics."Moreover, the unanticipated results in Experiment 3 show that there were no significant differences between two modes of "algebraic" and "geometric representations" in worked-out examples.The participants were forced to mentally integrate the solution steps with the information from "geometric representations", which quickly exhausted their cognitive capacity.
It looks like presented multimedia information in both verbal and non-verbal formats did not process in a way that it could expand the learner's mental schemata.This is contrary to the assumption of the multimedia effect.We determined that since the learning context did not require high-order thinking processes, providing visual text instruction may be just as useful as added graphics (e.g., pictorial annotations) into the visual text.In other words, spending time manipulating the presented multimedia messages by reading information in visual text and graphics would distract learners who are accustomed to learning through visual text format only (Kim & Kim, 2010).Chen, Hsieh, and Kinshuk (2008) also note that providing too many pictorial annotations leads to increased cognitive load when students do not have high verbal and visual abilities.
In this session, we would like to look more closely at our results of inhibited multimedia design effects using our original questions: Why do our results differ from the findings of existing multimedia design studies?What causes the different results?Why does our multimedia not work as design principles promise?In order to answer these questions, we propose a possible explanation that cognitive processes would be influenced by the characteristics of learner such as prior knowledge, affection, and metacognition while the learner builds mental representation in the human cognitive architecture as Mayer and other researchers promised.Although basic multimedia design principles that are implemented in light of the cognitive theory of multimedia learning provide an important foundation for multimedia learning, the study of student characteristics has often been neglected.According to Eysenck (2001), most human cognition involves a mixture of bottom-up processing (stimulus-attention-perception-thought processes-decision-response or action) and top-down processing.In top-down processing, we think that adding these three student characteristics to multimedia learning considerations can create more coherent explanations that better illuminate why and how students learn differently.Because it is not the media matter, but how they are used to select, organize, and make sense of new information to be learned.

Prior Knowledge
Students need prior knowledge to make sense of new content in learning.Mayer's cognitive theory of multimedia learning(CTML) demonstrates how the multimedia learner builds mental representations from the presented multimedia messages in cognitive processes (i.e., selecting relevant information, organizing it into coherent representations, and integrating it with prior knowledge) that foster meaningful learning.In Experiments 1 and 2, we found that multimedia design effects (e.g., multimedia and modality principles) do not automatically occur when the information to be learned is not related to the learner's prior knowledge, which would reduce working memory load and increase schema construction and automation in cognitive processes.For instance, in the cognitive processes of learning the spelling, meaning, and pronunciation of the new English word "gorge," the learner unsuccessfully integrates information in working (short-term) memory with unfamiliar knowledge from long-term memory while trying to integrate the provided information in English and Korean in a manner that makes sense.In other words, high interactivity material in Korean and English in the provided multimedia messages using "visual text and added spoken text" or "reduced visual text and added spoken text" imposes a high working memory load in the form of cognitive loads (e.g., intrinsic cognitive load) that can lead to perceptual errors.Similarly, adding graphics to the visual text inhibited the multimedia design effects on mental constructions with unfamiliar information from long-term memory because the multimedia learners regard added information as excess complexity and incomprehensible information (e.g., Acha, 2009;Stokes, 2002).These findings are closely related to the cognitive load theory view that working memory limitations such as duration and capacity limitations are critical when dealing with unfamiliar information from long-term memory.Therefore, it is essential to take into account prior knowledge in multimedia learning research because it plays a significant role in text comprehension (Alexander & Murphy, 1999).

Affection
Having multimedia presentations with more personal devices (e.g., tablet-PCs, tablets, touch pads and smartphones) enhances learning experiences including social interaction with emotions in or out of classroom.
For instance, many students access an online learning environment to watch rich digital content (e.g., eTextbooks, TV news, YouTube videos, and interactive lecture notes) and participate in on-the-go written/voice discussion activities (e.g., online discussion and Webinar activities) literally anytime and anywhere.The use of innovative technology tools for much more than social communication also gives students more comfort in using technology overall and confidence to be multimedia learners.As a result, they may feel affection for multimedia learning.This affection refers to a variety of student emotions toward to learning, including belief, expectation, interest, and motivation.Each of these student emotions play a different role in the learning processes and some may not be related to learning.For instance, our findings in Experiment 2 indicate that although many students express satisfaction with and enjoyment of learning through short message services on their mobile phones (e.g., text messaging; Cavus & Ibrahim, 2009), information on the smaller screen created obstacles in cognitive processes for users who are habituated to scanning information quickly rather than reading word by word on bigger computer screens (Kim & Kim, 2010).Some researchers have focused on how prior knowledge and beliefs are interconnected in cognitive processes (e.g., see Dole & Sinatra, 1998).Others have indicated that prior knowledge and interests affect learning (e.g., see Alexander, Kulikowich, & Schulze, 1994;Hidi & Renninger, 2006).In the psychology literature, although learning is not necessarily dependent on student emotions such as expectation, higher student expectation has been found to result in higher performance (Schilling & Schilling, 1999).Similarly, based on the findings of our experiments, we assume that learner expectations and knowledge affect mental model construction in an inappropriate way such that added spoken text into visual text would have negative effects on the recognition of new English vocabulary.Students had difficulty knowing exactly how words were pronounced because they were accustomed to memorizing new English vocabulary without knowing exactly how the word was pronounced; spoken text created an unnecessary distraction for them (Kim & Gilman, 2008).In short, student affection does not directly impact human cognitive processes in multimedia presentation, though our findings demonstrate that affection may impact learning processes to select, organize, and make sense of new information.

Metacognition
In addition to prior knowledge and affection, one of the primary foci of multimedia learning should be metacognition because students must be able to think out their cognitive processes in multimedia learning.In other words, metacognitive awareness allows students to be aware of what they know and how they understand.
The effects of metacognitive processes on cognitive process have been found in mathematics learning (Goos, Galbraith, & Renshaw, 2002;Sfard, 2001) and reading comprehension (Palincsar & Brown, 1989;Pressley, 1995).However, few studies have been conducted on metacognition in multimedia learning (e.g., see Lee & Baylor, 2006;Tsai, 2009) and there has been little research to examine students' prior knowledge, affection, and metacognition simultaneously.As noted by some researchers for the dualistic relation between cognition and metacognition, students' expectations may affect the examining process in metacognition such that "if their understanding of a question is dominated by their expectations, it is possible that they will not understand the question in the way that was intended, since they are likely to stop generating alternative meanings if they believe this understanding is correct" (Crisp, Sweiry, Ahmed, & Pollitt, 2008, p. 97).In considering how to investigate multimedia learning, therefore, the cognition-metacognition dichotomy should be reconsidered to better understand cognitive processes.In order to integrate metacognition with cognition in multimedia learning, an initial strategy is to collect two different types of data for both multimedia learning achievements and metacognition (e.g., see Tsai, 2009 for developing a metacognition survey in details).After collecting these two different types of data, the metacognition data can be analyzed to make conjectures about the correlations between cognition and metacognition.For a better understanding of cognitive processes in multimedia learning, more research is needed on cognition, metacognition, and their relationship.
In sum, we assume that prior knowledge, affection, and metacognition of the learner may impact cognitive processes in such a way that adding spoken text or graphics into the visual text would prevent multimedia design effects as the promised on both the recognition of new English vocabulary and the ability to solve math problems.
In other words, we suggest that students learn what they want to learn through multimedia messages based on their affection and metacognition while the provided multimedia materials help students to select relevant information, organize it into coherent representations, and integrate it with prior knowledge as meaningful learning as Mayer and Moreno (2002) suggest.

Cognitive Synergy
Although our studies are a preliminary start showing why current design principles need to be reconsidered with respect to different learners, we also propose a normative model for effective multimedia learning to show how cognitive processes should occur with learner prior knowledge, affection, and metacognition as shown in Figure 1.Students do not learn only by reading and listening to media information, but also with an interconnected learning process.In Figure 1, the center circle represents cognitive processes that occur in multimedia learning.
The three surrounding circles represent three relevant factors of cognitive processes that may contribute to learning outcome differences.The dashed line distinguishes the internal and external components when individual learners receive the material they are meant to learn.In the model, provided material is an instructional message in various formats including visual alone (e.g., visual text), visual and auditory (e.g., visual text and added spoken text), and verbal and non-verbal formats (e.g., visual text and added graphics).Prior knowledge refers to one's current preconceptions developed informally and formally to make sense of new information (Bransford et al., 2006).The term affection can be used to refer to motivation, interest, attention, expectations, beliefs, and attitudes toward multimedia learning (Matthews, Zeidner, & Robers, 2006).
Metacognition is a self-awareness that regulates one's own thinking processes in certain patterns such as planning, monitoring, evaluating, and revising (Brown, 1987;Tsai, 2009).The normative model for effective multimedia learning is not a simple liner regression or correlation.Instead, it shows complex cognitive learning processes with interconnected factors to produce a learning outcome not independently obtainable.We refer to this phenomenon as Cognitive Synergy in learning, which occurs when the learners create a mental construction with the both external and internal components in ways that enhance or magnify one or more cognitive processes as higher cognitive processes.Cognitive Synergy appears to support Mayer's CTML that it may reduce heavy loads on the limited capacity of working memory in order to increase the efficiency of processing in working memory.The model shows when or how multimedia learning should be more effective, supportive of different students, and more communal.As has been mentioned above, cognitive synergy occurs under the specific condition that two or more components are functioning together to achieve higher cognitive processes as meaningful learning.Based on our experiments, we assume that cognitive synergy is not necessarily associated with the use of multimedia as researchers have expected.For instance, the provided instructional messages in visual formal only (e.g., "visual text") may cause a higher cognitive synergy with the learner's prior knowledge (e.g., the knowledge is unfamiliar to the learner, but the learner is experienced with similar visual format presentations), affection (e.g., the learner's preference for visual rather than auditory format), and metacognition (e.g., the learner's capability to organize and use given visual information) while other messages in multimedia format (e.g., "visual text and added spoken text" and "reduced visual text and added spoken text") may distract from the cognitive processes that impact the mental model construction in an inappropriate way.This higher cognitive synergy in the visual text mode may increase effective working memory while extraneous overload would be increased by the need to mentally integrate information to be understood in visual and spoken text mode.Extraneous cognitive load is likely to have a negative impact on learning compared to the visual text mode of presentation.For the same reason, the use of visual text may be just as useful as adding graphics into the visual text for the learner may be able to engage in a higher processing that is not exceed the learner's cognitive capacity and thus build meaningful learning outcomes.We suggest the potential for synergy in multimedia learning to explain why some findings are in contrast to accepted current multimedia design principles.We strongly believe that depending on cognitive synergy in multimedia learning, the content, quality, and results of the multimedia learning can be changed.By the term cognitive synergy, we do not mean that all factors described previously will merge into one big structure that eliminates the unique characteristics of each of them.Instead, we think that synergic effects can inform one another in multimedia learning and create more coherent and useful perspectives that will help researchers and developers better understand why and how students learn in given multimedia information.It could be expected that multimedia researchers and teachers would be able to optimally align pedagogical and practical approaches with learner characteristics.

Conclusion
Although we focused on contradictory findings from three experiments particularly to learn and retain new English vocabulary and solve mathematics problems, our findings revealed shortcomings of multimedia learning in general.First, this study was designed to investigate the difference in multimedia learning outcomes on Web-based self-instructions rather than interactive learning environments.Second, the presented material did not require high-order thinking and the participants had limited domain knowledge.Third, the data of this study were based on repeated measurements using the same multiple-choice questions.It is generally agreed that multimedia design principles should be qualified by the content to be learned, student characteristics, and the conditions which performance is measured (Fletcher & Tobias, 2005).
An analysis of individual factors that may contribute to instructional outcomes differences will be helpful in order to understand the relevant components in multimedia learning processes.However, the analysis is still limited in creating a holistic picture of how and why this cognitive synergy occurs to increase the efficiency of processing in working memory.Therefore, future researchers need to develop analysis methods to investigate not only individual factors, but also their combined synergic effects on multimedia learning.With respect to approaches to synergic effects for future study on multimedia learning, we make several key claims, as follows:  Synergic effects are combinations of three learner characteristics: prior knowledge, affection, and metacognition.


Synergic effects may differ depending on learning contexts.
 Synergic effects are dynamic and multidirectional processes.


The nature of synergic effects varies with the quality of prior knowledge, with the development of personal affection, with the level of metacognition, and even with the stage of development.

Figure 1 .
Figure 1.A normative model for effective multimedia learning

Table 1 .
Summary of methods and results in three experimental studies