Psychological Factors Affecting Medical Students ’ Learning with Erroneous Worked Examples

The acquisition of diagnostic competence is seen as a major goal during the course of study in medicine. One innovative method to foster this goal is problem-based learning with erroneous worked examples provided in a computer learning environment. The present study explores the relationship of attitudinal, emotional and cognitive factors for learning with erroneous worked examples. 72 medical students from a German university worked with six case-based examples in the domain of arterial hypertension. Domain-specific conceptual prior knowledge, anxiety of making errors, attitudes towards errors, and ambiguity tolerance were measured as independent variables before the students worked with the examples. Diagnostic competence was operationalized by measuring conceptual, strategic, and conditional knowledge, which were assessed as dependent variables after working with the learning environment. A cluster analytic approach yielded three clusters. For each, the relationship with the learning outcome was analysed. Cluster membership significantly influenced the learning outcome in strategic, but not in conditional knowledge. Furthermore, cluster membership had a significant effect on conceptual knowledge; there was also an increase in conceptual knowledge for all clusters when conceptual knowledge measured after the treatment was compared to prior conceptual knowledge. The results clearly indicate the importance of a certain pattern of psychological factors for learning with erroneous worked examples.


Introduction
Acquiring diagnostic competence in medicine is a complex learning target.Diagnostic competence can be divided into domain-specific conceptual knowledge, strategic knowledge, and conditional knowledge (Stark, Kopp & Fischer, 2011).Conceptual knowledge consists of factual knowledge about domain-specific concepts and their interrelations.Strategic knowledge comprises knowledge about problem-solving strategies and heuristics in the diagnostic process.Conditional knowledge is knowledge about the conditions of application of conceptual and strategic knowledge and knowledge about the rationale behind the selection of diagnostic decisions and procedures.Since the early 1950s, universities and medical schools apply problem-based learning (PBL) to support the development of diagnostic competence (Dochy, Segers van den Bossche & Gijbels, 2003).In Germany, PBL is integrated in nearly all medical faculties; to date, 92 % of medical faculties have integrated PBL in one or more course (Kruppa, Jünger & Nikendei, 2009).
However, in educational psychology the effectiveness of PBL is still discussed controversially (Kirschner, Sweller & Clark, 2006;Schmidt, Loyens, van Gog & Paas, 2007;Stark, Herzmann & Krause, 2010).Unsupported PBL is not effective (Kirschner et al., 2006) so specific measures of instructional support have to be integrated in PBL-programs (Krause, Stark & Herzmann, 2011;Schmidt et al., 2007;Stark et al., 2010).For instance, instructional support can be given by integrating worked examples (Atkinson, Derry, Renkl & Wortham, 2000;Paas & van Gog, 2006) in PBL learning environments, combining forms of direct instruction with methods enabling self-generated knowledge construction (Rothmeier & Mandl, 2006).Worked examples assist learners in focussing on relevant aspects of the task and therefore preserve cognitive resources (Cooper & Sweller, 1987;Sweller, Chandler, Tierney & Cooper, 1990;Sweller & Cooper, 1985).Learning with worked examples proved to be very effective in different domains (Stark, Mandl, Gruber & Renkl, 2002).In order to increase the effectiveness of worked examples, various instructional measures have been evaluated.One innovative and especially promising measure in the context of medical education consists in the presentation of erroneous examples (Große & Renkl, 2007;Siegler, 2002).In various studies, our research group demonstrated the effectiveness of erroneous worked examples in combination with elaborate feedback in the context of a computer-based learning environment (Kopp, Stark & Fischer, 2008;Kopp, Stark, Kühne-Eversmann & Fischer, 2009;Stark et al., 2011).
The effect of erroneous worked examples and elaborate feedback on the acquisition of diagnostic competence depended on the learners' level of prior knowledge.When no elaborate feedback was given, only learners with a high level of prior knowledge profited from erroneous worked examples.Prior knowledge therefore seems to be an important prerequisite to profit from the specific learning potential of errors in general (e.g., Große & Renkl, 2007) and specifically from the learning environment used in our studies (Stark et al., 2011).The aim of the present study is to explore the role of prior knowledge in conjunction with other potentially relevant psychological factors and their relations to the learning outcome.
Besides prior knowledge, the acquisition of diagnostic competence as learning outcome might also be influenced by psychological factors like emotional states, attitudes and personality dimensions (Stark et al., 2011).In the context of the present study, we focussed on anxiety of making errors, attitudes towards errors, and ambiguity tolerance as independent variables and the learning outcome as the dependent variable.
In the domain of medicine errors may have fatal consequences (Al-Assaf, Bumpus, Carter & Dixon, 2003).Hence it is obvious that the possibility of making errors in diagnostic decisions is an import psychological aspect for physicians and medical students.In particular, it was shown that medical students have anxiety of making errors even when the erroneous outcome is reversible, like errors occurring in protected learning situations (Muller & Ornstein, 2007).Hence, anxiety of making errors is highly important in a learning environment where errors are an inherent design feature.In such a learning situation a learner not only has to cope with his own errors, he also has to cope with the errors presented to him.Thus, it seems to be important to clarify the role of anxiety of making errors when erroneous worked examples are presented to learners.
Humans show the tendency to evaluate objects and situation as positive or negative (Wänke & Bohner, 2006).This evaluation is usually called an attitude towards the object or situation.In the context of the present study this attitude refers to the evaluation of the errors presented in the worked example.The result of an evaluation of errors as positive or negative usually leads to different coping mechanisms, information processing strategies or motivations (Maio & Olson, 2000).Furthermore, learning with erroneous worked examples confronts the learner with an ambiguous situation (Budner, 1962) as it is unknown if the information currently presented is correct or not and is corrected later on.In addition to the kind of ambiguity imposed by the learning environment, it is possible that in the domain of medical diagnostics coping with errors increases the learners' perceived ambiguity.Ambiguity tolerance (Frenkel-Brunswik, 1949) might therefore be an important learning prerequisite in this context and hence its role in the context of learning with erroneous worked examples has to be explored.In the next section we will provide definitions of these factors and clarify their importance in the learning process.
The dependent variable in the study is diagnostic competence as the primary learning outcome.Diagnostic competence in the present study is analysed separately in the components according to Stark. et al. (2011).In addition to the learning outcome two variables of interests in this study are subjective learning success and time on task.The objective learning outcome can differ from the subjective experienced learning success and the presentation of erroneous examples may prolong the time needed to work on the examples (Kopp et al., 2008).However, subjective learning success is an important condition for learning efforts to be persistent and therefore is vital for the sustainability of the learning process (Kopp et al., 2009).

Domain-Specific Conceptual Prior Knowledge
Prior knowledge describes a learners' knowledge of domain-specific facts and their interrelations that a learner already has (Krause & Stark, 2006).Prior knowledge is available to a learner before a new learning task begins (Dochy & Alexander, 1995).Learners` domain-specific prior knowledge is the most important resource and the best predictor for future learning (Dochy, 1992;Krause & Stark, 2006).Prior knowledge influences which information attention is paid to, which aspects are regarded as important and what is understood (e.g., Alexander, 1996).The importance of prior knowledge was shown in several studies on text learning as well as example-based learning (c.f.Kintsch, 1994;Stark, 1999).Große and Renkl (2004;2007) showed that prior knowledge is especially important for learning with erroneous examples because it supports effective self-explanation processes, thus resulting in higher learning outcomes.Only if the learner clearly understands what is wrong in a given situation and why it is wrong can the potential of learning from errors fully unfold (Siegler, 2002).The presentation of erroneous examples provides opportunities for learning only when the student understands the error (Oser & Spychiger, 2005).

Anxiety of Making Errors
Anxiety refers to an intraindividually stable tendency to evaluate situations as threating and to exhibit an emotional reaction (Krohne, 1996).Errors therefore provide an emotional frame for a student's learning.As a matter of fact, emotions in general are very important for learning processes (Croskerry, 2005;Norman & Eva, 2010).The Yerkes-Dodson-Law indicates an inverted U-shaped relation between arousal and performance, stating that if the arousal is too low or too high, the person is neither challenged nor overwhelmed, respectively (Yerkes & Dodson, 1908).Arousal on a medium level leads to optimal performance.This relationship applies to all human cognitive processes, especially learning, and it is particularly associated with anxiety (Frenzel, Götz, & Pekrun, 2009).The optimal level of emotional arousal varies from learner to learner (Jansen & Streit, 2010).As a consequence, high anxiety of making errors does not form an optimal basis for learning with erroneous worked examples.Too low anxiety of making errors might be dysfunctional for learning as well because the absence of anxiety results in an inadequate processing of erroneous information Therefore it is hypothesized that a medium level of anxiety of making errors forms the optimal basis to profit from studying erroneous information (Spychiger, Kuster & Oser, 2006).

Attitudes towards Errors in the Learning Process
Attitudes are defined as positive or negative evaluations of an object or a situation (Fiske & Taylor, 1991).When learning with erroneous worked examples the evaluation aims at the provided errors.According to Weinert (2004), errors can be evaluated as frustrating or stimulating and hence hinder or foster the learners' motivation, respectively.The evaluation of errors further depends on the context in which they occur (Dweck, 1996).If errors occur in learning situations, they can be seen as a chance to avoid them in the future.But for errors to be productive, it is necessary that the learner realizes, analyses and corrects the errors accordingly and elaborates strategies to avoid similar errors in the future (Oser, Hascher & Spychiger, 1999, Heinze, 2005).Thus, if errors are perceived as stimulating and scrutinized thoroughly learners can profit from them.The evaluation of errors therefore forms the attitudinal prerequisite to learn from erroneous information presented in the learning process.

Ambiguity Tolerance
Ambiguity tolerance denotes a stable and domain-general skill of a person to cope with ambiguous situations (Dalbert, 1996, Frenkel-Brunswik, 1949).According to Frenkel-Brunswik (1949) ambiguity tolerance allows to predict in which way a person will cope with an ambiguous situation.While persons with high ambiguity tolerance tend to evaluate unknown, inconsistent, and complex situations as a positive challenge, those with low ambiguity tolerance are inclined to interpret ambiguous situations as a source of threat.This different evaluation of a situation leads to different coping strategies.In threatening situations, persons with high ambiguity tolerance ask significantly more often for further information than those with low ambiguity tolerance.The influence of ambiguity tolerance on information processing is moderated by the personal relevance of the situation (Sorrentino, Bobocel, Gitta, Olson & Hewitt, 1988).Under highly relevant conditions, persons with high ambiguity tolerance tend to process information rather systematically and deeply, whereas those with low ambiguity tolerance show a rather heuristic and superficial manner of information processing.In conditions with low relevance, this pattern is reversed.In medical education, the development of diagnostic competence is not only important for passing exams; in addition, it is highly recognized as one of the key competencies for expert practitioners.Therefore, it can be expected that most learners will evaluate the learning situation as relevant leading to the information processing described above.So, ambiguity tolerance should affect the learning process and its outcomes.

Research Questions
1) Which learner profiles can be differentiated on the basis of the psychological factors, that is domain-specific conceptual prior knowledge, anxiety of making errors, attitudes towards errors and ambiguity tolerance?
2) What is the relationship between the learners' profiles and the learning outcome in strategic and conditional knowledge?
3) To what extent is an increase in domain-specific conceptual knowledge fostered by the learning environment and how far is this increase moderated by the learner profiles?4) What is the relationship between learner profiles and subjective learning success and time on task?

Sample
Seventy-two advanced medical students (age: M = 24.15,SD = 1.94;50 women) from two medical faculties in Munich, Germany, participated voluntarily in the present study.All participants were clinical students in their third to fifth year.

Learning Environment
The learning environment was integrated into the CASUS software (Fischer, 2000).The learning environment consisted of six worked examples on secondary arterial hypertension.In each example a clinical case was presented (e.g., a 62 year old male patient with hypertension who is feeling tired and exhausted; see Appendix A).When working through these examples, the learner saw him-/herself in the role of an advanced medical student making diagnostic decisions.The character drew conclusions for the working and differential diagnoses, and derived further diagnostic steps from this preliminary diagnosis.Subsequently, a short feedback from a domain expert about the result followed (consideration, conclusion or further procedure are classified right or wrong).Every example consisted of three to four of such sequences until the final diagnosis was reached.In the example case shown in the appendix, the error consisted in premature closure: the patient was referred to an appointment in three months without excluding the relevant differential diagnosis.Time to work on the worked examples was not restricted to ensure ecological validity.

Instruments Measuring Variables for Clustering
The instruments measuring the variables for determining the learner profiles were composed of a multiple-choice test capturing domain-specific conceptual prior knowledge and three scales to measure emotional and attitudinal prerequisites.
Domain-specific conceptual prior knowledge was measured with a multiple-choice test containing 21 items (Appendix B).The test was designed to cover a broad range of aspects concerning the topic arterial hypertension.The responses were dichotomously coded, Cronbach's α was .53.This rather small Cronbach's α was expected as the items aimed to measure a broad range of knowledge and were therefore very heterogeneous; in addition, the coding procedure pointed towards a restricted variance.Both facts usually account for a low estimate of reliability.
The variables anxiety of making errors, attitudes towards errors and ambiguity tolerance were measured by scales consisting of several items.All items had to be answered on a six-point rating scale ranging from 1 = completely disagree and 6 = completely agree Anxiety of making errors was measured with six items (e.g., "I get afraid if an errors happens in a lecture.")adapted from a scale developed by Spychiger, Mahler, Hascher, and Oser (1998).Subjects with high scores in this scale are afraid of making errors in learning situations, of interactions with the teaching staff, and feel shame when making errors in front of peers.Cronbach's α was .90.
Attitude towards errors was measured with scale developed by Spychiger et al. (1998).The scale measures how a person is going to handle errors during the learning process, i.e., if a person analyses her errors and if she sees errors as an opportunity to learn from.Higher scores indicate that a subject evaluates errors as a learning opportunity.Subjects with higher scores are also more inclined to try out several alternative solutions.The scale consists of nine items (e.g., "Making errors during lectures are helpful to prevent such errors in the future."),Cronbach's α was .78.
Ambiguity tolerance was measured with a scale developed by Dalbert (1996).This scale captures the subjects' overall preference for ambiguous situations and their ability to cope with these situations.High scores indicate a tendency to seek ambiguous situations and cope with such situations.The scale consists of eight items (e.g., "I'm waiting for something exciting to happen.").Cronbach's α was .80.

Instruments Measuring Dependent Variables
Domain-specific conceptual knowledge.The same multiple-choice test used to measure domain-specific conceptual prior knowledge was also used to measure the domain-specific conceptual knowledge.The 21 items had to be answered a second time, Cronbach's α for the second measurement was .43.
Strategic knowledge was measured by 10 key feature cases (see Appendix B).For every key feature case three questions had to be answered and according to the quality of the answer points were assigned.A maximum of three points for each key feature case could be assigned, so the maximum number of points was 30.Cronbach's α was .67.
Conditional knowledge was measured by three problem-solving tasks with domain-specific topics (see Appendix B).Every task consisted of several questions.For a correct answer points were assigned, the maximum number for all tasks to obtain was 20 points.Cronbach's α was .72.
Subjective learning success was measured with a 13-items scale from Krause (2007).The scale measures the subjects' self-reported experience and learning outcomes.Cronbach's α for this scale was .88.Time on task was automatically registered by the learning environment, it is reported in minutes.

Procedure
After a short introduction into the learning environment, domain specific conceptual prior knowledge, attitudes towards errors, anxiety of making errors, and ambiguity tolerance were assessed.
Afterwards, all participants had to study six worked examples.After the learning phase, a 15-min pause was scheduled.Then subjective learning success was measured.At the end of the session, the three knowledge tests measuring domain-specific conceptual knowledge strategic knowledge, and conditional knowledge, were administered.

Identification of Learner Profiles
To determine the learner profiles the k-means cluster method was used.Differences in variance were eliminated by z-transformation.The F max -statistic (Bacher, 1996) was applied to fix the number of clusters.A 3-cluster solution was retained; Figure 1 shows the cluster profile.
Cluster 1 (N = 23).Domain-specific conceptual prior knowledge was quite low in this cluster (M = -.37,SD = .91).Anxiety of making errors (M = .89,SD = .53)and the attitude towards mistakes were pronounced in this group (M = .68,SD = .74).Also pronounced was ambiguity tolerance (M = .89,SD = .84).This cluster could be named as half-knowing/anxious learners because it is characterized by the high anxiety of making error, their level of domain-specific conceptual prior knowledge is on a middle level in comparison with the other clusters.Furthermore this cluster is characterized by a high score of attitude towards errors and high score ambiguity tolerance.
Cluster 2 (N = 32).Persons in this cluster showed the highest value in domain-specific conceptual prior knowledge compared with all other clusters (M = .79,SD = .52).Anxiety of making errors (M = -.10,SD = 0,78) and attitudes towards mistakes (M = -.16,SD = .74)were moderate.Persons in this cluster had low values in ambiguity tolerance (M = -.45,SD = .67).The second cluster could be named as knowing/unremarkable learners, as their level of domain-specific conceptual knowledge is high but all other variables show a medium level.
Cluster 3 (N = 17).This cluster showed the lowest value of domain-specific conceptual prior knowledge compared with the other clusters (M = -.98,SD = .56).Anxiety of making errors (M = -1, SD = .82)and attitudes towards mistakes (M = .61,SD = 1.22) were also lowest in this cluster.Ambiguity tolerance was approximately at the same level as in cluster 2 (M = -.36,SD = .95).The last cluster is characterized by a low level of domain-specific conceptual knowledge and low level of anxiety of making errors and attitudes towards mistakes, so this cluster could be interpreted as the not-knowing/fearless learners.

Relationship between Learner Profile, Strategic, and Conditional Knowledge
This research question was examined by means of univariate analysis of variance with cluster membership as independent and strategic and conditional knowledge as dependent variable.Table 1 shows the descriptive statistics for strategic and conditional knowledge.Concerning strategic knowledge, the clusters differed significantly (F(2,69) = 6.49, p = .003,partial η 2 = .16).In order to examine the source of this difference a Tukey post-hoc test was performed (see table 2).The post-hoc examination yielded homogenous groups for cluster 1 and cluster 3, for which no significant differences in strategic knowledge could be found.However, cluster 2 showed a significant higher strategic knowledge than cluster 1 and 3.For conditional knowledge no significant difference between the three clusters was found (F(2,69) = 2.16, p = .12,partial η 2 = .059).

Extent of Increase of Domain-Specific Conceptual Knowledge and Moderation by Learner Profile
To answer this research question a paired t-test comparing the performance in domain-specific conceptual knowledge before and after the learning phase for all clusters was calculated.The correlation between the two measurements was .71(p < .001)for the whole sample.The difference between both measurements was significant and substantial (t(71) = 11.64,p < .001,d = 1.71).Table 3 indicates that persons from cluster 3 showed the highest increase in knowledge in comparison with the other clusters.A univariate analysis of variance revealed a significant effect of cluster membership on the increase in domain-specific conceptual knowledge (F(2,69) = 7.12, p = .002,partial η 2 = .17).A Tukey post-hoc test showed that cluster 1 and cluster 2 are forming a homogenous group.Subjects in cluster 3 showed the highest knowledge increase.

Relationship between Learner Profile, Subjective Learning Success and Time on Task
This research question was examined by means of univariate analysis of variance.Cluster membership constitutes the independent variable; subjective learning success and time on task were dependent variables.
Table 5. Descriptive statistics for subjective learning success (* One subject had a missing value in subjective learning success, so only 16 subjects were in cluster 3 this time) For subjective learning success a significant effect for cluster membership (F(2,68) = 6.17, p = .003,partial η 2 = .15)was found.Table 5 gives the cluster means for subjective learning success.No effect was found for time-on-task (F(2,69) = .10,p = .90,partial η 2 = .003),however.
A Tukey post-hoc test for subjective learning success revealed that the significant effect in the analysis of variance was due to a significant difference between cluster 1 and 3. Table 6 shows the differences between the clusters.

Discussion
The resulting learner profiles could be described straightforwardly as half-knowing/anxious, knowing/unremarkable and not-knowing/fearless learners.One remarkable result is that subjects with a high score in attitudes towards errors also show a high score of anxiety of making errors and vice versa.This finding could be interpreted from a trait perspective.Learners with a high anxiety of making errors are more likely to scrutinize the errors made more thoroughly in order to avoid similar errors in the future.
The effect size of the relationship between learner profile and strategic knowledge was large (Cohen, 1988, Richardson, 2011).This effect was due to a significant difference between the knowing/unremarkable profile and the other profiles.The main difference between the knowing/unremarkable learners is made up by domain-specific conceptual prior knowledge.Persons belonging to the knowing/unremarkable profile also showed a medium level in anxiety of making errors.The results show that students with this specific profile profited most from erroneous worked example in regard to strategic knowledge.This finding is also in accordance with the assumption that medium anxiety levels foster learning processes in general.
For conditional knowledge no relationship with the learner profiles was found.It is not clear why there was an effect for strategic but not for conditional knowledge.One possible explanation is a method artifact.Learner profiles are aggregated measures over four continuous variables and this meant a loss of information about individual differences.The use of cluster membership as a nominal variable could also cause a loss of effect size and power as well as the occurrence of spurious significant main effects (MacCallum, Zhang, Preacher & Rucker, 2002).Especially the loss of power is a possible explanation for the missing effect of conditional knowledge.
Domain-specific conceptual knowledge was substantially higher after working with the learning environment for all learners.Especially learners with the not-knowing/fearless profile showed the highest increase, but this is a surprising result.examples has positive effects for the learners with little knowledge regardless of the disadvantageous combination of psychological factors.Not-knowing/fearless learners profit the most from using the learning environment; the unfavourable combination of learner perquisites notwithstanding.However, as the same items were used in both measurements and the learning phase was rather short, recall effects have to be taken into consideration, too.
The subjective learning success was influenced by cluster membership this effect was also large.Subjective learning success follows a curvilinear function with regard to domain-specific conceptual knowledge.Subjects showing a medium level of domain-specific conceptual knowledge apparently perceived the highest success in learning.Persons with the high and low level of domain-specific conceptual knowledge indicated a lower subjective learning success.A possible explanation for this finding could be that students with a high domain-specific conceptual prior knowledge did not perceive any increase in knowledge whereas subjects with low domain-specific conceptual prior knowledge did not have enough factual knowledge to perceive an advantage form using the learning environment.
Learner profiles obviously had no influence on time on task.This is in accord with results demonstrating the effectiveness of erroneous worked examples where time on task did not have any effect on the learning outcome (Kopp et al., 2008, Stark et al., 2011).Besides this subject matter interpretation, the result could also be a consequence of a method artefact similar to the one described above on conditional knowledge.
The study demonstrated the effectiveness of learning with erroneous worked examples for the acquisition of diagnostic competence expect for conditional knowledge.Especially the result that domain-specific conceptual knowledge was fostered regardless of the learner's domain-specific conceptual prior knowledge is relevant for a practical implementation.This finding indicates that learners can profit from using the learning environment regardless of the level of their prior knowledge.For not-knowing/fearless learners, with a low level of conceptual prior knowledge, working with erroneous worked examples yielded a distinct increase in domain-specific conceptual knowledge.Although there was no effect for strategic knowledge for these learners, the gain in conceptual knowledge will be beneficial for other learning activities in the same domain.For the knowing/unremarkable learners, with a high level in conceptual prior knowledge, strategic knowledge was fostered.Only the half-knowing/anxious learners had a marginal benefit from the learning environment according to these results in terms of an increase in domain-specific conceptual knowledge and strategic knowledge.However, irrespective of the learner profile, no negative effects of the learning environment are to be expected.
In future studies, it should be investigated if the missing effect of conditional knowledge is simply a method artifact or if other causes explain the missing effect.In addition, it has to be investigated if the missing effect of conditional knowledge is due to the measurement method.In order to check the generalizability of this study, other domains than arterial hypertension should be considered, too.
Furthermore, larger sample sizes should be used and the results should be validated with other methods.Besides the cluster analytic approach that was chosen here, the influence of psychological factors on learning can be assessed with other statistical procedures like regression analysis and latent variable models.
are no abnormal findings on physical examination.The blood tests reveal a sodium of 140 mmol/l and a hypokalaemia of 3.2 mmol/l.The rest of the lab results including creatinine and urea are normal, as is the urinalysis.Please name causes for the hypokalaemia in this case, and the underlying pathomechanisms for each reason.
3) Abdominal ultrasound reveals a mass in the area of the right adrenal gland.You suspect primary hyperaldosteronism.Please name the diagnostic tests you would like to order and which results you expect from these tests and why.

Figure
Figure 1.Learner profiles

Table 1 .
Descriptive statistics for strategic and conditional knowledge

Table 2 .
Mean differences for strategic knowledge

Table 3 .
Mean increase in domain-specific conceptual knowledge

Table 4 .
Mean differences for increase in domain-specific conceptual knowledge

Table 6 .
Mean differences for subjective learning success