Using Student Reflections to Explore Curriculum Alignment

The concept of curriculum alignment is held as a guiding principle of good curriculum design in higher education. Curriculum alignment can be mapped using a variety of strategies and tools. This paper reports on a project that expands the horizons of curriculum review by applying a novel methodology, word clouds, to investigate the use of student reflections for exploring curriculum alignment. Students, from Australia and Denmark, engaged in written reflections about their learning in a Business Brand Marketing subject. These reflections provide the data that is analysed for its alignment with the subject’s learning outcomes. The word cloud analysis is found to be useful in providing evidence of curriculum alignment and indicators for directing deeper textual analysis.


Introduction
The concept of curriculum alignment (Biggs & Tang, 2007) is held as a guiding, research-based principle (Angelo, 2012) of good curriculum design in higher education.A curriculum which is intentionally designed (Kift & Field, 2009) to be 'aligned' encourages student engagement in a deep, or at least deeper, approach to learning (English, Luckett & Mladenovic, 2004) where all students are supported to use higher-order learning processes (Biggs, 2003).Curriculum alignment refers to an overt alignment between the course content, learning activities, teaching strategies and assessment of a subject which are in place to achieve the intended learning outcomes.
Alignment of curriculum can be explored and evaluated through a range of strategies and tools, many of which have evolved from, or been informed by, reflective practice.These strategies may span multiple reflective perspectives or lenses (after Brookfield, 1995) ranging from the autobiographical, collaborative and student perspective through to the learnings offered by the research literature.Examples of strategies for exploring curriculum alignment include individual reflection on the curriculum before, during and after teaching analogous to reflection for, in, and on action (after Schön, 1983); peer review of the curriculum or peer observation of teaching (Rowe, Solomonides &Handal, 2010 andBell, 2012) and the use of student reflections, evaluations, consultations and partnerships (Bols & Freeman, 2011).
A range of tools have been developed to guide subject curriculum review with a focus on evaluating curriculum alignment (for example, the tasks offered by Biggs and Tang, 2011).Recent technologies for learning and teaching offer new possibilities for exploring curriculum alignment.This paper reports on a pilot project that applies a novel methodology, computer generated word clouds, to expand the horizons of curriculum alignment.It aims to do this by investigating the use of student reflections, on what students have learned, to explore the alignment of the curriculum with the intended learning outcomes.Learning outcomes "describe what a learner is expected to know, understand and be able to do after successful completion of a process of learning" (European Communities, 2009, p.11).The question posed therefore is: can new technologies be used as a methodology for providing evidence of an alignment between the intended learning outcomes of a subject and student reflections on student learning?

Student Reflection for Learning
As part of an extensive study on the use and role of student reflection in and for learning, this study uses data from two countries to investigate if there is an alignment between student reflections and intended learning outcomes for a key business/marketing subject (Brand Management) and if word clouds can be used to explore and evaluate curriculum alignment.
Student reflection is integral to the Brand Management curriculum that provides the case for this project.The students' engagement with reflection is specifically linked to a research assignment students undertake as a group activity.Reflection is a scaffolding strategy to enable students to make sense and meaning of their learning experiences (Rarieya, 2005) in relation to their research assignments.All students are required to document their reflection on learning as a part of their group research assignment, however, the submission of their summary reflections is optional.While the project assignment is assessed, the reflective writing is deliberately not assessed (Stewart & Richardson, 2000).
Reflection literature reminds us of the need for a curriculum to clearly articulate intent, purpose, meaning and expectations (Boud and Knights, 1996;Bringle & Hatcher, 1999) if we are to create a climate and context conducive for reflection.Further, transparent alignment (Harvey, Coulson, Mackaway & Winchester-Seeto, 2010) of reflective practice with the curriculum is achieved as student reflection on the group research activity adheres to the principles for aligning reflection for learning for curriculum design (op cit).Firstly, the intent of reflective practice is clearly articulated through an alignment of the research project assignment with the reflective tasks; secondly through an articulation of expectations, in this subject (Brand Management) the students know that the reflective learning task is both voluntary and non-assessable; and finally the authenticity of the task is ensured as it is a work-integrated learning task.This curriculum can be considered to have a Work Integrated Learning (WIL) framework as the content, learning activities and the assessment (the group research project), "integrate theory with the practice of work within a purposefully designed curriculum" (Patrick, Preach, Pockneer, Webb, Fletcher & Pretto, 2008, p. v).
The intent of the student reflection is focused through the use of a reflection template, a model developed specifically to scaffold the students' reflection on, and for, their learning.Students are introduced to the model and the roles of reflection in their first class by the research team.This model directs their reflection through four semi-structured questions on their learning on marketing, research, teamwork and areas for improvement.The reflective prompts of the model are: The students' written reflections, in response to the above prompts, provide the rich textual data that is analysed for this project.As the fourth reflective prompt of "What could I have done better?" did not directly align with a learning outcome, but potentially with a graduate attribute, the data from this subset was not used for the purposes of this project.A critically reflective approach underpinned the conceptual framework for the methodology developed for this project.The definition of critical reflection that resonates with our learning and teaching philosophy is of ...a deliberate process when the candidate takes time, within the course of their work, to focus on their performance and think carefully about the thinking that led to particular actions, what happened and what they are learning from the experience, in order to inform what they might do in the future (King, 2002, p.2).

Methodology
Two student cohorts, studying in the same year, provide the data for this study.Both undergraduate cohorts of students were taught the same subject (Brand Management) by one of the authors.The larger cohort was based in a large metropolitan university in Australia and the second in a large metropolitan university in Denmark.Following human ethics approval from both institutions involved, all students were invited to participate in this project which was voluntary and did not require students to undertake any additional tasks or activities.The students' reflections would provide one significant lens to the project.Brookfield's (1995) four reflective lenses: the lens of autobiography, of colleagues, of students and of the literature, provided the different data sources to enable triangulation, a method used to improve reliability (Silverman, 2000) and strengthen validity (Yin, 2011).The textual data used for this project therefore consisted of written student reflections, the autobiographical and collaborative reflections of the authors, together with the unobtrusive data of the subject outlines that provided the text on learning outcomes, and the lens of the research literature to inform the data analysis and interpretation.
Anonymous student reflections, focusing upon the four key reflection questions, were submitted with ethics consent forms.While all students were invited to submit their reflections about their learning, this participation was both voluntary and not graded.An overall response rate of 51% was attained.The response rate for the Danish cohort was higher (at 37 of the 50 students, or 74%) than that for the Australian cohort (with 65 of the 150 students, or 43%), resulting in a data set of 102 student respondents.All responses were entered into a spread sheet, which enabled easy uploading into the data analysis software utilised for this study, the NVivo content analysis software.
A large textual data set of the reflective responses by the 102 participating undergraduate students was achieved for this study.Responses from the two cohorts were aggregated, as the purpose of the analysis was an investigation of curriculum alignment, not of international comparisons.Prior to data aggregation, separate analyses were conducted by country cohort.The word frequency results were nearly identical, supporting our decision to aggregate the data.
The non-obtrusive data of the subject guide was used to identify the learning outcomes.These learning outcomes would represent the curriculum and its content.The student learning outcomes for this subject state that by the end of this subject students would be able to: 1) work effectively in teams to analyse and prepare presentations on brand management issues, 2) increase understanding of the important issues in planning branding strategies and in evaluating brand strategies, and 3) apply appropriate theories, models, and other tools used to make better branding decisions.
The methodology devised was aiming to expand the horizons of curriculum review.The steps developed for the process of using word clouds of students' reflections to explore curriculum alignment were: 1) conduct a word frequency analysis of the student reflection text , 2) present results as a word cloud, and 3) review the most frequent words for their alignment with the learning outcomes.

Word Frequency Analysis in Research
With the textual data sets prepared, a decision was needed regarding the method of analysis to be employed.The approach had to be compatible with the qualitative data and based on a method that had a proven validity and legitimacy.Content analysis, specifically word frequency analysis, has an established history (Krippendorff & Bock, 2009).
Computer-assisted analysis of qualitative data (CAQDAS) provides the advantages of speed and rigour (Seale, 2000).The word frequency query function of NVivo, a content analysis software, was used to generate reports on the word frequencies (Krippendorff and Bock, 2009) of each of the three sets of student textual data, representing student responses to the three reflective prompts (on learning about marketing/branding; on research and on teamwork).NVivo also generates a visual representation of the results as a word or tag cloud (QSR International, 2010).In this visual representation of the data, the higher the word frequency, the larger and denser the font in the graphic that is produced.
The "with minimum length setting" was set at four letters to avoid frequent words such as I, we and do (QSR International, 2010).Display was set to report the 100 most frequent words.The "finding matches" command was set to "including synonyms" resulting in for example, the word "branding" representing the synonyms of: brand, branding and brands.The novel approach conceptualised for this project was that of using word clouds for curriculum review, exploring the patterns of word frequencies of student reflections to determine if there was any alignment between the reflections and the learning outcomes for the subject under investigation.
A search of the literature identified limited empirical evidence for the role or contribution of word frequency analysis, word tags / word cloud technology to research (Hearst & Rosner, 2008).The majority of studies focused on the most useful placement of words or tags (Hearst & Rosner, 2008;Lohmann, Ziegler & Tetzlaff, 2009) however conjecture abounds on the applications of this relatively new technology.
The technology offers new ways of working with both quantitative and qualitative data.Cloud technology, by providing the means to analyse massive amounts of data (petabytes), makes the scientific method of hypothesise, model and test obsolete (Anderson, 2008).Petabytes allow us to say: "Correlation is enough."We can stop looking for models.We can analyse the data without hypotheses about what it might show.We can throw the numbers into the biggest computing clusters the world has ever seen and let statistical algorithms find patterns where science cannot" (p.109).
Although empirical evidence is not yet offered, the literature suggests that the two major types of visualisations, word frequency tag clouds and thematically organised word trees, may be useful in qualitative analysis for identifying core variables and theorising.They may also be used for validating research findings through exploration of observations and assumptions in researchers' reflective journals and checking for bias (Weisgerber & Butler, 2009).These collaborative uses do however present privacy and confidentiality issues preventing transcripts from being uploaded directly.Providing pseudonyms and obscure titles for data sets are methods suggested by Weisgerber & Butler (2009) to overcome these issues.Other applications for the visualisations suggested by the authors are as a pedagogical tool to encourage academic discussion and as a research presentation tool to condense complex arguments and data into a manageable and engaging format.These applications were successfully realised through an investigation into graduate attributes and how they have changed over a fourteen year period (Bosanquet, Winchester-Seeto & Rowe, 2010).
Whilst there is little published research on word tags or word clouds, the opportunities for collaborative group learning are many, suggesting possibilities for researchers to collaborate in data analysis, "encouraging diving deeply into the data set, finding patterns, challenging each other, making guesses about causal relationship, noting outlying data points, building on the findings of others and having fun at the same time" (Wattenberg & Kris, 2006, p. 551).This project utilises this approach for its data analysis.

Results and Discussion
The data produced by the student reflections on what they have learned about marketing and branding, research and team work provide the focus for this paper.Results for each of these three reflective prompts is now presented and discussed.

Reflecting on Marketing and Branding
Two learning outcomes were subject specific, focusing upon branding strategies, theories and models.These learning outcomes anticipated that students would: "increase (their) understanding of the important issues in planning branding strategies and in evaluating brand strategies" and "apply appropriate theories, models, and other tools used to make better branding decisions".Student reflections on "what have I learned about marketing/branding?" were used to explore curriculum alignment.
A look at the word cloud produced using the student reflections on what they had learned about marketing presents a clear image of the importance of branding and marketing (Figure 1), with the frequency of these words being 99 and 75 respectively (Table 1).This image indicates an alignment between what the students have learned and the intended learning outcomes for this subject, suggesting the achievement of an aligned curriculum.
Figure 1.Student reflections on marketing as a word cloud More 20 Management 20 Much 20 Insights about the student learning of marketing and branding can be gleamed from the written reflections that provided these word frequencies.Verbatim quotes from the student reflections are provided to illustrate the context of the frequent words.Where quotes are provided, the student's country where they studied is provided.Student comments range from the appreciation of a general knowledge base, for example: To reflection upon their depth, breadth and currency of learning, with comments such as: Learnt about current issues in marketing that have not been covered in other subjects -e.g.green marketing, branding of universities.Australia I have learnt that marketing is a constantly evolving strategy.It never remains still or it will become irrelevant compared to competitors.Australia I have learned to see the differences related to marketing and branding --> that branding is an important tool in gaining satisfied consumer and creating customer loyalty.Denmark And reflection on specific skills gained: With this being my 3rd and final year of studying marketing/branding I have developed a range of skills and a knowledge base of marketing concepts to fall back on.In this unit I have found these established concepts and skills have been taken to new level.By focusing on the one or two single areas, in my case within market research.I have gained a greater range of points of view, as well as how to deal with these concepts in a more practical context.Australia Alignment with the learning outcome that students would be able to "apply appropriate theories, models, and other tools used to make better branding decisions" can be supported with the following examples of student reflections: I was familiar with Aaker's theory before the start of the course, but I did learn to see branding from the perspective of Keller.Furthermore, I have learned more about the ways to maintain brand equity.

Denmark [Aaker and Keller are the two leading textbook authors in the field of branding]
Mainly involves following: -Marketing strategy for emerging market.-MNCs trend to focus on emerging market and shift their business and investment.-Emerging market such as China/India etc. assist MNCs to increase their market share and growth.Australia

Reflecting on Research
Students undertook a group research project and reflected on their learning from that research experience.This aligns with the first learning outcomes of students being able to "work effectively in teams to analyse and prepare presentations on brand management issues".A viewing of the word cloud derived from students reflections on the reflective prompt for research (Figure 2) highlights the words research and learned, followed by key research related terms such as information (cited 27 times) and articles (with a frequency of 20, refer to Table2).And on the value of these skills to their discipline of business/marketing: Research is essential to understand expectations of consumers, the dynamic nature of market demands.Australia

Reflecting on Teamwork
The third student reflective prompt supported students to consider "what have I learned about teamwork during this assignment?"Student reflections as presented in the word cloud (Figure 3) provide a strong indication of an alignment between their reflections and the subject's learning outcome on working effectively in teams.The words: work, team and members are followed by "good" (Table 3).Students reflected on many benefits of working in a team including: I have learnt to negotiate effectively and to delegate tasks to appropriate students.For e.g., giving a highly research based task to students who were not comfortable with public speaking so they had a small speaking role.Australia You have to know your team mates, otherwise it can be difficult to work together.It takes time to make a team work and it helps if everybody is at the same level.The good thing about is that you get others view on your work and you get to discuss things.Denmark Team is working together, discuss together.From the team work we know each other's and we have become good friends.Australia Trust and understand what the other members of the group said.The points of view of each member is important and must be respected.Australia It would be too simplistic to consider that the frequency of the word "good" with team work could be interpreted that team work in general was good.The students' writing reveals a more complex reasoning around the word "good", illustrated by: I am a good leader.Been able to outsource the work and remain in control.Denmark That it is a very good way to work with brand management.Denmark We had a very good co-operation in our group.Time has been short to make the assignment but I think we have handled the stress quite well.Denmark

Conclusion
The use of word clouds, generated by student reflections, was explored as a means of expanding the horizons of curriculum review.Three sets of student reflective data were used, reflections about learning of their subject, on research and on teamwork.For each data set the word cloud indicated an alignment of student learning with the planned or intended curriculum.The word clouds, their accompanying word frequency tables and the student reflective text suggest an entrapment in a "web of consistency" (Ramsden, 2011), wherein the intended learning outcomes, subject content and student assessment, in this case a group research project, all are aligned to ensure optimal learning.
This study makes three distinct contributions.Firstly, it demonstrated that the methodology of analysing word clouds can be used as a research method to explore curriculum.The word clouds generated from the student reflections provided deep insights into the student experience of the course.Secondly, the student reflective prompts, developed for students to reflect on their learning on marketing, research and teamwork have been demonstrated to successfully elucidate student engagement with reflective practice.The student written reflections were related to the three reflective prompts.Finally, using data derived from the model of reflective prompts in tandem with a word cloud analysis allowed an exploration of curriculum alignment, which in the case of the subject under investigation was found to be high (i.e.strong alignment between the units' outcomes and student reports of learning).Given that we tested our student reflection prompt model and the word cloud approach in two distinct countries (Australia and Denmark) and found no apparent differences between the two countries in our analysis, we can conclude that the reflection prompt model and the word cloud approach work universally, at least in Western university settings.
The word cloud method has the advantage of very quickly providing a visual representation of word frequency, and a speedy and sharp account of student learning.Using this technique, one can quickly make an estimation of the level of curriculum alignment, providing important curriculum review feedback to the subject convenor.This method offers potential as a tool for expanding the horizons of curriculum review and as a pedagogical tool to encourage academic discussion (Weisgerber & Butler, 2009).The word cloud technique can now benefit from emergent empirical studies that further explore the value and potential of the methodology and of the role of reflection for learning.
What have I learned about marketing/branding?What have I learned about conducting research?What have I learned about teamwork during this assignment?and What could I have done better?
have learned how marketing and branding really can be used in the real life.Furthermore I have learned how important marketing and branding is.Denmark I've learned a lot about branding and the important role it plays for companies.Branding is what makes customers favour one company from another.Branding makes the difference and after this course I've become familiar with techniques companies use and I'm much more aware of the advertisement and techniques used in them.Denmark

Figure 2 .
Figure 2. Student reflections on research as a word cloud Table 2.The top 10 words from student reflections on research Word Count research 77 have 61 learned 47 information 27 find 24 need 31 articles 20 important 21 about 20 topic 20 Students reflected on the specific research skills they have developed: I have learned about Google Scholar and to find hits [articles] that are heavily cited.Also I have learned to find more scientific journals than I could before.Denmark Learnt how to write a literary review, i.e how to research background information about an author.Australia How to effectively incorporate qualitative and quantitative research into projects.Critically analyse articles.Australia How to find good quality articles and how to use databases right.Furthermore I learned a lot about citation.Denmark I have learnt that research needs to be started early and be broader at the beginning -more focused

Figure 3 .
Figure 3. Student reflections on teamwork as a word cloudTable 3. The top 10 words from student reflections on teamwork Word Count Work 89 Have 62 group 41 Team 37 good 32 members 23 Well 27 important 20 Time 18 people 17

Table 1 .
The top 10 words from student reflections on marketing