Assessment and Standards for Graduate Outcomes

,


Introduction
The development and assurance of academic standards are an important part of the reform agenda in Australia, Europe and the United States of America (DEEWR, 2009).In the context of globalisation and the massification of education, students, governments and employers are requiring that educational institutions demonstrate quality processes and outcomes assured against national and global standards.Academic standards are important for shaping effective teaching and learning so that higher education institutions can be accountable to employers and in order for them to be nationally and internationally competitive (AUQA, 2009;James, 2003).In business education, the accreditation of business schools through associations such as the Association to Advance Collegiate Schools of Business (AACSB) and the European Quality Improvement System (EQUIS) is driving a focus on standards of learning outcomes.
There is a particular need for academic standards pertaining to the development and assessment of graduate attributes and skills.The Australian Universities Quality Agency (AUQA) states that one of the main foci of the standards agenda should be the development of standards for graduate attributes and generic skills (AUQA, 2009).Although most universities have statements on graduate attributes few, if any, have formalised academic standards against which to measure students' achievements.This paper discusses the necessity for graduate skills to be assessed and that academic standards for these skills are needed.The paper puts forward a framework within which standards for graduate skills can be developed and provides one example of standards for a graduate skill using this framework.Strategies as to how the standards could be used as a measure within assessment are given.

Graduate skills and assessment
"The continual focus on graduate skills is really part of a bigger, as yet unresolved, debate about the purpose of university education and how to develop educated persons who are both employable and capable of contributing to civil society."(Business Higher Education Round Table, cited in James, Lefoe, & Hadi, 2004, p. 175) The significance of graduate attributes as indicators of employability has been recognised since the early 1990s (Mayer Committee, 1992).Graduate skills are the skills that enable graduates to adapt in an ever-changing world and be effective in their professional lives and as engaged citizens (Bowden et al., 2000).We will use the term "graduate skills" in this paper to refer to the skills, attributes, attitudes, values, dispositions, capabilities and competencies that we want all graduates to have when they leave university.It is necessary to expand the concept of graduate skills beyond solely skills in order to incorporate a range of skills, attributes and capabilities.For example, there is a problem with thinking of sustainability and ethical practice as skills or attributes.While critical thinking and teamwork are readily conceptualised as enabling skills, sustainability and ethical practice are more about a graduate's disposition (see also Petocz & Dixon, in this Issue).
There is a lack of shared understanding about what graduate skills are and how they should be integrated and developed within a curriculum (Barrie, 2004).Freeman et al. (2008) found that there was little agreement about which graduate skills were important, whose responsibility they were to teach and how they should be assessed.It is difficult for academics to assess graduate skills without a common understanding.As the Graduate Attributes Project (GAP) found, assessment of these skills "remains problematic" (GAP, 2009).
Graduate skills need to be developed incrementally throughout a curriculum from first through to third year.Skills taught in first year need to be integrated and applied through second and third year.This suggests that there is a need for "alternative forms of assessment such as portfolios and capstone tasks that extend beyond or across traditional course boundaries" (GAP, 2009, p. 15).Assessment that is authentic -and includes self-regulated learning as well as engaging students actively with others and with their learning -will help to develop graduate skills (Green et al., 2009).
Once the assessments have been designed to develop the graduate skills, there still remains the issue of how one measures those skills.The Association of American Colleges and Universities (AACU) have developed a range of rubrics for graduate skills including critical thinking, teamwork and ethical reasoning.These rubrics provide criteria for each of the skills and how they could be assessed (AACU, 2009).

Academic standards for graduate skills
Higher education in Europe has been undergoing significant changes as a result of the Bologna Agreement.The aim of the agreement was to create a framework of comparable and compatible qualifications within the different higher educational systems that exist within the European Union (Berlin Communiqué, 2003).The process is also concerned with identifying what students have achieved in terms of graduate skills: "If the 'transparency instruments' designed to clarify what students have achieved for their (comparable) degrees are to be effective then there needs to be some agreement of what is being assessed and how, and against what criteria."(Quality Assurance of Student Assessment, 2008, p. 5).
Standards are being used as a means of measuring and reporting on student achievement as well as assuring the quality of higher education for all stakeholders.In Australia, AUQA was established in 2000 as an independent national agency to monitor and report on quality assurance in higher education (DEEWR, 2009).The 2008-2009 AUQA audit of higher education identified student assessment, alignment of learning outcomes and academic standards as areas of risk.AUQA (2009, p. 7) argues that, "unless academic achievement standards become a definite focus, the value of grades as 'warrants' of demonstrated intellectual and professional learning cannot be substantiated." The Australian government has recently replaced AUQA with a new body called the Tertiary Education Quality and Standards Agency (TEQSA).This Agency will seek to ensure that higher education institutions meet national standards for discipline areas and graduate skills.Performance funding from 2012 will be linked explicitly to academic standards (DEEWR, 2009).Sadler (2005, pp. 189-9) defines standards as: standard n.A definite level of excellence or attainment, or a definite degree of any quality viewed as a prescribed object of endeavour or as the recognized measure of what is adequate for some purpose, so established by authority, custom, or consensus.

Defining academic standards
The definition highlights two main characteristics of a standards, namely, that there is a qualifying threshold for a standard, and that the standard should be agreed upon by the relevant community (Sadler, 2005).National standards for higher education in Australia will need to show what level of achievement students are expected to demonstrate after they have completed their university studies (James, McInnis, & Devlin, 2002).The federal Government has indicated that discipline standards will need to be developed and agreed upon by stakeholders for the particular discipline.The Australian Governments' report "Transforming Australia's higher education system" (DEEWR, 2009, p. 32) states that, "Discipline communities will 'own' and take responsibility for implementing academic standards (working with professional bodies and other stakeholders where appropriate)." For TEQSA to be able to meet its objectives, national standards will need to be developed.The Australian Learning and Teaching Council (ALTC) have been overseeing a pilot project developing standards for different discipline areas.These standards integrate and apply the graduate skills within various discipline areas such as Accounting, Creative Arts, Science, History and Engineering (ALTC, 2010).

Standards and assessment
Sadler (2005) identifies four challenges for using standards, namely: conceptualising standards; setting standards; communicating standards to students and academic staff; and becoming proficient in the use of standards.The reform agenda around standards has highlighted the need for higher education institutions to rethink assessment practices and use assessment, "not only as a mechanism for making standards more concrete and explicit, but also as a more sophisticated and strategic tool for helping shape effective teaching and learning processes."(James, 2003, p. 198) The dominant method of determining students' level of attainment through their assessments currently is based on the use of criteria rather than standards (Sadler, 2005).Criterion-based assessment is both useful and valid, but different academics and institutions will have varying ideas as to how to apply this in theory and in practice.Sadler (2005) argues that, "To realise on the aspirations for criteria-based grading, a major shift in orientation is required towards 'standards-referenced' grading.Criteria-based grading begins with a focus on the criteria, leaving the standards to be implied or experienced incidentally.Criteria form an essential element of the evaluation and communication process, but ultimately it is the students' appreciation of quality, set against a background of external standards, that is of significance.A more appropriate approach would be to focus on the standards as the primary reference points against which student submissions are judged."(Sadler, 2005, p. 190)

Developing a standard for a graduate skill
In Australia, university courses (subjects) are graded using a five-point scale: High Distinction (HD), Distinction (D), Credit (C), Pass (P) and Fail (F).In terms of grade points, HD and D are 4, C is 3 and P is 2. Grade Point Averages (GPAs) are used for prizes and for entry into postgraduate programs, and employers use GPAs for some selection processes.
The challenge for educators is to accurately grade discipline-specific graduate skills according to a particular standard and to communicate these standards to students and other stakeholders, such as industry groups.

Developing the standards
For our research we selected 35 high-achieving students, five from each of seven universities around Australia; thus we had a cross-section of different universities from rural and metropolitan Australia.All the students were studying a business discipline such as marketing, accounting, tourism, economics, actuarial studies and finance.The students were given a pre-test to ascertain their conceptions of the four graduate skills we investigated: teamwork, critical thinking, ethical practice and sustainability.(These are described in detail in other articles in this Issue.)The students then participated in a three-day workshop and completed a post-test.
The pre-and post-test answers were randomised so that the project team did not know which the pre and post answers were or which answer corresponded to a particular student.The project team of seven experienced academics then graded the test answers into qualitatively different responses for each of the skills.We were able to do this in a reasonably consistent manner.
These qualitatively different responses became our grades.They link well with the SOLO taxonomy of Biggs and Collis (1982), which is not surprising as the taxonomy was developed from student responses.The levels also correspond well to other findings such as those of Reid et al. (2005).We adapted the work of Billett (2009) to introduce standards for each domain of knowledge: conceptual, procedural and professional.
The draft standards were then reviewed by the project reference group consisting of academics and industry representatives.The standards for sustainability were also reviewed by the sustainability committee at one university.At each iteration the document was clarified.The standards were then used in professional development with academics who used them to develop marking rubrics for assignments.This was difficult but the standards really assisted in the development of clear rubrics.Therefore we are confident that we have found robust standards that can be applied to Australian graduates by their lecturers.
Table 1 shows the standards of achievement for teamwork.These are the levels we would expect of a graduate from a three-year undergraduate degree at an Australian university.The aim is that these would be developed over the degree, with a range of learning tasks which increase in complexity with each year.We would expect that first year students should be able to demonstrate all levels on more structured tasks and that third year students would also demonstrate all levels.Over a degree we have shown that more students show higher levels of achievement.
Standards of achievement for critical thinking, ethical practice and sustainability were developed and tested (see Table 1).A general model is also available at the Graduate Skills website (www.graduateskills.edu.au).

Conclusion
Assessment of graduate skills is important in helping students develop those skills as well as in providing evidence that the skills have been acquired.Assessment should be designed to encourage students to actively participate and to measure all learning outcomes including outcomes for graduate skills.Attention should be paid to the types of assessment that are used so that students are able to integrate and evaluate their skills.This means that assessment should include peer assessment, self-assessment, authentic assessment and student interaction.
Assessment should be positioned strategically in the teaching and learning process.Academic standards for achievement of graduate skills are required to assure the students' learning.The standards we have developed are applicable to assessing across knowledge domains.This is significant as we move towards education that is transformational, that is, about what a student is becoming rather than focussing solely on what they know and can do.The generic skills, graduate attributes and dispositions agenda is gaining pace and is acknowledged by students, academics and industry representatives as a fundamental initiative.
Our work on developing standards that are grounded in theory and tested with students offers a powerful and robust template for the university community.Demonstrates ability to lead, and to develop relationships and procedures for working with others in novel ways.
Demonstrates a deep commitment to the team's goals through substantive contributions both during and outside of team meetings.
Demonstrates the ability to take a strategic view to lead and work with others to enable innovative outcomes in complex situations.
Demonstrates the ability to directly address destructive conflict, supporting the management and resolution of conflict in a professional manner that strengthens team cohesiveness and effectiveness.

Level 3 D
Demonstrates awareness that teamwork involves coordinating individual efforts to maximise outcomes to achieve a shared goal.
Understands that teamwork and effective team building requires communication, leadership, social and interpersonal skills.
Demonstrates an awareness of the group dynamics that are inherent in the team-building process.
Unifying co-operator.Demonstrates ability to articulate a shared goal and to collaborate with others to maximise the outcomes from shared work.
Demonstrates substantial individual contributions to team's goals outside of meetings/sessions.Demonstrates capability to take the initiative in working cooperatively to maximise the outcomes of a shared goal.
Demonstrates the ability to identify and acknowledge conflict, but engages and responds constructively to it.Resolutions are sought and the team is focused on shared goals and tasks.

Level 2 C
Demonstrates awareness that teamwork is a combination of individual effort designed to achieve a given goal.
Conceptualises teamwork as a team-building process that requires cohesion.
Cooperative follower.Some guidance required.Demonstrates ability to work with others to achieve a given goal.

Demonstrates satisfactory contributions.
Demonstrates capability to adapt to a given professional situation and people in order to work towards a given goal.
Demonstrates the ability to identify conflict.However, does not constructively address it and focusing the team on the shared goal and task.Unable to contribute constructively to group processes.
May act as a disunifying or disruptive influence and retard process of reaching a shared goal.
Unable to demonstrate an understanding of the need to act cooperatively.
Cannot recognise when this is appropriate or required in the work context.Standards of achievement for teamwork (from www.graduateskills.edu.au)

Table 1 .
Teamwork standards of achievement