Direct Measurement and Evaluation for Mechanical Engineering Programme Outcomes : Impact on Continuous Improvement

Universiti Kebangsaan Malaysia (UKM) is a research university that continuously undergoes an audit and accreditation process for the management of its courses. The Faculty of Engineering and the Built Environment (FKAB) is subjected to such processes, one of them is the auditing conducted by the Engineering Accreditation Council (EAC), which gives recognition to engineering programmes in Malaysia. The criteria that have been set by the EAC requires the faculty to measure and evaluate graduates based directly on the course outcomes (CO) and programme outcomes (POs) to ensure that they are able to achieve the programme and course objectives. This paper discusses a method developed for the measurement and evaluation of POs which is a direct measurement for assessing the courses offered in the mechanical engineering programme. Assessment templates are developed using a Microsoft Excel based on the data presented on the grade coordination meetings conducted at the end of each semester. The data used for this study were semester 2 2010/2011 session, and semester 1 session 2011/2012. Based on the results obtained, the majority of the courses achieves more than 80% for course level monitoring for their individual PO achievement except for five subjects. For programme level monitoring, from the average POs marks, all POs were scored more than 80%, which indicates that the Mechanical Programs offered by the department are satisfied with the department standard.


Introduction
All programmes in institutes of higher learning and universities are required to implement an outcome-based education curriculum set by the Ministry of Higher Education to ensure quality graduates are produced (Shahrir et al., 2008).The Faculty of Engineering and the Built Environment (FKAB) has initiated the concept of Outcome-Based Education (OBE) in 2005, particularly because the FKAB has to ensure that programmes are recognized by authorized accreditation bodies.The Engineering Accreditation Council (EAC) is one of Malaysia's accreditation bodies and is responsible for ensuring quality engineering programmes at local universities.The EAC has set certain standards to be achieved, emphasizing the measurement of student achievement through the OBE and in a manner that is consistent with the requirements of the ministry (Siti Aminah et al., 2011).
Programme Outcome can be measured by a variety of methods either on questionnaires, grading courses, observation of student achievement, portfolio analysis and interviews.All of these methods can be divided into two approaches, namely the direct or indirect measurement (Breslow, 2007).Allen (2008) explains that the direct measurement methods to assess the students' work in the form of creative assignments and systematically can be assessed based on rubric.While the indirect measurement methods can assess students' perception of learning and teaching environment.The result of this method is often combined with the direct method results.According Wojtczak (2002), Programme Outcome measurement is useful to assess student achievement to meet the needs of external examiners and also can provide the information for the purpose of improving the curriculum, as well as to document the success or otherwise for a programme.
Based on the OBE approach, the measurement of student achievement upon completion of the course is a requirement to determine the ability of the students based on their cumulative grade point average measurement (CGPA) for each semester.Many of researcher assess the Programme Outcome and Course Outcome based on their student achievement (Ishak et al.,2011, Nurina Anuar et al,.2011, Juridah Johari et al, 2010, Hamimi Fadziati et al,.2010, Mohd Faizal et al, 2012) .OBE also specifies that the assessment results for each course [course outcomes (COs)] will have to depend on what has been stated in the programme handbook.In addition, CO measurement is directly dependent on the achievement of students in each assessment determined in each course as scores from quizzes, tutorials, laboratories, and examinations.Thus, to ensure that the end result is achieved with quality assured, teaching and learning plans should be designed and provided by the course instructor at the beginning of each semester (Rozeha et al., 2007).
Each course offered throughout the mechanical engineering programme has its own COs to be achieved by the students upon course completion.Every course is designed to relate the results and contribute to the achievement of programme outcomes (POs).A detailed syllabus for each course as well as the respective results of the course and its relationship with the PO can be seen in FKAB undergraduate handbook for each academic session.
Figure 1 shows the process flow of the presentation and evaluation of student performance and achievement of COs as practiced in the department.
Figure 1.Process of delivery and assessment of student performance and achievement as practiced in the department Figure 1 shows that every course has specific delivery and evaluation methods for student achievement.Each element of assessment must be performed, and the course instructor must ensure the measurement results of the course evaluation achieve a predetermined threshold at the departmental level (EAC Document JKMB 2012).In the course assessment, each CO contribution to that particular course is mapped to the relevant PO.This enables the evaluation and its contribution toward achieving the PO for the four years of study.Table 1 shows the list of POs for the mechanical engineering programme beginning in semester 2010/2011 (FKAB Undergraduate Handbook 2011).
For each programme, such as the mechanical engineering programme for the second semester of 2010/2011, approximately 20 courses are offered including optional elective subjects.The general role of the programme coordinator is to monitor whether every course offered achieves the COs and PO according to prescribed standards.This paper discusses a template developed to help the coordinator monitor the performance of each subject offered within a semester.The huge number of courses makes it difficult for the programme coordinator to conduct an overall monitoring.However, this template aids in the process of monitoring the expected PO performance of each subject and prevents any course from being overlooked.

Methodology
The measurement procedure was developed using Microsoft Excel format based on the standards established by the department.The template contains the course code column representing the entire departmental programme as well as the percentage fields pertaining to the achievement of outcomes for each course, as shown in Table 2.
Coordinator for each course has to determine his/her CO relationship to POs as stated in the Undergraduate Handbook.They have to define the percentage workload or assessment for every COs towards the particular POs marks.The relationship between POs and COs are static, but the percentage contribution from each COs are varies depending on the instructional planning.Figure 2 shows an example of the weightage for each CO to the PO for subject KM 1914.The correlation between each PO and CO for each course is outlined in the FKAB undergraduate handbook for each semester.For instance, PO3 consist of 50% assessment marks from CO4 and C06 respectively.PO scores were obtained from the scores that have been determined during the grade coordination meetings using scoring templates developed by previous researchers (Shahrir, 2007).Every student's assessment marks such as from examination, tutorial or quiz will contribute to their COs marks depending on the weight distribution between the assessments.Then, the individual POs marks will generate based on the individual COs marks and the weight dissemination between each CO for the particular PO.For programme monitoring at the individual level, the department has set that every student should get more than 50 marks to be considered archiving that particular PO.Next, from the individual PO marks achieved by each student, the percentage scoring for the PO measurement of each course measure was calculated based on the total percentage of students who achieve at least 50 marks over the total number of students taking the course.
Figure 3 shows the final marks achieved by the course at the end of the programme.
The scores for each course will then be transferred to a monitoring template to determine the PO achievement for the whole programme.For programme monitoring at the course level,the department has designated that a PO is achieved if the percentage of students who scored at least 50 marks in a subject is 80%.Any subject that fails to achieve 80% will be marked in yellow, as shown in Table 2, to alert the course lecturer and programme coordinator about the PO-related achievement of the course so that further actions could be taken.
However, to measure the PO for programme monitoring at the programme level, the average of these data, excluding the POs for elective subjects, is calculated.In addition, to obtain a more accurate record, the average PO for selected subjects is also calculated.

Results and Discussion
The department offers two programmes: mechanical engineering and manufacturing engineering.To accomplish the self-EAC 2012 report, the department requires an approach for monitoring every course for each semester and for both programmes for 2010/2011 and for semester 1 2011/2012.However, in this paper, only the mechanical engineering programme is discussed with a focus on departmental compulsory courses and faculty, excluding compulsory university courses.PO assessments are correlated to the CO evaluation based on measurements that have been specified in the delivery and evaluation of the course at the beginning of each semester.Lecturers for courses provide a certain percentage of each element of evaluation (e.g., assignments, examinations, etc.) that contribute to the COs.Percentage weighting may be found in the teaching files for each semester.The percentage of students who obtained a minimum score of 50 marks is derived from the grading template and then transferred to the monitoring template.
This type of monitoring can provide an overview of each subject offered to ensure the quality of PO delivery.Tables 2 and 3 show the PO performance of individual courses offered by the mechanical engineering programme over two consecutive semesters.The listed courses were offered in the specified semesters.For almost all courses, 80% of students met the required standards by scoring a minimum of 50 marks for individual PO.However, five subjects, KKKM1324, KKKM2324, KKKJ3124, KKKM1513, and KKKJ4553, failed to meet the departmental standards.The small marks on POs, was related to the average score grade achieved by the student.If the average marks were low, it also indicates a low POs marks.All the marks were depended with the course assessment and low average may because of a lot of factors such as the type of assessment of the course, level of question being asked, the attitude of the students, the successfully of course delivery and much more.The PO achievement results will be discussed during the meeting of the quality unit (UP3) each semester (six-month cycle) after grading is completed for the course.The department, through the UP3 committee, identifies the causes of low PO achievement and informs the respective lecturers so that corrective action could be taken.Every course subject need to report their POs and COs course achievement in the course file for the department and faculty record.They also need to discuss the reason behind their low POs and COs marks (if applicable) and the action taken to counter back the deficit or the improvement can be made in the future.Changes and improvements in the delivery and assessment methods for the course are undertaken as part of a continuous quality improvement process for the programme in JKMB.
For programme level monitoring, a measurement of each PO for every course can be conducted.However, selecting certain core courses that represent the measurement of a certain PO is more appropriate.The choice of courses is determined by the department based on the courses' importance in terms of contribution to a particular PO.For example, to represent PO1, the courses considered are the KM1344, KM2114, KJ3124, KJ3933, and KJ4953, whereas for PO9, KJ4703 and KJ4953 are considered.For each PO, at least two courses are selected to provide a cumulative measurement.
Figure 4 shows the average percentage of student achievement for each PO based on a particular subject or all subjects, excluding elective courses.PO is measured based on the CO's the relationship with and contribution to the PO (CO attainment directly measured for each course).From figure 4, it shows that all POs exceeded the target of 80%.PO1 recorded the lowest achievement at 86.9% for selected subjects and 90.1% for all all subjects.

Conclusion
All POs have generally scored over 80% which are satisfied for programme level monitoring.The results are based on the theoretical aspect (class lectures) and the practical aspect (exposure through participation in competitions and association.However, for course level monitoring, a number of courses can further enhance their PO achievement.This comprehensive monitoring system ensures that each course's PO achievement is determined by the department.The department can identify the elements of underachievement and then inform concerned lecturers to take action for the purpose of continuous quality improvement, as intended for all EAC report documentations.

Figure 2 .
Figure 2. Sample entry weights that determine the relationship between COs and POs

Figure 4 .
Figure 4. Percentage of student achievement for each PO (Achievement for 50 marks and above)

Table 1 .
List of POs of the mechanical engineering and manufacturing engineering programmes

Table 2 .
Percentage of students who scored a minimum of 50 marks for specific courses for the mechanical engineering programme for semester 2 2010/2011

Table 3 .
Percentage of students who scored a minimum of 50 marks for specific courses for the mechanical engineering programme for semester 1 2011/2012