A Comparative Analysis between Direct and Indirect Measurement of Year I Integrated Project

The Integrated Project (IP) has been practised in the Department of Chemical and Process Engineering (JKKP) since the 2006/2007 session. Initially, the IP is only implemented for the Year II students for both Chemical (KK) and Biochemical Engineering (KB) programmes. Previously, the Year 1 curriculum was only based on the common faculty courses. Starting in the 2010/2011 session, a new curriculum was introduced for Year I with some reshuffling of the courses offered, in which the programme syllabuses are tailored to the offered departmental offerings starting in Year I. Based on the new curriculum, the students registered under the KK and KB programmes have undergone the IP since Semester I of their first-year studies. This implementation project encourages and gives the opportunity to students to take part in the open-ended problem-solving project. The students are trained to obtain information independently through lifelong learning, communication skills in terms of written and oral communications and identify the current issues and involve team work. The training can reduce the phenomenon of student dependence on lecturers or “spoon-feeding” to obtain information in order to understand a theory of learning. The exposure could also transform the learning scenario from lecture-centred to student-centred learning, which requires students to participate more actively in the learning process. The student achievement of the programme outcomes (POs), which has been set in terms of lecturer assessment (direct measurement) and student perception through questionnaires (indirect measurement), was compared at the end of the implementation of the IP in Semester II of the 2011/2012 session. From the comparison, it was found that both measurement methods gave very similar results for all the outlined POs with insignificant differences detected between the POs. This proves that the students gain benefits through the implementation of the IP which are in line with the lecturers’ expectations.


Introduction
The Integrated Project (IP) has attracted many attentions in recent years.The project has been implemented for several engineering undergraduate students at Northern Arizona University (Larson et al. 1996), University of Akron (Qammar et al. 2005) and Elizabethtown College (McBride et al. 2009).Chemical engineering associated with the operation, design, construction and management of commercial products through industrial processes (Abbass and Romagnoli 2007).Meanwhile, the profession of chemical engineer is broad and may be found across a large sector of the professional community.Thus, the appropriate learning and teaching methods as in IP are required so that the learning outcomes can be achieved to produce good chemical engineers.Through the integrated project, it could help to develop students' skills i.e., to work both independently and in teams, be able to communicate, could take responsibility for their own learning and, be able to solve tasks given independently (Yague et al. 2008).
The Department of Chemical and Process Engineering (JKKP) implemented an Integrated Project (IP) at department level in the 2006/2007 session for Year II students (Abdullah et al. 2008;Takriff et al. 2007).The implementation was based on the Outcome Based Education (OBE) which was initially introduced in the Faculty of Engineering and Built Environment (FKAB), Universiti Kebangsaan Malaysia (UKM), in Semester I of the 2005/2006 session (Mohd Nor et al. 2006).The IP project is similar to the plant design projects that were implemented for the Chemical and Biochemical Engineering programmes for final-year studies.The IP is a group work assignment that is implemented in every semester for Years II and III, which integrates all departmental courses in each semester.The purpose of introducing this project is so that the students can integrate the knowledge and theories they obtained within each semester and translate this into an assignment, and it can also reduce the students' burden as they only have to complete one project every semester.The method of IP implementation and improvement has been described in detail in Abdullah et al. (2008;2011a;2011b).
Starting from the 2010/2011 session, a new curriculum for both programmes offered at JKKP was introduced starting in Year I, by reshuffling the offered faculty courses according to the requirements of the departmental programme level.Based on the previous curriculum, Year 1 studies only involve general faculty courses offered to all programmes registered under FKAB.After a curriculum review, the faculty courses were created as department courses in order to meet the requirements of the Chemical and Biochemical programmes.Thus, according to the new curriculum, the students registered under both programmes have to undergo the IP from Semester I of Year I, compared to the previous curriculum which had implemented the IP in Semester I of Year II.The early exposure gives an opportunity to students to get involved in an open-ended project at an earlier stage.The students are trained to have the ability to obtain information through lifelong learning skills, to have good communication skills in terms of written and oral reports, to have the ability to identify current issues and have the opportunity to work in a team.These exercises are expected to eliminate the phenomenon of "spoon-feeding" where students depend entirely on the lecturers to obtain information in order to understand a theory (Abdullah et al. 2011c).Moreover, the exposure is expected to change the scenario of lecture-centred learning to student-centred learning, which requires students to participate actively in the learning process.
The Year I IP combines two courses in Semester I and three department courses in Semester II (Table 1).To implement the project, the coordinator of the Year I IP will divide the students into a few groups, with each group consisting of three to four members.Each group will be given a project related to the manufacturing of chemical and biochemical products.In the 2011/2012 session, there were six groups for the KK and five groups for the KB programmes.Table 2 lists the topics delivered for students in the 2011/2012 session.Even though the topic is the same for Semester I and II, the details of the IP tasks in Semester I and II are different.This is because in Semester I, the IP project combines different courses compared with the courses offered in Semester II (Table I).The topics and groups are maintained in order to allow the students to work in the same group for longer periods, to give an opportunity to each group to understand the IP topics and also to give them an opportunity to revise and make improvements on the report that was completed in Semester I.

Production of formaldehyde -
The IP is created as a platform to measure the students' achievement towards programme outcomes (POs) as outlined by the department of KK and KB programmes.Based on the new curriculum, there are nine POs (Table 3) which have been agreed at the department level.Besides the IP, all of the Pos are also measured through Research Project, Design Project and Open-Ended Lab.Seven POs are measured through the implementation of the IP, i.e., PO1 (application of basic information), PO2 (solution of engineering problem), PO3 (identification of current issues), PO6 (application of modern engineering equipment), PO7 (communication), PO8 (teamwork) and PO9 (lifelong learning).
From the 2010/2011 session until 2011/2012, two batches of Year I students that underwent the new curriculum completed the IP assignment in their first year.As practised previously, the IP evaluation is performed through direct and indirect measurement (Figure I).The direct measurement consists of an oral presentation, reports from lecturers and peer assessment, while indirect measurement is conducted through a survey that is distributed to the students at the end of the semester (Abdullah et al. 2011a).The lecturer will evaluate the students' achievement for PO1, PO2, PO3, PO6, PO7 and PO9.However, the students will evaluate the teamwork (PO8) through peer assessment.The survey distributed to the students will cover the response of students towards all of the PO achievements (PO1, PO2, PO3, PO6, PO7 and PO9).In Semester II of the 2011/2012 session, a survey was distributed to analyse the perception of Year I students of IP implementation.The students' PO achievement from the students' perceptions was compared with the direct measurement carried out by the lecturers to evaluate the effectiveness of IP implementation for Year I.

PO9
Ability to recognize and acquire the need to undertake life-long learning.

Methodology
To evaluate the effectiveness of the implementation of the Year I IP in Semester II of the 2011/2012 session, the students' achievements (PO1, PO2, PO3, PO6, PO7, PO8 and PO9) through the IP assignments were analysed in detail by comparing the scores obtained from direct and indirect measurement as shown in Figure 1.The indirect measurement is students' perception through survey, while the direct measurement is a lecturer's assessments through oral presentation, reports and team members' evaluation.Each group of Year I students is distributed with the respective IP topics (Table 2) on the third week of lectures.In week eight, the student groups have to submit an initial report which contains a literature review about the product that will be produced by the designed plant.The students should explain how and why the product needs to be produced by including the issues of usage and product market ability (demand and supply), the chemical/biochemical reaction involved and current issues related to the safety of the plant and environment.In week ten, lecturers will give comments to the students based on the preliminary reports.From week 10 to 12, the students have to improve their reports based on the feedback obtained from all the involved lecturers and continue the IP assignments with a detailed calculation of the mass and energy balance of the plant.The students also have to justify the process system chosen for the production.This part is a requirement of the KKKR1234 Physical Chemistry for Engineers course for KK students or KKKB123 Microbiology for Engineers course for KB students.For the KKKR1244 Chemical Engineering Thermodynamics II course, the students need to calculate the phase balance in the product separation process.Based on the understanding of the theoretical knowledge obtained from the KKKR1223 Basic Chemical and Biochemical Engineering course, the students were asked to draw an engineering drawing for a process flow chart of a plant using standard and established symbols.The full IP reports will be submitted in week 12 and the students have to present their IP tasks in week 13.For the direct measurement, the courses' lecturers for each programme involved had to assess the reports and oral presentation.The evaluation forms for the lecturers were developed based on the PO achievements by the students (PO1, PO2, PO3, PO6, PO7 and PO9).At the same time, the students were given with a Peer Assessment Form to evaluate the level of teamwork (PO8) in each group.
After the oral presentation (week 13), the coordinator of the Year 1 IP will arrange a review session in order to give feedback about the groups' performance during the IP implementation.The session will also allow students to give feedback toward the IP implementation.In this session, the students were provided with a survey to express their perceptions about the implementation of the IP.The questions asked in the survey are related to PO achievement, in which the students are required to give a level of agreement to all the given statements based on the Likert scale (Table 4).Examples of the lecturer evaluation form, peer assessment form and survey can be referred to in Abdullah et al. (2010).The scores given by the students in the survey (indirect measurement) were converted into percentages.These results were compared with the direct measurement of lecturer assessment through reports, oral presentation and team member evaluation in order to determine the weaknesses and strengths of the PO achievements among the students.Strongly agree

Statistical Evaluation
The results were analysed using an independent samples t-test with a confidence level of 95% or p < 0.05.Statistical calculations were executed with SPSS software for Windows, version 16.0 (SPSS Inc.USA).An independent samples t-test evaluates whether there is a significant difference in PO scores between indirect and direct measurement.

Student Demography
In the 2011/2012 session, about 40 students registered for the IP course, with 23 students from the KK and 17 students from the KB programmes as depicted in Figure 2a.From the demographic, the female students are 13% and 10% for the KK and KB programmes, respectively.The students came from various races such as Malay (62.5%),Chinese (22.5%),Indian (5%) and others (10%) (Figure 2b).

Comparison between Direct and Indirect Measurement
Figures 3 and 4 show the comparison between direct and indirect measurement of PO achievement through Year 1 IP by KK and KB students, respectively.Assessment of the KK programme (Figure 3) resulted in no significant difference between direct and indirect measurement for PO1, PO2, PO3, PO6, PO7 and PO9.However, a significant difference was apparently observed for PO8 and was proven through statistical analysis (Table 5) which gave a p-value below 0.05 (F-ratio of 33.8).The percentage achieved in all POs (PO1, PO2, PO3, PO6, PO7 and PO9) was in the range of 70-85 % assessed by either lecturers or students through the survey.A similar trend was also obtained by KB students (Figure 4) whose marks ranged from 70 to 85 %.Meanwhile, the marks for PO8 achieved by KB students which were evaluated thorough indirect and direct measurement were significantly different (p < 0.05, F-ratio = 25.1) with percentages of 65 and 93 %, respectively.The marks indicate that the PO assessment of the IP was acceptable.Reported by Yague et al. (2008), the students have developed general and transferable skills such as searching for information, working in teams, alternative analysis, decision making, written and oral presentations when a new teaching method was applied for the Chemical Engineering degree in the University of Valladolid.
Basically, direct and indirect assessments for PO8 (teamwork) were both evaluated by students, where the PO8 was one of the most importance skills to be achieved by students (Savage et al. 2007).The direct measurement was done after the oral presentation of the IP, and every student had to evaluate their group members, which contributes 20% of the overall IP marks.About 65% of the mark was obtained from the survey but almost 95% was obtained from the colleague evaluation.This happens because the students are usually generous during the evaluation and most of them gave full marks to their group members.This was different to the survey evaluation which does not have any relation to the marking, thus the feedbacks given by the students through the survey were more transparent.Therefore, this explains why the scores of students for PO8 through the survey were lower than the marks given to the group members.A similar trend was also seen in another comparative study of direct and indirect measurement of students' PO achievements for the Year III IP (Abdullah et al. 2010).
Overall, even though there were score differences between direct and indirect measurement for the KK and KB programmes, from Figures 3 and 4 it can be seen that the differences were not significant, indicating that the fixed POs were achieved successfully with average scores between 70 and 80 % for both programmes.Parthasarathy and Jollands (2009) found that by implementing integrated project methods, a high satisfaction level of 82% was achieved in 2008, showing that the progressive development of technical and generic skills is successful achieved.

Conclusions
A comparison of PO achievement through the Year I Integrated Project between indirect and direct measurement found that both assessment methods showed a similar trend of results and did not present a significant difference between the measured POs of PO1, PO2, PO3, PO6, PO7 and PO9.This illustrates a positive outcome of the implementation of the Year I IP.However there is a significant difference between direct and indirect measurement for PO8 due to generous attitude of students during the evaluation and most of them gave full marks to their group members.From the survey, the students felt that the IP had helped them a lot to understand the theory and basic information from the courses, which eventually could be applied to complete the IP task.Moreover, the assessment from lecturers of the PO achievement of students was also in line with the students' perception.Thus, the results support and strengthen the JKKP decision to implement the IP for Year 1 students.

Figure 1 .
Figure 1.Flow chart of programme outcomes assessment through direct and indirect measurements

Figure 2 .
Figure 2. Demographic of students for KK and KB programmes a) based on gender b) based on race

Figure 3 .
Figure 3. Achievement of programme outcomes through (PO) direct and indirect measurements for Chemical Engineering programme

Figure 4 .
Figure 4. Achievement of programme outcomes (PO) through direct and indirect measurements for Biochemical Engineering programme

Table 2 .
List of topics for Year 1 IP for 2011/2012 session

Table 3 .
Programme outcomes of IP for Chemical and Biochemical Engineering programmes in Department of PO8Ability to function effectively as an individual and in a group with the capacity to be a leader or manager.

Table 4 .
Likert scale for student survey in PO assessments in the IP

Table 5 .
Statistical evaluation of comparative analysis between indirect and direct measurement