Subjective Judgments and Overall Perception Dominate The Accreditation Processes-A Case Study from India

An effective accreditation mechanism should be able to demarcate between a good programme and a bad programme without any bias or preconceived notions of the assessor. This should be achieved through the analysis of variability of performance of programmes in various well-defined dimensions of quality. Moreover, the process should be as objective as possible to avoid the criticisms about the correctness of the decision. Considering these factors, a study has been planned to analyze the effectiveness of a well-established accreditation process. Accreditation process of National Board of Accreditation (NBA), India is taken as a case for the analysis. The study reveals the dominance of subjective judgments and overall perceptions of evaluators in the decision of accreditation status of a programme.


Introduction
According to most of the leading experts on quality, attaining quality goals through a process of continuous improvement over time depends critically upon a firm's ability to define in specific performance terms what it means by quality and then to measure these performance variables objectively (Krishnan et al, 1993).Definition of indicators of quality and the objective measurement of these indicators are critical in the assessment of quality of engineering programmes.A measurement system gives the ability to communicate meaningfully about what quality means.It provides the basis for unambiguous debate and decision-making by the stakeholders on quality on all aspects of the education programmes.Adopting a clear quality measurement also allows specific goals to be established and specific results to be predicted.Subsequently, the measurement process allows faster identification of quality problems, and interventions to solve those problems, greatly facilitating the achievement of programme's overall objectives.It is clear that the direct tie between quantifiable objectives, outputs, measurement and accountability that complete the quality loop and drive a continuous quality improvement process (Lau and Anderson, 1998).

Views on quality on engineering education
What is quality, quality of education especially engineering education, and how it can be achieved are of great interest to the stakeholders of engineering education.The quality of education is becoming important, particularly so in engineering education, where the output of the system, can have a direct impact on the quality of their employer organizations.Reeves and Bednar (1994) contend, 'the search for a universal definition of quality has yielded inconsistent results.Such a global definition does not exist; rather, different definitions of quality are appropriate under different circumstances'.Mukhopadhyay (2001) remarks that, depending on the goals, the term 'quality in education', has been defined by different researchers as excellence in education, value addition in education, fitness of educational outcome & experience for use, conformance of education output to planned goals, specifications & requirements, defect avoidance in education process, and meeting or exceeding customer's expectations of education.Quality in education has been defined variedly even as, 'fitness for purpose' (Tang and Zairi, 1998).Some studies concentrate on the institutional inputs and outputs.Mortimore and Stone (1990) draw attention to the 'normative and comparative' element inherent in quality, emphasizing quality in education as 'an attribute or defining essence; a degree of relative worth; a description of something good or excellent; and a non-quantified trait'.Any definition of quality should be expected to change over time, because 'it necessarily reflects a society's interpretation of educational needs and the intensity of its moral and financial commitment to fulfilling them'.The concept of quality when applied to higher education has been inconclusive (Cheng and Tarn, 1997;Pounder, 1999).Education quality can be viewed as the combination of the quality of input, process, and output of the education system.Eriksen (1995) argued that the primary input is the student (before exposure to a value-added service) who is subjected to a transformation (the application of a value-added service), which, in turn, produces an output (the student after exposure to a value-added service).It regards education quality as a multi-dimensional concept that cannot be assessed by only one indicator.LeBlanc and Nguyen (1997) identified curriculum, physical evidence, responsiveness and access to facilities as the factors, which explain service quality.These factors are related directly to the process of service delivery (Nelson, 1974) and reinforce the belief that quality control measures could be applied along the service delivery system.Education services need specific practices based on the three defining features of services: intangibility, concurrent learning and student participation in the service (Chung et al, 2001).According to Harris (1994) there are three generic approaches to TQM.First is a customer-focus approach, where the idea of service to students is fostered through staff training and development.Second is a staff-focus approach that is concerned to value and enhance the contribution of all the members of staff to the effectiveness of the school.The third approach that takes a service agreement focus, seeks to ensure conformity to specification at certain key measurable points of the educational process.Cheng (1996) defines education quality as the character of the set of elements in the input, process, and output of the education system that provides services that completely satisfy both internal and external strategic constituencies by meeting their explicit and implicit expectations.This definition involves the characteristics of input, process, output and multiple constituencies of an education institution.It regards education quality as a multi-dimensional concept that cannot be assessed by only one indicator.Many opinions can be observed in the literature about the factors promoting quality and excellence in engineering education.Some of them are teaching process (Cropley, 2003), University -Industry collaboration (Natarajan, 2003), role of management (Gopalan, 2003), student intelligence & interest (Mouly and Padmaja, 2003), excellence of teachers (Shrivastava. 2003), accreditation standards (Prem vrat ,2003), e-education (Maji, 2003) and proper documentation of activities (Jagdeesh, 2001).According to Dhend and Biradar (2000), they are Personnel, Policies, Functions, Activities, Procedures, Systems, Facilities & Infrastructure, Services, Environment and Funds.The quality assurance process should address specific academic issues involving faculty development & collaboration, strengthening of research programmes, curriculum development etc and in this connection the networking of institutions and the accreditation policy can play a vital role (Sirohi and Sinha, 2003).
It is clear from the above literature review that the Quality of Engineering Education cannot be defined by any single factor or dimension.The authors have viewed quality as the combination of various factors.Hence, the assessment of quality, and the accreditation process based on this assessment, should not be one-dimensional -one based only on the overall-perception/preconceived-notions of the assessor.

Scope of the study
The study is planned to check the above-discussed factors with respect to a well-established accreditation process.As far as Indian engineering education system is concerned, NBA is the official performance assessment mechanism.Through a series of workshops and seminars of academicians, industrialists, and administrators, NBA has finalized the criteria and procedures for the accreditation process (Manual for NBA Accreditation, 2000).Hence, the study is focused on the NBA accreditation process of engineering programmes from India.
The All India Council for Technical Education (AICTE), India; set up to oversee the growth and quality of technical education; established the National Board of Accreditation (NBA) in 1994 .The NBA is charged with the task of evolving a procedure for quality assessment in the technical education sector based on the specified guidelines, norms, benchmarks and criteria.By accreditation, NBA aims to 'recognize and acknowledge the value addition in transforming the admitted raw student into a capable engineer, having sound knowledge of fundamentals and acceptable level of professional and personal competence for ready employability in responsible engineering assignments'.The criteria or standards, by which individual programmes in any institution will be judged, are formulated to give an indication of the strengths and weaknesses of the programmes.These are classified into indices that measure the quality of different aspects of the programme.Under each index, typical criteria are included for measurement of performance.The points assigned for each of the criteria and the basis of accreditation decision is illustrated in tables 1 and 2.
The institutions who wish to accredit their programmes should submit the information and data according to the Performa provided by the NBA.The NBA will identify the Chairperson and the members of the visiting team with regard to the accreditation of a programme.The expert team selected by NBA will visit the institute and conduct discussions with the management, principal, faculty, supporting staff, students and other stakeholders to assess the performance parameters of the programmes.NBA uses a scoring system based on 8 criteria with a maximum score of 1000 points.The final decision of accreditation of the programme is based on the summated scores allotted for these 8 criteria variables by the experts visiting the institute.

Methodology
The study is planned to examine whether the accreditation/ assessment process of engineering education programmes are multidimensional and not based on the prejudice/preconceived-notions of the assessor about the performance of the programmes.The accreditation process of NBA, India is taken as a case and hypotheses are formulated as follows

Main Hypothesis:
Accreditation status (Accredited/Not-accredited) allotted to a programme by NBA is the outcome of an objective assessment based on all the 8 criteria of NBA.

Sub hypotheses:
None of the eight criteria of NBA process is correlated.

Assessment of NBA is multidimensional.
Only the combined score in all the eight criteria of a Programme have the discriminatory power in segregating between the 'Accredited' and 'Not-Accredited' status.

4.1Data description
The study requires the collection of score sheets prepared by the NBA expert team during their visit to the Institute/College for accreditation.The score sheets filled up by the experts during these visits are confidential documents, which are not accessed by the public.Pure random sampling is difficult when dealing with such confidential data.With the special permission from the NBA, accreditation scores of 160 programmes that have undergone NBA accreditation process during the period 2000 -2003 have been collected for the study.The selected samples of 160 engineering programmes (from 13 states) represent a cross section of Indian engineering education system.Different expert teams assessed the programmes.All these factors ensure the randomness of the samples.Criteria scores are tabulated and outliers are removed.

Testing of sub-hypothesis 1: Correlation analysis of NBA criteria
A Correlation analysis has been conducted to find out the relationships among the eight criteria.The Correlation matrix is shown in Table 3. Twenty-eight (8C 2 ) t-tests have been conducted to identify the significant pair wise correlations between the criteria.There is a probability of incorrectly rejecting the null hypothesis for any of the pair wise comparisons in the family of eight criteria that does not exceed ALPHA (0.05).To avoid this situation, multiple comparison methods that control the family wise error rate need to be used.Hence, the Holms-adjusted multiple comparison p-values are used to determine the significant correlations between the criteria.The matrix of Holms adjusted p-values is given in Table 4.

Testing of sub-hypothesis 2: Principal component analysis on NBA criteria
As the quality of engineering education is multi-dimensional, our interest is to find out whether the NBA assessment process is also multidimensional.We have to check whether the eight criteria can be reduced to a smaller number of linear functions (principal components) of these criteria, which can best summarize the original process.Principal component analysis (PCA) is used for this purpose of data reduction and summarization.In deciding when to stop factoring (that is, how many components to extract), 'latent root (Eigen value) criterion' is followed.To support this decision, 'Scree test criterion' and 'Percentage of variance criterion' are also used.The rationale for the latent root criterion is that any individual component should account for the variance of at least a single variable if it is to be retained for interpretation.Each variable contributes a value of one to the total Eigen value.Thus, only the components having latent roots or eigen values greater than one are considered significant, all components with latent roots less than one are considered insignificant and are discarded.The scree test is derived by plotting the latent roots against the number of components in their order of extraction and the shape of the resulting curve is used to evaluate the cutoff point.The point at which the curve begins to straighten out is considered to indicate the maximum number of factors to extract.The percentage of variance criterion is an approach based on achieving a specified cumulative percentage of total variance extracted by successive factors.The purpose is to ensure practical significance for the derived factors by ensuring that they explain at least a specified amount of variance.No absolute threshold has been adopted that would be valid for all applications.In the social sciences (as in the present research), where information is often less precise, a solution that accounts for 60 percent of the total variance (and in some instances even less) is considered satisfactory (Hair et al, 1998).The components with eigen value greater than one are extracted.Result of PCA on the eight criteria of NBA is given in Table 5.The Scree plot of the PCA is shown in Figure 1.

Testing of sub-hypothesis 3: t-tests on the criteria scores of the programmes
Among the 160 programmes for which data were collected, 138 scored 650 and more points in the accreditation process.The remaining 22 programmes scored only less than 650 points.Hence, 138 programmes got the status 'Accredited' and the other 22 are declared as 'Not Accredited'.Eight set of t-tests have been conducted to test the hypothesis 'Only the combined score in all the eight criteria of a Programme have the discriminatory power in segregating between the 'Accredited' and 'Not-Accredited' status'.The results of t-tests are displayed in the Table 6.

Results and Discussion
Correlation matrix (Table 3) and Holms adjusted p-values (Table 4) of the criteria show that most of them are highly correlated.Hence, the hypothesis 'None of the eight criteria of NBA process is correlated' is rejected.
Principal Component Analysis resulted in a single component (Table 5).The Scree plot (Figure 1) also emphasizes this finding.Around 63 % of variability is explained by this component alone.This result indicates that the eight criteria considered to be the essential dimensions of NBA evaluation process actually represent only a single component.Hence the hypothesis 'Assessment of NBA is multi-dimensional' is rejected.
With reference to the Table 6, at the 0.05 level of significance, the null hypothesis of 'equal criteria scores' is rejected.All the criteria scores are significantly different for the 'Accredited' and 'Not-Accredited' programmes.Moreover, all the eight criteria scores are significantly higher for the 'Accredited' programmes than the 'Not-accredited' ones.Hence, each of the criteria has discriminatory power in segregating between the 'Accredited' and 'Not-Accredited' programmes.
The discriminatory power of all of the criteria of NBA is already tested in a previous study (Viswanadhan et al, 2004) through another statistical technique, logistic regression.A model has been developed that contains the minimum number of variables and capable of making the prediction with maximum accuracy.It is found out that a model, which is made up of only a single criterion variable, can predict the accreditation status with good accuracy (86%).This result indicates that with the determination of a single criterion score, we can predict the accreditation chance of the programme with sufficient accuracy.
Results of the studies pointed out that: The eight criteria of NBA are highly correlated NBA process, in actuality, is one-dimensional Through a detailed procedure, NBA assesses only the overall performance of the programmes.

Conclusions, practical implications and limitations
An effective accreditation mechanism should be able to demarcate between a good programme and a bad programme without any bias or preconceived notions of the assessor.This should be achieved through the analysis of variability of performance of programmes in various well-defined criteria of quality.However, the study revealed that whatever elaborate procedures are adopted for the assessment, in reality, the final decision solely depends on the subjective judgment and overall perception of the evaluator.This finding has the following implications: Necessary steps may be initiated to eliminate (or to reduce) subjectivity from the assessment and accreditation processes or As the subjective judgments control the assessment process, elaborate and tedious scoring systems may be avoided from the accreditation procedures.
The scope of the present study is limited to the accreditation system and processes of NBA, India.The methods used in this paper can be extended to any quantitative accreditation processes.However, they cannot be directly applied to the assessment process of many of the accreditation boards around the world as they adopted non-quantitative methods for accrediting the programmes.

Figure 1 .
Figure 1.Scree plot of Principal Component Analysis on NBA criteria

Table 1 .
Accreditation decision of NBA

Table 6 .
t-tests on the criteria scores of the Accredited and Not-accredited programmes